Model-lane validation: GLM + deepseek-aider on real company tasks — quality report + token-ROI evidence (Gate 0b)

[kokevidaurre]

Founder direction 2026-07-10 (launch Gate 0b — launch-founding spec §6): no provider is claimed in public copy until its lane runs real company work with good results.

What

1. Run real squad tasks (not toys — actual company work items) through the GLM lane and the deepseek-aider lane.
2. Per lane, produce a quality/problem report: output quality vs the Claude lane, failure modes (tool gaps, format drift, silent losses — see reference: deepseek/aider = code-only no-web; web-scanners need claude-haiku+), retry/salvage behavior, wall-clock.
3. Token-ROI evidence in the same runs: cost per LANDED outcome per lane vs Claude-direct. Token ROI is the business wedge — "this really makes good use of tokens" is what makes a business buy squads instead of spending on CC tokens directly.
4. Output: a lane scorecard (green/yellow/red per task class) → green lanes may be claimed publicly; yellow/red get issues.

Depends on honest outcomes data — cli#1060 (outcomes hardcoded zero) should land first or the ROI numbers will be garbage.

[agents-squads]

Gate 0b lane scorecard — round 1 (2026-07-10/11)

7 runs on real backlog issues (matched ux/bug class): 2× deepseek-v4-flash (aider), 4× glm-4.7 (2 + 2 retries), 1× claude-sonnet baseline. All dispatched via squads run from hq with the #1060 outcomes fix live.

Lane	Task	Result	Wall	Ledgered cost	Landed
claude-direct	#914	Full solve: edits, tests, build, PR #1072, suite 2237/2238	12m47s	$8.55 (actions 83, edits 12)	PR #1072 ✅
deepseek	#913	Real, mergeable fix; provider died mid-run (API disconnect); harvested clean → PR #1075	10m50s	$0.03	PR #1075 ✅
deepseek	#912	CATASTROPHIC: replaced 826-line run.ts with a 6-line fragment; run reported ✔ (#1076)	18m19s	$0.007	❌
glm	#911	Correct line-exact root-cause spec, ZERO edits — lane is write-denied (#1073); first attempt stalled mid-stream	6m32s + 4m55s	$0.00 (telemetry gap #1077)	❌
glm	#915	Plan-and-ask ('Ready to proceed?') in one-shot mode; retry hit the write wall, produced spec only	1m02s + 5m32s	$0.00	❌

Verdicts (per task class)

• claude-direct: GREEN (execution) — only lane that lands end-to-end unattended. $8.55/landed PR.
• deepseek: YELLOW (scoped code fixes) — 1/2 runs produced mergeable-grade work at $0.04/landed PR (~200× cheaper than Claude-direct), but 1/2 was a catastrophic whole-file truncation that the harness reported as success. NOT green until harvest gate: detect catastrophic file truncation — aider replaced an 826-line file with a 6-line fragment and the run reported success #1076 (truncation guard) lands; every deepseek branch needs inbox review before merge. RED for large files / unattended use.
• glm: RED (execution), promising (analysis/spec) — structurally read-only today (GLM lane is read-only by construction — claude --print with no permission flags denies every file write #1073: claude --print with no permission flags). Model quality itself looked good: 2/2 correct root-cause diagnoses with line-exact fix specs. Re-validate after GLM lane is read-only by construction — claude --print with no permission flags denies every file write #1073.

Token-ROI (the wedge metric)

Cost per LANDED outcome: deepseek $0.04 vs Claude-direct $8.55 on same-class tasks — with the honest caveat that the deepseek land required orchestrator shepherding (harvest review, junk-stripping, independent verification) that itself costs Claude tokens, and a 50% severe-failure rate absorbed by the gate.

Public-claim ruling (per founder rule: green on OUR work only)

• Claude: claimable (already the default).
• deepseek: not yet claimable — needs harvest gate: detect catastrophic file truncation — aider replaced an 826-line file with a 6-line fragment and the run reported success #1076 + a clean multi-run record.
• GLM: not claimable.

Harness gaps filed this round

#1070 (harvest commits its own .agents copy), #1073 (GLM lane write-denied), #1076 (no truncation guard), #1077 (GLM cost/outcomes not ledgered). The #1060 fix is confirmed working in production: baseline run ledgered real outcomes (83 actions / 12 edits / 1 commit / 1 PR).