SDD Review-Dispatch Evals — grouped by experiment

Baseline — dev branch as shipped · 5 runs

Reference numbers the whole campaign is measured against: fractals $16.07 / 64.9 min, svelte $20.98 / 79.7 min.

Design iterations 1–5 — task-scoped review dispatch (the PR itself) · 15 runs

One task reviewer per task (spec + quality verdicts), review-package files, scope and escape-hatch tuning across five iterations on the branch. Converged on the frozen e355795 config: fractals −27% $, svelte −25% $ vs baseline. 'branch state' = the branch as it stood at launch time.

Planted-defect scenario development (v1 → v3) · 11 runs

Building the quality gate itself: a scenario whose plan plants an assertion-free test with a lying name plus a DRY violation. Failed through five distinct suppression mechanisms during development (controller pre-judging, severity pre-rating, reviewer calibration, implementer framing, and a wrong eval bar) — each fixed in the prompts generally, not by teaching to the test.

Guardrail scenarios — validation matrix · 6 runs

sdd-rejects-extra-features ($1.31–1.37 vs $1.88 baseline) and spec-reviewer-catches-planted-flaws: quality gates every config iteration had to keep passing.

E1 — final-review package · 2 runs

Hand the FINAL whole-branch reviewer a package file too. WIN: 33 turns/23 tools → 6 turns/3 tools at opus prices; opus bill −19%. Landed.

E2 — model-line decay · 1 run

Guidance-only model selection decays: dispatches from Task 3 onward inherited opus (+$5). Led to the REQUIRED model: template line. Landed.

E3/E3b — pipelined reviews · 2 runs

Parallel-call pipelining DEAD (0 paired dispatches in 29 — the controller emits exactly one tool call per message). Background pipelining works mechanically (7/28 dispatches) but benefit below run-to-run noise. Declined.

E4 — durable progress ledger · 1 run

Ledger in <git-dir>/sdd/progress.md, maintained well, run cost in-band; guards against post-compaction re-dispatch (worst real-session failure: 269 dispatches for ~22 tasks). Landed.

Combo gates — winners merged, re-validated · 3 runs

All surviving changes together (briefs, packages, model lines, ledger, recipes, risk budget): gates green, fractals band $11.67–14.84 over repeated runs.

A — lean vs combo (brief ablation) · 4 runs

Does the brief/report mechanism cost money? NO: lean $11.61/$12.27/$13.25 vs the combo band + the $14.10 same-night control — fully overlapping. Briefs stay, justified by fidelity + compaction durability. Also resolves: the 44.4-min e355795 run was a tail draw, not a better config.

B — crisp-plan ceiling (strict-cost L1) · 3 runs

Hand-rewritten 7-task plan + Global Constraints header + per-task Interfaces lines: $9.51/$12.65/$12.65, dispatches 28 → 20-24 (−21%), fix waves flat, gates 3/3. VALIDATED in effect — the follow-up is making writing-plans elicit this shape.

C — sonnet controller (strict-cost L2 recon) · 3 runs

Controller session on sonnet: $6.68/$8.05 (~−40% vs band), no turn inflation, clean judgment trail (caught go-mod-tidy silently dropping cobra), heavier-and-sane haiku tiering. Zero BLOCKED/⚠️ events arose, so escalation behavior is untested — full N=5 + judgment-audit gates still owed. The $0.13 fail was a quorum seeding bug (API-key picker defaults No) — found, fixed, now superseded by upstream's equivalent fix.

D — haiku task reviewers (strict-cost L3) · 6 runs

DEAD as pre-registered: planted-defect 2 pass / 1 indet / 2 fail (baseline 5/5); 0 of 10 defects cleanly flagged — haiku ADVOCATES for defects (DRY praised as YAGNI; assert-nothing test 'plan-compliant'; found-then-downgraded with the exact prohibited rationale). The ~$2-3/run saving in the fractals run and the quality failure are the same mechanism: lax reviews trigger fewer fix waves.

E — svelte n=2 · 1 run

Second combo svelte run: $20.30 (9 fix waves — review-strictness variance; all 34 dispatches model-disciplined). PR claim becomes the honest range $14.99–20.30 vs $20.98 baseline — time/tokens clearly better, cost overlaps at the top.