SDD Experiments — Decision Log
Each card: what changed, measured against the config it built on AND the original baseline. Coding-agent dollars on go-fractals unless noted. 63 runs total, ~$633.
Lineage:
baseline (dev) — fractals $16.07 · svelte $20.98 →
redesign iters 1–5 — one task-reviewer per task →
+ E1 package, E2 model line, E4 ledger, file handoffs = combo (PR #1717) — $11.67–14.84
probes off combo: A briefs ablation — neutral
B crisp plan — $9.51–12.65
C sonnet controller — $6.68–8.05, gates owed
D haiku reviewers — quality fails
E svelte n=2 — claim widened
killed along the way: E3 pipelining turn batching "don't restate" prohibition
The shipped config (PR #1717)
REFERENCE
Baseline — dev branch as shipped
5 runs
Two reviewers per task (spec, then quality), each re-deriving the diff with git commands; no model discipline in dispatches.
| scenario | cost | time | tokens |
|---|
| go-fractals | $16.07 | 64.9 min | 21.2M |
| svelte-todo | $20.98 | 79.7 min | 27.3M |
| rejects-extra-features | $1.88 | |
WIN
Redesign (iterations 1–5) — task-scoped review dispatch
landed · the PR's core · 15 runs
Changed: one task-reviewer per task carrying both verdicts (spec + quality); reviewer reads a generated review-package file instead of re-running git; scope budget + escape-hatch tuning. Built on: baseline.
| scenario | this | vs baseline |
|---|
| go-fractals | $12.81–14.31 | −11 to −20%（−32% time, −37% tokens at freeze) |
| svelte-todo | ≈$15.74 | −25% $ (−21% time, −28% tokens) |
| all 5 gate scenarios | pass | quality held |
Why it wins: one reviewer dispatch instead of two, and the package file turns reviews from ~9 turns / 6.4 tool calls into ~3 turns / 1 read.
WIN
E1 — give the FINAL reviewer a package too
merged into combo · 2 runs
Changed: final whole-branch reviewer gets review-package MERGE_BASE HEAD instead of re-deriving the branch diff. Built on: redesign.
| metric | this | vs parent |
|---|
| final-review turns / tools | 6 / 3 | was 33 / 23 |
| opus (controller-model) bill | | −19% |
WIN
E2 — REQUIRED model: line in dispatch templates
merged into combo · 1 run
Changed: nothing yet — this run exposed the failure: with guidance-only model selection, dispatches from Task 3 onward silently inherited opus. Built on: redesign + file handoffs.
| metric | guidance only | with REQUIRED line |
|---|
| explicit-model dispatches | decayed to (default)=opus, +$5 | 26/26, 31/31, 34/34, 57/57 across later runs |
WIN
E4 — durable progress ledger
merged into combo · 1 run
Changed: controller appends task completion to <git-dir>/sdd/progress.md; checks it at skill start. Built on: redesign.
| metric | result |
|---|
| ledger adoption / run cost | maintained unprompted, cost in-band |
| failure it insures against | real session re-dispatched ~22 done tasks post-compaction (269 dispatches) |
SMALL WIN, KEPT
File handoffs — task-brief + report files
merged into combo · 1 run + ablation A
Changed: implementer reads a per-task brief file and writes a report file; ≤15-line final messages. Built on: redesign.
| metric | modeled | measured |
|---|
| resident-context savings | ~50% | ~$1/run (dispatch prompts −9%, tool-result chars −16%) |
Kept anyway: exact values live in one file (fidelity), enables conflict detection, survives compaction. Confirmed cost-neutral by ablation A below.
WIN
Combo — everything merged (this is PR #1717's final config)
shipped · 3 gate runs + 9-run band
Changed: redesign + E1 + E2 + E4 + file handoffs + dispatch recipe + reviewer risk budget. Built on: all of the above.
| scenario | this | vs baseline | vs redesign |
|---|
| go-fractals (n=9) | $11.67–14.84 | −8 to −27% | ≈flat $; adds durability + discipline |
| svelte-todo (n=2) | $14.99–20.30 | −3 to −29% | range overlaps |
| planted-defect + guardrails | pass | quality held |
Killed along the way (negative results, recorded so nobody re-buys them)
DEAD
E3 — pipelined reviews
declined · 2 runs
Changed: asked the controller to dispatch next implementer + reviewer in one message / in background. Built on: redesign.
| variant | result |
|---|
| parallel tool calls | 0 paired dispatches in 29 — controller emits exactly one tool call per message, prompt-immune |
| run_in_background | adopted 7/28 but $14.84 (band top) — reviews only ~30-60s, benefit below ±20% run noise |
DEAD
Turn batching & "don't restate the brief" (micro-tested)
declined · measured inside E-runs + 25 API micro-samples
Changed: guidance to combine bookkeeping calls; prohibition on restating spec values in dispatches.
| variant | result |
|---|
| turn batching | 0 multi-tool messages ever, with or without guidance; 46% of controller turns are thinking-only |
| "do not restate" prohibition | 4.4 values re-typed vs 3.6 with NO guidance — backfired |
| positive 5-part recipe instead | 3.0, zero variance — this is what landed |
Batch A–E — probes off combo (overnight, strict-cost ladder)
NEUTRAL
A — do the brief/report files cost money?
ablation, settles a question · 4 runs
Changed: combo MINUS the brief/report mechanism (everything else kept). Built on: combo.
| scenario | lean (no briefs) | combo same night | combo band |
|---|
| go-fractals ×3 | $11.61 / 12.27 / 13.25 | $14.10 | $11.67–14.84 — fully overlapping |
Verdict: briefs are free. They stay for fidelity + compaction durability. Also resolves "was the 44.4-min run a better config?" — no, tail draw.
WIN
B — crisp plan (strict-cost L1)
validated → future writing-plans PR · 3 runs
Changed: the PLAN, not the skill: 10 tasks → 7 right-sized, ## Global Constraints header, per-task Interfaces: lines. Same combo config. Built on: combo.
| metric | crisp plan | vs combo (10-task plan) | vs baseline |
|---|
| go-fractals cost ×3 | $9.51 / 12.65 / 12.65 | mean −$1.5 (−12%) | −21 to −41% |
| dispatches | 20 / 21 / 24 | was 28 (−21%) | |
| quality gates | 3/3 pass, fix waves flat | no review-granularity loss | |
Open half: getting writing-plans to PRODUCE such plans (this used a hand-written one).
WIN PENDING GATES
C — sonnet controller (strict-cost L2)
recon positive · 3 runs (1 infra fail)
Changed: the controller SESSION runs sonnet instead of opus (subagent tiers unchanged, chosen by the controller). Built on: combo.
| metric | sonnet controller | vs combo | vs baseline |
|---|
| go-fractals ×2 | $6.68 / $8.05 | ≈−40% | ≈−55% |
| tokens | 12.6M / 16.1M | inside band — no turn inflation | |
| judgment | loops + omnibus + re-reviews held; caught go-mod-tidy dropping cobra | |
Not shipped: zero BLOCKED/⚠️ events arose, so escalation judgment is untested. Needs a broken-plan scenario + N=5 + judgment audit. (The $0.13 fail found a real quorum seeding bug, since fixed.)
DEAD
D — haiku task reviewers (strict-cost L3)
disproven, as pre-registered · 6 runs
Changed: Model Selection forces task reviewers onto the cheapest tier. Built on: combo.
| metric | haiku reviewers | combo |
|---|
| planted-defect scenario ×5 | 2 pass / 1 indet / 2 fail | 5/5 pass |
| defects cleanly flagged at severity | 0 of 10 | scenario's whole point |
| fractals cost | $9.61 (−$2-3) | the "saving" IS the laxness: fewer findings → fewer fix waves |
Sharpest finding: haiku doesn't miss silently — it advocates for defects (DRY praised as YAGNI; assert-nothing test "plan-compliant"; found-then-downgraded with the exact prohibited rationale).
CLAIM CALIBRATION
E — svelte second run
honesty, not optimization · 1 run
Changed: nothing — re-ran combo on svelte to turn a single-run claim into a range.
| metric | run 1 | run 2 | baseline |
|---|
| svelte cost | $14.99 | $20.30 (9 fix waves) | $20.98 — overlaps at top |
PR now claims the range. Time/tokens stay clearly better; cost is honestly "usually better, sometimes ≈flat".
Infrastructure runs (not win/lose experiments)
TOOLING
Planted-defect scenario development (11 runs) + guardrail matrix (6 runs)
Building and tuning the quality gates themselves: the planted-defect scenario went v1→v3, fixing five distinct ways the defect got suppressed (controller pre-judging, severity pre-rating, reviewer calibration, implementer framing, a wrong eval bar). The guardrail scenarios (rejects-extra-features, planted-flaws) ran repeatedly to confirm each config kept passing.
Batch A–E wall-clock ran under up to 7-way parallel load — dollars and tokens are the trustworthy columns. Full run-level detail: evals-matrix.html (chronological) / evals-by-experiment.html (grouped).