# SDD Experiments — Decision Log

Each card: what changed, measured against the config it built on AND the original baseline. Coding-agent dollars on go-fractals unless noted. 63 runs total, ~$633.
Lineage:
baseline (dev) — fractals $16.07 · svelte $20.98 → redesign iters 1–5 — one task-reviewer per task → + E1 package, E2 model line, E4 ledger, file handoffs = combo (PR #1717) — $11.67–14.84
probes off combo: A briefs ablation — neutral B crisp plan — $9.51–12.65 C sonnet controller — $6.68–8.05, gates owed D haiku reviewers — quality fails E svelte n=2 — claim widened
killed along the way: E3 pipelining turn batching "don't restate" prohibition
The shipped config (PR #1717)
REFERENCE

## Baseline — dev branch as shipped

5 runs
Two reviewers per task (spec, then quality), each re-deriving the diff with git commands; no model discipline in dispatches.

scenario | cost | time | tokens
--- | --- | --- | ---
go-fractals | $16.07 | 64.9 min | 21.2M
svelte-todo | $20.98 | 79.7 min | 27.3M
rejects-extra-features | $1.88 |

WIN

## Redesign (iterations 1–5) — task-scoped review dispatch

landed · the PR's core · 15 runs
Changed: one task-reviewer per task carrying both verdicts (spec + quality); reviewer reads a generated review-package file instead of re-running git; scope budget + escape-hatch tuning. Built on: baseline.

scenario | this | vs baseline
--- | --- | ---
go-fractals | $12.81–14.31 | −11 to −20%（−32% time, −37% tokens at freeze)
svelte-todo | ≈$15.74 | −25% $ (−21% time, −28% tokens)
all 5 gate scenarios | pass | quality held

Why it wins: one reviewer dispatch instead of two, and the package file turns reviews from ~9 turns / 6.4 tool calls into ~3 turns / 1 read.
WIN

## E1 — give the FINAL reviewer a package too

merged into combo · 2 runs
Changed: final whole-branch reviewer gets `review-package MERGE_BASE HEAD` instead of re-deriving the branch diff. Built on: redesign.

metric | this | vs parent
--- | --- | ---
final-review turns / tools | 6 / 3 | was 33 / 23
opus (controller-model) bill |  | −19%

WIN

## E2 — REQUIRED model: line in dispatch templates

merged into combo · 1 run
Changed: nothing yet — this run exposed the failure: with guidance-only model selection, dispatches from Task 3 onward silently inherited opus. Built on: redesign + file handoffs.

metric | guidance only | with REQUIRED line
--- | --- | ---
explicit-model dispatches | decayed to (default)=opus, +$5 | 26/26, 31/31, 34/34, 57/57 across later runs

WIN

## E4 — durable progress ledger

merged into combo · 1 run
Changed: controller appends task completion to `<git-dir>/sdd/progress.md`; checks it at skill start. Built on: redesign.

metric | result
--- | ---
ledger adoption / run cost | maintained unprompted, cost in-band
failure it insures against | real session re-dispatched ~22 done tasks post-compaction (269 dispatches)

SMALL WIN, KEPT

## File handoffs — task-brief + report files

merged into combo · 1 run + ablation A
Changed: implementer reads a per-task brief file and writes a report file; ≤15-line final messages. Built on: redesign.

metric | modeled | measured
--- | --- | ---
resident-context savings | ~50% | ~$1/run (dispatch prompts −9%, tool-result chars −16%)

Kept anyway: exact values live in one file (fidelity), enables conflict detection, survives compaction. Confirmed cost-neutral by ablation A below.
WIN

## Combo — everything merged (this is PR #1717's final config)

shipped · 3 gate runs + 9-run band
Changed: redesign + E1 + E2 + E4 + file handoffs + dispatch recipe + reviewer risk budget. Built on: all of the above.

scenario | this | vs baseline | vs redesign
--- | --- | --- | ---
go-fractals (n=9) | $11.67–14.84 | −8 to −27% | ≈flat $; adds durability + discipline
svelte-todo (n=2) | $14.99–20.30 | −3 to −29% | range overlaps
planted-defect + guardrails | pass | quality held

Killed along the way (negative results, recorded so nobody re-buys them)
DEAD

## E3 — pipelined reviews

declined · 2 runs
Changed: asked the controller to dispatch next implementer + reviewer in one message / in background. Built on: redesign.

variant | result
--- | ---
parallel tool calls | 0 paired dispatches in 29 — controller emits exactly one tool call per message, prompt-immune
run_in_background | adopted 7/28 but $14.84 (band top) — reviews only ~30-60s, benefit below ±20% run noise

DEAD

## Turn batching & "don't restate the brief" (micro-tested)

declined · measured inside E-runs + 25 API micro-samples
Changed: guidance to combine bookkeeping calls; prohibition on restating spec values in dispatches.

variant | result
--- | ---
turn batching | 0 multi-tool messages ever, with or without guidance; 46% of controller turns are thinking-only
"do not restate" prohibition | 4.4 values re-typed vs 3.6 with NO guidance — backfired
positive 5-part recipe instead | 3.0, zero variance — this is what landed

Batch A–E — probes off combo (overnight, strict-cost ladder)
NEUTRAL

## A — do the brief/report files cost money?

ablation, settles a question · 4 runs
Changed: combo MINUS the brief/report mechanism (everything else kept). Built on: combo.

scenario | lean (no briefs) | combo same night | combo band
--- | --- | --- | ---
go-fractals ×3 | $11.61 / 12.27 / 13.25 | $14.10 | $11.67–14.84 — fully overlapping

Verdict: briefs are free. They stay for fidelity + compaction durability. Also resolves "was the 44.4-min run a better config?" — no, tail draw.
WIN

## B — crisp plan (strict-cost L1)

validated → future writing-plans PR · 3 runs
Changed: the PLAN, not the skill: 10 tasks → 7 right-sized, ## Global Constraints header, per-task Interfaces: lines. Same combo config. Built on: combo.

metric | crisp plan | vs combo (10-task plan) | vs baseline
--- | --- | --- | ---
go-fractals cost ×3 | $9.51 / 12.65 / 12.65 | mean −$1.5 (−12%) | −21 to −41%
dispatches | 20 / 21 / 24 | was 28 (−21%) |
quality gates | 3/3 pass, fix waves flat | no review-granularity loss |

Open half: getting writing-plans to PRODUCE such plans (this used a hand-written one).
WIN PENDING GATES

## C — sonnet controller (strict-cost L2)

recon positive · 3 runs (1 infra fail)
Changed: the controller SESSION runs sonnet instead of opus (subagent tiers unchanged, chosen by the controller). Built on: combo.

metric | sonnet controller | vs combo | vs baseline
--- | --- | --- | ---
go-fractals ×2 | $6.68 / $8.05 | ≈−40% | ≈−55%
tokens | 12.6M / 16.1M | inside band — no turn inflation |
judgment | loops + omnibus + re-reviews held; caught go-mod-tidy dropping cobra |

Not shipped: zero BLOCKED/⚠️ events arose, so escalation judgment is untested. Needs a broken-plan scenario + N=5 + judgment audit. (The $0.13 fail found a real quorum seeding bug, since fixed.)
DEAD

## D — haiku task reviewers (strict-cost L3)

disproven, as pre-registered · 6 runs
Changed: Model Selection forces task reviewers onto the cheapest tier. Built on: combo.

metric | haiku reviewers | combo
--- | --- | ---
planted-defect scenario ×5 | 2 pass / 1 indet / 2 fail | 5/5 pass
defects cleanly flagged at severity | 0 of 10 | scenario's whole point
fractals cost | $9.61 (−$2-3) | the "saving" IS the laxness: fewer findings → fewer fix waves

Sharpest finding: haiku doesn't miss silently — it advocates for defects (DRY praised as YAGNI; assert-nothing test "plan-compliant"; found-then-downgraded with the exact prohibited rationale).
CLAIM CALIBRATION

## E — svelte second run

honesty, not optimization · 1 run
Changed: nothing — re-ran combo on svelte to turn a single-run claim into a range.

metric | run 1 | run 2 | baseline
--- | --- | --- | ---
svelte cost | $14.99 | $20.30 (9 fix waves) | $20.98 — overlaps at top

PR now claims the range. Time/tokens stay clearly better; cost is honestly "usually better, sometimes ≈flat".
Infrastructure runs (not win/lose experiments)
TOOLING

## Planted-defect scenario development (11 runs) + guardrail matrix (6 runs)

Building and tuning the quality gates themselves: the planted-defect scenario went v1→v3, fixing five distinct ways the defect got suppressed (controller pre-judging, severity pre-rating, reviewer calibration, implementer framing, a wrong eval bar). The guardrail scenarios (rejects-extra-features, planted-flaws) ran repeatedly to confirm each config kept passing.
Batch A–E wall-clock ran under up to 7-way parallel load — dollars and tokens are the trustworthy columns. Full run-level detail: evals-matrix.html (chronological) / evals-by-experiment.html (grouped).