Build-Loop Autoresearch — 25 Experiments

Method: mine existing run artifacts first (free), micro-test phrasings with controls next (~$1–5), spend full eval runs only where the cheaper tiers couldn't answer (~$7–15 each). Every automated score got a manual inspection pass — which caught and corrected three measurement bugs mid-loop (a grep counting template echoes as findings, a harness that never inlined the diff it claimed to test, a scorer regex blind across newlines). One retracted verdict was re-measured clean. Quality gates (planted-defect scenario) guard every config change; negative results get equal billing.

Mining the corpus — ten experiments from runs we already had

E02Do Interfaces blocks cut implementer exploration?REFUTED

What it tested: Whether the per-task Interfaces blocks (exact signatures consumed/produced) reduce how much implementers poke around the codebase before editing.
How: Mined per-subagent tool-call profiles from completed runs: elicited plans (with Interfaces) vs control plans (without), complete code held constant.
Pre-registered prediction: Elicited implementers make fewer pre-edit reads.
What happened: No reduction: 1.9 pre-edit explores with Interfaces vs 1.3–1.5 without. But the mining surfaced something bigger — the transcription effect quantified: implementers on complete-code plans run 17–24 turns with 1.3–1.9 explores, vs 36.7 turns and 5.6 explores on prose plans. The plan's code, not its interface annotations, is what keeps implementers on rails.
So what: Interfaces earn their keep elsewhere (fix-wave reduction, cross-task consistency) — not navigation.

E04Does the controller waste money reading report files?ALREADY OPTIMAL

What it tested: The brief/report file design intends to keep bulk text out of the controller's context. Does the controller defeat it by reading the reports anyway?
How: Counted controller Read calls touching report files across five runs.
Pre-registered prediction: Controller reads report files in under 30% of tasks.
What happened: 0–1 report-file reads per run across all five runs. The handoff design works exactly as intended.
So what: Lead closed — no saving available.

E07Is the implementer self-review step worth its turns?REFUTED (THE GREP LIED)

What it tested: Implementers self-review before handoff. Does that step catch enough to justify its cost?
How: Grepped 87 implementer transcripts for self-review findings — then manually read every match, per the methodology rule.
Pre-registered prediction: Self-review catches something in >25% of tasks.
What happened: The grep claimed 9 catches in 87. Manual inspection killed 8 of them as template echoes in dispatch prompts. True catch rate: 1 in 87 — an unanchored .gitignore entry that would have blocked staging. In the complete-code regime there is almost nothing for self-review to catch.
So what: Kept anyway: one averted fix wave (~$0.75) per ~87 tasks roughly pays for the turns, and catching before review is far cheaper than after. No change.

E08Can fix subagents run on the cheapest tier?CONFIRMED AS-IS

What it tested: Post-review fixes are usually mechanical. Do haiku fixers hold up?
How: Mined every fix dispatch across the whole run corpus: model used, run outcome.
Pre-registered prediction: Holds; small saving.
What happened: 34 haiku fix dispatches across passing runs — controllers already tier fixes down on their own. The one bad incident (a fixer's go mod tidy silently removed cobra) was caught by the existing fix-report contract.
So what: No skill change needed; the behavior already exists and the safety net already works.

E14Is dispatch cache behavior healthy?ALREADY OPTIMAL

What it tested: If subagent dispatches were cache-missing, reordering could save money.
How: Mined token-usage records from controller transcripts.
Pre-registered prediction: cache_read dwarfs uncached input; no win available.
What happened: Cache reads 7.7–30.7M vs only 19–31k uncached input tokens. Cache health is excellent; the cost axis is resident-context size, not hit rate.
So what: Lead closed.

E15Can reviewers be squeezed further?ALREADY AT FLOOR

What it tested: The reviewer risk budget was tuned earlier. Is there anything left?
How: Mined reviewer subagent profiles across complete-code runs.
Pre-registered prediction: No win available; close the lead.
What happened: Reviews run 4.7–5.0 turns / 2.7–3.0 tools everywhere packages exist (prose-plan runs: 10.1 turns / 6.8 tools). The package did the work; the budget binds.
So what: Lead closed.

E18Does coding-agent verbosity drive QA-agent cost?REFUTED

What it tested: If the eval QA agent's cost tracked the coding agent's message volume, terse final messages would cut eval costs too.
How: Correlated gauntlet token spend against coding-run duration across 30 fractals runs.
Pre-registered prediction: QA tokens correlate with coding final-message volume.
What happened: QA cost tracks duration (r=0.43) — the QA agent polls the screen. Message volume is irrelevant.
So what: Faster runs cut eval costs automatically; nothing else to do.

E19Is TodoWrite + ledger double bookkeeping?PREMISE REFUTED

What it tested: The skill asks for todos and a progress ledger. Dropping todos might save controller calls.
How: Counted TodoWrite calls in controller transcripts.
Pre-registered prediction: Todos redundant with the ledger for ≤10-task plans.
What happened: Zero TodoWrite calls in any measured run — controllers already run ledger-only. Bonus: controller tool calls scale with task count (49 on a 7-task plan vs 74 on a 10-task plan).
So what: No duplication exists to remove.

E20Is the TDD-evidence block in reports load-bearing?CONFIRMED

What it tested: Implementer reports carry RED/GREEN test transcripts. Do reviewers actually use them?
How: Searched 87 reviewer transcripts for references to the TDD evidence.
Pre-registered prediction: Reviewers cite it; keep.
What happened: 41% of reviewer transcripts reference the TDD evidence.
So what: Keep. Removing it would be felt.

E23Does the controller still re-derive context per dispatch?FIXED ALREADY

What it tested: Before this campaign, real sessions had 42k-char dispatch prompts of pasted history. Did the recipe + briefs actually fix it?
How: Measured dispatch-prompt sizes across runs.
Pre-registered prediction: Measurable but small.
What happened: Implementer dispatch prompts now average 2.5k chars on elicited plans (4.6k on prose; 42k in the pathological pre-fix sessions). The remaining mass is the prompt template skeleton itself.
So what: Re-derivation is gone. Any further trim is a template-minification question (E13, backlog).

Micro-tests — phrasings against controls, dollars at a time

E11Can sonnet write the plans?DIED ON STRUCTURE

What it tested: Plan-writing is judgment-dense and pre-registered to stay at opus. How exactly does a cheaper planner fail?
How: 5 sonnet plans from the same design + guidance, scored against opus baselines.
Pre-registered prediction: Sonnet plans miss constraints/interfaces fidelity.
What happened: Fidelity held perfectly (constraints 5/5, gradient verbatim, interfaces on every task) — mechanical guidance adoption is model-robust. But 4 of 5 plans collapsed to 3 tasks (sonnet mean 3.6 vs opus 5.8): the planner chose coarser review gates. The judgment loss showed up in structure, not in copying.
So what: Plan-writing stays at opus, with sharper evidence for why.

E17Do plan word-budgets help or hurt?CONFIRMED HARMFUL

What it tested: A length target might trim plan flab cheaply.
How: 5 plans with a 1,200-word prose budget (code explicitly exempted) vs 5 baseline.
Pre-registered prediction: Test-assertion count falls before prose does.
What happened: Severe: test signals fell 53.8 → 20.6 (−62%) and code fences 39 → 14 — even though baseline prose was already under the budget and code was exempted. The model read 'economy' as licence for austerity everywhere, cutting exactly the load-bearing content.
So what: Never add length targets to writing-plans. Same mechanism as the earlier nuance-clause regression.

E21Can reviewer reports lose their preamble?WIN −41%

What it tested: Reviewer final messages narrate process before delivering verdicts. A 'your final message IS the report' contract might trim that.
How: Micro-test on a real brief+report+diff fixture, 5 reps per arm, sonnet. First attempt had a harness flaw (the diff never inlined) — verdict retracted and re-measured clean.
Pre-registered prediction: −10–20% reviewer output tokens, verdicts intact.
What happened: −41% (1,160 → 687 tokens), spec verdicts 5/5 in both arms. Accidental bonus: the fixture mispaired Task 2's brief with Task 3's diff — and both arms correctly flagged the mismatch. Reviewers genuinely cross-check brief against diff.
So what: Adopted into the E27 stack.

E25Can reviewers work from the diff package alone?DANGEROUS — BRIEF IS LOAD-BEARING

What it tested: Dropping the brief and report from reviewer inputs would simplify dispatches.
How: Micro-test: same fixture with brief and report removed, 5 reps.
Pre-registered prediction: Both needed; close the lead.
What happened: Package-only reviewers delivered confident spec verdicts 5/5 — but silently redefined 'spec' to mean just the global constraints. Zero of five flagged that the brief was missing. The confidence survived; the basis didn't.
So what: Keep all three inputs. Same failure family as cheap-reviewer rationalization: silent scope-shrinkage.

E26Can a recipe shrink controller narration?WORKS, SMALL

What it tested: 46% of controller turns are thinking/narration. Message count is prompt-immune — but per-message length might not be.
How: Micro-test of a mid-workflow controller moment with tools provided, with and without a one-line-narration recipe; opus, 5 reps each.
Pre-registered prediction: Small win or nothing.
What happened: 162 → 74 narration chars per turn (−54%) with zero variance — and the healthy verify-the-implementer's-claims behavior survived inside the one line. Worth ~$0.1–0.3/run.
So what: A freebie; adopted into the E27 stack.

Full runs — where only real execution answers

E01What do critical-sections-only plans cost to execute?REFUTED — CONFOUNDED

What it tested: Jesse's question: what if plans carry only the critical code instead of everything inline?
How: Generated critical-only plans (only 1 in 5 genuinely complied — opus resists withholding code), executed the genuine specimen twice.
Pre-registered prediction: Between complete-code and prose: $9–12, 3–6 fix waves.
What happened: $5.21/$5.11 — the cheapest runs ever measured. But the genuine specimen had also self-merged to 3 tasks: per-implementer effort did double exactly as predicted (35 turns vs 17–24), there were just fewer implementers. Task count swamped code policy; the price is coarser review gates.
So what: Spawned E28 to isolate code policy at constant structure. Also: eliciting this regime at all is hard (1/5 compliance).

E28Code policy at constant structureREFUTED — BODIES ARE MARGINAL

What it tested: The clean version of E01: the same 7-task elicited plan with implementation bodies stripped to one-line descriptions — tests, interfaces, and structure untouched.
How: Controlled fixture edit (10 bodies stripped, 19 test assertions kept), 2 runs.
Pre-registered prediction: Costlier than $6.34–8.49; implementer effort doubles across 7 tasks.
What happened: $7.78/$8.54 — inside the with-bodies range. Implementer turns rose only 7%: the test code pins the work; writing code to satisfy given tests is nearly as guided as transcribing. Fix waves ticked up (1 → 3–4).
So what: The plan-content hierarchy, measured: tests + interfaces + structure are load-bearing; implementation bodies are ~40% of plan length for roughly nothing. A defensible writing-plans relaxation.

E03Haiku implementers under complete-code plansVIABLE, SMALLER THAN HOPED

What it tested: If implementation is transcription, the cheapest tier should manage it — with sonnet reviewers as the quality gate.
How: Conditional guidance ('when the plan text contains the complete code…'), 2 runs on the elicited plan.
Pre-registered prediction: Gates hold; −$2–3/run.
What happened: $6.34/$7.33 vs sonnet implementers' $6.34/$8.49 — overlapping; the real saving is ~$0.5–1 because haiku's turn inflation (30 vs 17–24) ate half the price gap. Gates held; one classic haiku sloppiness (committed a build binary) was caught by review. One drift: the final review ran sonnet.
So what: Worth keeping in the stack with the final-review tier pinned; not the big rock.

E22Does the conditional tier guidance discriminate?CONFIRMED

What it tested: The E03 wording licenses haiku only when the plan carries complete code. On a prose plan, does the controller correctly refuse?
How: Same config, prose 10-task fixture, 1 run.
Pre-registered prediction: Controller picks sonnet implementers; if it picks haiku anyway, expect a blowup.
What happened: 9 of 10 implementers went to sonnet — the one haiku was the genuinely mechanical task. $12.59, normal prose band, gates green.
So what: The conditional wording is shippable: it gates on exactly the right condition.

E05Narrower review-package context (-U3 vs -U10)NEUTRAL

What it tested: Does shrinking diff context push reviewers back into reading files?
How: review-package patched to -U3, 1 run.
Pre-registered prediction: -U3 forces file reads back up; keep -U10.
What happened: Reviewer profile byte-identical (5.0 turns / 3.0 tools), cost in-band. On small diffs the width doesn't matter — and the package is read once by sonnet, so there was no spend to recover anyway.
So what: Keep -U10 as large-diff insurance. Lead closed.

E06Cap the controller's thinking budgetBACKFIRED

What it tested: Mining showed ~2/3 of controller output tokens are invisible (redacted thinking) — the single biggest controller spend. Cap it?
How: MAX_THINKING_TOKENS=1024 injected into the launcher (verified in the binary first; careful contamination window), 2 runs.
Pre-registered prediction: −15–25% controller cost; judgment must stay clean.
What happened: Costs in-band ($7.15/$8.10) but controller messages rose 92 → 128/138 and output tokens nearly doubled (66k → 118–123k). Thinking buys turn efficiency; suppress it and the controller compensates with more steps and re-derivation. Gates held — it isn't harmful, just useless.
So what: Do not cap thinking. The turn floor rises to meet you.

E09Controller surveys the plan instead of reading itPARTIAL — HABIT WON

What it tested: With briefs delivering tasks, the controller only needs headings + constraints: a task-brief --list mode.
How: Implemented --list (smoke-tested on real plans), guidance added, 1 run.
Pre-registered prediction: −$0.5–1/run of resident context.
What happened: The controller ran --list once… then read plan.md twice anyway. Cost in-band, saving unrealized. The read-the-plan habit beat one guidance line.
So what: Dropped for now — the ceiling (~$1) doesn't justify more runs; phrasing iteration is future micro-work.

E24Fixture realism at svelte scaleCONFIRMED −24%

What it tested: Fractals showed hand-written fixture plans execute ~2× costlier than real skill output. Does it replicate on the bigger scenario?
How: Generated an elicited svelte plan — discovering first that the hand fixture had gold-plated an E2E suite the design never asked for; amended the design so scopes match. 2 runs incl. Playwright.
Pre-registered prediction: $10–13 vs the hand fixture's $14.99–20.30.
What happened: $12.40/$14.38 (mean $13.39) vs ~$17.6 — a −24% replication, milder than fractals' 2× because heavier intrinsic UI/E2E work dilutes plan effects.
So what: Fixture-realism finding stands across scales. Also: baseline svelte numbers always included unrequested scope.

The composition

E27The integrated ship candidateCANDIDATE ESTABLISHED

What it tested: Do the loop's wins compose? Opus controller + elicited plan + conditional haiku implementers + terse reviewer contract + narration recipe + final-review tier pin.
How: Stack clone, 2 fractals runs + 3 planted-defect quality gates.
Pre-registered prediction: All green; fractals ≤ $6.
What happened: Fractals $6.24/$6.60 (combo on hand plans: $11.67–14.84). Quality gates 2 of 3 — the one fail was a clean reviewer miss on DRY plus a stricter judge reading; inspecting the reviewer's 2,620-char report exonerated the terse contract as the suppressor.
So what: The candidate exists and costs half the shipped config. The N=5 gate battery is still owed before any of it becomes skill text.

Backlog — registered, not run (the 25 minimum was reached without them)

E10 Report contract 15→8 lines · E12 Coarser step granularity · E13 Prompt-template minification · E16 Best-stack composition on svelte

Where it nets out: the ship-candidate stack runs go-fractals at $6.24–6.60 against the shipped config's $11.67–14.84 — but most of that gap belongs to executing real, complete plans rather than the hand-written prose fixtures all earlier numbers were measured on. The durable lessons: tests+interfaces+structure are what make a plan cheap to execute; thinking is load-bearing, don't cap it; word budgets eat tests; cheap models fail by advocating, not by going quiet; and half of all leads close as "the current design already had this right" — which is exactly what the log is for.

Build-Loop Autoresearch — the 25 Experiments