# Build-Loop Autoresearch — the 25 Experiments

2026-06-11 · opus as coordinator throughout · every hypothesis pre-registered with a prediction · ~$165
Method: mine existing run artifacts first (free), micro-test phrasings with controls next (~$1–5), spend full eval runs only where the cheaper tiers couldn't answer (~$7–15 each). Every automated score got a manual inspection pass — which caught and corrected three measurement bugs mid-loop (a grep counting template echoes as findings, a harness that never inlined the diff it claimed to test, a scorer regex blind across newlines). One retracted verdict was re-measured clean. Quality gates (planted-defect scenario) guard every config change; negative results get equal billing.

### Mining the corpus — ten experiments from runs we already had

## E02Do Interfaces blocks cut implementer exploration?REFUTED

What it tested
  Whether the per-task Interfaces blocks (exact signatures consumed/produced) reduce how much implementers poke around the codebase before editing.
How
  Mined per-subagent tool-call profiles from completed runs: elicited plans (with Interfaces) vs control plans (without), complete code held constant.
Pre-registered prediction
  Elicited implementers make fewer pre-edit reads.
What happened
  No reduction: 1.9 pre-edit explores with Interfaces vs 1.3–1.5 without. But the mining surfaced something bigger — the transcription effect quantified: implementers on complete-code plans run 17–24 turns with 1.3–1.9 explores, vs 36.7 turns and 5.6 explores on prose plans. The plan's code, not its interface annotations, is what keeps implementers on rails.
So what
  Interfaces earn their keep elsewhere (fix-wave reduction, cross-task consistency) — not navigation.

## E04Does the controller waste money reading report files?ALREADY OPTIMAL

What it tested
  The brief/report file design intends to keep bulk text out of the controller's context. Does the controller defeat it by reading the reports anyway?
How
  Counted controller Read calls touching report files across five runs.
Pre-registered prediction
  Controller reads report files in under 30% of tasks.
What happened
  0–1 report-file reads per run across all five runs. The handoff design works exactly as intended.
So what
  Lead closed — no saving available.

## E07Is the implementer self-review step worth its turns?REFUTED (THE GREP LIED)

What it tested
  Implementers self-review before handoff. Does that step catch enough to justify its cost?
How
  Grepped 87 implementer transcripts for self-review findings — then manually read every match, per the methodology rule.
Pre-registered prediction
  Self-review catches something in >25% of tasks.
What happened
  The grep claimed 9 catches in 87. Manual inspection killed 8 of them as template echoes in dispatch prompts. True catch rate: 1 in 87 — an unanchored .gitignore entry that would have blocked staging. In the complete-code regime there is almost nothing for self-review to catch.
So what
  Kept anyway: one averted fix wave (~$0.75) per ~87 tasks roughly pays for the turns, and catching before review is far cheaper than after. No change.

## E08Can fix subagents run on the cheapest tier?CONFIRMED AS-IS

What it tested
  Post-review fixes are usually mechanical. Do haiku fixers hold up?
How
  Mined every fix dispatch across the whole run corpus: model used, run outcome.
Pre-registered prediction
  Holds; small saving.
What happened
  34 haiku fix dispatches across passing runs — controllers already tier fixes down on their own. The one bad incident (a fixer's go mod tidy silently removed cobra) was caught by the existing fix-report contract.
So what
  No skill change needed; the behavior already exists and the safety net already works.

## E14Is dispatch cache behavior healthy?ALREADY OPTIMAL

What it tested
  If subagent dispatches were cache-missing, reordering could save money.
How
  Mined token-usage records from controller transcripts.
Pre-registered prediction
  cache_read dwarfs uncached input; no win available.
What happened
  Cache reads 7.7–30.7M vs only 19–31k uncached input tokens. Cache health is excellent; the cost axis is resident-context size, not hit rate.
So what
  Lead closed.

## E15Can reviewers be squeezed further?ALREADY AT FLOOR

What it tested
  The reviewer risk budget was tuned earlier. Is there anything left?
How
  Mined reviewer subagent profiles across complete-code runs.
Pre-registered prediction
  No win available; close the lead.
What happened
  Reviews run 4.7–5.0 turns / 2.7–3.0 tools everywhere packages exist (prose-plan runs: 10.1 turns / 6.8 tools). The package did the work; the budget binds.
So what
  Lead closed.

## E18Does coding-agent verbosity drive QA-agent cost?REFUTED

What it tested
  If the eval QA agent's cost tracked the coding agent's message volume, terse final messages would cut eval costs too.
How
  Correlated gauntlet token spend against coding-run duration across 30 fractals runs.
Pre-registered prediction
  QA tokens correlate with coding final-message volume.
What happened
  QA cost tracks duration (r=0.43) — the QA agent polls the screen. Message volume is irrelevant.
So what
  Faster runs cut eval costs automatically; nothing else to do.

## E19Is TodoWrite + ledger double bookkeeping?PREMISE REFUTED

What it tested
  The skill asks for todos and a progress ledger. Dropping todos might save controller calls.
How
  Counted TodoWrite calls in controller transcripts.
Pre-registered prediction
  Todos redundant with the ledger for ≤10-task plans.
What happened
  Zero TodoWrite calls in any measured run — controllers already run ledger-only. Bonus: controller tool calls scale with task count (49 on a 7-task plan vs 74 on a 10-task plan).
So what
  No duplication exists to remove.

## E20Is the TDD-evidence block in reports load-bearing?CONFIRMED

What it tested
  Implementer reports carry RED/GREEN test transcripts. Do reviewers actually use them?
How
  Searched 87 reviewer transcripts for references to the TDD evidence.
Pre-registered prediction
  Reviewers cite it; keep.
What happened
  41% of reviewer transcripts reference the TDD evidence.
So what
  Keep. Removing it would be felt.

## E23Does the controller still re-derive context per dispatch?FIXED ALREADY

What it tested
  Before this campaign, real sessions had 42k-char dispatch prompts of pasted history. Did the recipe + briefs actually fix it?
How
  Measured dispatch-prompt sizes across runs.
Pre-registered prediction
  Measurable but small.
What happened
  Implementer dispatch prompts now average 2.5k chars on elicited plans (4.6k on prose; 42k in the pathological pre-fix sessions). The remaining mass is the prompt template skeleton itself.
So what
  Re-derivation is gone. Any further trim is a template-minification question (E13, backlog).

### Micro-tests — phrasings against controls, dollars at a time

## E11Can sonnet write the plans?DIED ON STRUCTURE

What it tested
  Plan-writing is judgment-dense and pre-registered to stay at opus. How exactly does a cheaper planner fail?
How
  5 sonnet plans from the same design + guidance, scored against opus baselines.
Pre-registered prediction
  Sonnet plans miss constraints/interfaces fidelity.
What happened
  Fidelity held perfectly (constraints 5/5, gradient verbatim, interfaces on every task) — mechanical guidance adoption is model-robust. But 4 of 5 plans collapsed to 3 tasks (sonnet mean 3.6 vs opus 5.8): the planner chose coarser review gates. The judgment loss showed up in structure, not in copying.
So what
  Plan-writing stays at opus, with sharper evidence for why.

## E17Do plan word-budgets help or hurt?CONFIRMED HARMFUL

What it tested
  A length target might trim plan flab cheaply.
How
  5 plans with a 1,200-word prose budget (code explicitly exempted) vs 5 baseline.
Pre-registered prediction
  Test-assertion count falls before prose does.
What happened
  Severe: test signals fell 53.8 → 20.6 (−62%) and code fences 39 → 14 — even though baseline prose was already under the budget and code was exempted. The model read 'economy' as licence for austerity everywhere, cutting exactly the load-bearing content.
So what
  Never add length targets to writing-plans. Same mechanism as the earlier nuance-clause regression.

## E21Can reviewer reports lose their preamble?WIN −41%

What it tested
  Reviewer final messages narrate process before delivering verdicts. A 'your final message IS the report' contract might trim that.
How
  Micro-test on a real brief+report+diff fixture, 5 reps per arm, sonnet. First attempt had a harness flaw (the diff never inlined) — verdict retracted and re-measured clean.
Pre-registered prediction
  −10–20% reviewer output tokens, verdicts intact.
What happened
  −41% (1,160 → 687 tokens), spec verdicts 5/5 in both arms. Accidental bonus: the fixture mispaired Task 2's brief with Task 3's diff — and both arms correctly flagged the mismatch. Reviewers genuinely cross-check brief against diff.
So what
  Adopted into the E27 stack.

## E25Can reviewers work from the diff package alone?DANGEROUS — BRIEF IS LOAD-BEARING

What it tested
  Dropping the brief and report from reviewer inputs would simplify dispatches.
How
  Micro-test: same fixture with brief and report removed, 5 reps.
Pre-registered prediction
  Both needed; close the lead.
What happened
  Package-only reviewers delivered confident spec verdicts 5/5 — but silently redefined 'spec' to mean just the global constraints. Zero of five flagged that the brief was missing. The confidence survived; the basis didn't.
So what
  Keep all three inputs. Same failure family as cheap-reviewer rationalization: silent scope-shrinkage.

## E26Can a recipe shrink controller narration?WORKS, SMALL

What it tested
  46% of controller turns are thinking/narration. Message count is prompt-immune — but per-message length might not be.
How
  Micro-test of a mid-workflow controller moment with tools provided, with and without a one-line-narration recipe; opus, 5 reps each.
Pre-registered prediction
  Small win or nothing.
What happened
  162 → 74 narration chars per turn (−54%) with zero variance — and the healthy verify-the-implementer's-claims behavior survived inside the one line. Worth ~$0.1–0.3/run.
So what
  A freebie; adopted into the E27 stack.

### Full runs — where only real execution answers

## E01What do critical-sections-only plans cost to execute?REFUTED — CONFOUNDED

What it tested
  Jesse's question: what if plans carry only the critical code instead of everything inline?
How
  Generated critical-only plans (only 1 in 5 genuinely complied — opus resists withholding code), executed the genuine specimen twice.
Pre-registered prediction
  Between complete-code and prose: $9–12, 3–6 fix waves.
What happened
  $5.21/$5.11 — the cheapest runs ever measured. But the genuine specimen had also self-merged to 3 tasks: per-implementer effort did double exactly as predicted (35 turns vs 17–24), there were just fewer implementers. Task count swamped code policy; the price is coarser review gates.
So what
  Spawned E28 to isolate code policy at constant structure. Also: eliciting this regime at all is hard (1/5 compliance).

## E28Code policy at constant structureREFUTED — BODIES ARE MARGINAL

What it tested
  The clean version of E01: the same 7-task elicited plan with implementation bodies stripped to one-line descriptions — tests, interfaces, and structure untouched.
How
  Controlled fixture edit (10 bodies stripped, 19 test assertions kept), 2 runs.
Pre-registered prediction
  Costlier than $6.34–8.49; implementer effort doubles across 7 tasks.
What happened
  $7.78/$8.54 — inside the with-bodies range. Implementer turns rose only 7%: the test code pins the work; writing code to satisfy given tests is nearly as guided as transcribing. Fix waves ticked up (1 → 3–4).
So what
  The plan-content hierarchy, measured: tests + interfaces + structure are load-bearing; implementation bodies are ~40% of plan length for roughly nothing. A defensible writing-plans relaxation.

## E03Haiku implementers under complete-code plansVIABLE, SMALLER THAN HOPED

What it tested
  If implementation is transcription, the cheapest tier should manage it — with sonnet reviewers as the quality gate.
How
  Conditional guidance ('when the plan text contains the complete code…'), 2 runs on the elicited plan.
Pre-registered prediction
  Gates hold; −$2–3/run.
What happened
  $6.34/$7.33 vs sonnet implementers' $6.34/$8.49 — overlapping; the real saving is ~$0.5–1 because haiku's turn inflation (30 vs 17–24) ate half the price gap. Gates held; one classic haiku sloppiness (committed a build binary) was caught by review. One drift: the final review ran sonnet.
So what
  Worth keeping in the stack with the final-review tier pinned; not the big rock.

## E22Does the conditional tier guidance discriminate?CONFIRMED

What it tested
  The E03 wording licenses haiku only when the plan carries complete code. On a prose plan, does the controller correctly refuse?
How
  Same config, prose 10-task fixture, 1 run.
Pre-registered prediction
  Controller picks sonnet implementers; if it picks haiku anyway, expect a blowup.
What happened
  9 of 10 implementers went to sonnet — the one haiku was the genuinely mechanical task. $12.59, normal prose band, gates green.
So what
  The conditional wording is shippable: it gates on exactly the right condition.

## E05Narrower review-package context (-U3 vs -U10)NEUTRAL

What it tested
  Does shrinking diff context push reviewers back into reading files?
How
  review-package patched to -U3, 1 run.
Pre-registered prediction
  -U3 forces file reads back up; keep -U10.
What happened
  Reviewer profile byte-identical (5.0 turns / 3.0 tools), cost in-band. On small diffs the width doesn't matter — and the package is read once by sonnet, so there was no spend to recover anyway.
So what
  Keep -U10 as large-diff insurance. Lead closed.

## E06Cap the controller's thinking budgetBACKFIRED

What it tested
  Mining showed ~2/3 of controller output tokens are invisible (redacted thinking) — the single biggest controller spend. Cap it?
How
  MAX_THINKING_TOKENS=1024 injected into the launcher (verified in the binary first; careful contamination window), 2 runs.
Pre-registered prediction
  −15–25% controller cost; judgment must stay clean.
What happened
  Costs in-band ($7.15/$8.10) but controller messages rose 92 → 128/138 and output tokens nearly doubled (66k → 118–123k). Thinking buys turn efficiency; suppress it and the controller compensates with more steps and re-derivation. Gates held — it isn't harmful, just useless.
So what
  Do not cap thinking. The turn floor rises to meet you.

## E09Controller surveys the plan instead of reading itPARTIAL — HABIT WON

What it tested
  With briefs delivering tasks, the controller only needs headings + constraints: a task-brief --list mode.
How
  Implemented --list (smoke-tested on real plans), guidance added, 1 run.
Pre-registered prediction
  −$0.5–1/run of resident context.
What happened
  The controller ran --list once… then read plan.md twice anyway. Cost in-band, saving unrealized. The read-the-plan habit beat one guidance line.
So what
  Dropped for now — the ceiling (~$1) doesn't justify more runs; phrasing iteration is future micro-work.

## E24Fixture realism at svelte scaleCONFIRMED −24%

What it tested
  Fractals showed hand-written fixture plans execute ~2× costlier than real skill output. Does it replicate on the bigger scenario?
How
  Generated an elicited svelte plan — discovering first that the hand fixture had gold-plated an E2E suite the design never asked for; amended the design so scopes match. 2 runs incl. Playwright.
Pre-registered prediction
  $10–13 vs the hand fixture's $14.99–20.30.
What happened
  $12.40/$14.38 (mean $13.39) vs ~$17.6 — a −24% replication, milder than fractals' 2× because heavier intrinsic UI/E2E work dilutes plan effects.
So what
  Fixture-realism finding stands across scales. Also: baseline svelte numbers always included unrequested scope.

### The composition

## E27The integrated ship candidateCANDIDATE ESTABLISHED

What it tested
  Do the loop's wins compose? Opus controller + elicited plan + conditional haiku implementers + terse reviewer contract + narration recipe + final-review tier pin.
How
  Stack clone, 2 fractals runs + 3 planted-defect quality gates.
Pre-registered prediction
  All green; fractals ≤ $6.
What happened
  Fractals $6.24/$6.60 (combo on hand plans: $11.67–14.84). Quality gates 2 of 3 — the one fail was a clean reviewer miss on DRY plus a stricter judge reading; inspecting the reviewer's 2,620-char report exonerated the terse contract as the suppressor.
So what
  The candidate exists and costs half the shipped config. The N=5 gate battery is still owed before any of it becomes skill text.

### Backlog — registered, not run (the 25 minimum was reached without them)

E10 Report contract 15→8 lines · E12 Coarser step granularity · E13 Prompt-template minification · E16 Best-stack composition on svelte

Where it nets out: the ship-candidate stack runs go-fractals at $6.24–6.60 against the shipped config's $11.67–14.84 — but most of that gap belongs to executing real, complete plans rather than the hand-written prose fixtures all earlier numbers were measured on. The durable lessons: tests+interfaces+structure are what make a plan cheap to execute; thinking is load-bearing, don't cap it; word budgets eat tests; cheap models fail by advocating, not by going quiet; and half of all leads close as "the current design already had this right" — which is exactly what the log is for.