# Autoresearch: superpowers build-loop cost-efficiency
Constraint: opus stays the coordinator (no controller-tier change — L2 is out of scope per Jesse).
Quality invariant: planted-defect + end-to-end gates; any regression kills the change.
Method tiers: MINE (existing artifacts, free) / MICRO (API batch + control, $1-5) / FULL (quorum runs, $7-15 ea).
Every experiment pre-registers a prediction. Negative results get equal billing.
Durable results copy to evals docs/experiments/; this file is the working log.

| # | Hypothesis | Method | Prediction | Status | Verdict |
|---|---|---|---|---|---|
| E01 | Critical-sections-only plans execute worse than complete-code plans (more implementer turns, more fix waves) | FULL ×2 | between complete-code and prose: $9-12, 3-6 fix waves | done | **REFUTED (confounded)**: $5.21/$5.11, cheapest ever — but the genuine critical-only specimen also self-merged to 3 tasks; per-implementer effort DID double (35 turns vs 17-24). Task count swamped code policy. Gate-coarseness caveat. → E28 isolates code policy at constant structure |
| E02 | Interfaces blocks cut implementer exploration tool-calls (complete-code constant) | MINE elicited vs control runs | elicited implementers make fewer pre-edit reads | done | **REFUTED**: explores 1.9 vs 1.3-1.5 — no reduction; interfaces value = fix waves, not navigation. Bonus: transcription effect quantified (impl 17-24 turns complete-code vs 36.7 prose; explores 1.3-1.9 vs 5.6) |
| E03 | With complete-code plans, implementation is transcription → haiku implementers + sonnet reviewers hold all gates at much lower cost | FULL ×2 | gates hold, −$2-3/run | done | **VIABLE, SAVING BELOW PREDICTION**: $6.34/$7.33 vs sonnet-impl $6.34/$8.49 (means 6.84 vs 7.42 — overlapping, ~$0.5-1 real). Turn inflation 30.3 vs 17-24 ate half the price gap; 7/7 haiku, reviews sonnet, gates green; 1 sloppiness fix (committed binary) caught by review. Caveats: final-review tier drifted to sonnet; planted-defect gate not yet run on this config |
| E04 | Controller spends materially on reading implementer report files it doesn't need (briefs/reports designed to keep it out) | MINE | controller reads report files in <30% of tasks | done | **CONFIRMED (design works)**: 0-1 report reads per run across 5 runs; no saving available |
| E05 | Review-package context width: -U3 vs -U10 changes reviewer Read-escapes and verdict quality | FULL ×1 | -U3 forces file reads back up; keep -U10 | done | **NEUTRAL at small-diff scale**: $7.54 in-band, reviewer profile identical (5.0 turns/3.0 tools). No spend to recover (package read once by sonnet); keep -U10 as large-diff insurance |
| E06 | Controller MAX_THINKING_TOKENS reduction cuts the 46%-thinking turn floor cost without judgment loss | FULL ×2 | -15-25% controller $; judgment audit must stay clean | done | **BACKFIRED**: $7.15/$8.10 (in-band, no saving) but controller msgs 92→128/138 and output tokens 66k→118-123k — thinking buys turn efficiency; suppressing it raises the turn floor. Gates held both runs. Do not cap thinking. Companion finding: ~2/3 of controller output tokens are billed thinking invisible in transcripts |
| E07 | Implementer self-review step's catch-rate justifies its turns | MINE across runs | self-review catches something in >25% of tasks | done | **REFUTED-ish**: true catch rate ~1/87 (grep said 9/87; manual inspection killed 8 as template echoes). Keep anyway: one averted fix wave ≈ pays for the turns; insurance asymmetry. No change |
| E08 | Fix dispatches are mechanical post-review → haiku fixers with named tests hold re-review pass rate | MINE | holds; small saving | done | **CONFIRMED as-is**: 34 haiku fix dispatches across passing runs; controllers already tier fixes down organically; the one regression (cobra removal) was caught by the fix-report contract. No change needed |
| E09 | task-brief --list mode (controller reads headings+constraints, never full plan body) saves resident context | FULL ×1 | −$0.5-1/run controller side | done | **PARTIAL ADOPTION, saving unrealized**: controller ran --list once, then read plan.md 2× anyway ($7.02 in-band, no harm). Habit beat guidance; ceiling was ~$1. Drop for now; phrasing micro is future work |
| E10 | Report contract 15→8 lines loses nothing the controller acts on | MINE + MICRO | controller acts only on status/commits/concerns | backlog | not run — 25-experiment minimum reached; queued for a future loop |
| E11 | Plan generation by sonnet from same design degrades plan quality (judgment-dense; pre-registered to die) | MICRO 5v5 | sonnet plans miss constraints/interfaces fidelity | done | **PARTIAL — died on structure, not fidelity**: GC 5/5 + fid 3/3 + ifc 100% (mechanical adoption is model-robust), but 4/5 plans collapsed to 3 tasks (mean 3.6 vs opus 5.8) — coarser gates chosen by the planner. Plan-writing stays at opus |
| E12 | Step granularity: coarser steps (per-file not per-action) don't change execution cost/quality with complete code | MICRO + FULL ×1 | no measurable difference | backlog | not run — 25-experiment minimum reached; queued for a future loop |
| E13 | Subagent prompts' SKILL.md re-reads: implementer/reviewer templates can shrink 30% without behavior change | MICRO adherence battery | adherence equal | backlog | not run — 25-experiment minimum reached; queued for a future loop |
| E14 | Cache hit rates across dispatches are healthy; no ordering fix needed | MINE token usage | cache_read >> input; no win available | done | **CONFIRMED**: cache_read 7.7-30.7M vs 19-31k uncached; cost axis is resident-context SIZE (E09/E13), not hit rate |
| E15 | Reviewer named-risk budget already binds (reviews ~3 turns); further tightening saves nothing | MINE | no win available; close the lead | done | **CONFIRMED**: reviews 4.7-5.0 turns / 2.7-3.0 tools across complete-code runs (prose-plan runs: 10.1/6.8). Lead closed |
| E16 | Best-stack composition (elicited plan + winning pieces) reproduces additively on svelte | FULL ×2 | svelte < $13 | backlog | not run — 25-experiment minimum reached; queued for a future loop |
| E17 | Plan length economy instruction (~1,200-word target) hurts: drops test cases before prose | MICRO 5v5 | test-assertion count falls vs C-full baseline | done | **CONFIRMED, severe**: test signals 53.8→20.6 (−62%), code fences 39→14 — though prose was ALREADY under budget (1161 mean) and code was exempted. Budget licensed austerity broadly (nuance-clause mechanism). NEVER add length targets to writing-plans |
| E18 | Gauntlet QA-agent cost scales with coding-agent verbosity → ≤15-line final messages also cut QA cost | MINE across run pairs | QA tokens correlate with coding final-message volume | done | **REFUTED**: QA cost tracks DURATION (r=0.43, n=30) — it polls the screen. Faster runs cut QA cost automatically; message volume irrelevant |
| E19 | Controller TodoWrite + ledger double-bookkeeping: ledger-only loses nothing | MINE | todos redundant with ledger for ≤10 tasks | done | **PREMISE REFUTED**: 0 TodoWrite calls in measured runs — controllers already run ledger-only. No win available. Bonus: controller tool calls scale with task count (49 vs 74) |
| E20 | Implementer TDD evidence block (RED/GREEN transcripts in report) is load-bearing for reviewer verdicts | MINE | reviewers cite it; keep | done | **CONFIRMED**: 41% of reviewer transcripts reference the TDD evidence. Keep |
| E21 | One-shot final message contract for reviewers (no preamble) trims reviewer output tokens measurably | MICRO 5v5 | -10-20% reviewer output tokens, verdicts intact | done | **WIN (clean rerun)**: 1160→687 out-tokens (−41%), spec verdicts 5/5 both arms. Bonus: both arms caught a brief/diff mismatch the fixture accidentally contained — reviewers cross-check brief vs diff (supports E25). In E27 stack |
| E22 | Conditional-tier guidance discriminates: on a PROSE plan the controller refuses haiku implementers | FULL ×1 prose fixture | controller picks sonnet (gates correctly) | done | **CONFIRMED**: 9/10 implementers sonnet (1 haiku = the mechanical README-ish task); $12.59 in prose band; gates green. Conditional wording is shippable |
| E23 | Dispatch prompt assembly: controller re-derives context it could copy from ledger+interfaces (post-E02 quantification) | MINE | measurable but small | done | **FIXED ALREADY**: impl dispatch prompts 2.5k chars on elicited (4.6k prose; 42k pre-fix real sessions). Remaining mass = template skeleton → E13 |
| E24 | Scenario-fixture realism: skill-generated svelte plan executes ~2× cheaper than hand svelte fixture (replicates fractals methodology finding) | FULL ×2 | svelte $10-13 vs 14.99-20.30 | done | **CONFIRMED directionally**: $12.40/$14.38 (mean 13.39) vs hand fixture 14.99-20.30 (~17.6), scope-matched incl. Playwright — −24% (milder than fractals' 2×; heavier intrinsic work dilutes plan effects). Plus pre-finding: hand fixture gold-plated E2E beyond the design (baseline svelte cost included unrequested scope) |
| E25 | Reviewer reads brief+report+package: package alone suffices for quality verdict; brief needed only for spec verdict | MICRO 5 reps | both needed; close the lead | done | **BRIEF IS LOAD-BEARING (dangerously)**: package-only reviewers gave confident spec verdicts 5/5 but silently redefined 'spec' as the global constraints; 0/5 flagged the missing brief. Silent scope-shrinkage, same failure family as haiku rationalization. Keep all three inputs |
| E26 | Controller narration volume responds to a terse-bookkeeping recipe (composition, positive) | MICRO 5v5 | small win or nothing (floor is thinking, not text) | done | **WORKS, small**: 162→74 chars/turn (−54%, zero variance); verify-claims behavior preserved. ≈$0.1-0.3/run. Freebie for E27 stack |
| E27 | Opus controller + elicited plan + stack (conditional haiku impl, terse reviewer contract, narration recipe, final-review tier pin): the ship candidate | FULL ×2 + planted ×3 | all green; fractals ≤ $6 | done | **CANDIDATE ESTABLISHED**: fractals $6.24/$6.60 (vs combo-on-hand-plans $11.67-14.84; vs elicited baseline $6.34-8.49); planted-defect gates 2/3 — the fail forensically attributed to reviewer variance + judge strictness, terse contract exonerated by report inspection. N=5 gates still owed before any skill change ships |

| E28 | Code policy at constant structure: the 7-task elicited plan with implementation bodies stripped executes worse than the original | FULL ×2 | costlier than $6.34-8.49 (implementer effort doubles ×7 tasks) | done | **REFUTED**: $7.78/$8.54 — overlaps with-bodies range; impl turns +7% only (tests-as-code pin the work); fix waves 3-4 vs 1. Hierarchy established: tests+interfaces+structure load-bearing, bodies marginal (~40% plan length for ~nothing) |

## Final tally (2026-06-11)

25 experiments closed (+4 backlog). Wins: E21 terse reviewer contract (−41% reviewer output), E26 narration recipe (−54% narration, zero variance), E03/E22 conditional implementer tiering (~$0.5-1, discriminates correctly), E24 fixture realism replicated, E27 candidate stack $6.24-6.60 fractals. Refuted/dead: E06 thinking cap (BACKFIRES — raises turn floor), E17 plan word budgets (slash test content), E02 interfaces-as-navigation, E11 sonnet planner (structure judgment), E28 implementation bodies (marginal — tests+interfaces+structure carry plans). Leads closed as already-optimal: E04 report reads, E08 haiku fixers, E14 cache, E15 reviewer floor, E19 todos, E23 dispatch re-derivation. Risk findings: E25 package-only reviewers silently shrink spec scope. Methodology: every automated score manually inspected; three scoring/harness bugs caught and corrected in-flight (E07 grep, E21/E25 diff inlining, E25 scorer regex).
