# Cost-pathologies eval campaign — design

Date: 2026-07-31
Status: drafted for Jesse's review; scope directives given in-session
(focus on cost, not wall-clock; multiple competing hypotheses per
problem; mechanical check for the two corpus-negative pathologies)
Budget: ~$580 remaining of the original $1000 envelope
Evidence base: three-host session mining, 2026-07-17..31 window, reports
in `_tmp/cost-pathologies-2026-07-31/` (local, remote-host-a,
remote-host-b — NEVER committed; aggregates only cross the line), plus
the 2026-07-28 audit and the fix-cycle's own 89-run corpus.

## Goal

Reduce the dollar cost of agent-built software by attacking the measured
cost sinks with competing candidate mechanisms, graded head-to-head by
pre-registered criteria. Wall-clock is explicitly out of scope this
campaign. Every experiment runs MULTIPLE hypotheses as parallel arms —
one-anointed-fix designs are a process failure here.

## Measured cost picture (what the mining established)

1. **Fork-tax / context inheritance**: 76.6% of one host's 11.9B codex
   tokens were full-history child forks replaying parent context
   (sqlite-verified); corroborated independently at 113B cumulative /
   99.2% duplicate content on another corpus. Structural, platform-level.
   The fix cycle's shipped dispatch guidance attacks the SDD slice;
   everything else still forks dirty.
2. **Review loops**: the dominant seat-count pathology on every corpus —
   ≥6 chains with 40+ rounds (9 reviewers on a 4-task plan); a
   32-dispatch unattended overnight chain (19 re-reviews of one file
   against a stale report, 17% clean rate); 65:30 review:impl ratio; a
   9-parallel-review wave. Reviewers do not run dry; convergence must be
   policy.
3. **Verification duplication**: 148 test invocations on one tree, one
   regression suite run 12×.
4. **Micro-dispatch overhead**: persona-style micro-dispatches paying a
   40–560K-token context floor each for tiny tasks.
5. **Invented requirements**: real, modest — 3 root-caused exemplars + 1
   reviewer finding later WITHDRAWN.
6. **Negative results (3 corpora, both harnesses)**: sidequests and
   advisory-detail escalation near-zero AS OPERATIONALIZED — with one
   mechanical check never run (M0 below) and interactive sessions
   undersampled. Jesse reports seeing both; the campaign treats the
   perception gap as a measurement question, not a dismissal.
7. **Out of campaign scope, flagged**: never-reset persistent-agent main
   loops (5.3B tokens / ~80× dominance on one host) — PA infrastructure,
   not superpowers skill text.

## Standing rules (carried from prior campaigns)

Append-only hypothesis log (`logs/2026-07-31-cost-pathologies.md`);
pre-registration before every battery; corpus-validated scorers with
manually inspected matches (non-circular); discrimination rule
(inconclusive-by-zero is a stop, and a control arm that won't fail
stops the battery); raw sessions never committed; correctness rides
beside cost in every verdict (a cheaper arm that parks defects loses);
privacy sweeps with disclosure on every commit; scrubbed hostnames in
everything published.

## M0. Mechanical closure of the negative results (pre-experiment)

Before any treatment work: (a) the sidequest check the mining never
finished — for every SDD-shaped session on all three hosts, diff the
files actually committed against the brief/task-declared file set;
out-of-scope commit ratio is the sidequest rate, with exemplars pulled
for anything >0. (b) Advisory-stall elicitation moves into X2's baseline
(seeded fixtures), since the field corpora cannot falsify it. If both
come back null again, the taxonomy entries close honestly and X2's
contract text is graded purely as a guardrail (must-not-harm), not a
fix.

## The experiments — competing hypotheses per problem

### X1. Review convergence policy (the dominant sink)

Problem: review chains that never converge because strong reviewers
always find something. Arms (each a distinct mechanism, run
head-to-head on identical fixtures):

- **X1-A criterion-backing**: a finding may block only if it cites the
  violated requirement (brief/spec/global-constraint line) or a
  concrete reachable failure path; everything else is advisory and goes
  to the ledger.
- **X1-B rising severity floor**: round N of a fix loop accepts only
  findings at or above a per-round severity floor (round 1: any
  Critical/Important; round 3+: Critical only).
- **X1-C marginal-value verdict**: every review ends with an explicit
  "would another round materially reduce risk? yes/no + what is at
  stake"; the controller stops on "no" regardless of finding count.
- **X1-D control**: current SDD policy (5-round cap + one fix wave) as
  shipped by the fix cycle.
- **X1-AB combined** if A and B both beat control independently.

Grading: rounds-to-terminal, novel-finding rate per round, cost per
task, AND defect-escape rate on fixtures with seeded known defects (the
E5 machinery) — an arm that converges fast by missing real bugs fails.
Tiers: MINE (chain-length / novel-finding-rate / severity-trend scorers
validated on the mined corpora) → MICRO (reviewer-prompt variants on a
fixed diff with known seeded-bug density; measures finding calibration
and stop-recommendation honesty per arm) → FULL (SDD runs on a buggy
fixture per arm).

### X2. Advisory plan details (Jesse's directive)

Problem statement (directive): plans carry filenames, identifiers, and
code to HELP the implementer; agents treating them as binding get stuck,
escalate, or contort. Corpus evidence thin → the fixture must elicit the
baseline. Arms:

- **X2-A plan-header contract**: writing-plans emits (and SDD briefs
  carry) an explicit "details are advisory; the spec is binding;
  deviations that preserve spec compliance are the implementer's call,
  noted in the report" block.
- **X2-B reviewer-lens fix only**: task-reviewer constraints text told
  to judge spec compliance, never plan-transcription fidelity.
- **X2-C implementer-empowerment only**: implementer template grants the
  deviation authority; plan and reviewer untouched.
- **X2-D control**: current text.

Fixture: an SDD plan seeded with advisory mismatches (a named file that
does not exist, a code snippet referencing a renamed identifier, a
stale signature) PLUS one genuine spec conflict. Criteria: advisory
mismatches produce zero BLOCKED/NEEDS_CONTEXT/human escalations and
zero reviewer findings, spec compliance preserved, AND the genuine
conflict still escalates (the discrimination guard — an arm that stops
ALL escalation fails).

### X3. Source-cited requirements

Problem: requirements invented from plausibility (pixel-equivalence,
1GB disk, AIX support — none requested). Arms:

- **X3-A reviewer citation rule**: every requirement a finding invokes
  carries its source line; uncited → suggestion, not defect.
- **X3-B implementer requirements inventory**: the report lists every
  requirement implemented with its source; unsourced entries are
  flagged by the controller before review.
- **X3-C adjudication default**: controller-side rule — uncited
  findings are ledgered as suggestions without a fix round.
- **X3-D control**.

Tiers: MICRO (baitable fixtures — an obvious "improvement" nobody
asked for) → FULL rider on X1's batteries (same runs, different
scorer). Grading: invented-requirement rate, plus a guard that real
spec requirements still get enforced.

### X4. Fork-tax field verification + universal isolation

Measurement study, not a text treatment (the treatment shipped with the
fix cycle): (a) instrument the mined corpora's signature (child bytes ÷
parent bytes; inherited-prefix duplicate ratio) into a reusable scorer;
(b) once the open PRs merge and field sessions accumulate, measure the
before/after on fresh trees; (c) audit every other skill that spawns
(dispatching-parallel-agents first) for full-history forks and propose
the isolation default there as a follow-up treatment if the audit finds
dirty forks. Cost: near-zero (scorer work + field data that accrues on
its own).

### X5. Verification leases (phase-2 item, now properly baselined)

Problem: identical suites re-run at identical tree state (12× worst
case). Arms:

- **X5-A receipts in report**: implementer report carries
  command+tree-SHA+result; reviewers and finishing honor receipts at
  unchanged SHA.
- **X5-B machine-checkable receipt file**: same, but as a workspace
  artifact the next seat verifies mechanically.
- **X5-C control**.

Prerequisite scorer work: the substring-aware duplicate-command counter
(the known `score_e3` upgrade). Guard: the invalidation probe — mutate
the tree mid-flow and the suite MUST re-run (an arm that skips a
required re-run fails regardless of savings).

### X6. Micro-dispatch floor

Problem: dispatching an agent costs a context floor regardless of task
size. Arms:

- **X6-A batching rule**: N small same-shape edits go to ONE subagent
  as a batch brief.
- **X6-B inline-when-trivial**: controller text — work that fits in a
  couple of tool calls is done inline, not dispatched (with an explicit
  boundary so it cannot swallow real tasks).
- **X6-C control**.

Tiers: MINE (dispatch-floor distribution from the mined corpora) →
FULL (fixture with a dozen small edits, cost per completed edit per
arm; quality guard: all edits reviewed and correct).

## Ordering and budget

1. M0 mechanical check (mining follow-up; ~$0 API, subagent time only)
2. X1 MINE + MICRO (~$30) → X1 FULL (~$150, the priority battery)
3. X2 fixture + baseline + arms (~$80)
4. X3 MICRO + rider (~$15)
5. X5 scorer upgrade + battery (~$60)
6. X6 MINE + battery (~$40)
7. X4 instrumentation (rides along; field data accrues post-merge)

Estimated total ≈ $375, reserve ≈ $200 for re-runs and surprises.

## Success criteria

- Every experiment ships either (a) a winning arm with pre-registered
  criteria met and correctness guards intact, or (b) an honest
  negative/inconclusive verdict in the log.
- Scorers validated against the mined corpora before grading any fresh
  run.
- No treatment ships text that a battery did not run (the fix cycle's
  433184c lesson, now a standing rule).
- Cross-arm comparisons reported even where all arms pass — the point
  is which mechanism wins, not whether any works.

## Out of scope

- Wall-clock/parallelism work (deferred portfolio exists).
- PA persistent-session hygiene (separate track, flagged to Jesse).
- RoboRev, token telemetry (still separate codebases).
- Adopting Drew's stack text (his PR evals proceed independently).

---

## Amendment 1 (2026-07-31): Drew's donated session — verified findings

A donated SDD session (verified against its rollout, analysis in the
_tmp corpus) adds two experiments, one arm, and two baseline
corrections:

**Verified narrative, with corrections:** (1) the task-sequencing
conflict WAS derivable from plan text, and the orchestrator declared
"preflight conflict scan is clean" 90 seconds before the Task 1
implementer hit it — the scan ran and missed it (or ran performatively);
(2) the contract conflict lived ~80 lines apart in ONE brief — a clean
compliance miss, not a capability gap; (3) the second-fix-wave approval
was legitimate (a cross-task regression only a live full-verification
run could find). New finding the donor didn't flag: a standing 8-hour-old
"fix it, then proceed" approval was reused to license an unrelated
substantive design decision without re-asking — approval-scope creep.

**Cost corrections that reshape priorities:** human-approval interrupts
cost ~zero tokens (the root thread goes fully dormant; the cost is
wall-clock and human attention — 8h48m, 2m, 24m here). The token driver
was the full gate suite restarting three times in under an hour (1.16M →
2.61M → 4.37M tokens) — prime X5 baseline material. And the "extra" fix
wave was CHEAPER than the routine first wave (3.98M vs 5.53M tokens);
the cap's real cost is the interrupt, and shipping the regression would
cost more than either.

### X7. Preflight that produces evidence (new)

Compliance failure, not capability gap → arms attack the compliance
mechanism:
- **X7-A evidence-bearing scan**: preflight must OUTPUT its pairwise
  checks (per task pair: interfaces consumed/produced compared; per
  brief: internal test-vs-code consistency) — "scan is clean" without
  the table is non-compliant.
- **X7-B mechanical consistency check**: a script over the plan's
  Files:/Interfaces: blocks (delete-vs-consume collisions, identifier
  mismatches between test and pseudocode blocks) run at sdd-workspace
  setup; prose scan only for what the script cannot see.
- **X7-C control**: current "scan the plan once" prose.
Fixture: plans seeded with (a) a delete-vs-need sequencing conflict,
(b) an intra-brief contract mismatch, (c) NO conflict (false-positive
guard — an arm that blocks clean plans on invented conflicts fails).
Criteria: seeded conflicts surfaced BEFORE Task 1 dispatch; clean plan
proceeds uninterrupted.

### X8. Approval scope (new, small)

Pathology: standing approvals reused for new substantive decisions.
Arms: (A) scope-bound approvals text (an approval covers the decision it
answered; new substantive decisions re-ask — mirrors the maintainer's
own explicit-confirmation norm); (B) approval-ledger discipline (each
approval logged with its scope; dispatches cite the covering entry);
(C) control. Fixture: a session shaped to tempt reuse (early broad
"yes, fix it" + later unrelated design fork). Guard: routine mechanical
decisions must NOT start re-asking (interrupt count on clean flows is a
criterion, not just reuse count).

### X1 addendum — the one-wave-cap arm set

Round-2 evidence: wave 2 cheaper than wave 1; interrupt is the cost.
New competing arms inside X1: (E) auto-second-wave permitted ONLY when
full verification (not a review) finds a regression, with scope shrunk
to the regression; (F) status quo (cap + human exception); (G) hard cap
(route residuals to finishing gates). Graded on regression escape,
human-interrupt count, and cost.

### X5 baseline upgrade

The 3-restarts-in-an-hour gate-suite pattern (escalating token cost per
restart) joins the 12× duplicate-regression baseline as X5's primary
targets; "by design" today, which is exactly what a lease would change.

---

## Amendment 2 (2026-07-31): SDD never stalls on a human (Jesse directive)

Principle, verbatim intent: subagent-driven development should never
stall out waiting for a human unless there is a catastrophic failure.
The donated session's 8h48m dormant overnight wait is the anti-pattern:
the interrupt cost ~zero tokens but a full working day of wall-clock and
human attention. Human input to a running SDD session is
ASYNC-ADVISORY, not blocking.

**Proposed catastrophic boundary (the only legitimate blocking stops):**
irreversible/destructive operations; security-sensitive actions;
external side effects under explicit-permission norms (merges, pushes to
shared branches, publishing); and plan defects so total that every
forward path is speculative. Everything else — plan conflicts, contract
ambiguities, cap exceptions, spec-vs-plan tensions — gets a controller
RULING: decide using the spec as the binding authority, record the
ruling and its reasoning in the ledger, proceed, and surface all rulings
prominently at session end (plus async notification where available) so
the human can override with rework rather than being required for
progress. Wrong-ruling rework is the accepted price; it is bounded and
measurable, while stalls are unbounded.

**Effects on the portfolio:**
- **X9 (new): non-blocking adjudication.** Arms: (A) rule-log-proceed
  (controller adjudicates non-catastrophic conflicts with recorded
  rulings; end-of-session rulings report); (B) rule-log-proceed + async
  surfacing (notification/comment at ruling time — human CAN intervene,
  never must); (C) control (current ask-and-wait text). Fixture: the
  donated session's conflict shapes re-seeded (sequencing conflict,
  contract conflict, cap exception) PLUS one seeded catastrophic case
  (e.g., a step requiring a destructive migration on shared state).
  Criteria: zero blocking waits on the non-catastrophic seeds, the
  catastrophic seed STILL stops (discrimination guard), rulings all
  ledgered and surfaced, and total cost (including any wrong-ruling
  rework) vs control.
- **X7 reframed:** preflight conflicts found by any arm produce
  RULINGS + ledger entries by default, not a blocking batched question;
  the current skill text's "present as one batched question" becomes a
  control-arm behavior. Evidence output (X7-A/B) doubles as the ruling's
  audit trail.
- **X8 reframed:** the anti-reuse arms surface new substantive decisions
  as recorded rulings with async notification — never blocking re-asks.
  The guard flips accordingly: the catastrophic class must still block;
  clean flows must not gain interrupts.
- **X1 wave-cap arms:** X1-E (auto-second-wave on verification-found
  regressions, scoped, logged) is now the directive-aligned default
  candidate; X1-F (human-exception) demotes to control.

Tension noted for the record: the maintainer's global collaboration
norms favor stop-and-ask in interactive work. This directive is scoped
to autonomous SDD execution, where the human has already delegated the
plan; the catastrophic boundary and the explicit-permission carve-outs
(merges/pushes) preserve the norms that must survive.

---

## Amendment 3 (2026-08-01): X5-A's honoring channel under codex (queue-campaign item 17)

The closed campaign's Task 11 battery (C1 correction,
`logs/2026-07-31-cost-pathologies.md`) found X5-A's receipt-issuing half
real and correctly formed but its honoring/invalidation half
structurally invisible under codex: the design's own text routes the
`LEASE-HONORED:`/`LEASE-INVALIDATED:` line through "the reviewer's
dispatch itself" — codex's inter-agent dispatch payload, encrypted at
rest (`"type": "encrypted_content"`; the same M0-documented harness fact
that left the sidequest scope check recoverable for only 17–24% of codex
work). X5-B's honoring/invalidation, routed instead through a plain
`custom_tool_call_output` (its receipts file read back via `cat`), was
hand-verified firing 3/3 in the same battery — proof the marker survives
codex's channel, just not the one X5-A currently uses.

**Mechanism.** X5-A's honoring/invalidating seat (a reviewer or
fix-round dispatch declining to re-run a command because an existing
receipt already covers it at the current tree_sha) additionally echoes
its `LEASE-HONORED:`/`LEASE-INVALIDATED:` line through a plain shell
command — e.g. `echo "LEASE-HONORED: command=<command_norm>
tree_sha=<sha>"` — so the line lands in the rollout as an ordinary
tool-call OUTPUT, not only inside the dispatch prompt's own encrypted
text. This is the same channel category X5-B's design already routes
its receipts-file reads through, and it is not speculative:
`score_x5_leases.py`'s `_text_sources()` already scans it
(`_collect_call_outputs()`, plus, per queue-campaign Task 2/item 10, the
exec CALL text itself) — the scorer needs no change.

**Why this doesn't change the treatment's behavior-shaping semantics.**
X5-A's actual decision — the controller recognizing an unchanged
tree_sha and skipping a redundant verification run — is untouched. The
echo is one additional, read-only shell command alongside the existing
dispatch-prose citation, not a replacement for it and not a new decision
point. X5-A still differs from X5-B in its primary design (receipts
carried in the report/dispatch prose, not a dedicated per-task file);
this amendment only makes that same prose-carried decision additionally
visible through a channel codex does not encrypt.

**What the scorer picks up.** The echoed line matches `_LEASE_LINE_RE`
directly (the strict grammar), so `_lease_events()` counts it as a real,
distinct `(kind, command_norm, tree_sha)` event — `receipts_honored`/
`invalidation_reruns` become non-zero under codex without relying on the
deliberately conservative prose detector (`_lease_events_prose()`, known
to miss 1 of 8 real exchanges by design). No battery runs this campaign;
this is a design amendment on record for whoever revisits X5-A next.
