# Test Plan: SDD Dispatch Structure Addition

## The Iron Law Applies

This is an EDIT to an existing skill. Same rule as new skills: **no change without a failing test first.** I must watch agents fail at dispatch-writing WITHOUT this addition before I can know whether the addition teaches the right thing.

## What kind of skill content is this?

This is a **technique** addition (how to structure a dispatch), but it has a discipline edge — agents under pressure tend to under-specify dispatches. So I test both: can they apply the structure, AND do they comply with it under pressure.

The real question the test must answer: **Does this 5-item structure actually fix a problem that exists in baseline dispatches, without introducing new failures?** If baseline dispatches are already fine, the addition is bloat and should be rejected.

---

## Method: RED-GREEN comparison with held-out evaluators

### RED Phase — Baseline (no addition)

**Setup:** Give subagents the realistic dispatcher's job: a plan with several tasks, brief files written, one task depending on interfaces from an earlier task, and one genuine ambiguity in the brief.

**Prompt the dispatcher subagent** to write the actual dispatch message it would send to an implementer subagent. Use the *current* SDD skill only.

**Then actually run the implementer** on that dispatch (no extra context) and have it attempt the task.

**Sample size:** 8–10 dispatcher runs across 3 different task scenarios (one with cross-task interface dependency, one with ambiguity, one "clean" task with neither). Vary the scenario so I learn which of the 5 items matters.

**Add pressure to a subset** (3 of the runs): time pressure ("the orchestrator is waiting"), prior context ("you already explained this earlier") — to see if dispatchers drop items they technically know they should include.

**Record verbatim:**
- Which of the 5 items appeared spontaneously
- Where implementers got stuck, asked questions, or built the wrong thing
- The dispatcher's reasoning for omissions

If baseline dispatches already include all 5 items and implementers succeed, **STOP — the addition has no failing test and should not be deployed.**

### GREEN Phase — With addition

Re-run the identical scenarios with the addition in the skill. Same sample size, same scenarios, same pressure subset.

---

## Controls

- **Same scenarios both phases** — only the skill text differs (the independent variable).
- **Blind scoring:** strip phase labels from transcripts before scoring so I don't favor GREEN.
- **Implementer is a fresh subagent** each run with no shared memory — measures whether the dispatch alone is sufficient.
- **Counterbalance scenario order** to avoid ordering effects.
- **One held-out scenario** never used during iteration, scored only at the end, to guard against me tuning the skill to the test set.

## Scoring

**Per dispatch (process):** binary presence of each of the 5 items → 0–5 score. Also flag *malformed* inclusions (e.g., brief introduced but not as "your requirements").

**Per implementer outcome (the outcome that actually matters):**
- Did it build the right thing without clarifying questions? (yes/no)
- Did it correctly use the earlier-task interface? (yes/no/n.a.)
- Did it resolve ambiguity the way the brief intended? (yes/no/n.a.)
- Off-track work products (count)

Process score is a leading indicator; **implementer outcome is the real test.** A dispatch can hit all 5 items and still fail — I weight outcome.

## Acceptance Criteria

Deploy only if ALL hold:

1. **Failing baseline confirmed:** baseline implementers failed (questions asked / wrong build / interface misuse) in a clear majority of runs — i.e., a real problem exists.
2. **Outcome improvement:** GREEN implementer success rate meaningfully exceeds baseline on the interface and ambiguity scenarios, including the held-out scenario.
3. **No regression on clean tasks:** the "clean" scenario shows GREEN dispatches aren't padded with empty/forced items (no items #3 or #4 invented when n.a.).
4. **Compliance under pressure:** pressure-subset GREEN dispatches retain the items; if dispatchers still drop them, the addition needs explicit "include even if you covered it earlier" language → REFACTOR, not deploy.

## REFACTOR triggers

- If agents force items #3/#4 when there's no dependency/ambiguity → add "omit if not applicable" guidance and re-test.
- If "read this first — it is your requirements" framing gets paraphrased into something weaker → tighten and re-test.
- New rationalizations for omission → rationalization table, re-test until the held-out scenario passes.

**Bottom line:** I won't deploy on the theory that more structure is better. I deploy only after a baseline failure proves the problem is real and the held-out GREEN run proves the addition fixes it without padding clean dispatches.