# Test Plan: Verifying Dispatch Format Addition to SDD Skill

## The Iron Law Applies Here

This is an EDIT to an existing skill. Same rule: **NO SKILL CHANGE WITHOUT A FAILING TEST FIRST.** I must establish baseline behavior BEFORE the change exists, then verify the change fixes the specific failures. If I can't demonstrate a baseline failure, the change isn't justified — I'd delete the draft and reconsider.

This is a technique skill (how-to: constructing a dispatch), so I test for **correct application**, not pressure resistance.

---

## What I'm Actually Testing

The draft makes a falsifiable claim: dispatching subagents WITHOUT this 5-part structure produces worse task execution than WITH it. The test must isolate that claim. Specifically I want to catch the failure modes the addition implies:

- Subagent doesn't treat the brief as authoritative (rewrites/second-guesses requirements)
- Subagent re-derives or violates interfaces from prior tasks it couldn't know
- Subagent re-litigates already-resolved ambiguity, or resolves it differently
- Subagent doesn't know where it fits / writes report to wrong place

---

## Method: RED → GREEN → REFACTOR with subagents

### RED Phase — Baseline (change NOT present)

Run the *current* SDD dispatch guidance against real-ish tasks and record verbatim what dispatching agents produce and what executing subagents do.

**Two-layer test**, because this addition affects two roles:
- **Layer A (authoring):** Does the dispatching agent, given a task + brief, write a dispatch that conveys fit/authority/interfaces/ambiguity/report-path *on its own*?
- **Layer B (execution):** Given a baseline dispatch, does the executing subagent fail in the predicted ways?

**Sample sizes:**
- 6 distinct task scenarios (varied so results aren't artifacts of one task). Each must contain at least: a dependency on a prior task's interface, and one genuine ambiguity in the brief.
- Run each scenario through **5 fresh subagents** per layer = 30 baseline authoring runs + 30 baseline execution runs.
- Fresh context every run. No memory of the skill discussion.

**Controls:**
- Identical task/brief inputs across RED and GREEN.
- Same scenario set, same scoring rubric, same scorer.
- Scenarios written before seeing the draft's wording, so I don't accidentally tailor tasks to the draft's strengths.
- Hold the rest of the SDD skill constant — change ONLY the dispatch section.

**If baseline doesn't fail:** The addition solves a non-problem. Stop. Don't deploy.

### GREEN Phase — With the change

Insert the draft. Re-run the **same 6 scenarios × 5 subagents** for both layers (30 + 30). Same scorer, same rubric, blind to which condition each transcript came from (label transcripts RED/GREEN only after scoring).

---

## Scoring

Binary-per-criterion rubric, scored per transcript. Each of the 5 dispatch elements maps to an observable outcome:

| # | Element | Layer A score (authoring) | Layer B score (execution) |
|---|---------|---------------------------|---------------------------|
| 1 | Task fit | Dispatch states where task fits | Subagent stays in scope, doesn't expand |
| 2 | Brief as requirements | Brief introduced as authoritative | Subagent treats brief as spec, doesn't rewrite reqs |
| 3 | Interfaces from prior tasks | Dispatch includes them | Subagent matches interface, no re-derivation |
| 4 | Ambiguity resolution | Dispatch states resolution | Subagent uses it, doesn't re-litigate or diverge |
| 5 | Report-file path | Path present | Report written to correct path |

Score = count of criteria met (0–5) per transcript. Track per-criterion pass rates, not just totals — I need to know *which* of the 5 elements actually moves the needle.

**Scoring integrity:** Transcripts shuffled and stripped of condition labels before scoring. A second scorer rates a 20% sample to check rubric reliability; disagreements force rubric clarification before final scoring.

---

## Acceptance Criteria

Deploy ONLY if all hold:

1. **Baseline failed:** Per-criterion baseline pass rate < 60% on at least the criteria the addition targets (otherwise no problem to solve).
2. **Material improvement:** GREEN execution-layer mean ≥ 4/5 AND each individual criterion's pass rate ≥ 80%.
3. **No regression:** No criterion drops vs. baseline. No new failure mode introduced (e.g., dispatches becoming bloated/rote, subagents ignoring the brief because the dispatch is now too long).
4. **Authoring is reproducible:** Layer A pass rate ≥ 80% — i.e., agents reliably produce the 5-part dispatch from the guidance, not just when spoon-fed.

If criterion 4 fails but execution improves, the *format* helps but the *instruction* doesn't teach it — REFACTOR the wording, don't deploy yet.

---

## REFACTOR Phase

Capture every new failure verbatim from GREEN transcripts:
- Did "introduced as 'read this first — it is your requirements'" actually stop brief-rewriting? If subagents still second-guessed, the phrasing is too soft → strengthen.
- Did any subagent treat element (4) "your resolution" as optional/advisory?
- Did dispatches get padded with low-value "fit" sentences that crowded out interfaces?

Address each, re-run affected scenarios, repeat until acceptance criteria hold with no new holes.

---

## Why this shape

- **Two layers** because the addition is guidance for dispatch *authors* but the payoff is in *executor* behavior — testing only one would leave half the claim unverified.
- **6×5 = 30 per cell** gives enough runs to see per-criterion pass-rate differences without false confidence from n=1.
- **Blind scoring + held-constant inputs** are the controls that keep this from being "I wrote it, it reads well, ship it."

The total is ~120 subagent runs across baseline and change. That's the cost of not deploying untested process documentation.