# Validation Matrix

Use the matrix to decide what to probe before writing scripts. The goal is not exhaustive combinatorics; the goal is high-value coverage that is visible, explainable, and likely to reveal runtime surprises. The matrix should make the real news easy to scan.

## Minimum Columns

Use these columns unless the task clearly needs more:

- `case_id`: Stable identifier such as `S1`, `E3`, or `R2`.
- `scenario`: Short description of the behavior under test.
- `mode`: `single-shot`, `repeat-N`, or `warm-up + repeat-N`.
- `question`: The concrete runtime uncertainty this case is answering.
- `setup`: Inputs, environment, or preconditions required for the case.
- `observation_summary`: A compact summary of what actually happened.
- `result_flag`: `unexpected`, `negative`, `expected`, or `blocked`.
- `evidence`: Path, log reference, or `deleted`.

Add these columns when they materially improve the investigation:

- `comparison_basis`: The baseline, docs, or prior behavior you are comparing against.
- `variable_under_test`: The single factor that is intentionally changing in a comparison case.
- `held_constant`: Prompt shape, tool setup, model settings, or state rules that were intentionally kept the same.
- `output_constraint`: Any schema, length, or response-shape constraint used to keep the comparison fair.
- `status`: Use `pass`, `fail`, `unexpected-pass`, `unexpected-fail`, or `blocked` only when there is a credible comparison basis or control.
- `confidence`: `high`, `medium`, or `low`.
- `state_setup`: Fresh or reused state, cache strategy, unique IDs, and cleanup checks.
- `repeats`: Number of measured runs.
- `warm_up`: Whether a warm-up run was used and why.
- `variance`: Any useful spread or instability note across repeated runs.
- `usage_note`: Token, usage, or output-length note when it materially affects interpretation.
- `control`: Known-good comparison point for regression or behavior-change questions.
- `risk_profile`: `read-only`, `mutating`, or `costly` for live probes.
- `env_vars`: Exact environment-variable names the case plans to read.
- `approval`: `not-needed`, `pending`, or `approved` for cases that need user permission before execution.

Use `result_flag` as the fast scan field. It is what makes unexpected or negative findings jump out before the reader studies the full report.

Use `status` only when you have a real comparison basis. If the case is exploratory and there is no trustworthy baseline, prefer a strong `observation_summary` plus `result_flag` and `confidence` instead of pretending the result is a clean pass or fail.

## Choosing Execution Mode

Pick an execution mode before you run the case:

- Use `single-shot` for deterministic, one-run checks.
- Use `repeat-N` automatically when the question involves cache behavior, retries, streaming, interruptions, rate limiting, concurrency, or other run-to-run-sensitive behavior.
- Use `warm-up + repeat-N` when the first run is likely to include cold-start effects such as container provisioning, import caches, or prompt-cache population.

Use these defaults unless the task clearly needs something else:

- `repeat-3` for a quick screen of a repeat-sensitive question.
- `warm-up + repeat-10` for decision-grade latency comparisons or release-facing recommendations.
- For costly live probes, start at `repeat-3`, then expand only if the answer is still unclear.

If it is genuinely unclear whether extra runs are worth the time or cost, ask the user before expanding the probe.

## Phase The Matrix

When the question is comparative or benchmark-like, do not jump straight to the largest matrix.

Start with a pilot:

- One control.
- One or two highest-signal success cases.
- The smallest repeat count that can disqualify a weak candidate quickly.

Expand only when:

- The candidate survives the pilot.
- The results are close enough that more samples matter.
- A major runtime surface is still uncovered.
- The user explicitly wants decision-grade evidence.

## Coverage Categories

Try to cover at least one case from each relevant category:

- `success`: Normal behavior that should work.
- `control`: Known-good comparison such as `origin/master`, the latest release, or the same request without the suspected option.
- `boundary`: Size, count, or parameter limits near a plausible edge.
- `invalid`: Bad inputs or unsupported combinations.
- `misconfig`: Missing key, wrong endpoint, bad permissions, or incompatible local setup.
- `transient`: Timeout, temporary server issue, network breakage, or rate limiting.
- `recovery`: Retry behavior, partial completion, duplicate submission, or cleanup.
- `concurrency`: Overlapping operations when shared state, ordering, or isolation may matter.
- `quality`: A harder or more open-ended sample when the user is asking about model intelligence, not just workflow parity.

If time is limited, prioritize categories in this order:

1. Known-good control when the question implies regression or drift.
2. Highest-risk success case.
3. Most plausible user-facing failure.
4. Most likely edge case with ambiguous behavior.
5. Cleanup or retry semantics.
6. Lower-probability extremes.

## Matrix Template

Use this compact template:

    | case_id | scenario | mode | question | setup | state_setup | variable_under_test | held_constant | comparison_basis | observation_summary | result_flag | status | evidence |
    | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- |
    | K1 | Known-good control | single-shot | Does the baseline still show the expected behavior? | Same probe against baseline target | Fresh state | none | current probe shape | `origin/master` or latest release | pending | pending | pending | pending |
    | S1 | Baseline success | single-shot | What does the normal success path look like at runtime? | Valid config and representative input | Fresh state | none | representative input and setup | current docs or local expectation | pending | pending | pending | pending |
    | R1 | Cache or retry behavior | warm-up + repeat-N | Does behavior change after the first run or across retries? | Same request repeated under controlled settings | Cache key or retry setup recorded | reuse versus fresh state | prompt shape and tool setup | same request without reuse, or docs if available | pending | pending | pending | pending |
    | C1 | Model comparison pilot | warm-up + repeat-N | Does candidate B preserve the covered behavior while improving latency? | Same scenario across two models | Fresh state and stable IDs | model name | prompt shape, tool choice, and model settings parity | control model in the same probe | pending | pending | pending | pending |
    | E1 | Invalid input | single-shot | How does the runtime reject a realistic bad input? | Missing required field | Fresh state | invalid field value | same request with valid field | same request with valid field | pending | pending | pending | pending |
    | X1 | Concurrent overlap | repeat-N | Do overlapping runs interfere with each other? | Two or more overlapping operations | Unique IDs plus cleanup verification | overlap timing | same logical input | same request serialized, if available | pending | pending | pending | pending |

## Recording Results

Keep `question` unchanged after execution. Put the actual behavior in `observation_summary`, then mark the scan-friendly `result_flag`.

Use these `result_flag` values consistently:

- `unexpected`: The result diverged from the best current understanding in a surprising way.
- `negative`: The result exposed a user-relevant failure, risk, or sharp edge.
- `expected`: The result matched the current understanding and did not reveal new risk.
- `blocked`: The case did not produce a trustworthy observation.

Only fill `status` when there is a credible comparison basis. Otherwise use `observation_summary`, `result_flag`, and `confidence` to communicate what was learned without over-claiming certainty.

For comparison cases, use `observation_summary` and the final report to say whether the evidence supports pattern parity only or a broader quality claim.

If a case reveals a new branch of behavior, add a follow-up case instead of overloading the original one.

## Evidence Discipline

Treat a case as incomplete when:

- The observed output omits the key result you were testing.
- The script mixed multiple questions and the result is ambiguous.
- Hidden state, cache behavior, or previous runs may have influenced the result and were not controlled or documented.
- The question is whether behavior changed, but the case has no credible control or baseline to compare against.
- The case plans to read environment variables, but the exact variable names were not approved by the user before execution.
- The case was repeat-sensitive, but it ran only once without a clear rationale.

When this happens, narrow the probe and rerun. A smaller script with a cleaner result is better than a more complicated script that is hard to trust.
