# Benchmark Evidence Policy

Every number published in `benchmarks/index.json` (and therefore in the
generated README tables) must be traceable to an evidence document under
`docs/`. This policy is enforced by `scripts/validate_benchmark_index.py` and
the release gate.

## Rules

1. **One source of truth.** The README benchmark tables are generated by
   `scripts/render_benchmark_tables.py` from `benchmarks/index.json`. Hand-edited
   numbers in the READMEs are drift and fail CI.

2. **Evidence per row.** Every entry must reference an existing
   `evidence_document` under `docs/` that states how the numbers were produced
   (dataset, pages, machine, backend, scoring mode, aggregation mode).

3. **No fabricated precision.** Display values must equal raw `metric_result.json`
   values × 100 under the same aggregation convention (pooled vs page-level are
   never conflated). The validator rejects mismatches.

4. **Failure accounting.** `failed_pages == known_failures + unknown_failures`
   when all are reported. A row may only count a page as a *known* failure when
   the page is on the profile's `allowed_failed_page_stems` allowlist.

5. **Evidence levels.** Each row carries a `run_type`:

   - `smoke` — 10-page acceptance evidence only; never quoted as a benchmark.
   - `resumed` — a full run resumed from the repo's own artifacts with
     fingerprint + provenance verification ("validated resumed run"). This is
      the current status of the 2026-08-06 full-set row (and, earlier, the
      2026-08-03 row).
   - `clean-room` — a full 1651-page run executed from a fresh checkout of the
     release commit; requires the Release Gate and a committed evidence doc.
   - `independent` — a clean-room run executed by a second machine.

   A row may only be *upgraded* when the corresponding evidence exists; until
   then it stays at the lower level.

6. **No benchmark-claim laundering.** Raw metric values are never edited to
   "look consistent"; when a run differs from a previous run, both numbers stay
   in the index with their own evidence rows and dates.

7. **Hashes.** Clean-room and independent rows must record
   `prediction_tree_hash` (and, when available, `profile_hash` and
   `result_hashes`); the validator rejects unverifiable claims.
