// SPDX-License-Identifier: CC-BY-SA-4.0
// Copyright (c) 2026 Jonathan D.A. Jewell (hyperpolymath) <j.d.a.jewell@open.ac.uk>
= Hyperpolymath Testing & Benchmarking Taxonomy
Jonathan D.A. Jewell <j.d.a.jewell@open.ac.uk>
v1.0.0, 2026-04-03
:toc: macro
:toclevels: 4
:icons: font
:source-highlighter: rouge
:experimental:

[abstract]
--
Canonical specification for all test categories, aspect dimensions, proof systems,
and benchmark classifications across the hyperpolymath estate. This document is
the single source of truth — all per-repo TEST-NEEDS.md files, CRG assessments,
gitbot enforcement, Hypatia rules, and PanLL dashboards reference this taxonomy.
--

toc::[]

== Status

[cols="1,3"]
|===
| Version | 1.0.0
| Date | 2026-04-03
| CRG Grade | D (alpha — this document itself)
| Enforcement | Manual until gitbot-fleet integration (v1.1)
|===

== Scope

This taxonomy applies to every assessable component in the hyperpolymath estate.
A component must declare which categories apply and demonstrate evidence for each.
Categories that do not apply must be explicitly marked N/A with justification.


=== Provenance: draw from `proven-tests-and-benches` FIRST

IMPORTANT: Before writing a new test or benchmark for any category or aspect
below, look for an existing one in the *Idris2 suite* at
https://github.com/hyperpolymath/proven-tests-and-benches[`hyperpolymath/proven-tests-and-benches`].
That repository is the *first* place to draw from — ahead of this document's
examples, ahead of a sibling repo, and ahead of writing something new.

Why it ranks first:

* The suite is *proven*, not merely present. An Idris2 test that type-checks has
  discharged its obligation in the type system, so it cannot silently degrade
  into a test that passes without asserting anything.
* Copying a test is how a shared expectation stays shared. Writing a fresh one
  per repo is how 16 subtly different definitions of "the same" test appear —
  the one-template-copied-N-times failure this estate keeps paying for.
* It gives a category a single reference answer, so "does this repo satisfy
  category 3?" has one meaning estate-wide.

Order of preference when instantiating any category in Part I, any aspect in
Part II, or any benchmark in Part IV:

. `proven-tests-and-benches` — the Idris2 suite. Adapt it.
. The reference implementation in Appendix C.
. A sibling repo already assessed at CRG C or better.
. Write a new one — and contribute it *back* to `proven-tests-and-benches` so
  the next repo inherits it rather than reinventing it.

CAUTION: A test written to make a gate go green, rather than to establish a
fact, is worse than no test — it is read as evidence. If no honest test exists
for a category yet, mark it N/A with justification, as Scope requires above.
That is a truthful state; a passing-but-vacuous test is not.

== Part I: Test Categories

=== 1. Unit Tests

**Definition:** Single-function or single-module tests with dependencies mocked or stubbed.

**Purpose:** Verify that individual units of logic are correct in isolation.

**Tools:** Framework-native (`cargo test`, `mix test`, `deno test`, `gleam test`, etc.)

**Coverage expectation:** All public APIs, all error branches.

**Applies to:** Every repo with source code.

=== 2. Point-to-Point (P2P) Tests

**Definition:** Multi-component tests verifying correct interaction across a single seam
(e.g., caller→callee, client→server, FFI boundary, IPC channel).

**Purpose:** Verify that two components communicate correctly when connected.

**Tools:** Integration test suites, mock servers, test doubles at the far end only.

**Coverage expectation:** All significant data flow paths across seams.

**Applies to:** Any repo with inter-component communication.

=== 3. End-to-End (E2E) Tests

**Definition:** Full-pipeline tests from user-facing entry point to final observable output,
exercising the real stack (no mocks except external services).

**Purpose:** Verify that the system works as a user would experience it.

**Tools:** Playwright, Cypress, custom CLI test harnesses, `tests/e2e.sh`.

**Coverage expectation:** All release-candidate workflows.

**Applies to:** Any repo that ships a user-facing artifact.

=== 4. Build Tests

**Definition:** Verify that the project builds cleanly from a fresh checkout.

**Purpose:** Catch dependency rot, missing files, stale lockfiles, platform assumptions.

**Tools:** `just build`, `cargo build`, `deno task build`, CI matrix builds.

**Coverage expectation:** All supported platforms × all supported toolchain versions.

**Applies to:** Every repo.

=== 5. Execution & Runtime Tests

**Definition:** Verify that compiled/interpreted code executes correctly under runtime conditions
(VM behaviour, JIT correctness, bytecode verification, hot-reload stability).

**Purpose:** Catch issues that only manifest at runtime, not compile time.

**Tools:** Runtime test harnesses, `oo7 run`, interpreter self-tests.

**Applies to:** Interpreters, VMs, JIT compilers, transpilers, REPLs.

=== 6. Reflexive Tests

**Definition:** Tests where the system tests itself — self-diagnostics, health checks,
`just doctor`, internal consistency validation.

**Purpose:** Verify that the system can detect its own failures.

**Tools:** `just doctor`, `just heal`, self-test commands, health endpoints.

**Applies to:** Any repo with a runtime component.

=== 7. Lifecycle Tests

**Definition:** State machine and resource lifecycle correctness — creation, use, cleanup,
error recovery, graceful shutdown.

**Purpose:** Verify no resource leaks, no use-after-free, no orphaned state.

**Tools:** Custom lifecycle harnesses, property tests with state machines.

**Applies to:** Connection pools, file handles, database transactions, service lifetimes.

=== 8. Smoke Tests

**Definition:** Fast sanity checks that verify basic functionality ("does it start? does it respond?").

**Purpose:** Gate before running expensive test suites. Fail fast.

**Tools:** `just smoke`, `curl /health`, process start/stop scripts.

**Coverage expectation:** <30 seconds total. Tests the critical path only.

**Applies to:** Every repo with an executable.

=== 9. Property-Based / Generative Tests

**Definition:** Tests that generate random inputs according to type constraints and verify
that properties hold across all generated cases.

**Purpose:** Find edge cases that example-based tests miss. Shrinking finds minimal reproductions.

**Tools:** `proptest` (Rust), `StreamData` (Elixir), `fast-check` (JS/ReScript), QuickCheck (Haskell/Idris2).

**Coverage expectation:** 1000+ generated cases per property. Regression files committed.

**Applies to:** Parsers, serializers, type checkers, any function with a clear invariant.

=== 10. Mutation Tests

**Definition:** Systematically mutate source code (flip operators, delete lines, change constants)
and verify that tests catch each mutation.

**Purpose:** Measure test suite quality, not code quality. A surviving mutant = a testing gap.

**Tools:** `cargo-mutants` (Rust), `mutation_testing` (Elixir), `stryker` (JS).

**Metric:** Mutation score (killed / total mutants). Target: >80%.

**Applies to:** Repos at CRG C+ where test suite quality matters for confidence.

=== 11. Fuzz Tests

**Definition:** Feed malformed, random, or adversarial input to find crashes, hangs, and
undefined behaviour.

**Purpose:** Security and robustness. Finds issues that structured tests and property tests miss.

**Tools:** `cargo-fuzz` / `libFuzzer` (Rust), `AFL++`, `jazzer` (JVM), custom harnesses.

**CRITICAL:** No fake fuzz placeholders. If fuzz isn't implemented, say so honestly.
A placeholder fuzz file is worse than no fuzz at all (misleads reviewers).

**Applies to:** Parsers, network protocols, file format readers, any untrusted input boundary.

=== 12. Contract / Invariant Tests

**Definition:** Verify that declared contractiles (Mustfile, Trustfile, Dustfile, Intentfile,
K9, ADJUST) actually hold at runtime.

**Purpose:** The contract system is only as strong as its enforcement. Test the contracts.

**Tools:** `must check`, `trust verify`, `k9 eval`, `adjust` CLI, custom contract harnesses.

**Applies to:** Every repo with contractile files.

=== 13. Regression Tests

**Definition:** Tests for specific bugs that were found and fixed, ensuring they never recur.

**Purpose:** Prevent regressions. Each fixed bug becomes a permanent test.

**Tools:** Named test cases with issue/commit references. `tests/regressions/` directory.

**Applies to:** Every repo. If you fixed a bug, add a regression test.

=== 14. Chaos / Resilience Tests

**Definition:** Deliberately inject failures — kill services, corrupt data, drop network,
exhaust memory, slow clocks — and verify graceful degradation.

**Purpose:** Verify the system doesn't catastrophically fail under adverse conditions.

**Tools:** Custom chaos harnesses, network fault injection, service kill scripts.

**Applies to:** Distributed systems (Burble, PanLL service switching, identity replication),
any system with graceful degradation claims.

=== 15. Compatibility Tests

**Definition:** Verify backward and forward compatibility — can v0.2 read v0.1 data?
Can the new API serve old clients?

**Purpose:** Prevent breaking changes from silently corrupting user data or workflows.

**Tools:** Version matrix tests, schema migration tests, API compatibility suites.

**Applies to:** Any repo with persistent data (VeriSimDB), versioned APIs, or wire protocols.

=== 16. Proof Regression Tests

**Definition:** Verify that all formal proofs still compile and verify after code changes.

**Purpose:** Proofs can silently break when definitions change. Catch this in CI.

**Tools:** `coqc`, `idris2 --check`, `lean --run`, `agda --safe`, integrated into CI.

**Applies to:** Every repo with formal proofs.

== Part II: Aspect Test Dimensions

Aspect tests are cross-cutting non-functional concerns. Each aspect can be tested
at any level (unit, P2P, E2E, etc.). The aspect is the *what*, the test category
is the *how*.

=== The 14 Aspects

[cols="1,3,3"]
|===
| Aspect | Definition | Example Test

| **Dependability**
| System performs its function under stated conditions for a stated period.
Includes reliability, availability, fault tolerance, recoverability.
| Restart after crash preserves state. 99.9% uptime under load.

| **Security**
| Protection against unauthorised access, data breach, privilege escalation,
injection, and other OWASP Top 10 + supply chain threats.
| SQL injection attempt returns error, not data. Dependency audit passes.

| **Usability**
| Effectiveness, efficiency, and satisfaction with which users achieve goals.
| New user completes task in <5 minutes. Error messages suggest fixes.

| **Interoperability**
| Ability to exchange information with other systems and use it correctly.
Includes protocol compliance, format adherence, API contract honour.
| JSON output validates against published schema. gRPC client from different language connects.

| **Safety**
| Freedom from unacceptable risk of harm to people, property, or environment.
Includes fail-safe defaults, dangerous-pattern bans, formal verification.
| No `believe_me`/`sorry`/`Admitted` in safety-critical paths. Panic-attack assail passes.

| **Performance**
| Time behaviour, resource utilisation, capacity under expected and peak load.
| Startup <2s. Query response <500ms at p99. Memory <128MB steady state.

| **Functionality**
| Correctness and completeness of the system's stated features.
| Every documented API endpoint returns the documented response. Feature matrix matches README.

| **Versability**
| Versioning correctness, API stability, backward compatibility, migration paths,
semantic versioning adherence, changelog accuracy.
| v0.2 reads v0.1 persisted data. Breaking changes bump major version. CHANGELOG matches commits.

| **Accessibility**
| Conformance to WCAG 2.3 AA+ and ADJUST contractile invariants (54 checks).
Keyboard, screen reader, colour contrast, touch targets, motion, CLI, i18n.
| axe-core returns 0 violations. Tab order is logical. Screen reader reads all controls.

| **Maintainability**
| Modularity, analyzability, modifiability, testability of the code itself.
Can someone else fix a bug? Is the architecture documented?
| New contributor can build + test in <10 minutes. Deep annotation present per CRG C.

| **Privacy**
| Data minimisation, consent, no-tracking, no-logging, ephemeral data handling.
Distinct from security (which protects data; privacy questions whether to collect it).
| Burble relay stores zero data after connection. No analytics payloads. GDPR compliance.

| **Observability**
| Ability to understand internal state from external outputs — logging, tracing,
metrics, diagnostics, error reporting.
| `just doctor` identifies the problem. Structured logs include correlation IDs. Metrics endpoint works.

| **Reproducibility**
| Builds, tests, and proofs produce the same result every time, on any machine,
from the same inputs. Includes deterministic builds and proof replay.
| `nix build` produces identical hash. `coqc` replays all proofs. CI matches local.

| **Portability**
| Adaptability to new platforms, installability, replaceability of components.
Cross-platform builds, container compatibility, architecture independence.
| Builds on Linux/macOS/Windows. Runs in Podman. No hardcoded paths.
|===

== Part III: Formal Proof Systems

=== Proof Assistants

[cols="1,2,2,2"]
|===
| System | Language | Use Case | Banned Patterns

| **Rocq (Coq)**
| Gallina + Ltac/Ltac2
| Semantics proofs, type soundness, preservation
| `Admitted` (track in proof debt ledger)

| **Idris2**
| Idris2 (QTT)
| ABI proofs, dependent types, linear resources, L1-L10 Kategoria levels
| `believe_me`, `assert_total`

| **Lean4**
| Lean4
| Mathematical proofs, algebraic structures
| `sorry`

| **Agda**
| Agda
| CNO proofs, tropical semirings, category theory. Must use `--safe` flag.
| `postulate` (in --safe mode)
|===

=== Type Safety Levels (Kategoria Framework)

[cols="1,2,3"]
|===
| Level | Name | What It Proves

| L1 | Basic Types | Type errors caught at compile time
| L2 | Algebraic Data Types | Exhaustive pattern matching, Option/Result handling
| L3 | Parametric Polymorphism | Parametricity, cannot inspect type parameters
| L4 | Higher-Kinded Types | Functor/Monad abstraction correctness
| L5 | GADTs | Type-indexed DSL safety
| L6 | Dependent Types | Lengths in types, impossible states are impossible
| L7 | Linear/Affine Types | Resources used exactly once, no leaks
| L8 | Refinement Types | Value constraints in types (x > 0)
| L9 | Session Types | Protocol state machines enforced by types
| L10 | Homotopy/Cubical Types | Equivalence reasoning, univalence
| L11 | Tropical Cost-Tracking | Min-plus semiring resource bounds (typed-wasm extension)
| L12 | Epistemic Safety | Information-theoretic access control (typed-wasm extension)
|===

TypeLL specialises these for database queries. typed-wasm extends to L11-L12.
Kategoria L10 is the current hard wall (QTT cannot express cubical univalence).

== Part IV: Benchmark Taxonomy

=== Benchmark Categories

[cols="1,3"]
|===
| Category | What It Measures

| **Latency** | Time to complete a single operation (p50, p95, p99, p999)
| **Throughput** | Operations per second under sustained load
| **Memory** | Steady-state allocation, peak allocation, leak detection
| **Startup** | Time from process launch to ready-to-serve
| **Build** | Time to compile from clean checkout
| **Energy** | Power consumption per operation (platform-dependent)
| **FFI Overhead** | Cost of crossing language boundaries (Idris2→Zig, Rust→C, etc.)
|===

=== Six Sigma Classification

Benchmark results are classified using a severity scale inspired by Six Sigma
process capability analysis. The classification determines CI behaviour.

[cols="1,2,2,2"]
|===
| Level | Definition | Threshold | CI Action

| **Unacceptable**
| Result is outside any reasonable operating envelope.
Indicates a fundamental regression or broken implementation.
| >50% regression from baseline OR absolute threshold breach (e.g., startup >10s)
| **Hard fail.** Block merge. Escalate immediately.

| **Acceptable**
| Result is within operating envelope but below expected performance.
May indicate a real regression that needs investigation.
| 20-50% regression from baseline
| **Soft fail.** Warn in PR. Require explicit reviewer approval to merge.

| **Ordinary**
| Result is within normal operating range. No action needed.
Variance is expected and within statistical control limits.
| Within ±20% of baseline (±2σ of historical distribution)
| **Pass.** Record and track for trend analysis.

| **Extraordinary**
| Result is significantly better than baseline.
May indicate a genuine improvement OR a measurement error.
| >20% improvement from baseline
| **Pass with flag.** Investigate: real improvement or measurement artifact?
Update baseline if confirmed genuine.
|===

=== Baseline Management

- Baselines are established from the mean of the last 10 CI runs on main.
- Baselines are versioned — a new baseline is cut at each minor version bump.
- Baseline history is retained in VeriSimDB for trend analysis.
- Manual baseline override requires a comment in the benchmark config explaining why.

== Part V: Enforcement Map

=== Who Checks What

[cols="2,2,3"]
|===
| Tool | Checks | Integration

| **panic-attacker**
| Static analysis, dangerous patterns, security assault, weak points
| Runs `assail` pre-commit (standing rule). Feeds findings to Hypatia.

| **Hypatia**
| Neurosymbolic triage (safety triangle: Eliminate > Substitute > Control),
cross-repo pattern detection, confidence-gated dispatch
| Consumes panic-attacker scans + VeriSimDB. Dispatches to gitbot-fleet.

| **gitbot-fleet**
| Automated fixes with confidence thresholds. rhodibot (RSR compliance),
echidnabot (proof verification), sustainabot (dependency health),
glambot (documentation), seambot (interoperability), finishbot (completion).
| Receives dispatch from Hypatia. Creates PRs with confidence scores.

| **CRG v2.0**
| Grade assessment (X/F/E/D/C/B/A) based on accumulated evidence.
D requires RSR. C requires annotation. B requires 6+ diverse targets.
| Manual assessment recorded in READINESS.md. Automated evidence collection planned.

| **contractile CLI**
| `must check`, `trust verify`, `dust status`, `intend check`, `k9 eval`, `adjust`
| Contract/invariant test category. Enforcement of Must/Trust/Dust/Intend/K9/ADJUST.

| **V1-GATE**
| Hard gates for stable release: proofs, tests, benchmarks, build, aspects,
security, claim parity. Cross-Repo Blitz Rule (Section 4).
| Publication pre-flight. Blocks release until all evidence present.

| **PanLL**
| Dashboard visualisation of all test/bench/proof status across estate.
Event-chain import. Blitz panel.
| Consumes VeriSimDB data. Displays CRG grades, test gaps, benchmark trends.
|===

=== CRG ↔ Test Category Mapping

[cols="1,4"]
|===
| CRG Grade | Minimum Test Evidence

| **X** | None
| **F** | Documented evidence of harm/waste (may include test results showing failure)
| **E** | At least 1 test of any category. Documented failures.
| **D** | Unit + smoke tests. Test matrix showing successes and failures. RSR compliance.
| **C** | Unit + P2P + E2E + smoke + reflexive + build + contract + aspect tests. All passing in home context. Benchmarks baselined. Deep annotation.
| **B** | All of C plus: property-based, fuzz, compatibility, lifecycle tests. Benchmarks in Ordinary range across 6+ diverse targets. Mutation score >80%.
| **A** | All of B plus: chaos/resilience, regression suite comprehensive, benchmark baselines stable across releases, external user feedback incorporated.
|===

== Part VI: Cross-Repo Blitz Rule

Per V1-GATE Section 4: when a system claim spans multiple repositories, all in-scope
repos must be audited together. The weakest repo controls the system release stage.

A "test blitz" means running all applicable test categories across all repos in a
dependency chain, in a single coordinated pass. The blitz result is a matrix:

[source]
----
           unit  P2P  E2E  build  exec  reflex  life  smoke  prop  mut  fuzz  contract  regress  chaos  compat  proof-reg
repo-a      ✓    ✓    ✓     ✓      -      ✓      ✓     ✓     ✓    ✓     ✓      ✓        ✓       -       ✓        ✓
repo-b      ✓    ✓    ✗     ✓      ✓      -      -     ✓     -    -     -      ✓        ✓       -       -        ✓
repo-c      ✓    -    -     ✓      -      -      -     ✓     -    -     -      -        -       -       -        -
----

The system CRG grade is determined by the lowest grade in the matrix, not the average.

=== Cross-Repo Test Ideas (added 2026-04-17)

These tests span two or more repos and catch failure modes that single-repo blitzes
cannot. Added after the echidna TypeDiscipline transition (commit `8f573f1`) exposed
several drift classes that had been latent for weeks. Each idea lists: **what it tests**,
**where it lives**, **why single-repo tests miss it**, and **estate applicability**.

==== CR-1. Mirror-enum drift test

**What:** Property test asserting that any enum mirrored across repos stays in sync
with its upstream source of truth. Consumer generates its mirror at test-time from
a shared manifest (JSON / A2ML) that the upstream emits, then diffs against its own
hand-maintained copy. Fails fast when they diverge.

**Where:** In each consumer repo's integration-test directory, plus a companion
manifest at `src/<mirror-enum>.manifest.a2ml` in the upstream.

**Why single-repo misses it:** Upstream adds a variant, downstream mirror stays
stale, consumer keeps compiling because downstream has `_ =>` fallback — drift goes
undetected until a user hits the fallback path with unexpected results.

**Applicability:** `ProverKind` (echidna -> echidnabot + protocol-squisher-bridge);
`Severity` / `Tier` / `BotId` (gitbot-fleet/shared-context -> each bot); future
`TypeDiscipline` (echidna -> hypatia, panic-attack); `NodeKind` / `EdgeKind` in
GNN schemas (echidna -> any ML consumer).

**Concrete gap today:** `echidnabot` and `protocol-squisher-echidna-bridge` both
carry local mirror enums that already lag behind echidna main; silent today because
matches use `_ =>`, loud tomorrow when someone relies on a specific variant.

==== CR-2. Foreign-enum exhaustive-match lint

**What:** Custom clippy lint (or `cargo miri`-style analysis, or a grep-based CI
job as MVP) that flags every `match` expression over an enum imported from an
external crate that lacks an `_ =>` fallback. The grep MVP is one regex plus a list
of extern-enum names.

**Where:** `developer-ecosystem/standards/hypatia-rules/lint-foreign-exhaustive.a2ml`
as the declarative policy; enforced by hypatia in every downstream repo's CI.

**Why single-repo misses it:** An exhaustive match compiles fine today, but a
one-variant addition in the upstream turns the downstream into a compile failure
on the next `cargo build`. The issue surfaces as upstream's CI problem even though
the fault is in the downstream's coupling decision.

**Applicability:** Every Rust repo with a path-dep on another hyperpolymath repo.
Especially `echidna` consumers; any repo importing `katagoria::Discipline` once
that exists; `gitbot-shared-context` consumers.

**Concrete gap today:** Echidna's `src/rust/main.rs` had three exhaustive matches
over `ProverKind` that were silently wrong before the TypeDiscipline transition
(they never covered the 12 original HP ecosystem variants) and broke loudly the
moment 28 more were added. Nobody noticed for weeks.

==== CR-3. FFI roundtrip bijection property

**What:** For every FFI boundary that serialises a Rust enum to a numeric or
string representation, a property test asserting `decode(encode(x)) == x` for
every variant — not just a hand-picked subset. Run in both directions where
both are exposed.

**Where:** The originating crate's `tests/ffi_roundtrip.rs`, generated from the
enum's `ALL` const (see reference implementation in echidna
`src/rust/disciplines/mod.rs`).

**Why single-repo misses it:** When a table only tests the variants the author
remembers, it silently tolerates missing entries. echidna's `kind_from_u8` was
missing codes 49–60 for weeks before the TypeDiscipline transition; the
existing roundtrip test only covered codes 0–48 and explicitly *asserted*
`kind_from_u8(49).is_none()`, locking in the bug.

**Applicability:** Every `*_to_u8` / `*_from_u8` or `*_to_str` / `*_from_str`
pair. Every CBOR-encoded proof-state field. Every GraphQL / gRPC / REST enum
bridge. Every Zig ABI that names variants.

==== CR-4. Shared-fixture corpus contract

**What:** Canonical golden fixtures live in ONE upstream repo (`katagoria/`,
`standards/a2ml/examples/`, `echidna/training_data/`) and every downstream that
consumes that format runs its parser / validator / type-checker over the
*shared* corpus. When katagoria graduates a new discipline, the fixture is
added once and every downstream's next CI run exercises it.

**Where:** Upstream hosts `fixtures/` with an index.a2ml. Downstream
`tests/shared_corpus.rs` dereferences via path dep or git submodule.

**Why single-repo misses it:** Each consumer writes its own fixtures, they
drift from the upstream format, and a format extension in the upstream doesn't
fail any test until a real user trips over it.

**Applicability:** katagoria → echidna + typell + typed-wasm (type disciplines);
a2ml → every repo that emits a2ml; avow-protocol → every repo that uses proven
results; VeriSimDB → every repo that writes octads.

**Concrete gap today:** `katagoria/research/tropical/TropicalKleene.idr` is
standalone; no downstream currently uses it as a fixture, so when tropical
semantics shift, nothing breaks.

==== CR-5. Claim-grounder adversarial cross-test

**What:** Each claim-grounder (invariant-path, PLASMA, Hypatia, ECHIDNA,
panic-attacker) is run against proofs produced by the others, specifically
constructed to hide violations or to appear cleaner than they are. This makes
the dual-use posture (per the Akerlof memory note) operational: a grounder
that cannot detect another grounder's escape hatches is not actually grounded.

**Where:** `verification-ecosystem/<grounder>/tests/adversarial_cross/`
plus a meta-repo `verification-ecosystem/meta-adversarial-suite/` that
coordinates the round-robin and publishes a weekly summary.

**Why single-repo misses it:** Every grounder tests itself against
hand-picked fixtures by default. The adversarial signal — "tool X can hide
from tool Y" — is only visible when you explicitly set them against each
other.

**Applicability:** All five listed grounders; any future grounder (invariant-path,
PLASMA, Hypatia, ECHIDNA, panic-attacker) immediately on admission.

==== CR-6. Upstream-HEAD sentinel in downstream CI

**What:** Every downstream-of-a-path-dep repo runs a nightly CI job that
pulls the upstream's `main` branch (overriding its Cargo.toml path dep or
submodule) and runs the full test suite. Result emitted as a status check
on the upstream's recent PRs via Hypatia / gitbot-fleet webhook.

**Where:** `.github/workflows/upstream-head-sentinel.yml` in downstream
repos; Hypatia rule in `standards/hypatia-rules/upstream-sentinel.a2ml`.

**Why single-repo misses it:** The upstream's CI is green, the downstream's
CI against the pinned version is green, but a merge that would break the
downstream is invisible until the pin bumps — often weeks later.

**Applicability:** `echidna` consumers (echidnabot, panic-attack, hypatia,
protocol-squisher-echidna-bridge); `shared-context` -> every gitbot;
`a2ml-rs` -> every repo consuming A2ML; `proven` -> every repo consuming a
formally-verified alternative.

**Concrete gap today:** Today's echidna commit `8f573f1` added 28 new
ProverKind variants. Downstream echidnabot silently continues to compile
(for now) but its dispatcher is mis-calibrated. An upstream-HEAD sentinel
would have pinged immediately.

==== CR-7. Workspace-aware blitz matrix

**What:** The blitz matrix (§ Part VI) distinguishes four cells per (repo,
category): `in-isolation` (cargo test with this member only), `in-workspace`
(cargo test at workspace root), `workspace-root-with-bins` (cargo test
`--all-targets`), and `cross-workspace` (downstream workspace importing this
as a path dep). Many bugs only appear in one of the four.

**Where:** An enriched matrix format in `.machine_readable/testing/blitz.a2ml`
per repo, rolled up at the ecosystem level in `standards/dashboards/blitz.a2ml`.

**Why single-repo misses it:** `cargo test --lib` can pass while `cargo test
--all-targets` fails on the same enum, because bin / test-target exhaustive
matches are outside the lib's compilation unit. echidna hit exactly this today.

**Applicability:** All workspace-member repos (echidna, gitbot-fleet,
developer-ecosystem monorepo, nextgen-languages monorepo, nextgen-databases).

==== CR-8. Cross-prover proof-exchange roundtrip

**What:** For every proof-interchange format (OpenTheory, Dedukti, TSTP,
Alethe, DRAT/LRAT, custom CBOR), a roundtrip test: prove theorem T in prover
A, export to format F, re-import to prover B, re-check T. Must hold across
{Coq, Lean, HOL4, HOL Light, Isabelle} × {OpenTheory, Dedukti} pairs at
minimum.

**Where:** `echidna/tests/proof_exchange_roundtrip.rs` with the heavy lifting
delegated to the `src/rust/exchange/` module. Parallel suite in any future
proof-exchange host.

**Why single-repo misses it:** The exchange module has unit tests for each
direction independently; nothing today asserts semantic preservation across
the roundtrip.

**Applicability:** `echidna/src/rust/exchange/`; any repo that ships an
exchange format bridge.

==== CR-9. Schema-surface drift detector

**What:** Every public surface (Rust enum, JSON schema, A2ML schema, GraphQL
type, Zig ABI) emits a machine-readable schema snapshot at build time to
`.schemas/<surface>.a2ml`. Every PR's CI computes the new snapshot and diffs
against the merged-main version; any change is flagged with a SEMVER
classification (patch / minor / major).

**Where:** `just emit-schemas` recipe per repo; shared implementation in
`developer-ecosystem/schema-snapshot-rs` (TBD).

**Why single-repo misses it:** Today a `pub enum` variant addition in the
upstream is invisible to the downstream until someone imports the crate
again. A schema snapshot makes the surface change a visible PR artifact.

**Applicability:** Every public-facing crate / library / protocol.

==== CR-10. Stale-fixture-and-baseline decay alarm

**What:** Any test fixture, proof corpus sample, benchmark baseline, or
regression gold file that has not been regenerated in N days (default 90)
raises a decay alarm in the daily CI pulse. Bench baselines specifically
decay faster (30 days) because upstream prover performance shifts.

**Where:** `standards/hypatia-rules/fixture-decay.a2ml` + per-repo
`.machine_readable/fixtures.a2ml` manifest listing every fixture with its
last-regenerated timestamp.

**Why single-repo misses it:** Tests keep passing against stale fixtures
that no longer reflect real inputs. Benchmarks pass against baselines that
no longer reflect current performance. Drift is slow and undetected.

**Applicability:** echidna (`training_data/*.jsonl`, `premises_merged.jsonl`,
etc.); proven (Idris2 proof libraries); a2ml (grammar examples); any repo
with a `fixtures/` or `corpora/` directory.

=== Weak Points Spotted in the Current Taxonomy (2026-04-17)

Recorded at the same time as the cross-repo ideas above, things that feel
under-specified in this document itself:

- **No "coupling test" category.** Tests that assert drift-freedom between
  two independent artefacts (mirror enums, redundant parsers, serde + custom
  Display, etc.) don't fit cleanly into the 16 categories. Proposal: add
  Category 17 "Coupling / Drift-freedom" in v1.1.
- **Proof-regression is single-prover.** Category 16 catches "Coq no longer
  accepts proof P" but not "Coq and Lean give different counterexamples for
  the same formula". Cross-prover regression deserves its own sub-category.
- **Benchmark taxonomy is single-process.** Six Sigma classification
  (§ Part IV) applies to one binary's hot path. There is no equivalent for
  end-to-end pipelines (panic-attack → hypatia → echidna → verisim) where
  the interesting latency is the tail across all four.
- **CRG mapping doesn't include cross-repo rows.** § "CRG ↔ Test Category
  Mapping" maps repo-local categories to grades. A repo with mirror-enum
  drift or FFI roundtrip gaps has a visible-externally failure mode that
  its in-repo blitz will not catch. Proposal: a new "Ecosystem CRG" rollup
  grade that can only be A if all CR-* cross-repo checks pass.
- **Dogfooding is listed (§ Part VII) but not adversarial.** Dogfooding
  asks "does the author use their own tool?"; CR-5 (claim-grounder
  adversarial cross-test) asks the much harder "can the tool fool its
  peers?". The latter is closer to real robustness.
- **"Shared fixtures" are implied but never required.** Several repos
  (katagoria, a2ml, avow-protocol) should host canonical fixtures that
  every downstream consumes, but the taxonomy doesn't elevate this to a
  first-class obligation. CR-4 above proposes a fix.
- **"Run time" categories don't include warm-boot vs cold-boot separation.**
  Many bugs (FFI init, plugin registry, GNN client handshake) only appear
  once per process lifetime. A cold-start test category is missing.

== Appendix A: Test Category Quick Reference

[cols="1,1,1"]
|===
| Category | Abbreviation | Applies From CRG

| Unit | UT | E+
| Point-to-Point | P2P | D+
| End-to-End | E2E | C+
| Build | BLD | D+
| Execution & Runtime | EXE | D+ (if applicable)
| Reflexive | REF | C+
| Lifecycle | LCY | C+
| Smoke | SMK | D+
| Property-Based | PBT | B+
| Mutation | MUT | B+
| Fuzz | FUZ | B+
| Contract/Invariant | CTR | C+
| Regression | REG | C+
| Chaos/Resilience | CHS | B+ (if applicable)
| Compatibility | CMP | B+ (if applicable)
| Proof Regression | PRF | C+ (if proofs exist)
|===

== Appendix B: Aspect Quick Reference

[cols="1,1"]
|===
| Aspect | Abbreviation

| Dependability | DEP
| Security | SEC
| Usability | USA
| Interoperability | IOP
| Safety | SAF
| Performance | PER
| Functionality | FUN
| Versability | VER
| Accessibility | ACC
| Maintainability | MNT
| Privacy | PRI
| Observability | OBS
| Reproducibility | RPR
| Portability | PRT
|===

== Appendix C: Reference Implementation

The **K9 Coordination Protocol** (`standards/k9-coordination-protocol/`) serves as the
reference implementation for this taxonomy. It demonstrates:

- **Full blitz assessment** against all 16 test categories and 14 aspect dimensions
- **79 tests** across 8 test files covering 9 applicable categories (7 N/A with justification)
- **Mutation testing** with 15 mutations (9 killed, 6 documented equivalents, 100% applicable score)
- **Six Sigma benchmarks** baselined on real hardware
- **CRG grade C** achieved via systematic blitz

Test files:

[cols="2,2,1"]
|===
| File | Category | Tests

| `generator/generate_test.js` | Unit + Integration | 22
| `generator/smoke_test.js` | Smoke | 6
| `generator/contract_test.js` | Contract/Invariant | 7
| `generator/property_test.js` | Property-based | 4
| `generator/mutation_test.js` | Mutation | 17
| `generator/fuzz_test.js` | Fuzz | 11
| `generator/regression_test.js` | Regression | 6
| `generator/compatibility_test.js` | Compatibility | 6
| `generator/bench.js` | Benchmarks | 6
|===

See `standards/k9-coordination-protocol/READINESS.md` for the full assessment.

This implementation demonstrates that even a small CLI tool (single generator script,
~580 lines) can be thoroughly assessed against the full taxonomy. The key insight:
most categories either apply directly or can be honestly marked N/A with justification.
A blitz does not require artificial tests for inapplicable categories.

== Part VII: Dogfooding Tests

=== Definition

**Dogfooding tests** verify that a project correctly consumes, integrates, and produces
value from other hyperpolymath tools. This is distinct from interoperability (which
tests that two systems can exchange data) — dogfooding tests verify that the estate's
own tools are actually used, correctly wired, and producing real output.

=== The Dogfooding Test Checklist

Every repo must answer these questions. The test evidence goes into the standard
blitz matrix as a new column: **DOG** (Dogfooding).

[cols="1,3,2"]
|===
| Check | What It Tests | How To Test

| **DOG-01: CI gate fires**
| Do the dogfooding CI tools (Hypatia, panic-attack, dogfood-gate) actually
execute and produce findings, or do they silently skip?
| Verify the workflow run log shows non-empty output from each tool. A run
that produces `[]` with `continue-on-error` is a silent no-op, not a pass.

| **DOG-02: Annotation pipeline works**
| Do panic-attack findings produce GitHub Actions inline annotations (file + line)?
| Run `panic-attack assail . --output findings.json` locally, verify each finding
has non-null `file` and `line` fields. Check GitHub PR annotations appear.

| **DOG-03: Groove manifest conformant**
| If this repo serves HTTP, does its Groove manifest validate against the schema?
| `jq . .well-known/groove/manifest.json` passes. JSON Schema validation against
`gossamer/schema/groove-manifest.schema.json` passes. Ports match actual bind address.

| **DOG-04: K9 contracts enforced**
| If this repo has config files, do K9 contracts exist and validate?
| `k9iser build` exits 0. The number of K9 contracts >= the number of significant
config files. No config file changes without corresponding contract update.

| **DOG-05: VeriSimDB wired (if stateful)**
| If this repo stores state, is VeriSimDB integrated?
| Grep for `verisimdb` in source code (not just docs/comments). Verify the
VeriSimDB client is instantiated and called, not just imported.

| **DOG-06: Template placeholders resolved**
| No `{{OWNER}}`, `{{REPO}}`, `{{AUTHOR}}`, `[YOUR-REPO-NAME]` remain in
any workflow, manifest, or configuration file.
| `grep -rn '{{[A-Z_]*}}' --include='*.yml' . \| grep -v '${{' \| grep -v '.git/'`
returns zero matches.

| **DOG-07: Tool version currency**
| Are workflow copies (hypatia-scan, static-analysis-gate) current with the
rsr-template-repo source?
| Diff the repo's workflow against `rsr-template-repo/.github/workflows/` equivalent.
Action SHA pins should match. Job structure should match.

| **DOG-08: Consumer field contract**
| If this repo consumes output from another tool (e.g., panic-attack JSON),
does it use the correct field names?
| For each `jq` expression that parses tool output, verify the field names
match the tool's actual serialisation. Document the contract in a comment.

| **DOG-09: Iser applicability assessed**
| Has someone assessed which -iser tools apply to this repo?
| A `.machine_readable/dogfooding.a2ml` file exists listing applicable isers
with yes/no/deferred status and justification.

| **DOG-10: Naming currency**
| All references to renamed tools use current names. No stale binary names,
old subcommand names, or deprecated APIs.
| Search for known renames: `panic-attacker` → `panic-attack`, `xray` → `assail`,
`AGPL` → `PMPL`, `Dockerfile` → `Containerfile`, `docker` → `podman`.
|===

=== Enforcement

DOG-01 through DOG-10 are checked by:

- **dogfood-gate.yml** (CI): DOG-01, DOG-03, DOG-04, DOG-05, DOG-06
- **Hypatia**: DOG-06, DOG-07, DOG-10 (new rules needed — see below)
- **gitbot-fleet/rhodibot**: DOG-07 (workflow currency check — new dispatch target)
- **Manual blitz**: DOG-02, DOG-08, DOG-09 (until automated)

=== Hypatia Rule Additions for Dogfooding

The following new Hypatia rules are derived from the 2026-04-04 estate dogfooding audit.
Each rule corresponds to a real class of silent failure found during the audit.

[cols="1,3,1"]
|===
| Rule ID | What It Detects | Severity

| **HYP-DOG-001**
| Unfilled template placeholders in workflow files (`{{OWNER}}`, `{{REPO}}`, etc.)
| Critical — silently disables CI tools

| **HYP-DOG-002**
| Stale workflow copies — action SHA pins differ from rsr-template-repo
| Medium — may use outdated/broken tool invocations

| **HYP-DOG-003**
| Renamed tool references (panic-attacker binary, xray subcommand, AGPL license)
| High — broken invocations

| **HYP-DOG-004**
| Groove port mismatches — manifest advertises port X, server binds port Y
| High — consumers get connection refused

| **HYP-DOG-005**
| Docker/Dockerfile usage in non-third-party code (policy: Podman/Containerfile)
| Medium — policy violation

| **HYP-DOG-006**
| Consumer field name mismatches — jq parsing `.file` when source emits `.location`
| High — silent annotation failure

| **HYP-DOG-007**
| Silent skip pattern — `continue-on-error: true` on a tool that fails to install
| Medium — masks broken dogfooding

| **HYP-DOG-008**
| HTTP server without Groove endpoint
| Low — dogfooding gap, not a security issue

| **HYP-DOG-009**
| Stateful repo without VeriSimDB reference in source code
| Low — dogfooding gap

| **HYP-DOG-010**
| Unfilled `[YOUR-REPO-NAME]` placeholders in 0-AI-MANIFEST.a2ml
| Medium — a2ml-validate-action's allowlist masks this
|===

=== gitbot-fleet Dispatch Targets for Dogfooding

[cols="1,3"]
|===
| Bot | Dogfooding Responsibility

| **rhodibot**
| Workflow currency — detects stale copies of RSR workflows and creates PRs
to update them from the template.

| **echidnabot**
| Proof + contract verification — verifies K9 contracts validate, formal proofs
compile, and VeriSimDB schema matches.

| **seambot**
| Groove + interop — verifies Groove manifests are present, conformant, and
port numbers match actual bind addresses.

| **finishbot**
| Template completion — detects unfilled placeholders (`{{OWNER}}`, `[YOUR-REPO-NAME]`)
and creates PRs with the correct values.
|===

=== Feedback-o-tron Integration

The `feedback-o-tron` tool is the autonomous multi-platform bug reporter for AI agents.
When a dogfooding check fails, it should:

1. Create a structured finding (JSON) with the DOG-XX check ID
2. Submit to VeriSimDB as a dogfooding-specific modality
3. Optionally create a GitHub issue (via feedback-o-tron's platform adapters)
4. Feed back to Hypatia for pattern detection across the estate

This creates the **dogfooding feedback loop**: tool → dogfood-gate → finding → feedback-o-tron → issue → fix → tool.

== Part VIII: Soft Groove Strategy

=== What Is a Soft Groove?

A "soft groove" is a lightweight, non-blocking Groove integration. Unlike the full
Groove protocol (which requires a live HTTP server, capability manifest, and bidirectional
message passing), a soft groove is:

1. **Declarative**: A static `.well-known/groove/manifest.json` that describes capabilities
   without requiring a running server
2. **Consumable**: Other tools can read the manifest at build time or CI time
3. **Progressive**: Can be upgraded to a full Groove endpoint later

Soft grooves are appropriate for CLI tools, libraries, and batch processors that
don't serve HTTP but still need to be discoverable by the ecosystem.

=== Soft Groove Manifest Format

A soft groove manifest uses the same JSON format but with `"mode": "passive"`:

[source,json]
----
{
  "groove_version": "1",
  "service_id": "feedback-o-tron",
  "service_version": "0.2.0",
  "mode": "passive",
  "capabilities": {
    "bug_reporting": {
      "type": "bug-reporting",
      "description": "Autonomous multi-platform bug reporting for AI agents",
      "protocol": "cli",
      "invoke": "feedback-o-tron submit --platform github",
      "requires_auth": true,
      "panel_compatible": true
    }
  },
  "consumes": ["static-analysis", "octad-storage"],
  "invoke_patterns": {
    "ci": "feedback-o-tron submit --ci --format json",
    "local": "feedback-o-tron submit --interactive",
    "mcp": "boj feedback-o-tron submit"
  },
  "health": null,
  "applicability": ["individual", "team", "ci"]
}
----

=== Deploying feedback-o-tron as a Soft Groove

feedback-o-tron should be soft-grooved into every repo that has a CI pipeline:

1. Add `.well-known/groove/manifest.json` to feedback-o-tron itself (passive mode)
2. Add a `feedback-o-tron` step to `dogfood-gate.yml` that submits failed checks
3. Add a `feedback-o-tron.toml` config to repos that want auto-submission
4. Wire feedback-o-tron into Hypatia's dispatch pipeline (Hypatia finding → feedback-o-tron → GitHub issue)
5. Wire feedback-o-tron into gitbot-fleet (failed dogfooding → rhodibot PR + feedback-o-tron issue)

== Revision History

[cols="1,1,3"]
|===
| Version | Date | Changes

| 1.0.0 | 2026-04-03 | Initial release. 16 test categories, 14 aspect dimensions, 4 proof systems, 12 type levels, Six Sigma benchmark classification, enforcement map, CRG mapping.
| 1.0.1 | 2026-04-03 | Added Appendix C: K9 Coordination Protocol as reference implementation for the taxonomy.
| 1.1.0 | 2026-04-04 | Added Part VII (Dogfooding Tests — DOG-01 to DOG-10, 10 Hypatia rules, gitbot dispatch targets, feedback-o-tron integration) and Part VIII (Soft Groove Strategy) based on estate-wide dogfooding audit findings.
|===
