// SPDX-License-Identifier: CC-BY-SA-4.0
// Copyright (c) 2026 Jonathan D.A. Jewell (hyperpolymath) <j.d.a.jewell@open.ac.uk>

= TRG A-Grade LLM Panel — Composition and Process
:status: v1.0 — quarterly re-baseline
:date: 2026-04-12

This document defines the binding composition rules for the *six-LLM
panel* referenced from `TOOLCHAIN-READINESS-GRADES.adoc` §5.7.6. A
component cannot reach grade *A* unless six LLMs from this panel
unanimously sign off on the evidence in §5.7.1–§5.7.5, alongside the
six human reviewers in the same section.

== Why an LLM panel at all

The 6+6 (humans + LLMs) requirement is not a category error. LLMs in
the panel role serve a *specific* function: adversarial review of the
proof corpus and the methodology, at a scale and reproducibility that
human reviewers cannot match individually. Three load-bearing reasons:

. *Reproducibility.* Every LLM signoff is evidenced by a session
  transcript that any reviewer can re-run. Human signoffs are
  similarly archived, but LLM signoffs are *byte-reproducible* in a
  way human signoffs aren't.
. *Adversarial breadth.* Six LLMs from six different vendors will
  surface different objections to the same evidence, drawn from the
  different training distributions of the six labs. This is breadth
  no single human reviewer can replicate.
. *Refusal of capture.* The vendor-distinctness rule prevents any
  single lab from gating an A-grade signoff. Even if one lab were
  captured (politically, commercially, or technically), the other
  five would still need to agree.

The LLM panel does *not* replace human reviewers. It supplements them.
Six humans MUST also sign off independently (per §5.7.6). The two
groups are *both* required; neither is sufficient on its own.

== Binding composition rules (v1.0)

The six LLMs MUST satisfy *all* of the following:

. *Vendor distinctness.* One LLM per major manufacturer / lab. No two
  LLMs in the panel may be from the same vendor. The qualifying-vendor
  matrix below is binding for v1.0.

. *Composition by role.* The six LLMs must include:
+
[cols="1,1,4",options="header"]
|===
| # | Role | Definition

| 2 | *Reasoning / thinking models* | Models with explicit chain-of-thought / extended reasoning capability at inference time. As of 2026-Q2: Claude Opus 4.x with extended thinking, OpenAI o-series reasoning tier, Gemini Deep Think, DeepSeek-R1-class models. The qualifying list is re-baselined quarterly.
| 2 | *Specialist coding models* | Models trained specifically for coding tasks with documented benchmark performance on at least one of: SWE-Bench Verified ≥ 50%, HumanEval ≥ 90%, LiveCodeBench top quartile. As of 2026-Q2: Claude Opus / Sonnet (coding tier), GPT-class coding tier, Gemini Code, Qwen-Coder, DeepSeek-Coder.
| 1 | *Instruct model* | A model from the general-purpose instruction-following family, not specifically reasoning-tier and not specifically coding-tier. Provides the lay-reader-equivalent baseline.
| 1 | *Mixture-of-experts (MoE) model* | A model with explicit MoE architecture, drawn from a vendor not already used above. As of 2026-Q2: Mixtral, DeepSeek-V3, Qwen-MoE, etc. The MoE constraint exists to ensure architectural diversity in the panel, not just vendor diversity.
|===

. *No vendor reuse across roles.* Even if a vendor produces both a
  reasoning model and a coding model, the panel may use only one
  model from that vendor. Six vendors, six models.

. *Reproducible session transcripts.* Each LLM signoff MUST be
  evidenced by a complete session transcript:
+
.. The exact prompt sent to the model
.. The full response (including any extended-thinking trace)
.. The model identifier (vendor, model name, version, parameter
   count where disclosed)
.. The temperature and other inference parameters
.. The timestamp and signoff verdict (PASS / FAIL / ABSTAIN)
.. The cryptographic hash of the transcript for tamper-evidence
.. The transcript MUST be re-runnable: the same prompt under the
   same model + parameters should produce a substantially equivalent
   verdict. (Sampling variation is acceptable; substantively
   different verdicts under the same prompt indicates the panel
   process is unreliable and the signoff is invalid.)

. *Independent submission.* Each LLM is shown the evidence package
  *without* knowing the verdicts of the other five. Coordinated
  prompting (e.g. "the other panellists agreed; do you?") invalidates
  the signoff. The six are queried independently.

. *Unanimous required.* All six LLMs must return PASS. Any abstention
  or FAIL blocks A-grade. There is no "five out of six" path.

. *Evidence archival.* The six transcripts, the evidence package
  shown to each LLM, and the per-LLM verdicts are archived under
  `audits/<component>/A-grade-evidence/llm-panel/` and form part of
  the public record of the A-grade signoff.

== Qualifying vendor matrix (v1.0, 2026-Q2)

[cols="1,2,4",options="header"]
|===
| # | Vendor | Eligible models (one per panel)

| 1 | *Anthropic* | Claude Opus 4.x (reasoning tier with extended thinking, OR coding tier — pick one role)
| 2 | *OpenAI* | GPT-class reasoning tier (o-series), OR GPT-class coding tier — pick one role
| 3 | *Google DeepMind* | Gemini Deep Think (reasoning), OR Gemini Code (coding) — pick one role
| 4 | *Meta* | Llama-class instruct tier
| 5 | *Mistral* | Mixtral-class MoE
| 6 | *DeepSeek* | DeepSeek-R1 (reasoning), DeepSeek-Coder, or DeepSeek-V3 MoE — pick one role
| 7 | *Alibaba (Qwen)* | Qwen-class — reasoning, coding, or MoE per role
| 8 | *Cohere* | Command-class instruct tier
| 9 | *xAI* | Grok-class instruct tier
|===

The panel MUST draw six vendors from this list of nine (or its
quarterly successors). The exact six chosen for any individual A-grade
assessment are at the discretion of the assessment team but must
satisfy the role composition above.

== Quarterly re-baseline

This document is re-baselined every calendar quarter (i.e. every 90
days). At each re-baseline:

. The qualifying-vendor list is reviewed against the current frontier.
  Vendors that have ceased to produce competitive models are removed.
  New vendors that meet the qualifying criteria are added.
. The role definitions ("reasoning model", "specialist coding model",
  "instruct", "MoE") are re-evaluated against the current frontier.
. The eligible-model list within each vendor is updated to the
  latest qualifying release.
. Any LLM signoff older than 12 months is *re-validated* against
  the current panel composition. Components that depended on a
  removed vendor must re-do the relevant signoff.

The quarterly review minutes are archived at
`audits/llm-panel-review-<year>-Q<n>.adoc` in this repo.

== Why this is novel and why it's worth defending

The 6+6 panel composition is, to the author's knowledge, unique
among software-readiness rubrics. The closest analogues are:

* *Multi-vendor formal-verification cross-checking* (e.g. CompCert
  validated by multiple Rocq versions across decades) — humans only
* *Adversarial multi-LLM evaluation in red-teaming research* — but
  not as a release-gate signoff
* *Multi-reviewer journal publication* — humans only

The novelty is the *binding composition rule* combined with the
*reproducibility requirement* combined with the *unanimous
requirement*. Each of those three alone exists somewhere; the
combination is new.

The defence against "LLMs as voting members of an audit panel is a
category error" is:

. They aren't *the* audit. Six humans also sign off, independently.
  The LLMs supplement, they do not replace.
. The vendor-distinctness rule prevents capture. Even if one lab
  produces a flawed model, the other five would catch it in
  independent review.
. The reproducibility rule means any reviewer can re-run any LLM
  signoff and get the same verdict (modulo sampling). This is
  *more* auditable than human signoffs, not less.
. The role composition (reasoning + coding + instruct + MoE)
  ensures architectural diversity, not just vendor diversity.

== Open invitation

If you think the composition is wrong, propose a tightening. If you
think a vendor is missing from the qualifying list, propose its
addition with evidence. If you think the role definitions are
miscalibrated against the current frontier, propose a re-baseline.

The panel earns trust by surviving critique, not by avoiding it.
