Documentation overhaul: improve developer onboarding and AI agent discoverability

## Background A deep audit of the repo's documentation surface (2,608 markdown files, 1,139 READMEs, 65 RFCs, 103 changelog snippets, 16 narrative docs) reveals strong narrative pockets but hollow coverage at the foundational layer and zero AI-aware structure below the root. This issue tracks a coordinated effort to improve docs for two audiences at once: **new contributors** (faster onboarding, fewer "ask in Slack" moments) and **AI coding agents** (less wasted exploration, more deterministic answers, fewer wrong-because-stale recommendations). ### Headline diagnosis - Only **22 of 154** `src/lib/` directories have a README (~14%). - Missing READMEs are precisely the protocol-critical libs: `mina_base`, `transaction_snark`, `pickles`, `mina_lib`, `consensus`, `ledger`, `scan_state`, `mina_block`. - The good READMEs that *do* exist (`staged_ledger`, `transition_frontier`, `network_pool`) are markedly above OCaml-project average — they're a model to replicate. - "Coda"-era branding still in 5 doc files (the project was renamed years ago); the *glossary RFC* (`rfcs/0023`) is the worst offender. - 65 RFCs with no lifecycle status — agents surface abandoned plans (e.g. `rfcs/0029-libp2p.md` "Coda's networking today uses an unmaintained Kademlia library") as authoritative. - One root `CLAUDE.md`; nothing per-area; no machine-readable repo map. --- ## Quick wins (each < 1 day) - [ ] Sed-replace "Coda" → "Mina" in `docs/{demo,daemon,docker,environment-variables}.md`; revalidate `codaprotocol/*` Docker image links - [ ] Update `README-branching.md` to reflect post-Berkeley reality (`release/mesa` is the active release branch; retire `rampup`/`berkeley` mentions) - [ ] Delete or fill stub `## How ports are used` block in `docs/daemon.md` (lines 7-9); rename file to `docs/daemon-rpc.md` since the valuable content is the RPC catalogue - [ ] Add `changes/README.md` (10 lines): filename convention, who consumes the snippets, link to `buildkite/scripts/changelog.sh` - [ ] Add `rfcs/README.md` status table for the 65 RFCs (even before deep triage) - [ ] Fix `Constans` → `Constants` typo at `buildkite/src/README.md:149,152` - [ ] Fix orphan `2.` numbered list (no `1.`) at `README-dev.md:333` - [ ] Expand the `src/lib/*` directory listing in root `CLAUDE.md` (currently lists 7/154; add a one-line description per library) - [ ] Add an `AGENTS.md` at root (3-line pointer to `CLAUDE.md`) for tool-portability with non-Claude agents ## Strategic investments (multi-week) ### 1. Foundational library READMEs (effort: L) Write a 50-150 line `README.md` for each, modeled on `src/lib/staged_ledger/README.md`: - [ ] `src/lib/mina_base/README.md` - [ ] `src/lib/transaction_snark/README.md` - [ ] `src/lib/pickles/README.md` - [ ] `src/lib/mina_lib/README.md` - [ ] `src/lib/consensus/README.md` - [ ] `src/lib/ledger/README.md` - [ ] `src/lib/scan_state/README.md` - [ ] `src/lib/mina_block/README.md` Each should cover: (a) one-paragraph "what is this", (b) main entry-points (3-5 file:function pairs), (c) key types, (d) cross-links to relevant `docs/` narrative. ### 2. Canonical glossary (effort: M, **highest-leverage AI win**) - [ ] Create `docs/GLOSSARY.md` with one canonical definition per term: scan state, transition frontier, breadcrumb, full vs persistent frontier, snarked ledger, staged ledger, ledger mask, work statement, snark work, completed work, ledger proof, protocol state, transition vs block vs breadcrumb, pickles, recursive proof, VRF, slot/epoch/k, soft-fork vs hard-fork, mesa/berkeley/izmir, devnet/mainnet/lightnet/dev profile, compatible/develop/master branches, prefork - [ ] Add canonical-definition anchors (`<!-- canonical: scan_state -->`) for deterministic agent grep - [ ] Refactor existing docs to *link* to the glossary rather than redefine inline ### 3. Machine-readable repo map (effort: M) - [ ] Adopt convention: `;; Description: <one line>` comment in every `src/lib/*/dune` library entry - [ ] Generate `STRUCTURE.md` (or `repo-manifest.json`) from those descriptions, kept fresh in CI - [ ] Cross-link from root `CLAUDE.md` ### 4. RFC lifecycle pass (effort: M) - [ ] Add front-matter to all 65 RFCs: `status: implemented | abandoned | superseded-by | partial` - [ ] Where superseded, add `superseded-by: <path>` redirect (e.g. RFC → `docs/snark-worker-and-pool.md`) - [ ] Move clearly-abandoned RFCs to `rfcs/archive/` ### 5. Split `README-dev.md` (effort: M) The current 397-line file mixes MacOS/Linux setup, runtime, debian packaging, docker, dune internals, and genesis constants under one heading. - [ ] `docs/setup-linux.md` - [ ] `docs/setup-macos.md` - [ ] `docs/setup-docker.md` - [ ] `docs/build-debian-locally.md` - [ ] `docs/build-docker-locally.md` - [ ] `docs/genesis-constants.md` - [ ] Reduce `README-dev.md` to a clear index with a "first 30 minutes" recommended path ### 6. CI/Dhall onboarding (effort: M) - [ ] `buildkite/HOWTO-add-a-job.md` — annotated walk-through of adding a new test job (copy `Jobs/Test/X.dhall`, register in `Constants/Artifacts.dhall`, set `dirtyWhen` and `tags`, run `cd buildkite && make all`) - [ ] Document `JobSpec` fields, `dirtyWhen` semantics, and the `Constants/{Artifacts,DebianPackage,Profiles}.dhall` relationship ### 7. Per-area `CLAUDE.md` files (effort: M) - [ ] `buildkite/CLAUDE.md` — Dhall compilation gotchas, `make all` is mandatory, where Constants live - [ ] `scripts/debian/CLAUDE.md` — codename matrix, deb-s3 invocation, common error modes - [ ] `src/app/test_executive/CLAUDE.md` — local test spinup - [ ] `src/lib/pickles/CLAUDE.md` — `proof-systems` submodule mapping --- ## AI-specific conventions worth adopting These are non-obvious patterns that significantly help agents but cost little to maintain: - **Canonical-definition anchors** (`<!-- canonical: <term> -->`) — deterministic grep, lintable for uniqueness - **`;; Description:` line in every dune file** — drives the auto-generated repo manifest - **Glossary as one file, not many** — cache-friendly (agent reads it once per session, not five times) - **Branch-state block in `CLAUDE.md`** with `last-updated:` date so branching guidance doesn't drift like `README-branching.md` did - **`AGENTS.md` mirror** at root for non-Claude tool portability - **Error-message → doc tokens** — daemon errors referencing unfamiliar concepts include a grep token (`see GLOSSARY:scan-state`) so an agent debugging user issues can follow - **Convention header for machine-generated docs** so agents skip them (`opam.export` has this informally; formalize it) --- ## Suggested phasing 1. **Phase 1 (1 sprint)**: all quick wins + start on the canonical glossary 2. **Phase 2 (1-2 sprints)**: 8 foundational library READMEs + repo manifest 3. **Phase 3 (1 sprint)**: RFC lifecycle pass + split `README-dev.md` 4. **Phase 4 (ongoing)**: per-area `CLAUDE.md`s and CI/Dhall onboarding ## Audit methodology Each item above includes the absolute file path it was discovered in. Full audit notes available on request — happy to attach the source paths and line-number citations to any sub-task that gets picked up.