# GhciSchemaGen overview

`make trace-schemas-regenerate`

or the step-by-step commands:

`GHC_ENVIRONMENT=- nix develop -c cabal run cardano-node -- trace-documentation --config configuration/cardano/mainnet-config.yaml --output-namespace-list bench/trace-schemas/newNamespaces.txt --output-file bench/trace-schemas/trace-documentation.md`

`nix run .#schema-gen`

`nix run .#apply-schema-overrides -- --verbose`

`nix run .#validate-trace-schemas`

`nix run .#validate-trace-log -- --log-file run/.../stdout`

Additional checks:

`make trace-schemas-overrides-check`

`make trace-schemas-overrides-coverage RANGE=origin/master...HEAD`

`make trace-schemas-validate`

## What it does

- Generates JSON schemas for trace messages in `bench/trace-schemas/messages` and type schemas in `bench/trace-schemas/types`.
- Uses the trace-documentation command to generate `bench/trace-schemas/newNamespaces.txt` from `MetaTrace` namespaces.
- Parses `namespaceFor` and `forMachine` clauses in source files, then asks `cabal repl` (GHCi) for types of variables used in those `forMachine` patterns.

## High-level flow

1. **Find relevant Haskell files**
   - Scans `cardano-node/src`, `cardano-submit-api/src`, `cardano-tracer/src`, `trace-forward/src`, and several tracing-related directories in `ouroboros-network`, `ouroboros-consensus`, and `hermod-tracing`.
   - Warns when any configured source directory is missing, instead of silently ignoring it.
   - Keeps only `.hs` files that contain `forMachine` or `namespaceFor`.

2. **Build namespace mapping**
   - Parses `namespaceFor` clauses to map a constructor (e.g. `ReplayBlockStats`) to namespace parts (e.g. `["LedgerReplay"]`).
   - This tells it which constructor corresponds to each namespace in `bench/trace-schemas/newNamespaces.txt`.

3. **Parse `forMachine`**
   - Extracts:
     - The constructor pattern in each clause.
     - The JSON keys from lines like `"foo" .= ...`.
     - Which pattern variable supplies each key.
   - Literal string RHS like `"kind" .= String "X"` are treated as string fields.

4. **Ask GHCi for types**
   - For each file, it runs GHCi and issues `:t` queries for patterns.
   - Extracts variable types from the returned type signatures or error output.
   - This becomes a map `constructor -> variable -> type`.

5. **Build/update schemas**
   - For each namespace in `newNamespaces.txt`:
     - Finds the matching constructor via the namespace map.
     - Builds or updates the `data.properties` schema using:
       - key -> variable mapping from `forMachine`
       - variable types from GHCi
   - If `data.properties` is still empty, it optionally pulls `data` from `bench/trace-schemas/messages-hist`.

## Key heuristics

- Namespace matching uses suffix match so `Namespace [] ["LedgerReplay"]` can match `ChainDB.ReplayBlock.LedgerReplay`.
- If a field uses a literal string (`String "..."`), it is treated as `type: string`.
- If `data.properties` does not exist, it synthesizes an object schema and fills properties from `forMachine`.

## Where to look in code

- Entry point and flow: `main`
- Namespace mapping: `parseNamespaceMap`, `findCtor`
- forMachine parsing: `parseForMachineClauses`, `parseFieldVarMap`, `parseFieldLine`
- GHCi type extraction: `ghciTypesForFile`, `runGhci`
- Schema update: `updateSchemaForNamespace`, `updateData`, `buildDataFromFieldMap`

## Validation

- `ValidateTraceSchemas.hs` checks `meta.schema.json` with `check-jsonschema`, then validates every file in `bench/trace-schemas/messages` against that meta-schema.
- The Haskell script controls discovery and execution; the actual JSON Schema validation is delegated to `check-jsonschema` via `nix run nixpkgs#check-jsonschema`.
- Run it via the packaged flake executable: `nix run .#validate-trace-schemas`.
- `ValidateTraceLog.hs` validates a real cardano-node log file: it skips the non-JSON preamble, validates the common envelope against `TraceMessage.schema.json`, validates known namespaces against the matching schema in `bench/trace-schemas/messages`, and reports namespaces that do not have a corresponding message schema.

## Packaging and tests

- The scripts are packaged in `bench/trace-schemas/scripts/schema-gen/trace-schema-gen.cabal` as executables, which is why they can be invoked with `nix run .#...`.
- The smaller helper scripts also have a package test suite:
  - `nix develop -c cabal test trace-schema-gen-test`

## Human changes that survive regeneration

- Treat `bench/trace-schemas/messages` and `bench/trace-schemas/types` as generated outputs.
- Put manual edits in sidecar override patches under `bench/trace-schemas/overrides`.
- Apply overrides with `nix run .#apply-schema-overrides -- --verbose` after every generation.
- Enforce in CI with:
  - `make trace-schemas-regenerate`
  - `make trace-schemas-overrides-check`
  - `make trace-schemas-overrides-coverage RANGE=origin/master...HEAD`

See `bench/trace-schemas/overrides/README.md` for override format and file layout.
