---
title: "Add a new language"
description: "How to add a new language analyzer to mehen — pick a parser, implement the LanguageAnalyzer trait, and register the analyzer."
keywords: ["new language", "analyzer", "parser", "developers"]
---

Each language is owned by a single per-language analyzer crate at `crates/mehen-<lang>/`. The analyzer:

- Pins its own parser (a language-specific parser like Ruff / Oxc / Mago / Prism / `ra_ap_syntax`, or
  a tree-sitter grammar).
- Owns metric interpretation for that language's syntax — what counts as a decision, an operator, a
  method, a comment, etc.
- Returns `LanguageAnalysis` (owned, `Send + 'static`) so `mehen-engine` can analyze files in parallel
  and never holds onto parser arenas.

For tree-sitter-backed languages the analyzer also owns its `grammar.rs` kind enum, generated from the
pinned grammar's `node-kind` table.

## Choosing a parser

The first decision is the parser. mehen prefers a language-specific parser when one exists with mature
Rust bindings and rich AST/semantic coverage; tree-sitter is the default fallback when no such parser
is available.

| Parser style | Examples in mehen | When to pick |
|---|---|---|
| Language-specific | Ruff (Python), Oxc (TS/JS), Mago (PHP), Prism (Ruby), `ra_ap_syntax` (Rust), pulldown-cmark (Markdown) | A maintained Rust crate with a typed AST, semantic model, or rich grammar coverage exists. |
| ANTLR | Kotlin (official Kotlin spec grammar), Java (grammars-v4) | A high-quality, maintained ANTLR v4 grammar exists but no Rust-native parser does. Generated via [`ophi-dev/antlr-rust-runtime`](https://github.com/ophi-dev/antlr-rust-runtime); the CST is far more semantically named than tree-sitter's. |
| Tree-sitter | Go, C, PowerShell | No mature language-specific or ANTLR parser is available, or tree-sitter's grammar quality is the best fit. |

## Adding a language-specific parser

<Steps>
  <Step title="Add the parser to the workspace">
    Add the parser crate(s) to `crates/mehen-<lang>/Cargo.toml`. Pin an exact version (or a git
    revision tagged for the release) so mehen's behavior is reproducible.
  </Step>
  <Step title="Implement the analyzer">
    In `crates/mehen-<lang>/src/lib.rs`:

    - Define `<Lang>Analyzer` and implement `mehen_core::LanguageAnalyzer` for it.
    - Walk the parser's typed AST and emit metrics through
      `mehen_metrics::{State, MetricTreeBuilder, …}` and the per-metric helpers.
    - Make sure the parser's arena, source buffer, and parser state do not escape the `analyze` call
      — `LanguageAnalysis` must be `Send + 'static`.
  </Step>
  <Step title="Register the analyzer">
    Add it to `mehen-engine`'s registry (`crates/mehen-engine/src/registry.rs`) so
    `Language::<YourLang>` dispatches to it.
  </Step>
  <Step title="Add tests">
    Per-metric integration tests under `crates/mehen-<lang>/tests/` — typically one file per metric
    family, snapshotting the rendered metric JSON via `insta`.
  </Step>
</Steps>

## Adding an ANTLR-backed language

ANTLR is a first-class backend (`AnalysisBackend::Antlr`), used when a high-quality ANTLR v4
grammar exists but no Rust-native parser does. The shared crate `mehen-antlr` provides the runtime
re-export, token-span conversion, recovered-error diagnostics, and hidden-channel comment (CLOC)
extraction. The Rust lexer/parser modules are generated **offline**
from a vendored `.g4` grammar and checked in; a normal `cargo build` does not run the generator.

The generated modules and their vendored `.g4` grammar do **not** live in the analyzer crate.
They live in a standalone, publishable `crates/mehen-<lang>-parser/` crate that carries only the
generated lexer/parser (no mehen-specific logic and no dependency on `mehen-core`), so external
tools can depend on the parser alone. The analyzer crate `mehen-<lang>` then depends on
`mehen-<lang>-parser` and walks the resulting `ParsedFile` through its arena-backed
`Node`/`RuleNodeView`/`TerminalNodeView` borrowing views.

<Steps>
  <Step title="Vendor the grammar">
    Copy the ANTLR `.g4` files (lexer, parser, and any `import`ed grammars like `UnicodeClasses`)
    into `crates/mehen-<lang>-parser/grammar/`. Record the upstream source, commit, and toolchain
    versions in a `PROVENANCE.md` so regeneration is reproducible (see
    `crates/mehen-kotlin-parser/grammar/`).
  </Step>
  <Step title="Register the codegen target">
    Add an `AntlrTarget` to `xtask/src/antlr.rs::TARGETS` (slug, crate dir, grammar dir, lexer/parser
    filenames). The crate dir is the parser crate, e.g. `crates/mehen-<lang>-parser`. The same entry
    also carries the README's display metadata — human-facing `display_name`, `upstream_name` /
    `upstream_url`, the `entry_rule` used in the parse example, and a one-line `sample_source` — so
    the generated README stays in step with the grammar. Module and type names in the README are
    derived from the `.g4` filenames automatically, matching what the generator emits.
  </Step>
  <Step title="Cover the grammar's semantic helpers (if any)">
    Generation always runs with `--sem-unknown error --require-full-semantics`: a metrics tool
    cannot afford a parser whose semantic predicates were silently assumed true, so every
    `{ this.Helper() }?` predicate, `{ this.Helper(); }` action, and hook-implemented grammar
    option (e.g. `superClass=…`) in the grammar must be accounted for, or
    `cargo xtask antlr generate <lang>` fails.

    For a grammar with helper calls (like Java's `JavaParserBase` predicates):

    1. Write a `patterns.toml` next to the `.g4` files lowering each helper — either to a pure
       pattern expression the generator can inline (`token_index_adjacent`,
       `cmp(ne, la(1), token(NAME))`, …) or to `lower = "hook"` for anything needing real logic.
       Point the target's `sem_patterns` field at it.
    2. Acknowledge hook-implemented grammar options via the target's `option_hooks` field
       (e.g. `superClass=JavaParserBase`).
    3. For `hook` lowerings, port the upstream base class exactly in a hand-written
       `src/hooks.rs` implementing the generated `<Lang>ParserHooks` / `<Lang>LexerHooks`
       trait, and set the target's `parser_hooks` / `lexer_hooks` field (e.g.
       `hooks::JavaParserBase`) so the generated README shows the hook-correct construction.
       The analyzer must then construct via `with_typed_hooks` — a hook-less parser fails loud
       (`AntlrError::Unsupported`) at the first hooked coordinate rather than mis-parsing. Add
       behavioral tests for each hook (see `crates/mehen-java-parser/tests/hooks.rs`).

    A grammar with no helpers (like Kotlin's) needs none of this — the strict flags are still
    passed and simply have nothing to reject.
  </Step>
  <Step title="Generate the parser modules">
    ```bash
    cargo xtask antlr generate <lang>
    ```

    xtask calls the workspace-pinned `antlr-rust-codegen` library directly. No external generator
    binary is required; keep its exact workspace pin in lockstep with `antlr-rust-runtime`.

    The result lands in `crates/mehen-<lang>-parser/src/generated/`: the `<lang>_lexer.rs` and
    `<lang>_parser.rs` modules plus `decisions.json` and `semantics.json` sidecars emitted by the
    generator. All four are checked in and drift-checked. The generated Rust files carry their own
    lint and `rustfmt::skip` attributes, so the parser crate includes them with plain
    `pub mod ...;` declarations.

    The same command also renders the parser crate's `README.md` from the shared
    `xtask/templates/parser-readme.md` template, using the target's display metadata (see the
    previous step). The README shows how to consume the crate as a git dependency and how to
    parse a snippet; it is checked in and drift-checked alongside the modules. Set `readme =
    "README.md"` in the parser crate's `Cargo.toml` so it ships on the registry page when
    published.

    <Warning>
    Never edit the generated modules **or the generated `README.md`** — both are regenerated from
    the grammar and the template. `cargo xtask antlr check-generated` guards drift (modules and
    README) without a separately installed generator. Keep the README's parse example in step with
    the compile-tested `//!` example in the crate's `lib.rs`.
    </Warning>
  </Step>
  <Step title="Add the dependencies">
    The analyzer crate `mehen-<lang>` depends on `mehen-<lang>-parser` (the generated grammar) and
    `mehen-antlr` (the shared helpers). The parser crate re-exports `antlr4_runtime`, so the analyzer
    reaches the runtime through it; the runtime is pinned once in the workspace
    `[workspace.dependencies]`, so the pin stays in one place.
  </Step>
  <Step title="Implement the analyzer + walker">
    Parse via the parser crate's one-call `<lang>_parser::parse_with_parser` + `into_parsed_file`
    (or, when the grammar has hooks, explicit construction with
    `<Lang>Parser::with_typed_hooks` as `mehen-java` does), then walk the resulting
    `ParsedFile`'s root `Node` with your own recursive visitor (like
    `mehen-java`/`mehen-kotlin`). The `Node`/`RuleNodeView`/`TerminalNodeView` views have **no
    parent pointer**, so thread any parent-dependent context (e.g. else-if detection) top-down.
    Match `RuleNodeView::rule_index()` against the generated `RULE_*` constants and terminal
    `token_type()` against the token constants, using the runtime's `child_rule`/`child_token`/
    `has_token`/`first_rule` navigation helpers. Comments are on a hidden channel (absent from the
    tree) — recover CLOC from the eagerly-buffered token store via `mehen_antlr::loc_tokens`
    (feeding it `parsed.tokens()` directly — `&TokenStore` is `IntoIterator` since the 0.15
    runtime).
  </Step>
  <Step title="Register the analyzer">
    Same as the other backends: register in `crates/mehen-engine/src/registry.rs`.
  </Step>
  <Step title="Add tests">
    Per-metric integration tests under `crates/mehen-<lang>/tests/`.
  </Step>
</Steps>

## Adding a tree-sitter-backed language

**Prerequisite:** a `tree-sitter-<lang>` crate compatible with the `tree-sitter` version pinned in the
workspace (`Cargo.toml` `[workspace.dependencies]`).

<Steps>
  <Step title="Pin the grammar">
    Two files must stay in sync — both reference the grammar at compile time:

    - `xtask/Cargo.toml` — the kind-enum generator imports the grammar at codegen time.
    - `crates/mehen-<lang>/Cargo.toml` — the analyzer imports the grammar at runtime to drive
      `tree_sitter::Parser`.

    Workspace-managed grammars (those listed in root `Cargo.toml` `[workspace.dependencies]`) can be
    referenced as `{ workspace = true }` from both places. Inline-pinned grammars must be kept in
    lockstep manually.
  </Step>
  <Step title="Register the language for codegen">
    Add a `GeneratorTarget` to `xtask/src/tree_sitter.rs::TARGETS`:

    ```rust
    GeneratorTarget {
        slug: "go",
        enum_name: "Go",
        crate_dir: "crates/mehen-go/src",
        language: || tree_sitter_go::LANGUAGE.into(),
    },
    ```
  </Step>
  <Step title="Generate the kind enum">
    ```bash
    cargo xtask tree-sitter generate go
    ```

    The result lands at `crates/mehen-go/src/grammar.rs`.

    <Warning>
    Never edit `grammar.rs` directly. `cargo xtask tree-sitter check-generated` runs in CI and fails
    the build if the checked-in file doesn't match a fresh render.
    </Warning>
  </Step>
  <Step title="Implement the analyzer">
    Same pattern as a language-specific parser, but use the generated `crate::grammar::<Lang>` enum
    for kind-id matching — it deduplicates positional kinds and exposes mnemonic identifiers
    (`PLUS`, `EQ_EQ`, etc.).
  </Step>
  <Step title="Register the analyzer">
    Same step as above: register in `crates/mehen-engine/src/registry.rs`.
  </Step>
  <Step title="Add tests">
    Per-metric integration tests under `crates/mehen-<lang>/tests/`.
  </Step>
</Steps>

## Bumping a pinned grammar

When dependabot bumps a `tree-sitter-<lang>` version (or you do it manually):

1. Update both `xtask/Cargo.toml` and `crates/mehen-<lang>/Cargo.toml` to the new version. The
   `regenerate-grammars` workflow does this automatically for inline-pinned grammars.
2. Run `cargo xtask tree-sitter generate --all` and commit the regenerated `grammar.rs` files.
3. CI's `cargo xtask tree-sitter check-generated` will fail until the regenerated files are committed.

## Validation

```bash
cargo check --workspace
cargo nextest run --all-features
cargo insta test --workspace --all-features --check --unreferenced reject \
    --test-runner nextest --no-test-runner-fallback --disable-nextest-doctest
cargo clippy --all-targets --all-features --locked
cargo xtask tree-sitter check-generated
```

## See also

- [Update grammars](/developers/update-grammars) — bumping pinned tree-sitter versions.
- [Implement LoC](/developers/loc) — example of a metric trait implementation.
