# Parse Benchmark

This benchmark compares parse throughput for generated ANTLR parsers and
tree-sitter parsers on Kotlin, C#, and Java fixtures, with an explicit Trino
SQL fixture set for SQL-dialect Rust vs Go checks.

The harness is intentionally a standalone script instead of `cargo bench`.
`cargo bench` is useful for in-process Rust-only measurements, but this check
has to generate ANTLR parsers, build a Go binary, run Python parsers, and load
tree-sitter language libraries. Keeping that orchestration outside Cargo makes
the same command usable locally and in CI.

## Setup

Use the same ANTLR jar and `grammars-v4` checkout described in `AGENTS.md`.
The repository-local directory is ignored and survives operating-system
temporary-directory cleanup; `cargo clean` removes it. The benchmark defaults
to:

- `ANTLR4_JAR=target/antlr-cleanroom/tools/antlr-4.13.2-complete.jar`
- `GRAMMARS_V4=target/antlr-cleanroom/grammars-v4`

The sparse checkout must include C#, the modern Java grammar, and Trino SQL in
addition to Kotlin:

```bash
git -C target/antlr-cleanroom/grammars-v4 sparse-checkout set kotlin/kotlin csharp/v7 java/java sql/trino
```

Install the Python dependencies in the interpreter you will use to run the
benchmark:

```bash
python3 -m pip install -r tools/parse-bench/requirements.txt
```

## Run

Quick local smoke:

```bash
python3 tools/parse-bench/run.py --quick
```

SQL-only Rust vs Go smoke (with AST parity gate):

```bash
python3 tools/parse-bench/run.py \
  --languages trino \
  --runtimes rust-antlr,go-antlr \
  --ast-check \
  --quick
```

Longer local run with reports:

```bash
python3 tools/parse-bench/run.py \
  --iters 20 \
  --warmups 3 \
  --rust-generated-only \
  --json target/parse-bench/results.json \
  --markdown target/parse-bench/results.md
```

Generator experiments use the same harness through repeatable passthrough
arguments. Use separate work directories so generated sources and binaries do
not overwrite the baseline:

```bash
python3 tools/parse-bench/run.py \
  --languages kotlin,trino \
  --runtimes rust-antlr \
  --rust-generator-arg=--fixed-lookahead \
  --rust-generator-arg=3 \
  --work-dir target/parse-bench-fixed
```

The script regenerates parsers into `target/parse-bench`, builds:

- a Rust runner generated directly from the grammar source with this runtime,
- a Python ANTLR runner using `antlr4-python3-runtime`,
- a Go ANTLR runner using `github.com/antlr4-go/antlr/v4`,
- a tree-sitter runner using `tree-sitter-language-pack`.

Rust generation keeps the pinned `JavaParser.g4` unchanged, including its
`JavaParserBase` option and both semantic predicates. The generated benchmark
parser uses the source compiler's historical `assume-true` policy for those
target-language helpers, matching downstream generation while retaining the
predicate metadata that controls runtime routing; since #209 those
untranslated coordinates lower as generatable hook-backed templates, so the
predicate-bearing rules run generated bodies rather than the ATN interpreter.
Python and Go still use the equivalent portable rewrite because that
grammars-v4 revision does not provide `JavaParserBase` implementations for
those targets. The `issue-174-return-expression.java` fixture guards the
resulting Java method-body performance path. JSON rows include a
benchmark-variant tag, so the comparator skips only method changes such as
this legacy-to-predicate transition (or the #209 interpreter-to-generated
routing change) and resumes Java regression checks once both reports use the
same variant.

Rust generation also keeps the pinned C# grammar unchanged. Its semantic
helper names are described by `patterns/csharp.toml`, while the benchmark-owned
Rust support module implements interpolation and the `CSharpLexerBase`
preprocessor state. Generic codegen exposes typed lexer lifecycle hooks,
structural channel/mode constants, and reusable parser-predicate lowerings; it
does not contain C# grammar or rule names.

The output table reports `min` and `avg` parse time per fixture and a relative
ratio against `rust-antlr` for the same fixture.
Use `--rust-generated-only` for Adaptive LL delivery evidence so the Rust
generator fails if any parser rule lacks a generated body and the Rust runner
fails if a generated parser path falls back to the interpreter.

### Rust vs Go AST parity

Pass `--ast-check` with `--phase parse` and both `rust-antlr` and `go-antlr`
selected. Before timing, the harness dumps each fixture's parse tree from both
runners (same `Rule`/`Term`/`Err` format as the Kotlin parity dumper) and
requires:

- byte-identical dumps, and
- no error nodes on either side (Rust also rejects a non-zero syntax-error count).

Dumps land under `<work-dir>/ast-dumps/<language>/`.

Kotlin, C#, Java, and Trino fixtures pass this gate. The former C# divergence
was caused by the missing `CSharpLexerBase.NextToken` preprocessing behavior:
inactive `#if` sections were still lexed as ordinary C# by the Rust runner.
The Rust support module now evaluates directives and consumes inactive sections
before normal lexing resumes.

```bash
python3 tools/parse-bench/run.py \
  --languages kotlin,csharp,java,trino \
  --runtimes rust-antlr,go-antlr \
  --ast-check \
  --quick
```

### Lex-only measurements

Use `--phase lex` to time generated Rust lexing and token buffering without
constructing a parser:

```bash
python3 tools/parse-bench/run.py \
  --phase lex \
  --languages kotlin,csharp,java,trino \
  --runtimes rust-antlr \
  --iters 20 \
  --warmups 3
```

The source-derived fixtures cover ordinary Kotlin, C#, Java, and Trino input.
Two lex-only Kotlin fixtures add concentrated ASCII coverage for long
identifiers, strings, comments, whitespace, and punctuation, plus mixed-script
coverage for the Unicode fallback. Two lex-only Java fixtures isolate long
ASCII range classes for identifiers, numbers, and whitespace.

Use a detached checkout for same-machine baseline comparisons, and select the
compiler-level configurations explicitly:

```bash
python3 tools/parse-bench/run.py \
  --phase lex \
  --runtimes rust-antlr \
  --runtime-root /tmp/antlr-runtime-main \
  --rust-native \
  --rust-thin-lto
```

`--rust-native` adds `-C target-cpu=native`. `--rust-thin-lto` writes
`lto = "thin"` and `codegen-units = 1` in the generated benchmark workspace,
where Cargo profile settings control the final application and its
dependencies.

For profile-guided optimization, first build an instrumented runner and train
it on the selected fixtures:

```bash
PROFILE_DIR="$(mktemp -d /tmp/antlr-parse-pgo.XXXXXX)"
PROFILE_DATA=/tmp/antlr-parse-pgo.profdata

python3 tools/parse-bench/run.py \
  --languages kotlin,csharp,java,trino \
  --runtimes rust-antlr \
  --rust-generated-only \
  --rust-pgo-generate "$PROFILE_DIR" \
  --work-dir /tmp/antlr-parse-pgo-train

rustup component add llvm-tools-preview
SYSROOT="$(rustc --print sysroot)"
HOST="$(rustc -vV | sed -n 's/^host: //p')"
"$SYSROOT/lib/rustlib/$HOST/bin/llvm-profdata" merge \
  -o "$PROFILE_DATA" "$PROFILE_DIR"
```

Then rebuild and measure with the merged profile:

```bash
python3 tools/parse-bench/run.py \
  --languages kotlin,csharp,java,trino \
  --runtimes rust-antlr \
  --rust-generated-only \
  --rust-pgo-use "$PROFILE_DATA" \
  --work-dir /tmp/antlr-parse-pgo-use
```

The profile-generation timings are instrumented training data, not comparison
results. When testing PGO together with `--rust-native` or `--rust-thin-lto`,
pass the same options to both commands. Use separate work directories for the
ordinary, instrumented, and profile-use runners.

## Prediction Memory Counters

Set `ANTLR_PERF_DUMP=1` to build the Rust runner with performance counters.
Parse runs print prediction and context-store measurements; lex-only runs print
lexer direct-ASCII, generic-character, scalar-replay, and bulk-commit counts:

```bash
ANTLR_PERF_DUMP=1 python3 tools/parse-bench/run.py \
  --languages csharp \
  --runtimes rust-antlr \
  --iters 10 \
  --warmups 2 \
  --rust-generated-only
```

The dump includes canonical context counts, pooled and retained bytes, arena
and workspace capacities, merge-cache activity, and outer-context cache
hits/misses. It also reports learned parser-DFA warm hits/misses, ATN
fallbacks, state interning activity, dense/sparse row counts, edge-density
histograms, and hot/cold retained bytes. Parser token-set counters report
inline, dense, and interval selection; packed bitset bytes; membership
hits/misses; and interval probes eliminated or retained. Store statistics are
collected in a separate untimed parse after the benchmark loop, so walking the
stores does not affect reported timings.

## PR Watchdog

For CI, run the benchmark on the base checkout and the PR checkout on the same
runner, then compare JSON reports:

```bash
python3 tools/parse-bench/compare.py \
  --baseline base-parse-bench.json \
  --current head-parse-bench.json \
  --max-regression 1.15
```

By default the comparator checks `rust-antlr` only. Repeat `--runtime` to add
other runtimes.

## Fixtures

Fixture metadata lives in `fixtures/manifest.json`. The fixture files are
source-referenced benchmark inputs that point at independent upstream parser
stress patterns:

- Kotlin: JetBrains Kotlin, kotlinx.coroutines, Ktor.
- C#: dotnet/wpf, Mono.
- Java: Mojang DataFixerUpper, Bazel, Google Closure Compiler, Trino.
- Trino SQL: Trino benchmark TPC-DS/TPC-H queries with benchmark placeholders
  normalized to identifiers, including a curated TPC-DS grammar-stress suite
  selected by CTE/window/UNION/EXISTS/CASE/grouping feature density.
