# AGENTS.md

This file provides guidance LLM agents and contributors when working with code in this repository.

## Project Overview

Ark is an R kernel for Jupyter applications, primarily created to serve as the interface between R and the Positron IDE. It is compatible with all frontends implementing the Jupyter protocol.

The project includes:
- A Jupyter kernel for structured interaction between R and a frontend
- An LSP server for intellisense features (completions, jump-to-definition, diagnostics)
- A DAP server for step-debugging of R functions

## Repository Structure

The codebase is organized as a Rust workspace containing multiple crates:

- **ark**: The main R Kernel implementation
- **harp**: Rust wrappers for R objects and interfaces
- **libr**: Bindings to R (dynamically loaded using `dlopen`/`LoadLibrary`)
- **amalthea**: A Rust framework for building Jupyter and Positron kernels
- **oak_semantic**: Per-file semantic index for R (scopes, symbols, definitions, uses)
- **echo**: A toy kernel for testing the kernel framework
- **stdext**: Extensions to Rust's standard library used by the other projects

### External dependencies (R parser)

The `oak_semantic` and `ark` crates depend on the R parser from [posit-dev/air](https://github.com/posit-dev/air), pinned by git revision in the workspace `Cargo.toml`. The relevant crates are re-exported under `aether_` prefixes:

- **`aether_syntax`** (`air_r_syntax`): Typed CST nodes (`RIdentifier`, `RBinaryExpression`, `RFunctionDefinition`, etc.), `RSyntaxKind`, `RSyntaxNode`, `RSyntaxToken`.
- **`aether_parser`** (`air_r_parser`): `parse()` function that produces an `RRoot` CST.

Source code for these crates lives in `posit-dev/air` under `crates/air_r_syntax/` and `crates/air_r_parser/`. When looking up type definitions, AST node variants, or syntax kinds, consult that repository.

## Common Development Commands

### Building the Project

```bash
# Build the entire project
cargo build

# Build in release mode
cargo build --release

# On Windows: If Positron is running with a debug build of ark,
# Windows file locking prevents overwriting ark.exe. For interim "progress"
# checks during development, just check the specific crate you're working on
# instead:
cargo check --package ark
# You'll have to quit Positron to do `cargo build`, though, on Windows.
```

### Running Tests

We use `nextest` to run tests, which we invoke through our `just` runner. `just test` expands to `cargo nextest run`. Arguments are forwarded and nextest generally supports the same arguments as `cargo test`.

```bash
# Run all tests
just test

# Run specific tests
# Much better output than `cargo test <testname>`
just test <test_name>

# Run tests for a specific crate
just test -p ark
```

### Writing Tests

- Do not worry about cross-test hygiene. We use `nextest` to run tests, so each individual test runs in its own separate process and cannot affect other tests.

- Prefer simple assertion macros without custom error messages:
    - Use `assert_eq!(actual, expected);` instead of `assert_eq!(actual, expected, "custom message");`
    - Use `assert!(condition);` instead of `assert!(condition, "custom message");`

- Prefer exact assertions over fuzzy ones, especially when ordering and completeness matter:
    - Use `assert_eq!(names[0], "foo()");` rather than `assert!(names.contains(&"foo()"));`

### Placing Integration Tests

Put integration tests under `crates/<crate>/tests/integration/`, with a `main.rs` that declares each test file as a module (`mod library;`). Don't add loose `*.rs` files directly under `tests/`. Each top-level file there compiles as its own test binary, so consolidating them into one `integration` binary keeps build and link times down and saves on disk space.

### Running Clippy

```sh
just clippy
```

### Formatting

```sh
cargo +nightly fmt --all
```

This requires the nightly toolchain because `.rustfmt.toml` uses nightly-only options.

### Kernel and DAP Test Infrastructure

Integration tests for the kernel and DAP server live in `crates/ark/tests/` and use the test utilities from `crates/ark_test/`.

**Key components:**

- **`DummyArkFrontend`**: A mock Jupyter frontend that communicates with the kernel over ZMQ sockets. Use `DummyArkFrontend::lock()` to acquire it (only one per process).

- **`DapClient`**: A DAP client for testing the debugger. Obtained via `frontend.dap_client()` after starting the kernel.

**Stream handling:**

All `recv_iopub_*` methods automatically skip and buffer stream messages. This means:
- Non-stream assertions read cleanly without worrying about interleaved streams
- Stream content is accumulated in internal buffers
- Use `assert_stream_stdout_contains()` or `assert_stream_stderr_contains()` to check stream content
- **Stream assertions must be placed BEFORE `recv_iopub_idle()`** within each busy/idle window
- `recv_iopub_idle()` acts as a synchronization point that flushes stream buffers and panics if streams were received but not asserted

**Common patterns:**

```rust
// Lock the frontend and send an execute request
let frontend = DummyArkFrontend::lock();
frontend.send_execute_request("1 + 1", ExecuteRequestOptions::default());
frontend.recv_iopub_busy();
frontend.recv_iopub_execute_input();
frontend.recv_iopub_execute_result();
frontend.recv_iopub_idle();
frontend.recv_shell_execute_reply();

// Asserting on stream output (order-sensitive: assert in expected order)
frontend.send_execute_request("cat('hello\\nworld\\n')", ExecuteRequestOptions::default());
frontend.recv_iopub_busy();
frontend.recv_iopub_execute_input();
frontend.assert_stream_stdout_contains("hello");
frontend.assert_stream_stdout_contains("world");
frontend.recv_iopub_idle();  // Flushes stream buffers, panics on unasserted streams
frontend.recv_shell_execute_reply();

// Use drain_streams() to consume buffered stream output so `recv_iopub_idle()` won't panic
let streams = frontend.drain_streams();
assert!(streams.stdout().contains("expected"));
```

**Debugging tests:**

Log messages (from the `log` crate) are not shown in test output. Use `eprintln!` for printf-style debugging.

Enable message tracing to see timestamped DAP and IOPub message flows:

```bash
ARK_TEST_TRACE=1 just test test_name      # All messages
ARK_TEST_TRACE=dap just test test_name    # DAP events only
ARK_TEST_TRACE=iopub just test test_name  # IOPub messages only
```

### Required R Packages for Testing

The following R packages are required for tests:
- data.table
- dplyr
- rstudioapi
- tibble
- haven
- htmltools
- R6
- readr

### Installation

After building, you can install the Jupyter kernel specification with:

```bash
./target/debug/ark --install
# or in release mode
./target/release/ark --install
```

## Generated code

Some of the files below `crates/amalthea/src/comm/` are automatically generated from comms specified in the Positron front end.
Such files always have `// @generated` at the top and SHOULD NEVER be edited "by hand".
If changes are needed in these files, that must happen in the separate Positron source repository and the comms for R and Python must be regenerated.

## Coding Style

- Do not use `bail!`. Instead use an explicit `return Err(anyhow!(...))`.

- When a `match` expression is the last expression in a function, omit `return` keywords in match arms. Let the expression evaluate to the function's return value.

- For error messages and logging, prefer direct formatting syntax: `Err(anyhow!("Message: {err}"))` instead of `Err(anyhow!("Message: {}", err))`. This also applies to `log::error!` and `log::warn!` and `log::info!` macros. For logging errors specifically, use Debug formatting `{err:?}` to get more detailed error information.

- Use `log::trace!` instead of `log::debug!`.

- Use fully qualified result types (`anyhow::Result`) instead of importing them.

- You can log `Result::Err` by using the `.log_err()` method from the extension trait `stdext::ResultExt`. Add some `.context()` if that would be helpful, but never do it for errors that are quite unexpected, such as from `.send()` to a channel (that would be too verbose).

- Avoid `.unwrap()` and `.expect()`. For truly unrecoverable errors, use an explicit match with a `panic!` branch. For recoverable errors, use `.log_err()` or propagate with `?`.

- Avoid unnecessary `.clone()`. Prefer returning `&str` over `String` from accessors so callers only allocate when they need to. Reorder operations to avoid cloning (e.g., build a response before consuming the source data). For `Arc`, use `Arc::clone(&x)` instead of `x.clone()` to make the cheap clone obvious.

- Keep `Cargo.toml` dependencies in alphabetical order.

- All dependencies must be declared in the workspace root's `Cargo.toml`. Individual crates reference them with `dep.workspace = true`.

- When you extract code in a function (or move things around) that function goes _below_ the calling function. A general goal is to be able to read linearly from top to bottom with the relevant context and main logic first. The code should be organised like a call stack. Of course that's not always possible, use best judgement to produce the clearest code organization.

- Keep the main logic as unnested as possible. Favour Rust's `let ... else` syntax to return early or continue a loop in the `else` clause, over `if let`.

- Always prefer importing with `use` instead of qualifying with `::`, unless specifically requested in these instructions or by the user, or you see existing `::` usages in the file you're editing.

- Prefer file level or `mod tests` level `use` statements over function level `use` statements.

- When two code paths do analogous things, make them structurally parallel so the symmetry is visible.

- Don't let comments drift from the code. If behaviour changes, update nearby comments. If a file is renamed, update its header comment.

- Use the new async closure syntax, e.g. `async move || { ... }` instead of `|| async move { ... }`.

- Name closure and pattern bindings after what they hold, not by a single letter. Still use short names if possible, but disambiguate if needed. E.g. write `|root| root.path(db)` and `|(root_path, _)| path.starts_with(root_path)`, not `|r|` / `|(p, _)|`.
