# Zena Language Server / VS Code Integration

## Status: In Progress

## Goals

Provide IDE features for Zena in VS Code, starting with:

1. **Diagnostics** — show type errors and parse errors inline
2. **Go to Definition** — jump to where a symbol is declared
3. **Formatting** — auto-format Zena source files
4. **Hover** — show the type of an identifier on hover

These features exercise the core compiler pipeline (parse → check → query) and
the full VS Code integration path.

## Architecture

### Phase 1: Self-Hosted Compiler via WASM (Current)

The **self-hosted compiler compiled to WASM** runs inside the VS Code extension
host. The bootstrap compiler compiles the self-hosted compiler to a `.wasm`
module; the extension loads and calls it from JavaScript. No separate process,
no LSP protocol overhead.

```
┌─────────────────────────────────────────────────────┐
│  VS Code Extension (vscode-zena)                    │
│                                                     │
│  ┌───────────────────────────────────────────────┐  │
│  │  Extension Host (TypeScript/Node.js)          │  │
│  │                                               │  │
│  │  ┌─────────────────────────────────────────┐  │
│  │  │  lsp.wasm (from @zena-lang/language-     │  │
│  │  │           service)                       │  │
│  │  │                                          │  │
│  │  │  LanguageService                         │  │
│  │  │    ├─ SourceFileCache (parse/scope)      │  │
│  │  │    ├─ Program (immutable snapshot)       │  │
│  │  │    │    ├─ SourceFiles (shared)          │  │
│  │  │    │    ├─ ScopeResults (cached)         │  │
│  │  │    │    └─ CheckResults (on demand)      │  │
│  │  │    └─ Queries (diagnostics, types, defs) │  │
│  │  └─────────────────────────────────────────┘  │
│  │                                               │  │
│  │  ┌─────────────────────────────────────────┐  │
│  │  │  JS Glue (compiler-service.ts)          │  │
│  │  │  • WASM instantiation + string marshal  │  │
│  │  │  • CompilerHost (read_file import)       │  │
│  │  │  • VS Code providers (diagnostics,       │  │
│  │  │    definition, hover, formatting)        │  │
│  │  └─────────────────────────────────────────┘  │
│  └───────────────────────────────────────────────┘  │
└─────────────────────────────────────────────────────┘
```

**Why this approach:**

- Uses the self-hosted compiler from the start — no bootstrap compiler
  dependency in the extension
- WASM runs in-process in the extension host, so no separate process or
  JSON-RPC overhead
- The self-hosted compiler already has parser, scope analysis, and
  SemanticModel with diagnostics, node types, and name resolution
- VS Code's extension API is simpler than implementing the full LSP protocol

**WASM ↔ JS interface:** Strings (source code, file paths, type descriptions)
are passed as WASM GC `externref` values via `$stringCreate`, `$stringSetByte`,
`$stringGetByte`, `$stringGetLength` exports. The `@zena-lang/runtime` package
provides `createStringReader`/`createStringWriter` helpers that wrap these into
ergonomic JS functions. The host provides a `read_file` import so the WASM
compiler can load dependency sources from the filesystem.

### Phase 2: Standalone LSP Server

When the CLI is separated from Node/TypeScript, add a `zena lsp` command that
speaks LSP over stdio. The VS Code extension becomes a thin client. Other
editors (Neovim, Helix, Zed) get support for free.

---

## Current Implementation

### What's Built ✅

**Extension infrastructure** (`packages/vscode-zena/`):

- `extension.cts` — CJS entry point with dynamic ESM import
- `compiler-service.ts` — WASM compiler wrapper: loads `lsp.wasm`,
  marshals strings, exposes `checkDocument()`, `getDefinition()`,
  `formatDocument()`
- `zena-extension.ts` — Extension lifecycle, VS Code event wiring,
  diagnostic collection, definition provider, formatting provider
- Status bar with Zena indicator, output channel for logging

**JavaScript API** (`packages/language-service/src/index.ts`):

- `createLanguageService()` — loads `lsp.wasm` and wraps its exports:
  `check()`, `hover()`, `completions()`, `definition()`,
  `documentSymbols()`, `format()`, `compileToWasm()`
- Host-agnostic: the caller supplies the Wasm and a `readFile` hook, so the
  same code runs in Node, in a browser, and in a Web Worker.
  `createVirtualFileReader()` serves a bundled stdlib where there is no
  filesystem
- Used by `packages/playground`. `compiler-service.ts` in the extension
  predates it and still marshals strings itself — folding it onto this API
  would remove the duplication

**Language service** (`packages/language-service/`):

- `zena/lsp.zena` — WASM entry point compiled from Zena. Exports `init()`,
  `check()`, `format()`, `getDefinition()`, and diagnostic getter functions.
  Uses the full Compiler pipeline for cross-module import resolution.
- Integration tests exercising diagnostics, formatting, and definition lookup

**Working features:**

- Diagnostics (parse errors, type errors, unresolved names) with source
  locations on document open/change
- Go to definition (identifiers, type annotations, constructors, fields)
  including cross-module jumps
- Hover (type string + contextual label for variables, parameters, classes,
  members, `this`)
- Document formatting via the zena-formatter
- Multi-file diagnostics (errors in dependencies shown under the dep's URI)
- Persistent LanguageService — compiler and LibraryLoader kept alive across
  calls, per-file scope/compilation caches, incremental `checkCompilation()`

### Current Limitations

The `LanguageService` keeps a persistent `Compiler` and per-file caches,
but some gaps remain:

1. **No version tracking on the JS side** — every tab switch or
   `onDidChangeTextDocument` re-checks even if the source hasn't changed.
   Tracking `TextDocument.version` would let the extension skip redundant
   `check()` calls.
2. **No export-signature comparison** — editing inside a function body
   still re-checks all dependents even though exports are unchanged.
3. **Hover lacks doc comments** — type + label are shown, but doc comments
   from source are not preserved in the AST or surfaced to hover results.

---

## Incremental Architecture: Program + SourceFileCache

See `docs/design/self-hosted-compiler.md` **Section 9** for the full design.
Summary of the three key abstractions:

### SourceFileCache

Long-lived, mutable cache of parsed source files. Shared across `Program`
snapshots. Stores `(path, version) → SourceFile` plus the file-local
`ScopeResult` (which depends only on the AST, not on other modules).

Eviction policy: after each edit cycle, entries not in the current `Program`'s
file set are dropped (except stdlib, which is always retained).

### Program (Immutable Snapshot)

An immutable snapshot of the compilation state at a point in time. Created
fresh on each edit; shares `SourceFile`s and `ScopeResult`s via the cache.
`CheckResult`s for unchanged subgraphs are carried forward from the previous
`Program`.

Key properties:

- **On-demand checking** — type-checking is lazy, triggered only when
  `getDiagnostics()` or `getSemanticModel()` is called for a specific file.
- **Export signature comparison** — if a file's exports haven't changed
  (e.g., editing inside a function body), dependents' `CheckResult`s are
  still valid and don't need re-checking.
- **Immutable** — queries always see a consistent snapshot. No interleaved
  mutation issues.

### LanguageService

The stateful orchestrator. Replaces the current module-level globals in
`lsp.zena`. Tracks open files, creates new `Program` snapshots on edits,
delegates queries.

---

## Implementation Plan

### Step 1: Extension Infrastructure ✅ Complete

- CJS entry point, WASM loading, string marshaling, VS Code providers

### Step 2: WASM Entry Point ✅ Complete

- `lsp.zena` compiled to `lsp.wasm` with `init()`, `check()`, `format()`,
  `getDefinition()` exports
- Host `read_file` import for filesystem access

### Step 3: Basic Diagnostics ✅ Complete

- Parse errors, type errors, unresolved names reported with source locations
- Multi-file diagnostics (dependency errors mapped to their file URI)

### Step 4: Go to Definition ✅ Complete

- Byte-offset-based lookup via `ScopeResult.references` and
  `ScopeResult.declarations`
- Cross-module definition jumps (identifiers resolved to dependency symbols)

### Step 5: Document Formatting ✅ Complete

- `format()` export calls zena-formatter, returns formatted source
- VS Code `DocumentFormattingEditProvider` wired up

### Step 6: Persistent LanguageService ✅ Complete

`LanguageService` in `lsp.zena` keeps a persistent `Compiler` (and its
`LibraryLoader`) across calls. Per-file `HashMap`s cache `ScopeResult`
and `CompilationResult` for multiple open files. `checkCompilation()`
carries forward previous `ProgramCheckResult` for incremental checking.

### Step 7: SourceFileCache + Program

Extract the full incremental architecture:

1. **`SourceFileCache`** — extract `LibraryLoader`'s cache into a standalone
   cache keyed by `(path, version)`. Include `ScopeResult` in cache entries.

2. **`Program`** — immutable snapshot with lazy `getCheckResult()`. Carry
   forward `CheckResult`s from the previous `Program` for unchanged files
   whose dependencies haven't changed.

3. **Export signature comparison** — detect when a file's exports are
   identical despite a source change (function body edits). Skip re-checking
   dependents when exports are stable.

### Step 8: Hover Types ✅ Complete

Hover shows type string + contextual label (e.g., `let x: i32`,
`(parameter) a: i32`, `class Foo`, `(property) field: i32`,
`this: ClassName`). Implemented via `getHover()` in `lsp.zena`
and `HoverProvider` in `zena-extension.ts`.

### Step 9: Document Symbols / Outline ← Next

Walk the AST to return top-level declarations (classes, functions,
variable bindings, type aliases, enums) with their source spans.
No type checking required — only the parse tree.

### Step 10: Hover with Doc Comments

Preserve `/** ... */` doc comments in the parser/AST. Thread them
through `SymbolInfo` or `SemanticModel` and include in `HoverResult`.

### Step 11: Version Tracking (JS Side)

Track `TextDocument.version` in `zena-extension.ts`. Skip `check()`
when the version hasn't changed (e.g., switching tabs back).

### Step 12: Additional Features (Future)

- Find All References
- Completions / IntelliSense
- Rename Symbol
- Code Actions (quick fixes)
- Signature Help
- Semantic Tokens (richer highlighting than TextMate)

---

## File Structure

```
packages/language-service/
  package.json              # Wireit build: TS + WASM compilation
  zena/
    lsp.zena                # WASM entry point (LanguageService, exports)
  src/
    index.ts                # JS API over the WASM exports (published)
    test/
      lsp_test.ts           # Integration tests (load lsp.wasm in Node)
      language_service_test.ts  # Tests for the JS API
  index.js                  # Built JS API
  lsp.wasm                  # Built artifact

packages/playground/
  src/
    zena-playground.ts      # <zena-playground> element
    worker/
      compiler-worker.ts    # Runs the JS API in a Web Worker
    stdlib-data.json        # The stdlib, bundled for a browser (generated)

packages/vscode-zena/
  package.json              # Extension manifest, Wireit build
  src/
    extension.cts           # CJS entry point (dynamic imports ESM)
    lib/
      compiler-service.ts   # WASM loading, string marshaling, host imports
      zena-extension.ts     # Extension lifecycle, VS Code providers
  lsp.wasm                  # Copied from language-service at build time
  syntaxes/                 # TextMate grammar files
  language-configuration.json
```

---

## Open Questions

- **Multi-root workspaces**: How to handle multiple `zena-packages.json`
  roots? Start with single-root.
- **Stdlib bundling**: Currently resolved via `../stdlib/zena` relative to
  the extension path. Works for development; need a strategy for published
  extension (bundle stdlib sources or rely on npm).
- **Web extension**: Running in VS Code for the Web (vscode.dev) is natural
  since the compiler is already WASM — but the `read_file` host import
  needs shimming for a virtual filesystem.
