# Observability in EasyCat

EasyCat exposes four distinct layers for seeing what a voice bot is doing. They
overlap on purpose, but each answers a different question and has different
guarantees. Reach for the wrong one and you will either lose data, leak PII, or
accidentally couple your application logic to a diagnostic sink.

This guide is the "which layer do I use when" map.
From this repository, run `uv run easycat docs --audience operators` to see
the full operator-facing route slice, including deployment, observability, and
journal durability. Use `uv run easycat docs --audience operators --json` when
automation needs that same operator map with command hints.

## The four layers

| Layer | What it is | Use it for | Guarantees |
| --- | --- | --- | --- |
| **A — stdlib logging** | `logging.getLogger("easycat")`, controlled by `EASYCAT_LOG_LEVEL` | Human, ad-hoc diagnostics while developing or tailing a process | Lossy. Off by default in library mode. |
| **B — EventBus** | `easycat.events`, `session.subscribe_event(...)` | Driving application behavior in reaction to session events | Live, in-process. Not a durable record. |
| **C — ExecutionJournal** | `runtime/`, `session.journal.read()`, `export_debug_bundle()`, the `easycat` CLI | Structured record of a session | Full mode is durable and replay-complete. Light mode is bounded, omits per-frame spans, and reports any eviction. PII-bearing by design. |
| **D — OpenTelemetry facade** | `easycat._observability` | Production metrics and traces | PII-scrubbed, low-cardinality. No-op without an SDK. |

### A — stdlib logging

Standard Python logging on the `easycat` logger. All module loggers are
`easycat.*`, so configuring the `easycat` logger configures the whole package.

- **Default is silence.** As a library, EasyCat installs only a
  `logging.NullHandler()` on import and never calls `logging.basicConfig()`.
  Your application owns root logging.
- **Process owners** — the `easycat` CLI, `easycat.run()`, and `debug="light"`/
  `debug="full"` wiring — opt in to console output by attaching exactly one
  tagged handler to the `easycat` logger (never root) via
  `enable_console_logging()`. Enabling it also sets `propagate=False` on the
  `easycat` logger so records do not double-log through root handlers your app
  configured — those handlers stop receiving `easycat` records once console
  logging is enabled. If you want `easycat` records in your own root pipeline,
  do not enable console logging; configure the `easycat` logger yourself.
- Logging is **lossy**: messages are dropped below the configured level, and the
  format is meant for humans, not machines. Do not parse it. Do not depend on a
  specific message appearing — use the journal (C) for that.

### B — EventBus

`easycat.events` defines the event types; `session.subscribe_event(...)` lets you
react to them. The EventBus **drives application behavior** — it is how your app
learns that the user started speaking, a turn ended, the bot produced audio, etc.

- It is **not an observability sink.** Subscribing to events to "log" things is
  fine for app logic, but the bus is live and in-process; it is not a durable
  record and it is not replayable.
- `session.subscribe_event(...)` and `EventBus.subscribe(...)` return an
  idempotent subscription token. Keep that token when a component owns a
  callback lifecycle and call `token.unsubscribe()` during teardown.
- Event handlers run inline in subscription order. By default, handler
  exceptions are logged, counted on the bus, and do not stop later handlers from
  running. Set `EasyConfig(handler_error_policy="raise")` (or the equivalent
  `TextSessionConfig` / `create_text_session` option) in tests or strict app
  code when a handler failure should abort dispatch and propagate to the
  emitter. Direct `EventBus` construction accepts the same option.
- Sessions created from `EasyConfig` or `TextSessionConfig` warn when a callback
  takes at least 5 ms, because slow handlers can stall audio-critical paths.
  Tune this with `slow_handler_threshold_s=...`, or set it to `None` to disable
  the diagnostic. Direct `EventBus` construction leaves it disabled unless
  configured explicitly.
- If you want a durable mirror of what flowed across the bus, that is the
  journal's job (C), which records bus activity (via the session journal sink)
  plus per-stage internal detail the bus never carries.

### C — ExecutionJournal

The journal (`runtime/`) is the durable, structured, replayable record of a
session. Read it live with `session.journal.read()`, export a self-contained
bundle with `export_debug_bundle()`, or inspect a bundle with the `easycat` CLI.

- It is the **single source of truth** for "what actually happened." Full mode
  retains replay-complete stage detail. Light mode keeps turn, event, error,
  and control history while omitting per-frame Audio/VAD/STT spans and their
  artifacts.
- It is **PII-bearing by design**: it records transcripts, agent output, and tool
  arguments so a session can be faithfully replayed and debugged.
- `journal_redaction="secrets"` (the default) preserves that replay content
  while removing credentials. Set `journal_redaction="pii"` to irreversibly
  redact phone numbers, URLs, request IDs, home paths, prompts, transcripts,
  and provider text at write time. Redacted CLI views and coding-agent context
  packs apply their own PII policy regardless; raw journals and debug bundles
  should still be treated as sensitive.
- It is gated by `debug=` (see orthogonality below): `debug="off"` does not
  journal; `debug="light"`/`debug="full"` do.
- The light journal is bounded by `journal_capacity` (default `10_000`
  records). `session.journal.dropped_records`, the bundle manifest's
  `journal_dropped_records`, and `easycat bundles show` make any eviction
  explicit; increase the capacity when an unusually long or event-heavy
  session needs more in-memory history.
- `record_to="runs"` on `EasyConfig` or `create_text_session(...)` exports a
  timestamped debug bundle on clean shutdown when journaling is enabled.

To consume a journal while a session runs, give `follow(...)` an explicit stop
event and always join the tail task during teardown:

```python
import asyncio

from easycat import EasyConfig, JournalRecordKind, create_session


async def tail(session, stop_tailing: asyncio.Event) -> None:
    async for record in session.journal.follow(stop=stop_tailing):
        if record.kind == JournalRecordKind.EVENT:
            print(f"[{record.name}] {record.data}")


async def run_and_tail(config: EasyConfig) -> None:
    async with create_session(config) as session:
        stop_tailing = asyncio.Event()
        tail_task = asyncio.create_task(tail(session, stop_tailing))
        try:
            await session.wait_closed()
        finally:
            stop_tailing.set()
            await tail_task
```

For a durable journal, inspect the same record stream without application code:

```bash
uv run easycat inspect .easycat/journals/<session_id>.sqlite
```

#### Artifact lifecycle and storage budget

1. A running `debug="full"` session owns a live SQLite journal and artifact
   directory under `data_dir` (by default `.easycat`). Retention and crash
   sweeps never move or delete a journal that still holds its live-owner claim.
2. `await session.stop()` closes writable backends and records clean close. The
   session retains a read-only `session.journal` view and can still execute
   `session.export_debug_bundle(...)` for postmortem inspection.
3. Every persistent SQLite close runs an opportunistic retention sweep. The
   built-in budget keeps at most **50 journals**, **2 GiB**, and **14 days** of
   live journal history; whichever limit is reached first prunes the oldest
   clean session. `journal_retention="archive"` (default) moves that session's
   database and artifacts into a private `.tar.gz` under `data_dir/archive/`;
   `journal_retention="delete"` removes them instead.
4. `easycat bundles list` discovers live, clean, and
   unclean/crash-recovered artifacts from the default data root. Retention
   archives are not included in bundle discovery; inspect or extract them only
   in an isolated, operator-controlled location. Raw sources remain sensitive
   even when a rendered CLI view is redacted.

The count/byte/age values above bound the active journal set, not archived
tarballs, exported bundles, or copies in object storage. Deployments using the
archive policy must separately budget and expire `data_dir/archive/` alongside
their tenant, legal, encryption, and deletion policies; EasyCat cannot expire
copies it no longer owns.

- Error records preserve PEP 678 exception notes in `ErrorInfo.notes`. Runtime
  `Error` events attach `stage`, `provider`, `code`, `session_id`, and
  `turn_id` when known; stage wrappers also attach `elapsed_ms`, `sequence`,
  and `record_key` on re-raised provider failures so a traceback points back
  to the journal record that captured the failing input. When streaming agent
  and TTS branches both fail in one turn, EasyCat emits a pipeline
  `ExceptionGroup` error that preserves both child errors.
- CLI entry points:
  `easycat bundles list`,
  `easycat bundles show <path>`,
  `easycat debugger serve <path>`,
  `easycat inspect <path>`,
  `easycat replay <path>`,
  `easycat latency <path>`, and
  `easycat bundles export <path>`.
  `easycat latency <path>` rolls the critical-path milestone deltas up across a
  bundle's turns and reports `count`/`p50`/`p90`/`p95`/`p99` per segment; see
  [latency](latency.md).
  Add `--json` (`easycat bundles list --json`, `easycat bundles show <path> --json`,
  `easycat inspect <path> --json`, `easycat replay <path> --json`,
  `easycat latency <path> --json`,
  `easycat bundles export <path> --output DIR --json`) for a
  parseable summary. Add `--issues` to `easycat bundles show <path>` /
  `easycat inspect <path>` to render a severity-ranked rollup of detected
  problems (errors, tool failures, timeouts, empty transcripts, slow
  milestones, slow/missed barge-ins, and — when the bundle carries stored PCM
  artifacts — audio-health cards for clipping, near-silent caller capture, and
  dead air); the `issues` key is always present in the `--json` envelope.
- More journal CLI entry points:
  `easycat diff <path> <path>` diffs two bundles or journals turn by turn,
  surfacing milestone and transcript deltas between a baseline ("before")
  and a comparison ("after") run; restrict it with `--turn` and add `--json` for
  a parseable summary.
  `easycat journal grep <path> --query TEXT` runs a redacted full-text search
  over a journal or bundle, `easycat journal follow <path>` live-tails a SQLite
  journal as it grows (redacting every line), and
  `easycat journal promote <path> TURN_ID --out FILE` saves one turn as a
  replayable, self-contained regression bundle.
  `easycat tail <path>` is the short alias for `easycat journal follow <path>`.
- Optional debugger UI:
  install the extra with `uv sync --extra debugger --group dev` from this repo,
  or `uv add 'easycat[debugger]'` in an app. For post-call inspection, launch
  the first-class browser UI directly from the CLI:

  ```bash
  easycat debugger serve PATH --no-open-browser
  uv run easycat debugger serve runs/session.bundle --no-open-browser
  uv run easycat debugger serve .easycat/journals/<session_id>.sqlite
  ```

  The UI gives every captured call a timeline-first forensic workspace: an
  overview dashboard with recommended next steps, Live lanes for during-call
  event flow, per-turn waterfalls, transcript/audio playback, paged raw records,
  deterministic issue triage cards, replay controls, and
  live-session bundle export. You can also import `serve_bundle` for an offline
  bundle or `serve_session` for a live session:

  ```python
  from easycat.debugger import serve_bundle, serve_session

  serve_bundle("runs/session.bundle", port=8765)
  serve_session(session, port=8765, in_thread=True)
  ```

  The debugger is loopback-only by default and has no auth. Keep it on
  `127.0.0.1` unless you have a controlled, private debugging environment and
  explicitly choose `allow_remote=True`.

  **Dev mode (always-available dev timeline).** For the inner development loop,
  opt in to dev mode instead of launching the debugger by hand:

  ```bash
  EASYCAT_DEV=1 easycat serve
  ```

  ```python
  from easycat import VoiceApp

  VoiceApp(agent=agent, dev=True).run("browser")
  ```

  Dev mode (`EASYCAT_DEV=1` / `VoiceApp(dev=True)`) defaults to durable
  debugging when you have not set `debug=` explicitly, registers every live
  session in a process-local registry, and launches ONE loopback debugger UI per
  process. Registration flows through the single `create_session` funnel, so
  **every mode** populates the UI — including the per-connection
  browser/websocket/twilio sessions built downstream — and sessions
  auto-unregister when they stop (weakly held, so a stopped call is never pinned
  alive or left lingering in the list).

  The UI adds a **live session selector** that updates over the WebSocket as
  calls come and go (`GET /api/dev/sessions` lists them; `POST /api/dev/select`
  re-points every panel — switching a session resets the live-follow cursor so
  a session whose journal is "behind" still streams cleanly). It also surfaces a
  **cross-session overview strip** (`GET /api/dev/overview`) with one chip per
  session colored by error count, a "follow newest" toggle, `[`/`]` keyboard
  switching, and a filter box. If the default port (8765) is taken — e.g. a
  second dev process — the UI scans the next few loopback ports automatically
  (override with `EASYCAT_DEV_DEBUGGER_PORT`).

  Dev mode is **purely additive** over the autolaunch guard: it is a separate,
  explicit opt-in. `debug="full"` on its own still keeps a durable journal and
  never opens a browser tab — durable journaling and UI autolaunch remain
  distinct concepts. Dev mode also never opens a tab in CI / non-interactive
  shells (same loopback + interactive-terminal guards as the autolaunch path).

### D: OpenTelemetry facade

`easycat._observability` is a thin facade over the OpenTelemetry API for
production **metrics and traces**.

- It is a **no-op without an SDK**: if `opentelemetry-api` is absent or no SDK is
  configured, every span/metric call does nothing. OTel is an *optional*
  dependency; EasyCat never pulls it in as a hard dependency.
- It is **PII-safe and low-cardinality**: span and metric attributes are
  validated against an explicit allow-list (`easycat.*` and a small set of
  `gen_ai.*` keys). Any attribute that is on the forbidden list, or whose name
  *contains* a high-risk substring (`transcript`, `prompt`, `content`, `text`,
  `body`, `secret`, `token`), is rejected with a `ValueError`. This is
  defense-in-depth so a new PII-bearing attribute cannot silently leak into
  traces.
- Correlation ids (`session_id`/`turn_id`) are deliberately **kept out of OTel**
  attributes — they are logging-only correlation (see below) and would also be
  high-cardinality span attributes.

> **`easycat.server.*` server metrics are registered and emitted (M8).** The
> `easycat.server` process layer registers its five metric names
> (`easycat.server.requests.total`, `easycat.server.request.duration`,
> `easycat.server.sessions.rejected.total`, `easycat.server.connections.active`,
> `easycat.server.draining`) in `METRIC_DEFINITIONS` and its three new labels
> (`easycat.route`, `easycat.server_state`, `easycat.auth_result`) in
> `LOW_CARDINALITY_ATTRIBUTE_KEYS` — in the SAME change that first emits them,
> because emitting an unregistered name/key raises `ValueError`. Emission lives
> in `easycat.server.metrics` and routes through the same `sanitize_attributes`
> path (no bypass); it is a no-op without an SDK but still validates the
> names/labels. `easycat.route` is constrained to an **enumerated set of route
> templates** that is asserted before recording, so a raw path (which can carry
> `?token=` or user content) can never become a label. No token, PII, or raw
> path is ever a server-metric label.

## Golden rules

- **Logs are lossy; the journal is complete.** If you need to be sure something
  was captured, use the journal (C), not logging (A).
- **OTel is PII-safe; the journal is not.** Export OTel data to third parties
  freely. Treat journal bundles as sensitive — they contain transcripts and
  agent output.
- **The EventBus drives behavior; logs only observe.** Put application logic on
  the bus (B). Put human diagnostics in logs (A). Do not invert this.

### Why B and C both exist

The EventBus (B) is the live, in-process channel your application reacts to. The
journal (C) is the durable record. They are not redundant: the journal mirrors
the bus (via the session journal sink) *and* adds per-stage internal detail that
never crosses the bus. So B is for "act on this now," and C is for "reconstruct
exactly what happened later." You generally subscribe to B for behavior and read
C for forensics.

## Configuration and orthogonality

There are three independent knobs, and they control different things:

- **`EASYCAT_LOG_LEVEL`** — controls layer A only (the stdlib `easycat` logger
  level). Accepts `debug`, `info`, `warning`, `warn`, `error`, and `critical`
  (case-insensitive). When a process owner enables console logging, this
  resolves the level; the default is `INFO`, and `DEBUG` is used only when you
  explicitly request it. It has the same single meaning in `easycat.run()` and
  in `debug="light"`/`debug="full"`.
- **`EASYCAT_LOG_FORMAT=json|text|human`** — switches layer A's console handler.
  `json` renders single-line JSON, `text` renders plain non-Rich text for log
  collectors, and `human` renders the Rich-capable interactive formatter.
  This is an **explicit opt-in**: a TTY toggles *color* only for `human`, never
  JSON.

  The JSON field set is a **semi-public UNSTABLE schema** — do not build hard
  dependencies on it yet. Current fields:

  | Field | Meaning |
  | --- | --- |
  | `ts` | ISO-8601 timestamp |
  | `level` | log level name |
  | `logger` | logger name (e.g. `easycat.session._session`) |
  | `msg` | formatted message |
  | `session_id` | bound session id, or `-` |
  | `turn_id` | bound turn id, or `-` |
  | `exc` | formatted traceback (only present when an exception is attached) |

- **`EASYCAT_ENV=dev|prod`** — selects the default layer A renderer when
  `EASYCAT_LOG_FORMAT` is not set. `prod` / `production` uses single-line JSON
  for log pipelines; `dev` / unset keeps the human renderer. An explicit
  `EASYCAT_LOG_FORMAT` always wins.

- **`debug=`** (`"off"` / `"light"` / `"full"` on `EasyConfig`, default
  `"light"`) — controls journal and artifact capture (C).
  Journaling is on by default so sessions are always recorded, but the
  default `"light"` keeps turn/control history in memory and omits per-frame
  Audio/VAD/STT spans and artifacts. Opt into `"full"` for replay-complete,
  crash-survivable on-disk stage detail; set `debug="off"` to skip recording
  entirely.
  Debugger UI launch is an independent opt-in controlled by
  `debugger_autolaunch=True`. `debug=` is **orthogonal to log level**:
  `EASYCAT_LOG_LEVEL` decides how verbose the human console log is. Turning
  one up does not turn the other up.

- **Advanced observability knobs** are direct `EasyConfig` fields, for example
  `warmup=False`. The
  value is validated and preserved in safe debug-bundle config snapshots.
  `warmup=True` runs structural provider/model `warmup()` hooks during
  `Session.start()` before audio ingress and emits `warmup_completed` timing
  records. The bundled providers now implement those hooks — OpenAI TTS primes
  its HTTP pool, Silero VAD and Smart Turn prime their ONNX sessions, the OpenAI
  Realtime STT runs a connect-handshake-close cycle, and the OpenAI Agents
  bridge primes the SDK's shared client — so the first turn does not pay their
  cold-start cost.

### Correlation ids in logs

When a session/turn is active, log records emitted within that async context are
tagged with `session_id` and `turn_id` (via a `contextvars`-backed logging
filter on the console handler). The console formatter shows them as
`[session/turn]`, and the JSON formatter emits them as fields. Unbound records
show `-` in both formats.

The ids are captured at task-creation time: a task inherits the ids bound in the
context that created it. Short-lived per-turn work (agent, TTS) is created after
`bind_turn` and inherits the turn id; the long-lived audio-pipeline tasks are
created at session start, before any turn, so they re-bind the current turn each
loop iteration to stay correlated. `threading.Thread` workers do not inherit the
ids, but EasyCat avoids that boundary.

## Honesty caveats

- **The journal is still sensitive.** EasyCat scrubs safe config/environment
  snapshots, selected agent-bridge metadata, and obvious secret-like journal
  fields through `apply_write_filter`, but normal journal records and bundles
  still preserve transcript text, agent output, and tool-result text for replay.
  A pluggable full `RedactionPolicy` is still planned. Do not attach journal
  bundles to public issues or send them to third parties until you have manually
  scrubbed them. Config snapshots are diagnostic rather than lossless: cycles
  and values beyond fixed depth, item, node, scalar, and final-output safety
  budgets collapse to `...`, `<unavailable>`, or type/length markers so debug
  export cannot recurse forever or grow without bound.
- **Latency is reported, not gated.** Every pipeline stage records its
  `elapsed_ms` to the journal, and each turn emits a `turn_total_latency_ms`
  (voice, once first TTS audio is available) or `text_turn_latency_ms` (text)
  metric record. EasyCat does not reject or alert on slow turns at runtime; use
  the OTel latency histograms (D), the `easycat latency` bundle summary, and the
  `easycat validate latency` regression lane to observe real numbers and catch
  regressions in CI.
- **`gen_ai.*` attributes are development status.** The committed
  `gen_ai.operation.name`, `gen_ai.request.model`, and `gen_ai.system` span keys
  track the OpenTelemetry GenAI semantic conventions, which are themselves still
  evolving. Treat them as subject to change; do not build durable dashboards that
  assume their stability.
