# Chapter 11 — The Journal as Mental Model

<!-- BEGIN auto:navigation -->
**Progress: 12 of 16** · [← Chapter 10 — Cleaning the Signal](../10-cleaning-signal/) · [Ladder index](../) · [Progress worksheet](../PROGRESS.md) · [Exercises](./EXERCISES.md) · [Chapter 12 — Evals + the Latency Budget →](../12-evals-and-latency/)
<!-- END auto:navigation -->

> You will not write pipeline code. You will read three bundles
> with planted bugs and find them.

<!-- BEGIN auto:spaced-retrieval -->
## Recall before reading

> **Following the ladder? Spaced retrieval — Chapter 9 — Interruption / Barge-in**
>
> Close earlier chapters and answer from memory before reading further. If this
> chapter is your starting point, skip this block.
>
> **Answer from memory:**
>
> Does the triggering speech event belong to the interrupted turn, and what must finish before
> the next STT stream?
>
> After recording your answer, explain one way `barge-in cancellation` changes how you reason
> about `journal query coverage`. Keep the first answer visible.
>
> **Check only after answering:**
>
> ```bash
> uv run python docs/teaching/09-interruption/barge_in_turn_probe.py
> ```
>
> Cite one observed field, measurement, or behavior; repair only the part your
> evidence disproved.
<!-- END auto:spaced-retrieval -->

<!-- BEGIN auto:offline-checkpoint -->
> **Hardware-free checkpoint:** prove `journal query coverage` without a microphone,
> speakers, or provider credentials:
>
> **Predict first:** How can marginal query counts distinguish an empty filter intersection from
> a misspelled turn?
>
> ```bash
> uv run python docs/teaching/11-journal/query_coverage_probe.py
> ```
>
> **Evidence to find:** a zero-result intersection has marginal matches, while a misspelled turn
> has none.
>
> **Explain the result:** Use the marginal counts to decide whether to loosen a filter or fix an
> identifier.
>
> [See all 16 checkpoints](../#hardware-free-checkpoint-spine).
<!-- END auto:offline-checkpoint -->

## Prerequisites

- [Chapters 0-10.](../) This is a consolidation chapter.
- `uv sync --group dev`. No API keys needed — everything is
  offline against pre-recorded bundles.

> **Minimum to skip the ladder:** chapter 2 alone (the first
> chapter that writes a bundle). The investigation flow doesn't
> depend on you having built the pipeline, just on knowing what
> the records mean.

## Read-only by design

This is the one chapter where you do **not** produce audio.
There's nothing for a microphone to do. The "runnable at every
checkpoint" rule bends here because the chapter's whole point is
*reading* bundles, not making them. The fixtures under `bundles/`
are checked in so you can work offline, on a plane, without API
keys, without a mic — pick the constraint that fits.

If you do want to scrub a *live* bundle: any chapter from 2
onward dumps one to its own `runs/` directory. Chapter 9c
(`docs/teaching/09-interruption/runs/`) is the most interesting —
barge-in turns have the richest record set. Open one with
`investigate.py` in this chapter once you've finished the three
planted-bug investigations below.

## Diff from chapter 10

- **Added:** `investigate.py` — a tiny CLI bundle browser;
  `bundles/bug_*.bundle` — three planted-failure fixtures checked
  in (unlike the gitignored `runs/*` from earlier chapters);
  `generate_bundles.py` that builds those fixtures; an explicit
  walkthrough of the parallel `RunBundle` / `JournalView` query surface
  (`filter_by_stage`, `filter_by_turn`, `lookup_by_sequence`).
- **Removed:** the entire live pipeline. This chapter writes no
  audio.

## The journal is not logs

- **Logs** are unstructured prose. You grep them. Your search
  depends on your choice of words.
- **The journal** is a strictly ordered, typed, per-turn event
  stream. You *query* it: by stage, by turn, by sequence. Every
  record has a stable envelope schema; each emitter owns the schema
  inside `data`.

Every runnable chapter from chapter 2 onward has been dumping
bundles to `runs/`. This chapter teaches you what those bundles
are for.

> **Bundle safety.** Bundles are ZIP/JSON archives that EasyCat
> validates for structure and path traversal, but they are not signed
> or tamper-evident. Only open bundles you generated yourself or got
> from a source you trust. (The ones checked in under `bundles/` are
> generated by the companion scripts in this repo.)

## The three bugs

The bundles live in `bundles/*.bundle` — checked in, unlike the
gitignored `runs/*` from other chapters, because this chapter
can't work without them. You do not need to regenerate them to do
the lesson. To experiment with the companion generator without
rewriting tracked fixtures, send its chapter-shaped output to the
gitignored `.easycat/` directory:

```bash
uv run python docs/teaching/11-journal/generate_bundles.py \
    --output-root .easycat/teaching/11-journal
```

Maintainers intentionally refreshing the checked-in fixtures omit
`--output-root` and review the resulting bundle diff.

Each bundle records a session with a specific, planted failure.
Your job is to find the failure using only the journal.

## Your tool

Start with the chapter's canonical entry point. With no arguments it
opens the first planted-bug fixture, so this command works immediately
after setup:

```bash
uv run python docs/teaching/11-journal/main.py
```

`main.py` delegates to `investigate.py`. Call that script directly when
you want to choose a fixture or add filters:

```bash
uv run python docs/teaching/11-journal/investigate.py \
    docs/teaching/11-journal/bundles/bug_01_empty_final.bundle

# Filter by stage, record name, turn, or exact sequence:
uv run python docs/teaching/11-journal/investigate.py \
    docs/teaching/11-journal/bundles/bug_02_tts_stutter.bundle --stage tts

uv run python docs/teaching/11-journal/investigate.py \
    docs/teaching/11-journal/bundles/bug_03_ghost_interruption.bundle --name interruption.start

uv run python docs/teaching/11-journal/investigate.py \
    docs/teaching/11-journal/bundles/bug_03_ghost_interruption.bundle --turn ch11-bug03-turn-2

uv run python docs/teaching/11-journal/investigate.py \
    docs/teaching/11-journal/bundles/bug_03_ghost_interruption.bundle --sequence 9
```

Under the hood it is:

```python
from easycat.debug.testing import load_bundle

b = load_bundle(path)
for r in b.records():
    ...
```

`RunBundle` exposes `filter_by_stage`, `filter_by_turn`, and
`lookup_by_sequence` directly. A live session's read-only
`session.journal` is a `JournalView` with the same query names.
The vocabulary transfers, but the record representation differs:
bundle records are dictionaries (`r["name"]`), while live journal
records are typed objects (`r.name`). Both stage and turn filters
return materialized lists; stage filtering scans the records, while
an exact sequence lookup is bounded on a live journal backend.

## Turn isolation can hide session causes

`filter_by_turn` is intentionally strict: it excludes records whose
`turn_id` is `None`. That is right for reconstructing one turn, but
session-scoped configuration can explain why the turn behaved that
way. In the ghost-interruption fixture, sequence 1 is an unscoped
`audio.config` record with `aec: off`; a strict turn-2 query starts at
sequence 8 and hides the cause.

Use the investigator's opt-in context join:

```bash
uv run python docs/teaching/11-journal/investigate.py \
  docs/teaching/11-journal/bundles/bug_03_ghost_interruption.bundle \
  --turn ch11-bug03-turn-2 --stage audio --include-session-context
```

The join includes turn-less records from the target turn's same
session only. It never pulls unscoped records from another session,
and `--include-session-context` is rejected unless `--turn` identifies
the target scope.

Run the provider-free comparison:

```bash
uv run python docs/teaching/11-journal/session_context_probe.py
```

The strict result contains sequences 8–13. The context result adds
sequence 1, and an `audio` stage filter isolates the `aec: off`
configuration without mixing it into every ordinary turn query.

## Validate an empty query

`(no records matched)` is not self-interpreting evidence. It can mean
the stage never ran, but it can also mean a mistyped turn ID or an
impossible intersection such as “sequence 9 *and* stage agent” when
sequence 9 is an STT record.

`investigate.py` prints the active filters, composed match count, and
each filter's marginal count. On an empty query it also prints known
turns/stages/names or the available sequence range. Compare:

```bash
# Typo: the turn filter itself has zero marginal matches.
uv run python docs/teaching/11-journal/investigate.py \
  docs/teaching/11-journal/bundles/bug_03_ghost_interruption.bundle \
  --turn typo

# Impossible intersection: each filter matches alone, but not together.
uv run python docs/teaching/11-journal/investigate.py \
  docs/teaching/11-journal/bundles/bug_03_ghost_interruption.bundle \
  --sequence 9 --stage agent
```

For a human investigation, an empty result remains successful because
absence may be the finding. For a script or CI check, add
`--require-match` to exit non-zero when the composed query is empty.

Run the provider-free query probe:

```bash
uv run python docs/teaching/11-journal/query_coverage_probe.py
```

A filtered sequence gap is expected—the filter deliberately hides
records. Even an unfiltered gap can come from bounded retention or an
incomplete export. Verify source coverage and retention before calling
a gap an in-flight failure.

## Investigation 1 — guided

Open `bug_01_empty_final.bundle`. Work the evidence:

1. `turn.started` at t_ms=1,000,000. User is expected to speak.
2. `stt.final` at t_ms=1,001,100. Look at `data["text"]`.

What do you see? An **empty string**. The turn committed a final
but there was no transcript.

Now look for the agent:

```bash
... --stage agent
```

Nothing. The agent stage never ran.

Write down your hypothesis — *why did STT commit an empty final?*
— in a single sentence before opening
[`solutions.md`](./solutions.md).

## Investigation 2 — semi-guided

Open `bug_02_tts_stutter.bundle`. The user hears stuttering — fine,
gap, fine — between sentences. Hypotheses to walk:

- **Sentence splitter wrong?** Look at the `stage.tts.execute`
  records. How many? Three? That's correct for a three-sentence
  reply.
- **Agent stalls?** `agent.first_token` at t_ms=2,001,400 — fast.
  No other agent records; probably fine.
- **TTS slow?** Compare `elapsed_ms` on each `stage.tts.execute`.
  One is 5× the others. **Why?** The records between them —
  `ws.reconnect.*` — tell you.

```bash
uv run python docs/teaching/11-journal/investigate.py \
    docs/teaching/11-journal/bundles/bug_02_tts_stutter.bundle --stage tts
```

## Investigation 3 — unguided

Open `bug_03_ghost_interruption.bundle`. The bot *thinks* it was
interrupted but the user never spoke. Find the journal evidence
for:

- Which record told the coordinator a barge-in happened?
- What is *missing* from around that record that would confirm
  the interruption was a real user?
- What setting in the journal explains why the bot self-triggered?

Start with a strict turn query, then opt into session context when you
need the configuration that explains both turns.

Write your hypothesis in one sentence before peeking at
[`solutions.md`](./solutions.md).

## Why the journal has to be structured

Three properties make this kind of investigation tractable:

1. **Strict ordering by sequence number.** You can say "what
   happened after sequence 14" and mean it.
2. **A typed envelope plus emitter-defined payload schemas.** A live
   `JournalRecord` has stable fields such as integer `sequence`, string
   `name`, `JournalRecordKind`, and `data: dict[str, Any]`. The checked-in
   fixtures consistently put strings in `data["text"]` and numbers in
   `data["t_ms"]`, but that is their emitters' contract—not validation
   performed by the journal.
3. **Causal grouping by turn_id.** In multi-turn bundles, all
   records for one turn share the same id and can be isolated.

JSON records save you from regex-parsing prose, but structured does not
mean schema-validated. Validate fields at the boundary where a script,
dashboard, or SLO starts depending on their types and domain constraints:

```bash
uv run python docs/teaching/11-journal/payload_schema_probe.py
```

The provider-free probe appends two `demo.latency` records. The journal
preserves both `125.0` and the malformed string `"125.0"`; only the probe's
emitter-aware validator rejects the latter. A real shared record name should
have one documented payload schema. When compatibility matters, evolve that
schema deliberately rather than assuming `dict[str, Any]` made it safe.

### The teaching shape vs. the production shape

Chapters 2-10 and the fixtures here emit *composite events* of the
form `stage.<name>.execute` with a single `elapsed_ms` field — one
record per stage span. This is deliberate: the query layer stays
at the surface and the reader can read the bundle top-to-bottom.

The production journal in `src/easycat/runtime/` emits **paired
records**: `stage_start` at the beginning of a span, a matching
`stage_complete` at the end, with artifact refs and error blobs
in between. You reconstruct a span by matching start/complete on
a span correlation id carried in the record `data`. Paired records
buy you partial-span visibility on crashes and concurrent-span
overlap that single events can't express. Read the production
records after you've internalised the teaching ones.

Also be aware: `turn.state_changed` and `ws.reconnect.*` records
in these bundles are **representative synthetic records** — the
production pipeline emits similar signals, but chapters 2-10's
own scripts do not write them. They are here so the fixtures
tell a coherent story without running the whole framework.

## Try breaking it

Pick any bundle from your `runs/` directories — chapter 9 is
good — and write three hypotheses for what *could* go wrong on a
flaky day. For each hypothesis, name the record you'd query
first.

Then plant your own bug in chapter 9c's code. Dump the bundle.
Can a classmate (or your future self) find it?

<!-- BEGIN auto:practice-handoff -->
## Practice and self-check

Work through [the chapter exercises](./EXERCISES.md), then try their closing
self-check from memory. If an answer is weak, rerun the hardware-free
checkpoint or revisit the section that owns the gap.
<!-- END auto:practice-handoff -->

## What's next

[Chapter 12 — Evals + the latency budget](../12-evals-and-latency/). Reading
bundles is the foundation. Now we turn them into numbers you can
trend over time.
