# Matrix Event Cache Interaction Contract

This document defines the durable Matrix event-cache boundary and the reproducible evidence required to change it.

## Contract boundary

The cache retains conversation history observed in joined-room timeline events.

The point cache is intentionally broader than visible conversation history.

Every admitted joined-room timeline event with an event ID can be retained for point lookup, relation resolution, recent-event lookup, and redaction bookkeeping.

Visible thread history projects supported `m.room.message` events, collapses edits into their originals, and omits non-message events and still-opaque encrypted payloads.

Durable conversation history is distinct from ephemeral sync state.

Membership loss is a separate lifecycle boundary that fences access and purges previously retained history only for the departed principal.

## Admitted joined-room timeline families

The following treatment is covered against both SQLite and PostgreSQL backends by `tests/test_matrix_cache_interaction_contract.py`.

| Interaction family | Point cache | Visible thread history | Indexes and invalidation |
| --- | --- | --- | --- |
| Text, notice, emote, and location messages | Retained | Visible when they belong to the thread | Thread and reply relations map to the root and invalidate only that thread |
| Valid file, image, audio, voice, and video messages | Retained | Visible when they belong to the thread | Thread and reply relations map to the root and invalidate only that thread |
| Explicit thread children and relation-less replies | Retained | Visible | Event-to-thread mappings are retained and the known thread is invalidated |
| Root, child, and reply edits | Retained | Applied to the original rather than shown separately | Edit and event-to-thread indexes are retained and the known thread is invalidated |
| Message references | Retained | Visible | Message references resolve through the relation target and affect its known thread |
| Reactions | Retained until redacted | Not visible | Reactions do not alter thread snapshots |
| Sticker, poll start, poll response, poll end, beacon info, and beacon | Retained | Not visible | These families remain room-level and do not invalidate thread snapshots |
| Generic state and timeline events | Retained | Not visible | These families remain room-level |
| Member, name, topic, avatar, power, join-rule, history-visibility, guest-access, alias, encryption, and pin state | Retained when delivered in the joined timeline | Not visible | These families remain room-level |
| Call invite, candidates, answer, select-answer, reject, negotiate, and hangup | Retained | Not visible | These families remain room-level |
| RTC membership, focus, and notification events | Retained | Not visible | These families remain room-level |
| Encrypted relation-bearing events | Retained as opaque events | Not visible until decryption supplies message content | Thread, reply, edit, and message-reference relations gap-mark the known thread fail-closed; explicit relations stay point-indexed, the opaque payload never enters the snapshot, and the marker survives appends until a decryption-capable refresh replaces the snapshot |

Relation names such as `m.reference` and `m.thread` are reused by non-message families.

Only `m.room.message` and `m.room.encrypted` relations can enter event-to-thread bookkeeping.

Poll responses, poll ends, beacons, stickers, state, calls, and RTC events remain room-level even when they carry a relation-shaped payload.

Plaintext message replies and references cannot inherit visible thread membership through one of those non-message events; unresolved dependent membership invalidates the room's thread snapshots fail-closed.

Explicit snapshot writes apply the same event-family filter, and relation walks validate cached index targets before trusting their thread IDs.

Page-local, cached, and homeserver-scan root proof accepts only plaintext or encrypted message children.

Point-event payload quality is monotonic per event ID: clear normalized content may replace a retained opaque `m.room.encrypted` payload, but a later opaque payload never replaces already-clear content.
The upsert refuses the downgrade atomically inside the database conflict clause, so opposite arrival orders, duplicate IDs in one batch, and separate cache clients on one backing store all converge on the clear payload.
The same rule guards a thread snapshot row when an incremental append rewrites an event the snapshot already holds.
Payload-derived edit and relation index rows are written only for payloads the point store actually accepted.
Explicit thread snapshots still record authoritative event-to-thread membership for every eligible snapshot event, even when the point store retains an existing clear payload.

## Redactions

Redaction envelope events are intentionally omitted from the point cache.

A redacted target is durably tombstoned so a later sync replay cannot resurrect it.

Redacting a visible message removes the message and its event-to-thread mapping.

Redacting an original also removes dependent edits and their edit and thread indexes.

Redacting only an edit removes that edit and restores the next applicable visible state of the original.

Redacting a reaction removes only the reaction and leaves the visible thread snapshot unchanged.

When target metadata identifies a poll response, poll end, beacon, or other non-message event, redaction removes only that point event and leaves every thread snapshot unchanged.

A metadata-less redaction of an absent target is a thread-state no-op.

If target metadata is unavailable but a cached target is removed, the impact is unknown and every cached thread in that room is invalidated fail-closed.

## Deliberately excluded sync categories

Complete state under a joined-room sync response is excluded because it is a current-state snapshot rather than conversation history.

Invite-room and leave-room timelines are excluded because they are outside joined conversation history.

Typing and receipt events are excluded because they are ephemeral.

Presence is excluded because it is ephemeral and global.

Global and room account data are excluded because they are per-account state.

To-device events and device-list changes are excluded because they belong to encryption and device lifecycle rather than conversation history.

The exclusion tests prove that these categories cannot create point rows, thread rows, edit rows, or invalidation markers.

The exclusion tests prove that the leave timeline payload cannot create cache rows.

Authoritative membership handling separately fences and purges retained history for rooms in the sync leave section before timeline caching begins.

## Thread snapshot reads

A durable thread snapshot is usable when its rows are durably present and no gap marker is recorded against it.
A gap is detected and refetched rather than prevented, so there is no validation timestamp and no trust comparison.
Presence and the marker are separate questions, and both are asked: a never-cached thread carries no marker either, so a caller that checks only the marker treats every cold thread as a cache hit.

A room-scoped gap is a wildcard-thread marker.
It fans out across every thread in the room that already holds a `thread_state` row, and is also recorded once on `room_state`.
The room-level copy covers the thread the fan-out cannot reach: one whose first fetch is still in flight and therefore has no row to update.
Only a replacement reads it, so thread reads take no join.

A replacement keeps whichever gap its fetch does not cover, at either scope, and clears the rest.
Replacement is ordered by fetch start rather than arrival, because installing a snapshot deletes the events it omits: an older fetch is refused outright, and a successful append moves the same watermark forward so an in-flight scan cannot delete an event that landed while it was running.
This ordering is wall-clock based, knowingly: `fetch_started_at` is captured before the homeserver round-trip, so clock skew between workers sharing one PostgreSQL namespace can misorder a replacement.

A snapshot without its thread root or one still containing opaque `m.room.encrypted` payloads is rejected.

Only a rejected or absent snapshot enters an authoritative homeserver room-history scan and cache refill.

A cache-miss refill is single-flight per full read contract, so concurrent matching readers share one homeserver scan.
There is no admission gate, failure backoff, or cooldown.
A refill that completes without installing a snapshot still returns its homeserver history, so reads stay fail-open.

A degraded dispatch proof for an unproven thread candidate retries strict proof before it may demote the event to room level.

Startup thread prewarm scans outside the live write coordinator so its bulk room scan cannot starve dispatch reads.

A refill whose reconstruction contains still-opaque encrypted evidence for the requested thread, or whose scan holds an opaque relation with unresolved thread impact, gap-marks the thread and fails the read instead of certifying incomplete history.

Relation-less ciphertext and opaque annotations are excluded from that rejection because they cannot change any visible thread snapshot.

The thread-history version marker clears only previously certified thread snapshots and their state, preserves point and relation indexes, rotates the durable certification generation, and records a one-time storage marker transactionally.

A second unchanged read is served from cache and performs no homeserver scan.

The advisory read path may use a labelled stale-cache fallback when a required refill fails.

Dispatch reads reject stale fallback and propagate the refill failure.

Every completed labelled cache or source read emits its own `matrix_cache_thread_history_refreshed` event with `mode`, `cache_read_ms`, `homeserver_fetch_ms`, `homeserver_scan_parse_cpu_ms`, `resolution_ms`, `sidecar_hydration_ms`, `coordinator_queue_wait_ms`, `post_coordinator_read_ms`, `thread_read_total_ms`, `refill_singleflight_wait_ms`, `refill_singleflight_shared`, page and event counts, `cache_reject_reason`, `thread_read_source`, degradation state, and error state.
Refilled labelled reads add `cache_store_written` and `cache_store_failed`.
A rejected snapshot instead emits `Thread cache rejected for read` with `cache_gap_marked_at`, `cache_gap_age_ms`, and `cache_gap_reason` when its gap carries that metadata.
A dispatch read cut short by a coordinator timeout or a fetch timeout instead emits `matrix_cache_thread_read_degraded`.

## Disposable live audit

`tests/manual/matrix_event_cache_live_audit.py` creates a new private room owned by the test agent and never accepts access tokens as command-line arguments.

The harness uses UUID transaction IDs for every idempotent write.

The harness generates and decodes a real PNG, WAV, and WebM fixture before upload, downloads each MXC through the authenticated client media API, and verifies its SHA-256 digest before sending media events.

The harness emits the interaction matrix, client-controllable ephemeral categories, redaction cases, and opaque encrypted relations through authenticated Matrix client APIs.

Before redacting an original with a dependent edit, the harness waits through read-only service-cache observations until that edit index is present, so the redaction case cannot race sync ingestion.

The harness can invite and explicitly join a second test agent when its token is supplied through a second environment variable.

After room creation and any invited join, the harness queries authenticated joined membership, requires exactly the expected authenticated accounts, and records the sorted member IDs in raw evidence.

Evidence collection opens the service cache with SQLite `mode=ro`, enables `PRAGMA query_only`, starts one read transaction for a consistent snapshot, and records only IDs, counts, integrity state, hashes, and timings.

Strict thread reads use a separate new disposable SQLite database and refuse an existing path or the service-cache path.

The first strict read refills that isolated cache from the authenticated homeserver, the second proves a zero-fetch cache hit, and a redaction applied through the cache API proves rejection and refill without writing the live service database.

The second read must return the same visible event IDs as the first with zero homeserver time, pages, and scanned events.

The third read must reject the snapshot under the gap reason the redaction recorded, omit the redacted child, and refill successfully without degradation or error.

The backend-neutral owning-seam test `test_advisory_stale_fallback_is_labeled_and_dispatch_rejects_it` forces the same homeserver failure against SQLite and PostgreSQL.

It proves that an advisory read may return labeled degraded `stale_cache` history while the strict dispatch snapshot propagates the refill failure instead of serving stale context.

Every declared interaction expectation is compared with homeserver accounting, the read-only service snapshot, and the initial visible thread projection before evidence is written.

The CLI requires the service-cache path and strict-read mode, and the writer refuses output without complete passing expectation validation or with any homeserver-to-cache accounting gap.

An invited test agent requires both `--invite-user-id` and `--invite-access-token-env`, and the harness verifies that token's identity before the account can join.

The evidence writer rejects secret-shaped keys and either access-token value before writing JSON.

Use this local form after loading credentials from a secret store into the environment.

```bash
uv run python tests/manual/matrix_event_cache_live_audit.py \
  --homeserver "$MATRIX_HOMESERVER" \
  --evidence /tmp/matrix-event-cache-audit.json \
  --cache-db "$MINDROOM_STORAGE_PATH/event_cache.db" \
  --strict-read-cache-db "/tmp/matrix-event-cache-strict-$(uuidgen).db" \
  --strict-thread-reads
```

Use `--invite-user-id`, `--invite-access-token-env`, and `--trigger-user-id` together to exercise a real running agent's thread-history read path.

For hosted evidence, reread the remote `AGENTS.md` files, run only against the `mindroom-chat` instance, keep both tokens inside the remote shell, and use a new disposable private room.

Do not point hosted evidence at local port 8008.

Do not print tokens, edit the live database, deploy, restart, or use a pre-existing room.

The sanitized harness JSON contains the exact fetch source, timings, scanned pages, scanned event counts, visible event IDs, cache rejection reason, and executable expectation-validation count for all three isolated strict reads.

Hosted evidence describes the deployed service as a production baseline.

PR-specific classification remains proven by the SQLite and PostgreSQL owning-seam tests unless the PR itself has been deployed through a separately authorized release.

## Known storage and encryption gaps

Opaque encrypted thread evidence is owned by this contract track and fails closed.

An opaque `m.room.encrypted` child with a clear thread-affecting relation gap-marks only its thread, no later append clears that marker, and thread-history refresh rejects reconstructions that still contain the opaque evidence.

Thread snapshot replacement currently owns point-row deletion through the duplicated storage layout, which belongs to the storage normalization track.

A deterministic reproduction is to cache a thread child plus a relation-less reply, edit, and message reference, refill the snapshot, redact the child, and refill again.

The authoritative refill removes the now-unresolvable reply, reply edit, and reference from point storage even though the homeserver timeline still retains them.

This storage-owned reproduction is documented for coordination and is not inferred from any helper process writing the live service cache.

## Durable evidence

The checked-in live evidence records the disposable room and thread identifiers, fixture hashes, cache accounting, and three consecutive read outcomes: refill, verified cache hit, and rejection followed by refill.

The evidence must contain no credentials or message bodies.
