> Completeness critique across all four designs (2026-06-12). Its addition items
> are incorporated into ../REPLAY_PLATFORM_DESIGN.md.

The four designs are individually strong but disagree on load-bearing shared contracts (candidate packaging form, the compaction component and its window/prefix/manifest scheme, sink policy naming, recording_run_id semantics, and the gseq fix family), and the overall plan is missing several whole workstreams: secrets/PCI handling in recordings (sensitive test credentials and card payloads can be captured by the current demo recording flow), replay-time outbound egress control (lookup misses execute live against real connectors), replay DB-state provisioning beyond Redis (db fall-through hits live pg; the demo only works because replay shares the record-phase database), pipeline observability with named metrics/SLOs, cost/capacity engineering including the Kafka-retention/S3 footprint, and a public-packaging/release path that replaces private history and nested vendor paths.

ITEMS:
- CONTRADICTION: replay-pipeline specifies IN-TREE deja crates in the HS PR (path=../deja) while packaging explicitly rejects in-tree and specifies a rev-pinned git dependency; the compat gate (read candidate Cargo.lock for deja version) only works as designed under the git-dep form
- CONTRADICTION: three incompatible specs for the same compaction component — s3-store's deja-compactor binary (5-min windows, landing/windows/sessions, masking inside compactor) vs replay-pipeline's materializer (ingress-timestamp correlation selection) vs packaging's orchestrator ingest step (hour windows, raw/curated zones, masking as separate job); no single manifest JSON schema exists for the dashboard/catalog to pin to
- CONTRADICTION: sink policy appears as DEJA_SINK_POLICY=block|drop, DEJA_SINK_POLICY=fail_open|block, and DEJA_RECORD_DELIVERY=fail_open|block across the three designs; the reviewed writer-enqueue-layer drop fix is reflected in none of them
- CONTRADICTION: packaging redefines recording_run_id as a per-boot instance id while the demo/dashboard use it as a session id and the graph trace join is scoped on it — envelope needs distinct session_id/instance_id/window fields agreed by all four designs
- UNOWNED: four interacting gseq fixes proposed by separate reviews (emit-time allocation, sampling-gate-before-allocation, spawned-task flag propagation, multi-producer ordering/renumbering) have no single owner; dashboard still anchors diff rows on bare gseq which is ambiguous in any multi-pod staging window — and staging IS multi-pod, making this v1-critical
- GAP: production graph transport is a dashboard 'contract item' that no other design implements — s3-store has no graph/ prefix or graph handling in compactor/manifests/masking, and the producer side doesn't exist (graph.rs writes files only; the Kafka envelope hardcodes artifact_type deja_artifact_record at deja_record_sink.rs:34)
- GAP: runner-computed scoring must populate the Postgres divergence index, but the pull-based runner protocol has no scoring-results ingestion endpoint and no ordering guarantee before run completion
- VERIFIED SENSITIVE-DATA RISK: local demo recordings can capture credential-like fields and card-like payloads; recordings put Kafka, S3, lookup tables, Postgres divergence rows, and the auth-light browser UI in sensitive-data scope, so v1 needs redaction plus storage/IAM controls before production use
- VERIFIED EGRESS RISK: outbound HTTP replay substitutes on lookup hit but executes LIVE on miss (http_client.rs:33 replay_with fall-through; replay_boundary returns None -> live call), so novel candidate calls from runner VMs hit real connectors with recorded credentials; no runner network-egress policy is designed
- VERIFIED DB-STATE GAP: db boundaries fall through to live execution on Err arms and misses (deja/src/lib.rs:360-455), and the demo replay only works because hyperswitch-replay shares the record-phase pg (docker-compose.deja.yml:119-125); fresh-pg runner replays of S3 windows have no designed DB seeding/migration-version policy
- VERIFIED 10c AUDIT: kernel, lookup renderer, visualizer, and divergence scoring consume the S3-pulled events.jsonl chain (lifecycle/mod.rs:629-667, lookup/mod.rs:126, visualize-replay.py:72) and survive JSONL-sink removal; deja-tui (SEMANTIC_FILE_NAME, deja-tui/src/lib.rs:12) and the unlisted deja-semantic-metrics binary (deja-record/src/bin/deja-semantic-metrics.rs:33) read the producer JSONL and break
- VERIFIED: the Kafka sink is best-effort secondary with no-op flush exactly as decision 10a states (deja_record_sink.rs:19-24, flush at :111-113); the nested vendor path dep is real (router/Cargo.toml:128 path=../../../../crates/deja); the sampling injection site and Superposition client both exist (router/src/lib.rs:420-440, external_services/src/superposition.rs)
- MISSING WORKSTREAMS: pipeline observability with named metrics/SLOs (load-bearing under fail_open), cost/capacity engineering (~420GB/day, Kafka retention floor vs Vector worst-case lag, double storage landing+windows), runner build security (build-from-source = arbitrary code execution, free-form refs, spoofable actor header), gate noise/verdict policy (demo needs a human-set --expect-divergence flag today), unified Postgres schema ownership, and an integrated v1 critical-path plan (Superposition sampling is NOT on the v1 critical path; the axum rework and sink removal are)

DETAIL:
## Completeness critique — overall plan across s3-store / replay-pipeline / dashboard-ux / packaging

All file references are absolute under `{WT}` (abbreviated `{WT}` below).

---

### A. Cross-design contradictions (must be reconciled into shared contracts)

**A1. Candidate packaging form: in-tree vs git-dep (replay-pipeline vs packaging).**
replay-pipeline: "Candidate images built from the REAL Hyperswitch repo with IN-TREE deja crates (crates/deja* copied in by the integration PR, path=../deja)". packaging: in-tree explicitly REJECTED ("triples the PR to ~+13k lines, forks the source of truth"), rev-pinned git dependency chosen. These cannot both be the upstream PR. Knock-ons: (a) packaging's compat gate "read the candidate ref's Cargo.lock for its deja version BEFORE building" only works under git-dep (the lock carries `source`/`rev`; an in-tree path dep carries no usable rev); (b) the runner's build steps and the CI integration-check `[patch]` mechanism differ per form. Ground truth confirms the current layout is the local-only nested path hack: `{WT}/vendor/hyperswitch-deja-clean/crates/router/Cargo.toml:128` — `deja = { path = "../../../../crates/deja", optional = true, ... }`. **Addition item:** a single "candidate packaging form" decision record, with the compat-gate mechanism and runner build script specified against it; replay-pipeline's build design rewritten to match.

**A2. One compaction component, three incompatible specs.**
- s3-store: new `deja-compactor` binary; `landing/` → `windows/` + `sessions/`; 5-minute UTC event-time windows; dedup `(instance_id, global_sequence)`; seal = manifest PUT-last after grace + consumer-offset check; MaskingEngine INSIDE the compactor between sort and write.
- replay-pipeline: a "window materializer" that dedups by `(producer_instance_id, global_sequence)` and selects whole correlations by INGRESS timestamp + grace window.
- packaging: "the orchestrator's new ingest step"; `raw/` → `curated/` zones; HOUR-window prefixes; dedup `(recording_run_id, global_sequence)`; masking as a SEPARATE JOB between zones.
Disagreements: component ownership, window granularity (5min vs 1h), prefix taxonomy (landing/windows/sessions vs raw/curated), masking hook location (in-compactor vs between-zone job — both claim to be "the masking hook point" required by fixed decision 3), dedup identity field naming, and correlation grouping basis (event-time windows + correlation indexes vs ingress-timestamp selection). The dashboard schedules replays and enforces masking/completeness policy off "the catalog manifest" — whose field schema currently differs per design. **Addition item:** one canonical Recording Store Contract document (bucket/prefix layout, window granularity, manifest JSON schema with exact field names incl. masking state + completeness + manifest_revision, sealing semantics, who runs the compactor and where it gets Postgres/audit access), which pipeline, dashboard, and packaging all reference instead of restating.

**A3. Sink policy env contract: three names, three value sets.**
`DEJA_SINK_POLICY=block|drop` (s3-store), `DEJA_SINK_POLICY=fail_open|block` (replay-pipeline), `DEJA_RECORD_DELIVERY=fail_open|block` (packaging). Same intent, divergent contracts — this is the env var the HS PR, the demo overlay, and ops runbooks all hardcode. Additionally, the replay-pipeline review's finding (drop decision must also exist non-blockingly at the writer-enqueue layer, because `{WT}/crates/deja-record/src/writer.rs:320-335`'s untimed blocking send stalls request threads once the channel fills) is reflected in NONE of the three designs' decision lists. **Addition item:** one sink-policy spec (name, values, default per env, where the drop decision executes, loss-counter semantics) adopted verbatim by all three.

**A4. `recording_run_id` semantics fork.**
packaging redefines `recording_run_id` as a per-process instance id (`{service}-{host}-{boot_ns}`). But: the demo/local model uses it as the discrete SESSION id shared by every event of a recording (`{WT}/vendor/hyperswitch-deja-clean/docker-compose.deja.yml:83` sets `DEJA_RECORDING_RUN_ID: ${RECORDING_ID}`); the dashboard's mandatory graph-trace scoping joins on `(recording_run_id, node_id)`; s3-store instead adds a separate `instance_id` plus `capture.mode`. If packaging's redefinition lands, graph scoping changes meaning and local session parity (fixed decision 1: discrete sessions remain for local dev) breaks. **Addition item:** envelope v2 must carry three distinct fields — session/window selector, producer `instance_id`, and the graph-scoping id — with an explicit mapping table, and the dashboard's join spec updated to use the agreed field.

**A5. The gseq fix family has four interacting fixes and no owner.**
The three reviews independently demand: (1) emit-time gseq allocation (cancellation gaps — s3-store review), (2) sampling gate BEFORE sequence allocation (replay-pipeline review; packaging design claims this but the spawned-task propagation hole remains), (3) a correlation→decision registry or SpanCorrelation flag for spawned-task events (packaging review), (4) ingress-time drive ordering + producer-scoped/renumbered sequences threaded through lookup/observed/divergence artifacts (replay-pipeline review, multi-producer windows). These interact: emit-time allocation changes the same EventBuilder code path as the sampling gate; renumbering changes what the compactor's canonical sort and the manifest coverage check consume; and dashboard-ux still anchors deep links and diff rows "by gseq" (bare u64), which is ambiguous the moment a window spans two pods — and the FIRST deployment target (staging) is multi-pod with restarts, so this is v1-critical, not a prod-scale deferral. **Addition item:** a single "event identity & ordering" spec (allocation point, sampling gate point, spawned-task flag transport, compound key `(instance_id, gseq)` threaded through LookupEntry/ObservedCall/divergence rows/UI anchors, kernel drive order) that all four designs implement against, plus a regression test in the demo fixture.

**A6. Graph artifacts in the production store: a contract with no counterparty.**
dashboard-ux: "production graph transport rides the hardened Kafka sink as a second artifact_type routed by Vector to a graph/ prefix — a contract item with the pipeline designer." No other design implements it: s3-store's layout (landing/windows/sessions), compactor, manifests, masking, and retention cover semantic events only; packaging's envelope v2 and Vector config don't mention graphs. The producer side doesn't exist either: graph capture writes files via `DEJA_GRAPH_DIR` (`{WT}/crates/deja-record/src/graph.rs`), and the only Kafka envelope hardcodes a single artifact type (`{WT}/vendor/hyperswitch-deja-clean/crates/router/src/services/kafka/deja_record_sink.rs:34` — `ARTIFACT_TYPE: "deja_artifact_record"`; the writer pipeline handles `SemanticEvent` only). **Addition item:** either a full graph-transport workstream (producer envelope, Vector route, store prefix, completeness accounting, masking of graph payloads, retention) or an explicit v1 decision that production recordings have no graph data and the dashboard's trace feature is local-only — the current plan ships a UI whose production data path nobody is building.

**A7. Runner scoring → Postgres divergence index has no ingestion path.**
replay-pipeline puts scoring ON the runner (artifacts via presigned S3 PUTs; scorecard dual-homed). dashboard-ux requires per-divergence rows "persisted to Postgres at scoring time" for filtering. Runners are pull-based, outbound-HTTPS-only, with no Postgres access; the specified protocol (claim/lease/heartbeat/stage transitions/presigned PUTs) has no scoring-results ingestion endpoint, and there's no ordering guarantee that the index is ingested before the run is marked complete (dashboard-ux itself flags 'dormant ingest endpoints must be on the runner critical path' as a risk but the pipeline design doesn't own the fix). **Addition item:** add `POST /runner/runs/{id}/divergences` (or equivalent) to the runner protocol, with size limits for ~2000-request samples, idempotency, and a state-machine rule that `completed` requires index ingestion.

**A8. `manifest_revision` is not threaded through.**
s3-store requires scorecards to carry the `manifest_revision` they were computed against (window amendment problem) and says "the catalog notifies" on amendment. Neither replay-pipeline's scorecard/run schema nor dashboard-ux's scorecard/catalog views include revision display, an "amended since scoring" badge, or who consumes the notification. **Addition item:** manifest_revision in the catalog tables, the job document/config_snapshot, the scorecard schema, and a UI badge + manual re-run affordance.

---

### B. Missing workstreams (nobody owns these)

**B1. Sensitive data in recordings — capture risk.**
- The demo sends connector credentials in a request body: `{WT}/demo/workload.sh:155` (`"connector_account_details":{"auth_type":"HeaderKey","api_key":"$STRIPE_API_KEY"}`); `{WT}/demo/lib.sh:18` requires it; the compose passes it into the router env (`docker-compose.deja.yml:86`).
- The current recording flow can capture sensitive credential and card fields in recording artifacts (values intentionally not reproduced here).
Consequences none of the four designs addresses: the Kafka topic, S3 landing+windows, rendered lookup tables, observed files, the Postgres divergence index, and the auth-light browser UI can all become sensitive-data surfaces. "Staging-only until masking lands" does not cover this. **Addition items:** (i) a v1 credential-field redaction denylist applied at record time or in the compactor — distinct from and ahead of the full masking workstream; (ii) S3 SSE-KMS + bucket IAM design and Kafka topic ACL/encryption posture; (iii) UI-side redaction rules for known credential fields in the explorer; (iv) retention/deletion policy for derived artifacts (lookup tables, observed, divergence rows), not just recordings.

**B2. Replay-time outbound egress control.**
Outgoing HTTP substitutes on lookup hit but executes LIVE on miss: `{WT}/vendor/hyperswitch-deja-clean/crates/external_services/src/http_client.rs:33` (`replay_with = semantic_boundary::replay_response(...)`) and `{WT}/crates/deja-record/src/lib.rs:1296-1331` (`replay_boundary` returns `None` → "the caller executes + records as usual"). A candidate whose change alters outbound request args produces misses → the runner VM actually calls the real connector with recorded credentials substituted from the recorded DB rows. At 2000-request samples this can hammer Stripe test accounts or mutate their state; with imperfect masking it could one day carry live credentials. The dashboard treats "novel calls" purely as a UI classification; no design specifies runner network policy. **Addition item:** deny-by-default egress on replay containers (compose network without external route, or an explicit proxy allowlist), plus a distinct observed-call outcome ("novel outbound blocked") so blocked calls render as such instead of as connector timeouts/status-0 noise.

**B3. Replay environment state provisioning beyond Redis.**
DB boundaries replay from the lookup on hit, but recorded `Err` arms with unknown kinds and lookup misses FALL THROUGH TO LIVE execution (`{WT}/crates/deja/src/lib.rs:360-455`, the documented "V1 skip error arms" policy). The demo only works because the replay container shares the record-phase Postgres in the same compose project — `{WT}/vendor/hyperswitch-deja-clean/docker-compose.deja.yml:97-127` (`hyperswitch-replay` depends_on the same `pg`, no DB reset between record and replay), so fall-through queries see the state the record phase created. On a runner VM replaying an S3 window: pg is fresh and empty, every fall-through returns NotFound where the recording environment had rows, and migrations come from the CANDIDATE ref while the recording was made on another ref (schema drift). s3-store flags only the Redis half ("mid-stream window fidelity"). **Addition items:** runner DB provisioning policy (empty+migrations from which ref; behavior when candidate migrations diverge from the recorded window's schema), a divergence classification for fall-through-induced noise, and an explicit statement of the live-fallback semantics in the replay contract.

**B4. Pipeline observability and alerting.**
Every design defers it (packaging: "the alerting story ... is assumed, not designed"; replay-pipeline: "alerting on producer drop counters ... is operationally load-bearing"). Under fail_open + manifests-as-loss-detection, alerting is part of data integrity, not ops garnish. Missing: concrete metric names and export path for the producer (Hyperswitch's metrics infra vs deja's own), delivery/drop/loss counters, Vector consumer-group lag SLO, compactor seal-latency SLO + stuck-window alert, runner fleet/lease health, S3 PUT error alarms, and a recorded-vs-landed reconciliation job feeding a dashboard. **Addition item:** an observability design doc with named metrics, transports, SLOs, and the page/no-page policy; the seal-blocked-on-lag condition (s3-store) needs an explicit alert threshold.

**B5. Cost and capacity engineering.**
~420GB/day raw at 10^8 events/day (4.2KB/event observed); landing/ + windows/ stores ~2x until TTL; Kafka 12-partition sizing vs peak rates (review math: avg ~1.2k events/s, peaks far higher); broker disk = retention floor × rate; Vector instance count capped at partition count; S3 request costs from per-key batching (key cardinality = windows × instances); compactor compute and memory (external sort flagged but unsized). No budget guardrails and no feedback loop from volume/cost telemetry to the Superposition sampling policy (the natural cost lever — fixed decision 2 makes the strategy evolvable, but nobody designed the cost input to it). **Addition item:** a capacity/cost model with the Kafka retention floor explicitly derived from worst-case Vector outage tolerance (this number also bounds the only unrecoverable loss mode upstream of landing/).

**B6. Runner security model.**
Build-from-source is arbitrary code execution by design (build.rs and proc-macros run at compile time; the built router then runs on the VM), triggered by an auth-light API whose actor header is spoofable (acknowledged) and whose code ref is free-form. Missing: allowlisted repos/orgs for candidate refs; scoping of all runner-resident credentials (git fetch tokens, image registry, sccache backend, S3 recording READ — uploads are presigned but reads aren't specified); per-run workspace isolation/cleanup; disk quotas (a single build's target dir was observed at 4.6GB mid-build per packaging); and an egress policy for the BUILD phase (crates.io/git only) separate from the replay phase (B2). **Addition item:** a runner security/secrets section in the runner protocol design.

**B7. Gate signal quality (noise policy / control run).**
Today the demo's verdict needs a human-supplied expectation: `{WT}/demo/run-deja-demo.sh:34-46,121-125` (`--expect-divergence/--expect-pass` flip the exit criterion). The v1 manual gate gives a human a scorecard, but no design owns: known-noise suppression (id/time substitution edge cases, version-skew rank degradation, transport status-0 rows), a merge-base self-replay control run to subtract environmental noise from candidate-caused divergence, or what scorecard state means "safe to merge." At 2000-request samples, even small noise rates make the explorer unreadable and the gate untrusted. **Addition item:** a v1 noise-policy: control-run option, per-divergence triage states (the dashboard's deferred 'annotations' is adjacent but not the same), and documented verdict semantics.

**B8. Postgres schema ownership and machine actors.**
Three designs each define tables (orchestrator runs/schedules/runners; recording catalog recording_windows/+producers; audit_events; divergence index). No single migration set, owner, or compatibility process. Also: the audit model assumes a human `X-Deja-Actor`, but the compactor (catalog PUTs are "audit-logged") and the runner agent are machine principals with no defined actor convention. **Addition item:** one schema document + migration crate, and an actor convention covering service principals.

**B9. Integrated v1 sequencing / critical path.**
Cross-design dependencies are stated individually but never assembled: axum+tokio rework gates the dashboard (hard dependency, dashboard-ux); the Postgres store swap and runner protocol land concurrently (replay-pipeline flags the M0/M1 risk); sink removal (decision 10) breaks local consumers; envelope v2 gates the compactor; the compactor gates the catalog which gates the schedule-replay UI. Concrete sequencing facts established here: (i) Superposition sampling is NOT on the v1 critical path — staging can record at 100% while the gate ships; (ii) the sole-sink switch IS on the critical path and has verified local breakage (B10/S2); (iii) multi-pod staging makes the gseq/multi-producer fixes (A5) v1-critical. **Addition item:** a single dependency-ordered milestone plan with "demo stays green" as an explicit invariant at every step (it is the only end-to-end regression fixture the project has, and s3-store's design leans on it as such).

**B10. Decision-10c consumer audit — completed against ground truth (this was an open verification item).**
Survives JSONL-sink removal (these consume the S3-pulled `{root}/recordings/{id}/events.jsonl` chain): the orchestrator's pull step writes it (`{WT}/crates/deja-orchestrator/src/lifecycle/mod.rs:629-667`), the lookup renderer reads it (`{WT}/crates/deja-orchestrator/src/lookup/mod.rs:126`, `{WT}/crates/deja-orchestrator/src/lib.rs:122,157`), the kernel reads the lookup table, divergence scoring reads lookup/observed/http-diffs only (`{WT}/crates/deja-orchestrator/src/divergence/mod.rs:9-11`), and the visualizer globs `recordings/*/events.jsonl` (`{WT}/demo/visualize-replay.py:72`).
Breaks: **deja-tui** — hardwired to the producer-side filename `semantic-events.jsonl` (`{WT}/crates/deja-tui/src/lib.rs:12`; written only by the JSONL sink, `{WT}/crates/deja-record/src/lib.rs:423-429`; demo wires it via `DEJA_SINK=both` + `DEJA_ARTIFACT_DIR`, `docker-compose.deja.yml:81-82`) — this confirms the packaging review's finding; and **deja-semantic-metrics**, a consumer NO design or review lists (`{WT}/crates/deja-record/src/bin/deja-semantic-metrics.rs:33,124` requires a directory containing `semantic-events.jsonl`). **Addition item:** the sink-removal plan must materialize the pulled recording at `{state}/recording/semantic-events.jsonl` (or teach both binaries the `recordings/{id}/events.jsonl` layout) before JSONL removal.

---

### C. Spot-checks performed (claims vs repo)

1. **"HyperswitchKafkaRecordSink is best-effort secondary, swallowed failures, no-op flush" — TRUE.** `{WT}/vendor/hyperswitch-deja-clean/crates/router/src/services/kafka/deja_record_sink.rs:19-24` documents exactly this ("BEST-EFFORT ... composite swallows secondary failures, Kafka delivery reports are not awaited, and flush() does not flush"); `flush()` is `Ok(())` at :111-113. Bonus finding: the envelope hardcodes a single `artifact_type` (:33-34), so the dashboard's "second artifact_type for graphs" requires producer changes nobody scoped (A6).
2. **"Sampling injection site at router lib.rs:429 / Superposition client exists" — TRUE.** The `RequestIdentifier` middleware construction with the deja replay `IdReuse` logic is at `{WT}/vendor/hyperswitch-deja-clean/crates/router/src/lib.rs:~420-440`, and `{WT}/vendor/hyperswitch-deja-clean/crates/external_services/src/superposition.rs` exists. The packaging review's spawned-task propagation hole stands (correlation layer carries only the correlation id).
3. **"vendor nested layout is local-dev only" (decision 5 input) — confirmed as the CURRENT reality:** `{WT}/vendor/hyperswitch-deja-clean/crates/router/Cargo.toml:128` uses `path = "../../../../crates/deja"`, which cannot exist in the real HS repo — making the A1 contradiction (which packaging form replaces it) a blocking pre-v1 decision, since the runner build design depends on it.
4. **Secrets/PCI presence (B1) — sensitive-field capture risk:** prior demo artifact spot-checks found credential-like and card-like fields in recording JSONL; values intentionally are not reproduced in this repository.
5. **Live-fallback semantics (B2/B3) — verified:** redis boundaries use `replay_ok` (`{WT}/vendor/hyperswitch-deja-clean/crates/redis_interface/src/commands.rs:114-131` et al.), db uses `record_query_async` with documented fall-through to live execution (`{WT}/crates/deja/src/lib.rs:360-455`), http_outgoing falls through on miss (`http_client.rs:33`); the replay container shares the record-phase pg/redis (`docker-compose.deja.yml:97-127`).

### D. Suggested priority order for the addition items
1. A1 (packaging form) and A2 (recording store contract) — every other design writes against these.
2. A5 (event identity & ordering spec) — v1-critical because staging is multi-pod.
3. B1 (secrets/PCI redaction + KMS/ACL) — gates even the staging-only deployment posture.
4. A7 + B2 + B3 (runner protocol completeness: divergence ingestion, egress policy, DB provisioning) — the v1 gate produces wrong/empty/noisy results without them.
5. B9 (integrated sequencing with demo-stays-green invariant) and B10 (10c consumer fixes) — protect the only regression fixture.
6. A3/A4/A6/A8, B4-B8 — required before any prod rollout, schedulable after the v1 slice is consistent.