> **Archived.** This document records the preload/FFI-era correlation design (superseded by the macro/hook architecture and the 6-rank address ladder). It is kept for historical context and no longer matches the shipped system; the current reference is [DEJA_RECORDING_ARCHITECTURE.md](../DEJA_RECORDING_ARCHITECTURE.md).

# Déjà Correlation Architecture

This note captures the current correlation design, why it is the right foundation, why Postgres works, why Redis/fred does not fully work yet, and what questions should be revisited later.

## What we need from correlation

Déjà records external I/O at the syscall/socket boundary using LD_PRELOAD. The hooks can see bytes sent to Postgres, Redis, HTTP, etc., but a raw syscall does not naturally know which application request caused it.

Correlation adds that missing request context:

- group dependency I/O by request
- validate that replay exercised the same dependency paths
- explain what a request touched during debugging
- keep benchmark metrics tied to request execution, not background noise

The key requirement is: **when an application task performs I/O, the LD_PRELOAD hook must be able to read the current request/correlation ID from that task's execution context.**

## Current solution

### 1. `deja-actix` owns framework scoping, not app-specific parsing

`deja-actix` exposes a generic `CorrelationId` and middleware hook. It does not know about Hyperswitch internals.

For Hyperswitch, the bridge uses `router_env::RequestId` already parsed/generated by Hyperswitch's `RequestIdentifier` middleware. This avoids parsing request headers twice and respects Hyperswitch's configured ID reuse behavior.

Important config:

```toml
[log.trace_header]
id_reuse_strategy = "use_incoming"
```

This allows workload-provided `X-Request-ID: deja-concurrent-*` headers to survive as the canonical Hyperswitch request ID.

### 2. `deja-tokio::DEJA_CORRELATION_ID` is the source of truth

The old thread-local request scope was wrong for async Rust. Tokio tasks can yield, resume, and move between worker threads. A plain thread-local can leak scope across unrelated tasks or disappear when a task moves.

The replacement is:

```rust
tokio::task_local! {
    pub static DEJA_CORRELATION_ID: String;
}
```

Middleware wraps the request future:

```rust
DEJA_CORRELATION_ID.scope(correlation_id, inner_future).await
```

Tokio's task-local implementation swaps the value in/out on each poll with RAII semantics. That is the fundamental correctness property:

- while request task A is being polled, hooks see A's correlation ID
- when A yields, the value is removed/restored safely
- when task B is polled, hooks see B's ID
- work-stealing does not break correlation
- there is no manual set/clear bug on `Pending`

### 3. LD_PRELOAD hooks read the task-local through an FFI bridge

`deja-tokio` exports:

```rust
#[no_mangle]
pub unsafe extern "C" fn deja_correlation_id(buf: *mut u8, len: usize) -> usize
```

`deja-preload` resolves this symbol with `dlsym(RTLD_DEFAULT, "deja_correlation_id")` and reads the current ID at hook time.

This is the key bridge between Rust task context and C-level syscall interception.

### 4. Spawn boundaries need propagation

`DEJA_CORRELATION_ID` does not automatically flow into `tokio::spawn` or `spawn_blocking` tasks. We added wrappers:

```rust
deja_tokio::spawn(...)
deja_tokio::spawn_blocking(...)
```

These capture the current correlation ID and re-scope it inside the child task/closure.

This fixed Postgres/Diesel paths because `async-bb8-diesel` runs blocking Diesel queries inside `tokio::task::spawn_blocking`. Patching it to use `deja_tokio::spawn_blocking` preserved request context through the blocking boundary.

## In-band validation markers

Hook-level metadata can be wrong or missing, so we added an oracle that embeds the expected correlation ID inside the actual dependency protocol payload.

Postgres marker:

```sql
/* deja_expected_request_id=<id>;deja_backend=postgres;deja_operation=<op> */ SELECT 1
```

Redis marker:

```text
ECHO deja_expected_request_id=<id>;deja_backend=redis;deja_operation=<op>
```

The scanner compares:

- expected ID embedded in the payload
- actual `request_id` captured by the hook

This validates correlation from inside the real I/O stream instead of trusting only middleware state.

Marker helpers live in `deja-tokio`:

- `current_correlation_id()`
- `sanitize_marker_value()`
- `postgres_validation_marker_sql()`
- `redis_validation_marker_payload()`

All marker values are sanitized to `[a-zA-Z0-9_.:-]` before embedding.

## Why Postgres works

Postgres I/O stays in the request's execution path once the blocking boundary is patched.

Path:

1. Actix request enters `DejaScope`
2. `DEJA_CORRELATION_ID.scope(id, future)` wraps the request future
3. Diesel query goes through `async-bb8-diesel`
4. patched `deja_tokio::spawn_blocking` carries the ID into the blocking closure
5. marker SQL and real query execute with the ID scoped
6. LD_PRELOAD hook reads the same ID via FFI

Result observed in the pipeline:

- Postgres markers match hook request IDs
- no mismatches
- request/I/O pair completeness is high

This proves the core design is fundamentally sound for request-owned I/O.

## Why Redis/fred does not work the same way

Redis uses the `fred` client. fred does not write to the socket directly from the request task.

fred architecture:

1. request task calls a Redis API method
2. fred packages the command
3. command is sent into an internal channel
4. a long-lived routing task reads all commands from that channel
5. that routing task writes to the Redis socket

The routing task is spawned when the Redis pool is created, before any HTTP request exists. Therefore it has no `DEJA_CORRELATION_ID`.

So at hook time:

- the Redis payload is fully visible
- the ECHO marker contains the expected request ID
- but the socket write happens in fred's routing task
- LD_PRELOAD sees `request_id=<none>`

This is why upstream Tokio task hooks alone do not fix Redis. Even if `tokio::spawn` propagates context without a fork, fred's routing task was spawned at startup, outside any request scope. There is no request context to inherit.

## What is actually lost for Redis?

Not the bytes. Déjà still records Redis command/response payloads and timing.

What is missing is structured per-request metadata on the Redis socket event.

| Capability | Status |
|---|---|
| Record Redis bytes | works |
| Record Redis timing/order | works |
| Replay HTTP requests that cause Redis calls | works |
| Validate HTTP-level replay | works |
| Know directly which request caused each Redis socket event | missing |
| Infer Redis owner from in-band marker payloads | partially works |

For the current replay model, this is not a correctness blocker. We replay at the HTTP/application level: send the same request and let the application naturally execute Redis calls. We are not trying to drive Redis commands ourselves during replay.

It does matter for advanced debugging, dependency graphs, and exact per-request I/O diffing.

## Demo interpretation

The oracle should not show Redis as a generic failure. It should show:

```text
postgres 107/107 ✓
redis      0/932 ⚠ known: multiplexed routing task; markers present in payloads
```

That means:

- core correlation is correct for request-owned I/O
- Redis payload capture is complete
- Redis structured request tagging needs a client-aware propagation mechanism

## General solution direction

A general solution must handle both types of async I/O:

### A. Request-owned tasks

Examples:

- Actix handler future
- spawned child task created during request
- `spawn_blocking` DB query closure

Solution:

- task-local correlation
- spawn wrappers or runtime-level spawn instrumentation
- FFI bridge for hooks

This is what we have now.

### B. Shared long-lived I/O driver tasks

Examples:

- fred Redis routing task
- multiplexed database drivers
- background connection managers
- queues where request tasks enqueue work and another task owns the socket

Task-local inheritance is not enough. The context must travel with the operation, not the worker task.

A more general design:

1. Capture current correlation ID at API call/enqueue time
2. Attach it to the internal command/message metadata
3. When the routing task dequeues the command, temporarily scope that ID while writing the socket
4. Optionally preserve the ID until the matching response is read

Conceptually:

```rust
struct CommandEnvelope<T> {
    command: T,
    deja_correlation_id: Option<String>,
}

while let Some(envelope) = rx.recv().await {
    if let Some(id) = envelope.deja_correlation_id {
        DEJA_CORRELATION_ID.scope(id, write_command(envelope.command)).await;
    } else {
        write_command(envelope.command).await;
    }
}
```

For Redis/fred specifically, this means modifying fred or wrapping it at a layer where command metadata can be preserved into the router task.

## Correlation 2.0: replay disambiguation, not just request attribution

The Hyperswitch Redis/DB analysis showed that correlation is not only a debugging convenience. It is also one of the replay disambiguators needed when dependency reads are mutable.

A pure replay lookup table like this is unsafe for stateful dependencies:

```text
signature(command) -> recorded_response
```

Example:

```text
HSET payment_attempt pa_1 { status: Pending }
HGET payment_attempt pa_1 -> { status: Pending }
HSET payment_attempt pa_1 { status: Charged }
HGET payment_attempt pa_1 -> { status: Charged }
```

Both `HGET payment_attempt pa_1` reads have the same command signature, but the correct response depends on state/order.

The fresh Hyperswitch checkout at `vendor/hyperswitch-fresh` commit `bc39324410031bec3e8c3d0ba924d81841c0c341` contains at least:

| Category | Count |
|---|---:|
| Direct production Redis read primitive call sites | 31 |
| Direct app-level Redis read sites excluding generic wrappers | 27 |
| KV-store Redis read call sites via `KvOperation::{HGet,Get,Scan}` | 29 |
| Concrete Redis read sites that can become replay-ambiguous | 56 |
| KV-store Redis write call sites via `KvOperation::{Hset,HSetNx,SetNx}` | 21 |
| Direct Redis write/delete primitive call sites excluding pub/sub | 76 |

The highest-risk families are payment intent, payment attempt, refund, mandate, address, reverse lookup, volatile payment method records, retrieve-and-delete token flows, and cache populate-on-miss flows.

This means collision-free signatures are insufficient. They prevent accidental hash collisions, but they do not distinguish multiple valid responses for the same stateful read signature.

### Required replay identity

For dependency replay/diffing, each recorded operation should carry enough identity to support multiple match modes:

```text
global_index
connection_id / fd
protocol
operation_signature
resource_key
causal_scope_id
local_sequence_in_scope
resource_version_or_cursor
request_payload
response_payload
```

The minimum robust lookup for non-emulated mutable reads is:

```text
(causal_scope_id, operation_signature, local_sequence_in_scope)
```

A request id alone is not enough. One request can perform:

```text
GET key
SET key new_value
GET key
```

Both reads share the same request id and signature. They still need an occurrence index or resource version.

### Replay lookup hierarchy

Replay should prefer the strongest available evidence and degrade with warnings:

1. **State-seeded real dependency replay** — for owned DB/Redis, reconstruct the request's pre-state, seed isolated infrastructure, and let candidate code hit real dependencies.
2. **Stateful emulator result** — for protocols where Déjà can model enough semantics, e.g. Redis KV/hash commands.
3. **Exact causal lookup** — `causal_scope_id + local_sequence_in_scope + operation_signature`.
4. **Resource cursor lookup** — `resource_key + resource_version_or_cursor`.
5. **Connection/global sequence lookup** — useful for deterministic single-threaded recordings, risky under concurrency.
6. **Per-signature FIFO queue** — acceptable only when concurrency/order ambiguity is low.
7. **Signature-only fallback** — allowed only with an explicit ambiguity warning.

State-seeded replay changes the role of correlation. Correlation is no longer only a runtime lookup key for returning recorded DB/Redis responses; it becomes the mechanism for building the per-request fixture:

```text
correlation scope + local sequence
  -> identify pre-write reads
  -> derive initial DB/Redis facts
  -> identify request-produced writes
  -> validate ordered side effects and final state
```

The critical rule is that Déjà must seed **pre-request state**, not every value ever read during the request. If a request does `GET k -> nil`, then `SET k v`, then `GET k -> v`, only the first read contributes to initial state; the second read is an in-request observation. See `docs/STATE_SEEDED_REPLAY.md` for the complete model.

### Ambiguity diagnostics

Recording should detect repeated signatures with divergent responses:

```text
signature = HGET mid_m1_pid_p1 pa_a1
responses_seen = 4 distinct values
classification = ambiguous_mutable_read
recommended_disambiguator = state_version or causal_scope + ordinal
```

This converts a theoretical correctness issue into an inspectable artifact property.

### Consequence for shared driver tasks

The Redis/fred routing-task gap becomes more important once Déjà does dependency-level replay/diffing. HTTP-level replay can still work without perfect Redis request attribution because the application naturally executes its own Redis calls. But if Déjà wants to explain or replay dependency responses precisely, the Redis command must carry causal metadata from API/enqueue time into the routing task.

So Correlation 2.0 is:

- task-local scope for request-owned work,
- command/work envelopes for shared I/O drivers,
- per-scope operation ordinals,
- optional protocol-specific state emulation,
- and ambiguity diagnostics when the artifact cannot be matched safely.

See `docs/HYPERSWITCH_REPLAY_AMBIGUITY_REPORT.md` for the detailed Hyperswitch counts and examples.

## Future questions / discussion themes

Use this section as a reminder list.

### Product / demo

- Is HTTP-level replay sufficient for the first product demo?
- Should Redis be presented as a known limitation or hidden from the scorecard?
- What minimum correlation story is convincing: PG-only perfect match, or Redis partial with explanation?

### Benchmarking

- Should in-band oracle be a hard gate only for backends that are known request-owned?
- Should Redis markers be reported separately as `known_unscoped_driver_task`?
- Should the scorecard distinguish capture completeness from request attribution?

### Architecture

- Do we standardize on explicit runtime Builder construction with upstream Tokio task hooks?
- How do we make hook installation ergonomic for `#[tokio::main]` applications without patching Tokio?
- How should correlation propagate through channel-based worker pools?
- Do we need a generic `CorrelationEnvelope<T>` pattern for libraries?

### Redis/fred

- Should we patch fred to carry correlation metadata with commands?
- Is there a stable extension point in fred for command metadata?
- Can we wrap `RedisConnectionPool` APIs enough, or do we need to fork/patch fred internals?
- How do we associate Redis responses with the command ID under pipelining/multiplexing?
- **If Hyperswitch migrates to redis-rs:** redis-rs `MultiplexedConnection` has the same CommandBoundary gap (mpsc channel + PipelineSink). The patch would be simpler: add correlation to `PipelineMessage`, scope in `PipelineSink::start_send` (sync, not async). No auto-pipelining means no mixed-request batching problem. See `INTEGRATION_ROADMAP.md` §2a-alt for details.

### Replay semantics

- Are we doing application-level replay only, or do we eventually want dependency-level replay/diffing?
- If dependency-level diffing is required, how exact must per-request Redis attribution be?
- Should dependency events be matched globally by sequence/time, or grouped by request?

### Validation markers

- Should markers be enabled only in benchmark builds/config?
- Should marker helpers move from `deja-tokio` into a smaller `deja-correlation` crate?
- Should Redis validation use only marker events, or try to relate marker-to-next-command with pipelining support?

## Current recommendation

Do not solve Redis/fred correlation before the demo.

Instead:

1. keep task-local + FFI as the source-of-truth architecture
2. show Postgres marker matches as proof of correctness
3. show Redis as captured-but-untagged because of a shared routing task
4. document the general solution as operation-context propagation through command envelopes
5. revisit fred patching only when dependency-level Redis debugging becomes a product requirement
