# Internal Spec — 2026-05-21
# BYOK Encryption at Rest

> **Workflow convention** (internal only — do not duplicate into `AGENTS.md` / `CLAUDE.md`):
> 1. Write the spec first: save to `docs/internal/{date}-{content}.md`
> 2. Implement against the spec
> 3. Open the PR against the spec — PR description links back to this file as source of truth
>
> **Branch naming**: `feat/<topic>` etc. No PR numbers / phase labels / qualifiers in branch names.

---

## BYOK Encryption at Rest

**Goal**: encrypt all *new* provider API keys at rest. Existing plaintext rows are explicitly left untouched. This PR does not solve "DB dump leaks historical BYOK"; it solves "DB dump does not leak any new BYOK written after this PR ships."

### Problem

`provider_key.token` is stored as `text().notNull()` plaintext (`packages/db/src/schema.ts:594`). A DB dump leaks every customer's upstream provider credentials. Eight code paths currently read `providerKey.token` and forward it to upstream APIs.

This codebase is a fork of an open source project with an unknown number of existing deployments and users. Forcing key re-entry or running a one-shot backfill against unknown deployments is not viable.

### Threat model (explicit)

In scope:
- DB dump / read access via SQL injection, backup leak, compromised replica → must yield no usable provider credentials **for keys created after this PR ships**
- Cross-tenant ciphertext copy (write access copies row A's ciphertext into row B and tries to use B's identity) → must fail (AAD binding)
- Wrong / rotated master key → must fail loudly at first request, never silently fall back to plaintext

Out of scope:
- Memory dumps of the running gateway process
- Compromised gateway host with root
- Key rotation procedures (single-version `v1` only this PR; `kid=v1` in the format so future rotation does not require schema change)
- **Legacy plaintext rows written before this PR ships** — see "Accepted residual risk" below

### Accepted residual risk

Legacy plaintext rows remain in the database until the row owner updates their key (at which point natural rollover encrypts it).

We accept this residual risk because:
1. This codebase is a fork with an unknown number of existing deployments and users. Forcing key re-entry would break working production setups
2. The threat model for *new* keys is fully addressed by this PR
3. Future cleanup is an ops decision — e.g., a one-shot encryption pass after enough natural rollover, or contacting remaining users — **explicitly out of scope for this PR**

If Codex (or any reviewer) flags "forward-only is not safe": acknowledged. The alternative (forced re-entry or downstream-affecting backfill) is worse for a fork with unknown user base. This is the deliberate decision, not an oversight.

---

### Schema change

```ts
// packages/db/src/schema.ts — provider_key table
token:            text("token"),                  // change: drop notNull() so new rows can have NULL plaintext
tokenCiphertext:  text("token_ciphertext"),       // new — null for legacy rows, populated for new/updated rows
tokenMasked:      text("token_masked"),           // new — written at create/update time. UI reads this, never decrypts
```

`token` column type stays `text`, just becomes nullable. Existing rows are unaffected (their `token` stays populated, their two new columns stay NULL).

**Schema migration**: Drizzle generates multiple statements (DROP NOT NULL + ADD COLUMN for each new column) plus a snapshot. No data migration. Run `pnpm run setup` to regenerate; do not hand-edit the SQL.

**DB-level invariant**: add a CHECK constraint so a row can never have **both** `token` and `token_ciphertext` non-null at the same time (that would be ambiguous — which one wins on read?). Legacy rows have `token` non-null, ciphertext null. New/rolled-over rows have ciphertext non-null, `token` null. Both-null is allowed at the constraint level (deleted-soft / disabled states), and `readProviderKey()` handles it by throwing.

```sql
ALTER TABLE provider_key
  ADD CONSTRAINT provider_key_token_xor
  CHECK (token IS NULL OR token_ciphertext IS NULL);
```

If Drizzle doesn't generate this from schema annotations, update the schema definition so the generated migration includes the constraint, then regenerate via `pnpm run setup`. Do not write migrations manually.

---

### Ciphertext format

```
llmgw:v1:<b64url(iv)>:<b64url(ciphertext)>:<b64url(tag)>
```

- Magic prefix `llmgw:v1:` — boot/read sanity check
- `v1` = kid. Future rotation increments to `v2`
- All three fields base64url, no padding
- IV: 12 bytes from `crypto.randomBytes(12)` (CSPRNG, never reused)
- Tag: 16-byte GCM auth tag
- **AAD**: `provider_key|${row.id}|${row.organizationId}` — binds ciphertext to row identity. Cross-row / cross-org copy fails GCM tag

### Key derivation

```
master_key = utf8(GATEWAY_API_KEY_HASH_SECRET)
data_key   = HKDF-SHA256(master_key, salt=null, info="provider-key-token:v1", length=32)
```

No new env var: this reuses the existing `GATEWAY_API_KEY_HASH_SECRET`, which is already provisioned for gateway, API, and worker in every deployment. The HKDF `info` string domain-separates the encryption data key from the API-key fingerprint HMAC that uses the same secret. Like `getApiKeyHashSecret()`, the secret is only required in production; outside production a fixed dev secret is used so builds, CI, and local runs work with no configuration.

### Secret rotation groundwork

`GATEWAY_API_KEY_HASH_SECRET` accepts a comma-separated keyring, newest first (`getApiKeyHashSecrets()`). The first entry is the current secret: all fingerprint HMACs and all new encryptions use it. New ciphertexts are written as `llmgw:v2:<kid>:<iv>:<ct>:<tag>`, where `kid` is an HKDF-derived 8-hex-char id of the encrypting secret (`getSecretKeyId()`), so decryption picks the right keyring entry directly and reports precisely when a needed secret has been dropped. Legacy `llmgw:v1` ciphertexts (no key id) are trial-decrypted against every keyring entry.

To rotate: prepend the new secret to the keyring on all services, re-encrypt existing `token_ciphertext` rows (backfill tooling not built yet — decrypt-with-old/encrypt-with-new), then drop the old entry. NOT yet rotation-safe: persisted HMAC lookups (`master_key.tokenHash`, `scim_token.tokenHash`) and historical `log.usedApiKeyHash` attribution are matched against the current secret only; multi-secret lookup / lazy re-stamping is deferred until a rotation is actually needed.

---

### Write path

1. UI submits raw `token` to `POST /api/keys/provider`
2. Validate provider with **plaintext** first (existing flow at `apps/api/src/routes/keys-provider.ts:261`). Do not encrypt before validation
3. Resolve the row `id` **before** encrypting (so AAD can include it). The existing table uses `$defaultFn(shortid)` at `schema.ts:594` — switch this insert to generate the id in application code (`const id = shortid(); insert({ id, ... })`) so AAD is computable pre-insert
4. Compute `tokenMasked` from plaintext (reuse existing `apps/api/src/lib/maskToken.ts` behavior — first 12 chars visible)
5. Encrypt plaintext with `data_key` + fresh random IV + AAD → produce `llmgw:v1:...` ciphertext string
6. Insert / update with `token = NULL`, `token_ciphertext = <ciphertext>`, `token_masked = <mask>`
7. **Update of a legacy row** (rollover): exact same path. `token` flips from plaintext to NULL, the other two columns populate. The old plaintext is overwritten in the same UPDATE
8. Never log plaintext. Validation error messages must never include `token`, even on failure

---

### Read path

All read sites switch from `providerKey.token` to a single helper. The helper lives in `packages/actions` (a server-only package that already depends on db), not in `packages/shared` — `shared` is consumed by frontend packages, so Node `crypto` must not be in a root-exported module of a polyglot package, and `packages/db` already depends on `shared` which would risk a cycle.

```ts
// packages/actions/src/provider-key/read.ts
export function readProviderKey(row: ProviderKey): string {
  if (row.tokenCiphertext != null) {
    // Encrypted path — strict, no fallback
    if (!row.tokenCiphertext.startsWith("llmgw:v1:")) {
      throw new Error(`provider_key ${row.id}: unknown ciphertext version`);
    }
    // parse, derive AAD from row.id + row.organizationId, GCM decrypt
    // any failure (parse / tag / AAD mismatch) → throw, never fall back to row.token
    return decrypt(row.tokenCiphertext, aad(row));
  }
  // Legacy path — column is NULL, no ciphertext exists for this row
  // Note: this is NOT a fallback. It's an explicit dual-mode read keyed on column nullity,
  // not on content shape. Attacker tampering with token_ciphertext keeps the column non-null,
  // which forces the strict path → GCM failure → throw. There is no path where tampered
  // ciphertext silently uses row.token.
  if (row.token == null) {
    throw new Error(`provider_key ${row.id}: no token available`);
  }
  return row.token;
}
```

**Why this is not a downgrade attack**:
- Path selection is by **column nullity**, not by content inspection
- Attacker with DB write access flipping a byte in `token_ciphertext` leaves the column non-null → strict path → GCM tag fail → throw
- Attacker can only reach the legacy path by setting `token_ciphertext = NULL` — but that requires write access to a security-sensitive column. If they have that, they can also just write whatever they want into `token`. There is no new attack surface introduced by the dual-mode read
- A new row written by this PR's code has `token_ciphertext` populated. There is no path in the new code that writes both `token` and `token_ciphertext` for the same row simultaneously

Read sites to migrate:

Plaintext-token reads (call `readProviderKey()` and pass the decrypted result to the upstream call):
- `apps/gateway/src/chat/chat.ts:3027`
- `apps/gateway/src/chat/chat.ts:3126` (hybrid fallback path)
- `apps/gateway/src/chat/tools/resolve-provider-context.ts:233`
- `apps/gateway/src/chat/tools/resolve-provider-context.ts:261`
- `apps/gateway/src/moderations/moderations.ts:397`
- `apps/gateway/src/moderations/moderations.ts:412` (hybrid second read)
- `apps/gateway/src/embeddings/embeddings.ts:539`
- `apps/gateway/src/embeddings/embeddings.ts:562` (hybrid second read)
- `apps/gateway/src/videos/videos.ts:1112`
- `apps/gateway/src/videos/videos.ts:1208`
- `apps/gateway/src/videos/videos.ts:2165`
- `apps/worker/src/services/video-jobs.ts:158`
- `apps/worker/src/services/video-jobs.ts:273`

Truthiness check (special case, **not** a plaintext read):
- `apps/gateway/src/chat/chat.ts:3223` — currently uses `providerKey.token` as a truthy test to decide free-vs-paid model behavior. After this PR, encrypted rows have `providerKey.token === null`, so the check must switch to `providerKey.token != null || providerKey.tokenCiphertext != null` (or a small `hasProviderKey(row)` helper). Do not call `readProviderKey()` here — no decryption needed for a truthy check, and we don't want to pay the decrypt cost or expose plaintext just to ask "does this row carry a key?"

That's 14 sites total. Before merge, re-verify with `rg "providerKey\.token|key\.token|updatedProviderKey\.token|ctx\.providerKey\?\.token|nextContext\.providerKey\?\.token" apps ee packages` and assert the result set matches this list. Any new hit must either become a `readProviderKey()` call or a `hasProviderKey()` truthy check — no exceptions.

**Cache invalidation** (`apps/gateway/src/lib/cached-queries.ts`, `packages/cache/src/swr.ts`):

Gateway reads use `cdb` + SWR. **API create/update writes use uncached `db`**, which means the existing Drizzle and SWR invalidation hooks **do not fire automatically** on a provider-key write. SWR mirrors can hold a stale row for up to `SWR_DEFAULT_TTL_SECONDS = 14400` (4 hours) if Postgres is briefly unavailable.

This was acceptable for plaintext (worst case: gateway reads a recently-updated key 4h late) but is unacceptable for rollover from plaintext to ciphertext, where the gateway would keep using the old plaintext path while the DB row is fully encrypted.

**Required** (must be in this PR, not deferred):
1. `apps/api/src/routes/keys-provider.ts` create / update / delete handlers must explicitly invalidate the gateway provider-key cache for the affected `(organizationId, providerId)` tuple after the DB write succeeds
2. Invalidation must cover both the Drizzle TTL cache and the SWR mirror — use whatever helper `swrWrap` exposes (e.g., `invalidateSwrByTables(["provider_key"])`) or write one if missing
3. Acceptance test: write a new key via API → without waiting, fetch via gateway path → must see new value, not stale

For new rows the cache cannot have a stale entry by definition. For legacy rows (`token_ciphertext = NULL` forever until owner updates) the cache holds a legacy row with the legacy code path, which is correct. The race only exists at rollover — fix it once, here.

### API response handling

Current API code uses `...row` spread then `token: undefined` to suppress the field. After adding `tokenCiphertext`, the spread will leak the new column. Relying on `.omit({ token: true })` against the row schema is fragile — any future secret-bearing column added to `provider_key` (rotation kid columns, etc.) would silently leak until someone remembers to omit it too.

**Fix**: define a single dedicated **public response schema** for provider-key endpoints and have every handler return that shape, instead of spreading `row` and subtracting fields.

```ts
// apps/api/src/routes/keys-provider.ts (or a colocated _response-schemas.ts)
const providerKeyPublicSchema = z.object({
  id: z.string(),
  organizationId: z.string(),
  providerId: z.string(),
  status: z.string(),
  maskedToken: z.string(),
  createdAt: z.date(),
  updatedAt: z.date(),
  // ...any other explicitly-safe fields
});

function toPublicProviderKey(row: ProviderKey): z.infer<typeof providerKeyPublicSchema> {
  return {
    id: row.id,
    organizationId: row.organizationId,
    providerId: row.providerId,
    status: row.status,
    maskedToken: row.tokenMasked ?? maskToken(row.token ?? ""),
    createdAt: row.createdAt,
    updatedAt: row.updatedAt,
  };
}
```

Apply at all three sites:
- `apps/api/src/routes/keys-provider.ts:285` (create response)
- `apps/api/src/routes/keys-provider.ts:344` (list response)
- `apps/api/src/routes/keys-provider.ts:621` (update response)

The list-active route already uses explicit columns and is fine — verify before merge, don't break it.

`maskedToken` falls back to computing from `row.token` for legacy rows (which have `token_masked = NULL`). This is the only place legacy plaintext is touched in the API, and only to derive a mask — never returned raw.

---

### Logging redaction

Hard requirements (these are new tests, not just policy):
- `packages/actions/src/validate-provider-key.ts:268` (`log upstream errorMessage`) — upstream providers sometimes echo the submitted key in their error body. Redact any substring equal to the plaintext token from `errorMessage` before logging
- `packages/actions/src/validate-provider-key.ts:299` (exception path) — exception `.message` and `.stack` can carry the token if the fetch wrapper formats it in. Wrap so the catch block sees only `validation failed for provider <providerId>` with no token in scope. If the underlying error has a `.cause`, strip or sanitize it
- `apps/api/src/routes/keys-provider.ts:235` (`validationResult.error` forwarded to HTTP 400 response **and** to logs) — same `errorMessage` flows through. Run the same redaction before either logs or the client response body, so a malicious upstream can't echo the user's own key back to them via a 400
- Error responses to clients: never include the raw token, even on validation failure. Add explicit integration test that submits a key, forces validation failure, asserts the response body and logs contain the masked form (or no token reference at all), never the plaintext
- Add a single `redactToken(text, plaintextToken)` helper in `packages/actions/src/provider-key/` — any code path that touches both the plaintext and an error string must pass through it
- Sentry / Datadog / external observability: out of scope to *configure*, in scope to *not feed* the token to any logger. The redaction helper above is the choke point

---

### Out of scope (explicit)

- Branding, copy, nav changes
- Public fork repo + source links
- Key rotation procedures (kid `v1` is in the format; v2 rollout is a follow-up spec — single-version for now)
- Encrypting other secret fields (HKDF info string makes adding them trivial later)
- Backfill of legacy rows
- Payments
- **Adversary with DB write access**: an attacker who can `UPDATE provider_key SET token_ciphertext = NULL, token = 'attacker-controlled'` can force the legacy path on a previously-encrypted row, causing the gateway to forward an attacker-chosen key to upstream. This does not recover the original secret (the ciphertext is unreadable without the master key), but it can hijack traffic and burn the customer's upstream billing until detected. **Mitigation is DB access control + audit logging at the infra layer, not application code.** Calling this out so reviewers don't expect the dual-mode read to defend against arbitrary DB writes — it defends against *ciphertext tampering*, not against an attacker who can also touch the legacy column
- **Plaintext exposure outside the primary DB row**: rollover encrypts the row in the primary database, but replicas, WAL, point-in-time backups, Redis/SWR mirrors (until invalidated), and prior `pg_dump` snapshots may continue to hold the pre-encryption plaintext. Treat backup/replica retention as a separate cleanup task driven by ops policy

---

### Acceptance criteria

- [ ] Schema migration: `token` becomes nullable, `token_ciphertext` and `token_masked` columns added, CHECK constraint `provider_key_token_xor` enforces `token IS NULL OR token_ciphertext IS NULL`
- [ ] New rows (via UI create or API): `token = NULL`, `token_ciphertext` starts with `llmgw:v1:`, `token_masked` is populated. Verify with SQL after a create
- [ ] Updating a legacy row: the row in the primary DB has `token = NULL`, `token_ciphertext` populated, `token_masked` populated. (Replicas, WAL, backups, cache mirrors may retain prior plaintext — see "Out of scope".)
- [ ] Legacy rows untouched by this PR continue to work — gateway can call upstream successfully with them
- [ ] All 13 plaintext-token read sites use `readProviderKey()`; the 1 truthiness-check site (`chat.ts:3223`) uses `hasProviderKey()` or the equivalent inline check. Raw `.token` access on `providerKey` rows is gone (grep over `apps/` + `ee/` + `packages/` returns 0 hits outside the two helpers and outside tests)
- [ ] CHECK constraint test: attempting to write a row with both `token` and `token_ciphertext` non-null fails at the DB level
- [ ] API responses never include `token` or `tokenCiphertext` (integration test asserts the response JSON shape against the public schema)
- [ ] Cross-org AAD: ciphertext from org A row pasted into org B row fails decryption (test)
- [ ] Tampered ciphertext (flip 1 byte in iv / ct / tag) fails decryption with `throw` — does NOT fall back to legacy `token` even if the legacy column is populated (critical test for the dual-mode read)
- [ ] Wrong master key: gateway throws on encrypted-row reads; never falls back to `token` (test)
- [ ] `GATEWAY_API_KEY_HASH_SECRET` missing in production throws on the first encrypt/decrypt, with the same clear error already raised by the API-key hash path — no eager boot check, so non-production builds and CI need no configuration
- [ ] Random IVs: 1000 encryptions of the same plaintext produce 1000 distinct ciphertexts (test)
- [ ] Logging: validation failure for a fake provider key does not contain the plaintext token in any log line, error message, telemetry span, or HTTP 400 response body (test — including the case where the upstream provider echoes the submitted key in its error body)
- [ ] Cache invalidation: write a key via API → immediately fetch via gateway path → see new value, not stale. Test with both initial create (no prior cache entry) and update (rollover legacy → encrypted)
- [ ] `chat-reasoning.e2e.ts`, `chat-prompt-caching.e2e.ts`, `fallback.spec.ts` pass — gateway path works end-to-end with both encrypted (new) and plaintext (legacy) rows
- [ ] New e2e: create encrypted key → list shows masked → chat call succeeds → moderations call succeeds → embeddings call succeeds → videos call succeeds → worker video job succeeds

---

### Unit test list

- `encrypt(pt, aad) → decrypt(ct, aad) === pt` round-trip
- Tampered ciphertext byte → throws (GCM tag)
- Tampered AAD (wrong org id, wrong row id) → throws
- Tampered prefix (`llmgw:v2:...`, missing colon, `bare-plaintext`) → throws with distinguishable error
- IV uniqueness: 1000 encryptions → 1000 distinct IVs and distinct ciphertexts
- HKDF: same master + same info → deterministic data key; different info → different data key (proves future OAuth-token encryption can't cross-decrypt provider-key ciphertext)
- `readProviderKey({ tokenCiphertext: null, token: "sk-..." })` → returns plaintext (legacy path)
- `readProviderKey({ tokenCiphertext: <enc>, token: "sk-..." })` → ignores `token`, decrypts ciphertext (no downgrade)
- `readProviderKey({ tokenCiphertext: <tampered>, token: "sk-..." })` → throws (no fallback to plaintext, even though `token` is non-null)
- `readProviderKey({ tokenCiphertext: null, token: null })` → throws (no token at all)
- `hasProviderKey({ tokenCiphertext: null, token: null })` → false; with either column non-null → true (powers the truthiness check at `chat.ts:3223` without needing to decrypt)
- `maskToken` behavior identical between `tokenMasked` (stored at write) and computed-from-`token` (legacy fallback) for the same input
- Update path: writing a new ciphertext sets `token = NULL` in the same UPDATE; CHECK constraint allows the result
- `redactToken(text, plaintextToken)`: substring removal works for full token, partial token (last N chars), and is a no-op when token not present
- Missing secret in production: `encryptProviderKey` / `decryptProviderKey` throw `GATEWAY_API_KEY_HASH_SECRET is required in production`; outside production the shared dev secret is used and both round-trip

---

### Open questions to confirm before coding

1. **ID generation**: switch the `provider_key` insert to generate `id = shortid()` in application code (so AAD can include `id` pre-insert). Vote: yes — same lib already imported, ~2 lines
2. **Secret manager for prod**: which one? Document in PR description, even if it's "we used X today, formal policy TBD"
3. **Secret rotation procedure**: not built this PR, but write a 5-line runbook section in the PR description explaining how a future operator would do it (e.g., "deploy v2 keyring code, set both `_V1` and `_V2` env vars, re-encrypt by writing back through API on next request, retire `_V1`"). Note that rotating `GATEWAY_API_KEY_HASH_SECRET` now also invalidates provider-key ciphertexts, not just API-key fingerprints
4. **Test for "legacy plaintext rows still work"**: requires test fixture with a pre-seeded row that has `token = "sk-..."`, `token_ciphertext = NULL`. Confirm the e2e seed script supports this shape before merge

---

### Estimate

2.5 days. Breakdown:
- Crypto helper (encrypt / decrypt / HKDF / AAD / `llmgw:v1:` format) + `readProviderKey` + `hasProviderKey` + `redactToken` in `packages/actions`: 0.5 day
- Schema migration (DROP NOT NULL + 2 new columns + CHECK constraint): 0.25 day
- Write path (validate → app-generated id → encrypt → insert single statement) + cache invalidation on create/update/delete: 0.5 day
- 13 plaintext-read sites + 1 truthiness site migrated: 0.5 day
- Public response schema + 3 route handlers: 0.25 day
- Tests (unit + e2e + log redaction + CHECK constraint + cache invalidation): 0.5 day
- Total: 2.5 days focused. Pad to 3 if open questions surface unknowns

---

### Anticipated review objections + responses

Expected reviewer pushback and our responses:

- **"Forward-only doesn't solve DB-dump leak for existing rows"** — Acknowledged in "Accepted residual risk" section. This is the deliberate fork-compatibility tradeoff. Not a spec defect
- **"`token` column nullable could let new code accidentally write NULL plaintext + NULL ciphertext"** — Defended at two layers: (1) CHECK constraint allows both-null but `readProviderKey` throws on it; (2) write path always writes exactly one. Tested
- **"DB-writer can clear `token_ciphertext` and inject `token` to hijack traffic"** — True, listed in "Out of scope". This PR defends against ciphertext *tampering* (GCM tag fails), not against an attacker who can write any column they want. That mitigation lives in DB ACL / audit policy, not in this code change
- **"Cache won't be invalidated on rollover because API uses uncached `db`"** — Correct, handled explicitly: API create/update/delete invalidate the SWR + Drizzle cache for the affected `(organizationId, providerId)` after the DB write. Acceptance test covers write-then-immediate-read race
- **"Insert-then-update window for id-then-encrypt"** — Removed by generating `id` in application code before insert. Single INSERT, no window
- **"`packages/shared` would create cycles or ship Node crypto to frontend"** — Helper lives in `packages/actions` (server-only, already depends on db). Spec was updated
- **"Read-site list missed sites"** — Verified by re-running grep against the spec's listed line numbers. List is now 13 plaintext reads + 1 truthiness check = 14 total. Pre-merge check command included in spec
- **"Rotation isn't designed"** — Out of scope; `kid=v1` in the format makes a future rotation purely additive (new env var, keyring decrypt map, no schema change). Runbook stub in PR description
