# Architecture & Game Design

> AI provider integration, token management, and context compaction are
> documented in [AI_RUNTIME.md](AI_RUNTIME.md).
>
> Choosing a public **DNS name** for the website (criteria, shortlist, checklist)
> is covered in [DNS_REGISTRATION.md](DNS_REGISTRATION.md).

## Tech Stack

- **Framework:** Next.js 16 (App Router, TypeScript)
- **API Backend:** Node.js + Express
- **Styling:** Tailwind CSS v4
- **State:** Zustand
- **AI Providers:** Vertex AI (Anthropic SDK) + Azure OpenAI/Foundry (chat completions REST)
- **AI Runtime:** Provider-agnostic adapter in `backend/src/lib/ai-runtime.ts`
- **NLP:** compromise (sentence-level fact/hypothesis extraction for context compaction)
- **Tokenizer:** gpt-tokenizer (o200k_base BPE, used for compaction budget estimation)
- **Markdown:** react-markdown + remark-gfm
- **Icons:** Lucide React

## Gameplay Platform Boundary

`PlatformId` is a gameplay dimension stored on each simulated session. Supported
gameplay platforms are:

- `aro-classic`
- `aro-hcp`
- `aks`

This is intentionally separate from deployment concerns like
`CLUSTER_FLAVOR` or `PROD_CLUSTER_FLAVOR`. The simulator can run on AKS while
presenting an ARO Classic or ARO HCP session, and no runtime path should infer
gameplay platform from host cluster flavor.

---

## Project Structure

```text
SRESimulator/
├── CLAUDE.md                             # Design document and game spec
├── README.md
├── Makefile                              # Build, lint, dev, and CI targets
├── docker-compose.yml                    # Azure SQL Edge for local MSSQL testing
├── helm/sre-simulator/                   # AKS/ARO deployment manifests
├── knowledge_base/                       # Reference docs loaded into AI context
│   ├── sre-investigation-techniques.md
│   ├── Openshift-clusters-alerts-resolutions.md # ARO Classic only
│   ├── Community-reported-issues.md             # ARO Classic only
│   └── platforms/                        # Platform-specific gameplay bundles
│       ├── aro-classic/
│       ├── aro-hcp/
│       └── aks/
├── docs/
│   ├── ARCHITECTURE.md                   # This file
│   ├── AI_RUNTIME.md                     # AI provider, compaction, token management
│   ├── CONTENT_BOUNDARY.md               # Repo-owned runtime content contract
│   └── DNS_REGISTRATION.md               # Public site DNS name criteria & checklist
├── frontend/                             # Next.js application (UI only)
│   ├── src/
│   │   ├── app/
│   │   │   ├── page.tsx                  # Landing page (platform + difficulty selection)
│   │   │   ├── game/page.tsx             # Main game page
│   │   │   ├── leaderboard/page.tsx      # Hall of fame view
│   │   │   └── api/[...path]/route.ts    # Internal BFF proxy to backend service
│   │   ├── components/
│   │   │   ├── chat/                     # Chat panel, messages, input
│   │   │   ├── terminal/                 # Terminal output display
│   │   │   ├── dashboard/                # Cluster overview and alerts
│   │   │   ├── scoring/                  # Phase tracker, score breakdown
│   │   │   ├── layout/                   # Game layout, header, right panel
│   │   │   └── shared/                   # Code blocks, incident ticket
│   │   ├── hooks/                        # useChat, useCommand, useScoring
│   │   ├── stores/                       # Zustand game state
│   │   ├── lib/                          # API client helpers
│   │   └── types/                        # TypeScript type definitions
│   └── .env.local                        # Preferred local env file (Makefile falls back to .env)
└── backend/                              # Express API server
    └── src/
        ├── index.ts                      # API wiring and middleware
        ├── routes/                       # /api/chat, /api/command, /api/scenario, /api/scores, /api/ai
        └── lib/
            ├── ai-runtime.ts             # Provider-agnostic AI adapter (Vertex + Azure)
            ├── ai-config.ts              # Provider detection, readiness checks
            ├── context-compactor.ts       # Chat history compaction with retained-state
            ├── nlp-extract.ts            # Compromise-based fact/hypothesis extraction
            ├── profanity.ts              # Nickname profanity filter (static blocklist)
            ├── token-logger.ts            # Per-route token usage observability
            ├── rate-limit.ts             # Per-IP rate limiting for AI routes
            ├── knowledge.ts              # Knowledge base file loader
            ├── prompts/system.ts         # System prompt builder
            └── storage/                  # Pluggable persistence layer
                ├── types.ts              # ISessionStore, ILeaderboardStore, IMetricsStore
                ├── index.ts              # Factory (selects JSON or Azure SQL)
                ├── migrate.ts            # T-SQL migration runner
                ├── json-*-store.ts       # JSON/in-memory implementations
                ├── mssql-*-store.ts      # Azure SQL (T-SQL) implementations
                └── migrations/           # Numbered .sql migration files
```

---

## Final Infra Baseline (AKS default + Azure SQL)

For the final production-infra rehearsal, provision a dedicated environment
through `infra/` with:

- `owner_alias = "aaffinit"` so main Azure resources follow
  `aaffinit-test-*` naming (`aaffinit-test-rg`, `aaffinit-test` cluster).
- `cluster_flavor = "aks"` so new environments follow the lower-cost AKS path
  by default while keeping the ARO fallback available.
- Isolated Terraform state key (for example
  `aaffinit-test-sre-simulator.tfstate`) to avoid side effects on other stacks.
- Explicit `extra_tags.test = "true"` in addition to mandatory infra tags.
- Minimum-cost AKS sizing (`Standard_B2s`, autoscaled from `1` to `3` nodes).
- Static frontend public IP kept in the main resource group so the public
  endpoint stays alongside the other customer-managed resources.
- Acceptance of the AKS-managed node resource group as the one planned
  resource-group exception; Azure creates and owns that RG automatically.
- `enable_database = true` for Azure SQL-backed persistence.

Preflight gate:

```bash
make tf-preflight \
  OWNER_ALIAS=aaffinit \
  CLUSTER_FLAVOR=aks \
  TF_STATE_ACCOUNT=<state-account> \
  LOCATION=westeurope \
  TF_STATE_KEY=aaffinit-test-sre-simulator.tfstate \
  SQL_SERVER_NAME=aaffinit-test-sql-20260403
```

AKS preflight does not require Geneva confirmation. If you use the ARO
fallback path instead, add `GENEVA_SUPPRESSION_ACCESS_CONFIRMED=true` after
validating Geneva access.

Then initialize state safely:

```bash
make tf-init-isolated OWNER_ALIAS=aaffinit CLUSTER_FLAVOR=aks
```

For first-time runs, preflight prompts to create the Terraform backend when
`TF_STATE_ACCOUNT` is missing or not found, and writes backend defaults to
`infra/.tf-backend.env` for subsequent targets.

---

## Cluster Exposure Model

Short answer: **only the frontend is internet-exposed; backend stays private
inside the cluster**.

### What is exposed

- On the AKS default path, Envoy Gateway publishes a namespaced `Gateway` plus
  two `HTTPRoute` objects (HTTP redirect + HTTPS app route) on the existing
  static public IP.
- The AKS frontend `Service` stays `ClusterIP` in Gateway mode; the public edge
  lives on Envoy, not on the app `Service` itself.
- cert-manager owns the `sre-simulator-gateway-tls` secret via the
  `letsencrypt-azuredns-prod` `ClusterIssuer`, so HTTPS for
  `play.sresimulator.osadev.cloud` is cluster-managed.
- The Azure-generated hostname tied to the static public IP remains an
  operator-only fallback for DNS troubleshooting or propagation delays; the
  customer-facing URL is `https://play.sresimulator.osadev.cloud`.
- On the ARO fallback path, a single OpenShift `Route` serves the same role.
- `publicService` mode remains the explicit AKS rollback path when operators
  need to promote only the frontend back to a public `LoadBalancer`.
- There is **no** backend Route, Ingress, Gateway listener, or public
  `LoadBalancer` service (`/api` is not published directly).
- The backend remains a private `ClusterIP` service reachable only from inside the namespace network.

### How the internal proxy works

- Browser still calls same-origin paths like `/api/chat`.
- Those requests reach the frontend origin first.
- In AKS Gateway mode, Envoy terminates TLS and the HTTPS `HTTPRoute` forwards
  traffic to the frontend `ClusterIP` service.
- The `/api/scenario` rule has an explicit 30-second edge timeout. This leaves
  headroom for anonymous verification and session persistence around the
  backend's 12-second AI timeout and catalog fallback; other paths retain the
  Envoy default.
- Frontend route handler (`app/api/[...path]/route.ts`) proxies server-to-server to `http://<release>-backend:<port>`.
- Backend `NetworkPolicy` only allows ingress from frontend Pods on backend port.

### Request flow in cluster

1. User opens `https://play.sresimulator.osadev.cloud/`.
2. Envoy Gateway accepts the request on the static public IP and terminates TLS.
3. The `HTTPRoute` forwards the request to the frontend `ClusterIP` service.
4. Frontend calls `fetch("/api/...")`.
5. Frontend pod proxies the request internally to backend `ClusterIP`.
6. Backend responds to frontend pod; frontend returns response to client.

### Security outcome

- Backend is not directly reachable from the internet.
- External traffic terminates at the Gateway-managed frontend edge only.
- Frontend and backend `Service` objects both stay private inside the namespace.
- Backend remains isolated with least-privilege pod-to-pod access.

Runtime audit command:

```bash
make public-exposure-audit NS=sre-simulator
```

---

## Manual E2E with Azure SQL (optional)

The default manual E2E flow (`make e2e-azure-route-up`, which now targets AKS
unless `CLUSTER_FLAVOR=aro` is set) deploys without cluster database
credentials. To exercise **Helm database mode** in a throwaway E2E namespace,
reuse an existing Kubernetes `Secret` that already holds the ADO.NET
connection string (Helm default key: `connection-string` on
`database.secretConnectionStringKey`).

Prerequisites:

1. A source namespace containing the canonical DB secret (for example the
   stable production namespace `sre-simulator` or a dedicated shared namespace).
2. RBAC allowing the deployer identity to **read** the secret in the source
   namespace and **create/update** secrets in the target E2E namespace.
3. `jq` must be installed on the machine running the Make target because the
   secret-copy helper rewrites Kubernetes metadata client-side before
   `kubectl apply` or `oc apply`.
4. In `backend/.env.local` (preferred), `backend/.env` (fallback), or the
   environment, set `DB_SECRET_NAME` to the secret resource name. Optionally
   set `DB_SECRET_SOURCE_NAMESPACE` to the namespace that already holds that
   secret; if omitted, the automation copies from `PROD_NAMESPACE` (Makefile
   default `sre-simulator`).

Before `helm upgrade`, the selected deploy helper (`scripts/aks-deploy.sh` or
`scripts/aro-deploy.sh` via `scripts/select-deploy.sh`) copies the secret into
the new or refreshed E2E namespace using the active cluster CLI and `jq` only
(secret values are not printed). `make e2e-azure-route-up` and
`make e2e-azure-route-refresh` both run this step when `DB_SECRET_NAME` is set.
When it is unset, behavior matches the previous non-DB E2E path.

---

## Investigation Methodology

The game enforces the SRE "Scientific Method of Investigation":

| Phase | What to do | Example prompt from DM |
| --------------------- | ---------------------------------- | ------------------------------------ |
| **Reading** | Read the incident ticket carefully | _"What inconsistencies do you see?"_ |
| **Context Gathering** | Check dashboards, cluster history | _"Have you checked Geneva first?"_ |
| **Facts Gathering** | Run commands, collect evidence | `oc get nodes`, KQL queries |
| **Theory Building** | Form a hypothesis | _"What do you think is happening?"_ |
| **Action** | Apply the fix (safely) | _"Is this reversible?"_ |

The AI Dungeon Master enforces phase ordering and pushes back if you try to skip ahead.

---

## Scoring System

You start at **0/100** and earn points through good investigation practices.

### Dimensions

| Dimension | Max | What earns points | What loses points |
| ------------- | --- | --------------------------------------------- | ------------------------------------ |
| Efficiency | 25 | Focused, targeted investigation | Excessive or irrelevant commands |
| Safety | 25 | Checking dashboards first, suggesting backups | Running commands without context |
| Documentation | 25 | Following phases in order, thorough analysis | Skipping phases, jumping to action |
| Accuracy | 25 | Correct hypotheses, proper root cause | Wrong theories, misidentified causes |

### How scoring works

**AI-driven scoring:** The Dungeon Master evaluates your approach in real-time and awards/deducts points after each interaction using `[SCORE:dimension:+/-points:reason]` markers embedded in responses. These are parsed by the frontend and hidden from the user.

**Automatic scoring:** The system also tracks behavior directly:

- Opening the Dashboard tab before running commands → safety bonus
- Running commands without checking the dashboard → safety penalty
- Running commands during the Reading phase → documentation penalty
- Exceeding 5 commands → progressive efficiency penalties
- Command rate limit: each suggested command can only be run once (button shows "Ran" after execution)

### Grades

| Grade | Score |
| ----- | ----- |
| A | 90+ |
| B | 80+ |
| C | 70+ |
| D | 60+ |
| F | < 60 |

---

## AI Integration

The backend integrates with LLM providers for chat, command simulation,
and scenario generation. See **[AI_RUNTIME.md](AI_RUNTIME.md)** for full
details on provider abstraction, context compaction, token management,
per-route deployments, and environment variable reference.

Key design highlights:

- **Dual-provider support** — Vertex AI (Claude) and Azure OpenAI behind a
  single adapter, switchable via `AI_PROVIDER` env var.
- **Hybrid context compaction** — regex for structured markers + NLP
  (compromise library) for fact/hypothesis extraction, with BPE token
  estimation for accurate budget tracking.
- **Reasoning-model compatibility** — automatic o-series detection skips
  unsupported parameters and manages `reasoning_effort`.
- **Section-based KB retrieval** — only scenario-relevant knowledge base
  sections are sent (~10K tokens vs ~32K full KB).
- **Prompt caching** — static instruction block first in system prompt to
  maximize Azure OpenAI cache hits across players.
- **Per-route deployments** — separate Azure OpenAI deployments per route
  for independent rate-limit pools.
- **Graceful degradation** — retries with reduced reasoning effort, then
  falls back to mock output if the model exhausts its budget.

---

## Backend API Routes

In cluster deployments, browser requests hit the frontend at `/api/*`; the
frontend BFF proxy forwards them internally to this backend service.

### Frontend auth routes

GitHub sign-in is terminated in the Next.js app, not directly in Express. The
frontend owns:

- `GET /api/auth/github/login` — starts the GitHub OAuth redirect flow
- `GET /api/auth/github/callback` — exchanges the code, fetches the GitHub
  profile, and sets a signed HTTP-only viewer session cookie
- `GET /api/auth/session` — returns the current viewer plus access policy for
  the landing page
- `POST /api/auth/logout` — clears the viewer session cookie

Express reads the signed viewer session cookie on proxied requests and treats it
as the source of truth for GitHub-authenticated access.

### `POST /api/scenario`

Generates a scenario for the given platform and difficulty. Access is now gated by player
identity:

- **GitHub-authenticated players** can request `easy`, `medium`, and `hard`
- **Anonymous players** can request `easy` only, and only after passing
  Cloudflare Turnstile plus the browser fingerprint/IP anti-abuse check
- Anonymous Easy mode is limited to one run per 24 hours via a salted server-side
  claim key derived from browser fingerprint, IP, and user-agent signals

After access is approved, the route binds the session to a `PlatformId`, then
either selects a repo-owned scenario from `scenarios/<platform>/<difficulty>/`
or calls the configured AI provider with the shared methodology and the
platform-scoped knowledge file set to produce a realistic incident ticket and cluster
context. In `AI_MOCK_MODE=true`, it returns a deterministic platform-aware mock
scenario.

### `POST /api/chat`

Streaming chat endpoint. Builds a system prompt with Dungeon Master persona, methodology enforcement, active scenario, and knowledge base. Chat history is automatically compacted when token estimates exceed the budget. Returns SSE stream from the configured provider (or a mock stream in mock mode).

### `POST /api/command`

Simulates command execution. Given an `oc`, `kubectl`, KQL, or legacy Geneva
command alias and the current scenario, the configured provider generates
realistic output consistent with the incident. The stored session platform is
the source of truth for which cluster CLI is allowed. In `AI_MOCK_MODE=true`,
the route returns platform-aware mock command output. Falls back to mock output
when reasoning models exhaust the completion budget.

### `GET /api/ai/readiness`

Returns AI runtime readiness checks (safe diagnostics only, no secrets).

### `GET /api/ai/probe?live=true`

Performs an active live probe against the configured provider to validate in-cluster connectivity end-to-end.

### `GET /api/ai/token-metrics`

Returns per-route token usage totals and recent request entries. See [AI_RUNTIME.md — Token Observability](AI_RUNTIME.md#token-observability).

---

## Persistence & Storage Backends

The backend supports two storage modes, selected via the `STORAGE_BACKEND`
environment variable (`json` or `mssql`). `json` is the local/test fallback;
Azure SQL (`mssql`) is the intended deployed and production storage path.

### JSON mode (default for local/test)

Best for local development and test runs only.

- **Sessions**: In-memory `Map` with 24h TTL, now tagged with identity kind
  (`github` vs `anonymous`), selected gameplay platform, and score persistence
  eligibility. Lost on pod restart.
- **Players**: JSON file (`data/players.json`) storing GitHub player metadata
  used to normalize persistent identities locally.
- **Anonymous trials**: JSON file (`data/anonymous-trial-claims.json`) storing
  salted daily claim keys and expiry timestamps for guest-rate limiting.
- **Leaderboard**: JSON file on PVC (`data/leaderboard.json`). Writes are
  serialized through an in-process async mutex and keyed by GitHub user id per
  platform + difficulty.
- **Metrics**: Log-only (no persistent storage).

Constraints:

- Single backend replica only (PVC is `ReadWriteOnce`, sessions are in-process).
- No data survives pod restarts (except leaderboard file on PVC).
- Not a supported deployed or in-cluster runtime mode.
- `initStorage()` now refuses to start with `STORAGE_BACKEND=json` when either
  `NODE_ENV=production` or `KUBERNETES_SERVICE_HOST` is present.

### Azure SQL mode

Required for multi-replica deployments and the intended production path. Uses Azure SQL
Database free tier (100K vCore-seconds/month, 32 GB storage, $0/month).

- **Sessions**: Stored in `sessions` table. Shared across replicas.
  Stale entries (>24h) are cleaned up opportunistically. Session rows now record
  identity kind, GitHub identity, anonymous claim key, selected gameplay
  platform, and whether the run is eligible for persistent score submission.
- **Players**: Stored in `players` and upserted by GitHub user id so nickname
  changes do not create duplicate persistent identities.
- **Anonymous trials**: Stored in `anonymous_trial_claims`, keyed by salted
  claim digest with 24h expiry for free-trial enforcement.
- **Leaderboard**: Stored in `leaderboard_entries` table. Uses `MERGE`
  to atomically keep the best score per (GitHub user id, platform, difficulty,
  traffic source). The stored callsign/nickname corresponds to the best
  recorded run for that entry, not necessarily the player's latest callsign
  change.
  Per-difficulty trim to 10 entries happens after each insert.
- **Metrics**: Stored in `gameplay_metrics` table. Captures per-session
  analytics (commands executed, scoring events, AI token consumption) together
  with platform-aware session slices and an open JSON `metadata` column for
  future extensibility.

To enable:

```bash
STORAGE_BACKEND=mssql
DATABASE_URL="Server=<fqdn>;Database=sresimulator;User Id=sresimadmin;Password=<pwd>;Encrypt=true"
```

Helm sets these automatically when `database.enabled=true`, and startup now
fails fast instead of silently falling back to JSON in production-like runtimes.

Migrations run automatically on startup using `sp_getapplock` for
cross-replica serialization. Schema is managed via numbered `.sql` files
in `backend/src/lib/storage/migrations/`.

### Why Azure SQL free tier

- **Zero cost**: 100K vCore-seconds/month + 32 GB included permanently.
- **Zero maintenance**: HA, backups, patching handled by Azure.
- **Auto-pause**: Database sleeps after 60 min idle; resumes on next request.
- **Point-in-time restore**: Built-in backup retention (7 days default).

When the monthly free allocation is exhausted the database auto-pauses
until the next billing cycle (configurable to continue with pay-as-you-go
instead).

### Local MSSQL testing with Azure SQL Edge

Azure SQL Edge is a free, ARM64/AMD64-compatible Docker image that is
wire-compatible with Azure SQL Database. It lets developers validate
T-SQL queries and migrations locally without an Azure subscription.

```bash
# Start SQL Edge and create the sresimulator database
make dev-db

# Run all integration tests against the real SQL engine
make test-mssql

# Stop the container
docker compose down
```

`make dev-db` starts the container, polls for TCP readiness, waits for
the SQL engine to accept queries, and creates the `sresimulator` database.
`make test-mssql` depends on `dev-db` and runs the full integration test
suite with `STORAGE_BACKEND=mssql`.

CI runs the same tests automatically via an `integration-test-mssql` job
that uses Azure SQL Edge as a GitHub Actions service container. The same job
includes `make smoke-backend-mssql` to verify backend startup and DB-backed
route behavior (`GET /api/scores`) with `STORAGE_BACKEND=mssql`.

**Note:** The Azure SQL Edge `latest` image does not ship `sqlcmd`. All
readiness checks use Node.js (`net` module for TCP, `mssql` package for
SQL queries). The container healthcheck uses Python 3 (bundled in the
image).

### Tier 3: Real Azure SQL free tier (manual E2E)

This is a manual smoke-test procedure that validates Terraform provisioning,
Helm wiring, network connectivity, and actual Azure SQL free-tier behavior
(auto-pause, resume latency). It requires an Azure subscription and a running
cluster selected through `CLUSTER_FLAVOR` (AKS by default, ARO as fallback)
and is **not** automated in CI.

#### Prerequisites

- AKS or ARO cluster provisioned via `terraform apply` (the base infra must
  already exist).
- CLI tools: `terraform`, `az`, `helm`, and the platform CLI (`kubectl` for
  AKS, `oc` for ARO).
- A strong password that meets
  [Azure SQL complexity requirements](https://learn.microsoft.com/en-us/sql/relational-databases/security/password-policy).

#### Step 1 — Provision the Azure SQL Database

```bash
cd infra
terraform apply -var enable_database=true \
                -var sql_admin_password='<strong-password>'
```

Terraform creates the SQL Server, the `sresimulator` database on the
GP_S_Gen5_2 serverless SKU, enables the free-tier offer via `azapi`, and
opens the "Allow Azure services" firewall rule.

#### Step 2 — Capture the connection string

```bash
terraform -chdir=infra output -raw sql_connection_hint
```

The output is a template: replace `<PASSWORD>` with the password you used
in step 1.

#### Step 3 — Create the K8s Secret

```bash
NS=sre-simulator   # or your target namespace
kubectl -n "$NS" create secret generic sre-sql-creds \
  --from-literal=connection-string="Server=<fqdn>;Database=sresimulator;User Id=sresimadmin;Password=<pwd>;Encrypt=true;TrustServerCertificate=false"
```

Use `oc` instead of `kubectl` when running the ARO fallback path.

The secret key must match `database.secretConnectionStringKey` in the Helm
values (default: `connection-string`).

#### Step 4 — Deploy with database enabled

```bash
DB_SECRET_NAME=sre-sql-creds CLUSTER_FLAVOR=aks make prod-up
# or for an ephemeral namespace:
DB_SECRET_NAME=sre-sql-creds CLUSTER_FLAVOR=aks make e2e-azure-route-up
```

`helm_deploy_sre()` detects `DB_SECRET_NAME` and passes
`--set database.enabled=true --set database.existingSecretName=sre-sql-creds`
to Helm. The backend pod starts with `STORAGE_BACKEND=mssql` and runs
migrations automatically on first connect.

For the guarded final deployment flow (Geneva suppression + exposure/DB checks):

```bash
DB_SECRET_NAME=sre-sql-creds \
GENEVA_SUPPRESSION_RULE_ACTIVE=true \
CLUSTER_FLAVOR=aks \
make prod-up-final
```

#### Step 5 — Validate persistence across pod restarts

1. Open the app in a browser and play a game to completion.
1. Submit a score and verify it appears on the leaderboard.
1. Kill the backend pod:

```bash
kubectl -n "$NS" delete pod -l app.kubernetes.io/component=backend
```

Use `oc` instead of `kubectl` when running the ARO fallback path.

1. Wait for the replacement pod to become ready, then reload the
   leaderboard page. The score must still be present.

#### Step 6 — Validate auto-pause and resume

The free-tier database auto-pauses after 60 minutes of inactivity.

1. Leave the app idle for at least 60 minutes (no requests).
1. Make a new request (e.g. load the leaderboard).
1. The first request will take approximately 30 seconds while Azure
   resumes the database. Subsequent requests should respond normally.

If GitHub pipelines are unavailable, first verify the deployed backend is wired
for Azure SQL mode:

```bash
make db-mode-check NS=sre-simulator
```

This checks that the backend deployment is configured with
`STORAGE_BACKEND=mssql` and a `DATABASE_URL` secret reference. Production and
final deploy flows already run this check, followed by `make db-port-forward-check`,
so the repo has both deployment-time and startup-time protection against
accidentally serving production traffic on the JSON backend.

Then use the selected cluster CLI's port-forward support as an additional
reachability smoke test:

```bash
make db-port-forward-check NS=sre-simulator
```

For read-only inspection of live Azure SQL contents, use the local helper first:

```bash
make db-inspect NS=sre-simulator
# quick player-only completion stats by difficulty
make db-admin-stats NS=sre-simulator
```

If your local machine cannot reach Azure SQL directly because of firewall
rules, run the same inspector inside the deployed backend pod instead:

```bash
make db-inspect-live NS=sre-simulator
# quick player-only completion stats by difficulty
make db-admin-stats-live NS=sre-simulator
# or with a custom read-only query:
NS=sre-simulator SQL="SELECT TOP 5 * FROM leaderboard_entries ORDER BY created_at DESC" make db-inspect-live
```

For custom queries, prefer single quotes around the outer `SQL=...` value when
your statement contains shell-sensitive characters.

The inspector retries transient SQL connection timeouts so Azure SQL serverless
resume latency does not fail the first read-only inspection attempt immediately.

#### Teardown

The database is on the free tier ($0/month), so the recommended default
is to leave it running. If you need to remove it, the SQL Server and
database are protected by `prevent_destroy` lifecycle rules. You must
remove those blocks before Terraform will allow deletion:

1. Edit `infra/sql-database.tf` and delete the `lifecycle { prevent_destroy = true }`
   blocks from both `azurerm_mssql_server.main` and `azurerm_mssql_database.app`.
1. Run `terraform apply -var enable_database=false` from the `infra/` directory.

### Storage interface pattern

All persistence is abstracted behind three interfaces (`ISessionStore`,
`ILeaderboardStore`, `IMetricsStore`) in `backend/src/lib/storage/types.ts`.
The factory in `backend/src/lib/storage/index.ts` selects the implementation
at startup based on `STORAGE_BACKEND`.

---

## Multiplayer Scaling & Resilience

### Concurrency model

The backend is stateless per-request for AI calls (chat, command, scenario).
All conversation state lives in the browser and is sent with each request.
This means multiple users can play independently against the same backend
without session affinity.

### Scaling by storage mode

| Concern | JSON mode | Azure SQL mode |
| --- | --- | --- |
| Backend replicas | **1 only** | **N (horizontal)** |
| Session survival | Lost on restart | Survives restarts |
| Leaderboard consistency | Single-writer mutex | Database MERGE |
| Sticky sessions needed | No | No |
| Connection pooling | N/A | `mssql.ConnectionPool` |

When using Azure SQL, enable backend HPA or increase the backend replica floor
in Helm values as needed. JSON mode is local/test only and is not a supported
deployed path; Azure SQL mode is the supported path for safe horizontal
scaling. No sticky sessions or session affinity are required since all state is
in the database.

### Infrastructure

- **Azure SQL Database free tier** ($0/month):
  Provisioned via `infra/sql-database.tf` when `enable_database = true`.
  Protected by `prevent_destroy` lifecycle rule.
  Serverless General Purpose (GP_S_Gen5_2) with auto-pause after 60 min idle.
  Free offer includes 100K vCore-seconds/month and 32 GB storage.

### Rate limiting & throttle handling

See [AI_RUNTIME.md — Rate Limiting](AI_RUNTIME.md#rate-limiting--throttle-handling) for per-IP rate limits, Azure 429 retry strategy, and AOAI capacity sizing guidelines.
