RCA: perf-regression-predefined-throughput-steps-i8g-tablets failure (Apr 26) — transient read admission queue saturation

## Root Cause Analysis: 3-Run Comparison at 1,200,000 and 1,500,000 op/s Steps **Test:** scylla-enterprise-perf-regression-predefined-throughput-steps-i8g-tablets **Steps:** 1,200,000 op/s and 1,500,000 op/s **Instance Type:** i8g.4xlarge (3 nodes) **SCT Branch:** origin/branch-perf-v17 ## Runs Compared | Property | Failed Run (Apr 26) | Passed Run # 1 (Apr 28) | Passed Run # 2 (May 3) | |----------|--------------------|-----------------------|-----------------------| | **Run ID** | 5a958af9 | 70b4ef85 | 2fe61093 | | **Status** | FAILED | passed (with errors) | **PASSED** | | **Region** | us-east-1 | eu-west-1 | us-east-1 | | **Scylla Version** | 2026.2.0~dev | 2026.2.0~dev | 2026.3.0~dev | | **SCT SCM** | fac174555371 | 12656f0c1d17 | 12656f0c1d17 | | **AMI** | ami-0a1a27078c0e085a9 | ami-0a65fb0959e3517af | ami-01b0b632ab39b099d | | **Jenkins Build** | [#13](https://argus.scylladb.com/tests/scylla-cluster-tests/5a958af9-8c9b-4b22-aa58-a347c1a0f70c) | [#17](https://argus.scylladb.com/tests/scylla-cluster-tests/70b4ef85-b3aa-4a1b-bc90-846bf2be4f0d) | [#18](https://argus.scylladb.com/tests/scylla-cluster-tests/2fe61093-af40-42a5-8138-0613a9b8ba8f) | | **Events** | ERROR: 2 | ERROR: 1 | ERROR: 0 | ## Key Observation The **new passed run (# 18) is in us-east-1** with the same SCT SCM revision as the Apr 28 eu-west-1 run — and it **passed cleanly** with 0 errors. This eliminates "us-east-1 infrastructure" as the root cause and confirms the issue in the failed run was **transient/environmental**. ## HDR Histogram Summary Comparison (Step 1,200,000) | Metric | Failed (Apr 26) | Passed # 1 (Apr 28) | Passed # 2 (May 3) | |--------|-----------------|--------------------|--------------------| | **P95 (ms)** | 67.9 | -- | 1.10 | | **P99 (ms)** | **1754.27** | 5.27 | **4.86** | | **Throughput (op/s)** | 1,199,990 | 1,199,461 | 1,199,378 | ## Per-Interval HDR Breakdown (Step 1,200,000) ### Failed Run (Apr 26) | Interval | P95 | P99 | |----------|-----|-----| | 1 (05:54-06:04) | **1022.36** | **2671.77** | | 2 (06:04-06:14) | 1.20 | 5.88 | | 3 (06:14-06:24) | 1.45 | **192.94** | ### Passed Run # 2 (May 3) | Interval | P95 | P99 | |----------|-----|-----| | 1 (05:48-05:58) | 1.10 | 4.58 | | 2 (05:58-06:08) | 1.09 | 3.96 | | 3 (06:08-06:18) | 1.12 | 8.99 | ## Step 1,500,000 op/s — HDR Summary | Metric | Failed (Apr 26) | Passed # 1 (Apr 28) | Passed # 2 (May 3) | |--------|-----------------|--------------------|--------------------| | **P95 (ms)** | **20,535** | 7,353 | 1.71 | | **P99 (ms)** | **23,320** | 10,721 | 6.30 | ### Queued Reads (max during step 1,500,000) | Node | Failed (Apr 26) | Passed # 1 (Apr 28) | |------|-----------------|-------------------| | node-1 | **1,008** | 16 | | node-2 | 12 | **1,074** | | node-3 | **1,127** | 18 | ## Root Cause Deep Dive: The Mechanism 1. **Memory is at 99% capacity** (8.10 / 8.17 GB per shard). The nodes have essentially no headroom. 2. **Read concurrency admission control** kicks in: Scylla has an internal limit on concurrent active reads per shard (`scylla_database_active_reads`). When a node hits this limit, additional reads get **queued** (`scylla_database_queued_reads`). 3. On the **failing nodes**, active reads peak at 93-94 (hitting the internal concurrency limit), causing 1000+ reads to queue. On healthy nodes, active reads stay at 3-22. 4. The **Scylla P99 coordinator latency** (12-13ms reported) only measures time spent *processing* the read — it does NOT include time spent waiting in the admission queue. So Scylla reports "I processed it in 12ms" but the request actually waited 20+ seconds in the queue first. 5. **Cache hit rate is 100%** — this is NOT a cold-cache/disk-IO problem. The reads hit the cache, but some slow down enough (likely due to memory allocation pressure from evictions/re-caching) to hold their admission slot longer, creating a cascading queue buildup. ### Why Only Some Nodes? The failed run shows **uneven load distribution** — node-1 and node-3 are overwhelmed while node-2 is fine. This points to **uneven tablet ownership** or a topology-aware routing issue where certain nodes got more of the keyrange being stressed. ### Why Transient? - Same region, same instance type passes cleanly on May 3 - Likely caused by: slightly different tablet distribution after initial data loading, or a specific EC2 instance with marginally slower memory/NVMe performance that tips the system over the admission control threshold under full load ### Bottleneck Layer Summary | Layer | Problem? | Evidence | |-------|----------|----------| | Client (cassandra-stress) | No | It correctly reports the latency it observes | | Network | No | No transport blocking | | Scylla transport/CQL layer | No | 0 requests blocked by memory | | **Scylla read admission control** | **YES** | 1000+ queued reads, active reads at limit (93-94) | | Scylla cache/memory | Contributing | 99% memory used, evictions occurring, no headroom | | Disk IO | No | 100% cache hit rate | ## Conclusion The root cause is **Scylla's per-node read admission queue saturating** due to memory pressure leaving no headroom for the read concurrency limit to stay below threshold. It's an instance-level (server-side) issue, not a client problem. The failed run appears to be a **one-off infrastructure fluke** — a single node experienced severe read queuing due to memory pressure. The same test configuration passes reliably in the same region with the same (or newer) code. No systemic issue requiring code changes has been identified. ## Recommendation - **Mark the failed run as "not a product issue"** — the regression was transient and not reproducible. - **If recurrence is suspected**, add monitoring for `scylla_database_queued_reads > 100` as an alert during perf runs to detect single-node issues in real-time. - **Consider adding a retry mechanism** for perf steps that fail only in the first interval — a brief warmup or retry could avoid false failures from transient infrastructure events.