Mina Node Stuck in Bootstrap - Epoch Ledger Sync Fails After Num_accounts Query

## Summary A Mina plain node gets stuck indefinitely in bootstrap mode during epoch ledger synchronization. The node successfully initiates epoch ledger sync and receives `Num_accounts` responses from all peers, but never progresses to request additional ledger data needed to complete the synchronization. This has been confirmed to be a distinct failure mode from #18380. ## Environment - **Node Type**: Mina plain node (itn-standard-plain-1) - **Peer ID**: `12D3KooWQoNikmozrvEbE9me3PfpoJ4VzmP1tz3ZeUazYPfRDnX7` - **Network**: ITN testnet - **Log Timestamp**: 2026-03-25 07:24:00 – 09:30:00 UTC ## Background / Incident Context This issue was discovered during an ITN testnet incident on 2026-03-25. Several nodes (coordinator, plain-1, plain-3, plain-4, seed-1) were restarted by automated uptime management after running for over a week. Following restart, these nodes entered bootstrap and failed to recover, leaving the network SNARK-work starved. The coordinator restarted at 19:26 UTC on 2026-03-24 and entered a restart loop, unable to synchronize. Other nodes followed the same pattern. At the time of investigation, 5 of 10 nodes were in sync and 5 were stuck in bootstrap — including nodes that were connected to healthy peers, ruling out peer unavailability as the cause. ## Distinction from #18380 Issue #18380 describes a bootstrap failure where `download_best_tip` returns `None` due to zero peer responses, causing an early deadlock before epoch ledger sync begins. This failure diverges from that path: - The node **successfully retrieves best tips** from peers (multiple responses between 07:25:15–07:25:22) - The node **loads the frontier from disk** successfully at 07:26:16 - The node **enters epoch ledger sync** normally - Epoch ledger sync **receives valid `Num_accounts` responses** from all peers - Sync then **stalls mid-way** — not at startup The failure mode here is a mid-sync stall within the epoch ledger sync subsystem, entered via a working best tip and a disk-loaded frontier. The fix for #18380 will not cover this case. ## Observed Behavior ### What Works - ✅ Best tip download from peers (valid responses received) - ✅ Frontier loaded from disk - ✅ Epoch ledger sync initiated - ✅ `Num_accounts` queries issued and answered by all 10 peers - ✅ RPC connectivity confirmed (node visible in peer transaction gossip and rate limiting logs) ### What Fails - ❌ No `What_child_hashes` queries ever generated - ❌ No `What_contents` queries ever generated - ❌ No multi-item batch requests (`"n"` is always `1`) - ❌ No `handle_answer` or `handle_node` trace logs - ❌ Node loops on `Num_accounts` queries (~30 total across ~3 rounds) and never exits bootstrap ## Log Evidence ### Confirmed via grep All 30 downloader results in the log are `Num_accounts` — no other query type appears: ```bash $ grep '"result is $result' /tmp/itn-standard-plain-1-5fc4f48b85-ckvqn.prev.log \ | jq '.metadata.result.elts[0][1][0]' | sort | uniq -c 30 "Num_accounts" ``` No follow-up queries generated: ```bash $ grep -E "What_child_hashes|What_contents" \ /tmp/logs-2026-mar25/itn-standard-plain-1-5fc4f48b85-ckvqn.prev.log # (no output) ``` No batch requests (n ≥ 2) present: ```bash $ grep -E "downloading.*from.*peer.*n.*[2-9]" \ /tmp/logs-2026-mar25/itn-standard-plain-1-5fc4f48b85-ckvqn.prev.log \ | grep -v "Num_accounts" | wc -l 0 ``` ### Sample log entries Successful `Num_accounts` query (pattern repeated ~30 times): ``` "Downloader: downloading $n from $peer" "result is {"length":1,"elts":[["jwHCUmPYgfGRR8h9y73uvA6n4bcCZ7zBatKWmpkkrSqbfTcinHZ", ["Num_accounts"]]]}" ``` ## Expected Behavior 1. ✅ `Num_accounts` query → valid response received 2. ❌ `handle_num_accounts` → should call `handle_node` *(not happening)* 3. ❌ `handle_node` → should enqueue `What_child_hashes` / `What_contents` *(not happening)* 4. ❌ Follow-up queries → responses → sync complete *(not happening)* ## Files to Investigate - `src/lib/syncable_ledger/syncable_ledger.ml` — `handle_num_accounts` - `src/lib/syncable_ledger/syncable_ledger.ml` — `handle_node` - `src/lib/syncable_ledger/syncable_ledger.ml` — `main_loop` query generation - Bootstrap / frontier loading code — specifically the path where a frontier is restored from disk before epoch ledger sync begins ## Logs - Full `.mina-config` directories of nodes: `gs://tmp-hardfork-testing/logs-2026mar25-bootstrap-failure` - Recent log dumps: https://drive.google.com/drive/folders/1vbsL6QCqVWRHSz1V2m2EcLSTJ0k_DoH_ ## Impact - Nodes that have been running for over a week cannot successfully restart and rejoin the network - Affected nodes cannot participate in consensus or serve epoch ledger data - SNARK works stop being produced if all coordinators get affected - Issue is reliably triggered by automated uptime restarts after long-running sessions, making it a recurring operational hazard ## Severity **High** — Nodes cannot bootstrap after restart, with confirmed network-level impact on SNARK work availability. Likely to recur on any node restart after extended uptime.