Reduce per-block loading cost during `load_full_frontier`

This is initiated by https://o1-labs.slack.com/archives/C09GEF94J31/p1776204438811599 **Data source:** whale-2, whale-3, whale-4 internal traces — current `internal-trace.jsonl` files (the loading phase), NOT the rotated files (which contain pre-restart steady-state block production) ## Correction notice An earlier version of this document analyzed the **rotated** trace files (`internal-trace.jsonl.0` through `.jsonl.50`), which contained steady-state `add_breadcrumb_exn` traces from the previous 3-day run (Apr 11-14). Those traces included `Calculate_diffs` and `Move_frontier_root` — operations that **do not occur during `load_full_frontier`**. The loading path bypasses `Full_frontier.calculate_diffs` entirely and only emits `New_node` diffs (a hash table insertion, ~0.1ms). This version uses the correct data from the current trace files, which contain the actual `load_full_frontier` → `visit` → `Breadcrumb.build` traces identified by `Loaded_transition_from_storage` entries. **Evidence that the code paths are disjoint:** | Trace marker | Loading path | Steady-state path | |---|---|---| | `Loaded_transition_from_storage` | present | absent | | `Breadcrumb_integrated` | present | present | | `Add_breadcrumb_to_frontier` | **absent** | present | | `Calculate_diffs` | **absent** | present | | `Move_frontier_root` | **absent** | present | On whale-4 (the only node that completed loading and transitioned to steady-state), both phases appear in the same trace file but are temporally disjoint — zero `Calculate_diffs` or `Move_frontier_root` entries fall within the loading window (20:20-20:30 UTC). ## Summary The per-block cost during `load_full_frontier` is dominated by **unnecessary command verification** (`Verify_commands`), not frontier bookkeeping. Each block with user transactions spends ~3.2s in the verifier subprocess (signature checks, SNARK proof verification via RPC), despite these blocks having already been fully verified before persistence. Blocks without user transactions load in ~12ms. The `Full_frontier.apply_diffs` call that inserts the block into the in-memory frontier is 0.1ms — effectively free, because the loading path injects `New_node` diffs directly, bypassing `calculate_diffs` and never triggering root moves. ## Per-block cost breakdown (all whales) | Phase | whale-2 (2,153 blocks) | whale-3 (1,304 blocks) | whale-4 (360 blocks) | |---|---|---|---| | Load→Prediff (deserialization) | 26ms (1.5%) | 23ms (1.5%) | 19ms (1.7%) | | **Prediff→Apply_diff (verification)** | **1,198ms (70.2%)** | **1,094ms (71.6%)** | **773ms (68.0%)** | | Apply_diff→Diff_applied (ledger) | 475ms (27.9%) | 404ms (26.4%) | 340ms (29.9%) | | Apply_full_frontier_diffs | 0.1ms (0.0%) | 0.1ms (0.0%) | 0.1ms (0.0%) | | Extension notifications | 5.0ms (0.3%) | 4.5ms (0.3%) | 3.0ms (0.3%) | | **TOTAL per block** | **1,707ms** | **1,528ms** | **1,137ms** | | Aggregate | whale-2 | whale-3 | whale-4 | |---|---|---|---| | Total build time | 61.2 min | 33.2 min | 6.8 min | | Wall clock | 87.9 min | 47.5 min | 10.1 min | | Async overhead | 30.3% | 30.0% | 32.4% | | Inter-block gap (avg) | 743ms | 657ms | 546ms | ## Bimodal distribution: blocks with transactions vs empty blocks The per-block times follow a bimodal distribution. Blocks without user transactions are fast; blocks with transactions trigger the full verification pipeline. **whale-3 Prediff→Apply_diff distribution:** | Bucket | Count | % | |---|---|---| | 0-10ms (no txns) | 672 | 51.5% | | 10-100ms | 186 | 14.3% | | 100ms-1s | 50 | 3.8% | | 1-3s | 133 | 10.2% | | 3-6s (heavy txn blocks) | 262 | 20.1% | | >6s | 1 | 0.1% | **Split by block type (whale-3):** | | Empty blocks | Blocks with transactions | |---|---|---| | Count | 858 (65.8%) | 446 (34.2%) | | Avg Verify_commands time | 11ms | **3,151ms** | | Avg Apply_diff time | 6ms | **939ms** | | Avg total | 12ms | **4,094ms** | ## Representative traces ### Block with transactions (3.1s total) ``` + 0.0ms Loaded_transition_from_storage + 0.0ms Build_breadcrumb + 0.1ms Validate_staged_ledger_diff + 40.7ms Prediff + 53.9ms Verify_commands + 71.7ms Long_async_cycle +2544.3ms Verify_commands_done ← 2.5s in verifier subprocess +2544.8ms Apply_diff +2583.2ms Update_coinbase_stack +2583.4ms Update_ledger_and_get_statements +3072.0ms Update_ledger_and_get_statements_done ← 489ms ledger application +3072.6ms Fill_work_and_enqueue_transactions +3088.8ms Update_pending_coinbase_collection +3089.4ms Diff_applied +3091.4ms Apply_full_frontier_diffs +3091.6ms Apply_full_frontier_diffs_done ← 0.2ms frontier insertion +3093.9ms Breadcrumb_integrated ``` ### Empty block (12ms total) ``` + 0.0ms Loaded_transition_from_storage + 0.0ms Build_breadcrumb + 0.1ms Validate_staged_ledger_diff + 1.4ms Prediff + 1.4ms Verify_commands + 2.4ms Verify_commands_done ← 1ms, nothing to verify + 2.5ms Apply_diff + 4.3ms Update_coinbase_stack + 4.5ms Update_ledger_and_get_statements + 6.5ms Update_ledger_and_get_statements_done ← 2ms + 7.0ms Fill_work_and_enqueue_transactions + 7.5ms Diff_applied + 8.9ms Apply_full_frontier_diffs + 9.1ms Apply_full_frontier_diffs_done ← 0.2ms + 12.3ms Breadcrumb_integrated ``` ## Root cause: `Pre_diff_info.get` calls `check_commands` unconditionally The bottleneck is entirely within the `Prediff→Apply_diff` window. The code path: 1. `validate_staged_ledger_diff` (`validation.ml:471`) calls `Staged_ledger.apply` (`staged_ledger.ml:1215`) 2. `apply` emits `[%log internal] "Prediff"` at line 1234 3. `Pre_diff_info.get` is called at line 1236 with `~check:(Check_commands.check_commands t.ledger ~verifier ~transaction_pool_proxy)` 4. Inside `Pre_diff_info.get` (`pre_diff_info.ml:506`), `Staged_ledger_diff.validate_commands` at line 509 passes **all** user commands through `~check` 5. `Check_commands.check_commands` (`check_commands.ml:36-85`) does three expensive operations: - **VK loading** (line 41): `to_all_verifiable` loads verification keys from the ledger for all zkapp commands - **Transaction hashing** (line 49-58): `hash_command_with_hashes` serializes every command (bin_prot + Blake2), including `read_all_proofs_from_disk` for zkapps. A `PERF` comment at line 25 flags this as known-slow. During loading the hash is 100% wasted — the `dummy_transaction_pool_proxy` (line 14) always returns `None` - **Verifier RPC** (line 62): `Verifier.verify_commands` sends all commands to the verifier subprocess, which performs Schnorr signature verification and `Pickles.Side_loaded.verify` for zkapp proofs 6. `[%log internal] "Apply_diff"` is emitted at line 1247 only after all of the above completes ### What `skip_staged_ledger_verification:\`All` actually skips The loading path passes `~skip_staged_ledger_verification:\`All` (`persistent_frontier.ml:244`). But this flag only controls: - `check_completed_works` (line 1225) — skipped (before Prediff, saves SNARK work verification) - Staged ledger hash computation (line 524 of `validation.ml`) — skipped - `verify_scan_state_after_apply` — skipped It does **not** skip `Pre_diff_info.get` or the `check_commands` pipeline within it. The flag has no effect on the dominant bottleneck. ### Why a fix already exists but isn't used `Pre_diff_info.get_unchecked` (`pre_diff_info.ml:523`) skips `validate_commands` entirely and goes straight to `get_impl` (pure coinbase/fee arithmetic, sub-millisecond). It is used by `apply_diff_unchecked` (`staged_ledger.ml:1274`). But the loading path calls `Staged_ledger.apply` (the checked variant), not `apply_diff_unchecked`. ## Secondary costs in Apply_diff→Diff_applied (blocks with transactions) | Sub-phase | Avg time | Code path | |---|---|---| | `Update_ledger_and_get_statements` | 862ms | Transaction application to Merkle ledger | | `Fill_work_and_enqueue_transactions` | 28ms | Scan state enqueue | | `Update_pending_coinbase_collection` | 0.7ms | Coinbase bookkeeping | `Update_ledger_and_get_statements` is the core ledger mutation — applying each transaction's account updates to the Merkle tree. This work is necessary to reconstruct the staged ledger state. Within it, per-transaction overhead includes: - `Ledger.merkle_root` called at `staged_ledger.ml:494` before each transaction (for source hash assertion) - `Sparse_ledger.of_ledger_subset_exn` at line 497 constructs a sparse ledger witness for each transaction — needed for SNARK statement generation but not for the ledger update itself ### Body reference computation before Prediff `validation.ml:484-487` unconditionally calls `Staged_ledger_diff.Body.read_all_proofs_from_disk` to compute a body reference hash. This deserializes all SNARK proofs from LMDB. Currently costs ~23ms (within the Load→Prediff window). Not the bottleneck but unnecessary during loading. ## Recommendations ### 1. Skip command verification during loading (eliminates ~3.2s/block for blocks with transactions) When `skip_staged_ledger_verification = Some \`All`, the `apply` function in `staged_ledger.ml` should bypass `Pre_diff_info.get` and use the unchecked path instead. **Concrete change** in `staged_ledger.ml`, lines 1234-1244: ```ocaml (* Current code — always calls checked path *) [%log internal] "Prediff" ; let%bind prediff = Pre_diff_info.get witness ~constraint_constants ~coinbase_receiver ~supercharge_coinbase ~check:(Check_commands.check_commands t.ledger ~verifier ~transaction_pool_proxy) |> ... (* Proposed — skip verification when loading from persistence *) [%log internal] "Prediff" ; let%bind prediff = match skip_verification with | Some `All -> Pre_diff_info.get_trusted witness ~constraint_constants ~coinbase_receiver ~supercharge_coinbase |> Deferred.return | _ -> Pre_diff_info.get witness ~constraint_constants ~coinbase_receiver ~supercharge_coinbase ~check:(Check_commands.check_commands t.ledger ~verifier ~transaction_pool_proxy) |> Deferred.map ~f:(Result.map_error ~f:(fun error -> Staged_ledger_error.Pre_diff error)) ``` This requires adding a `get_trusted` variant to `Pre_diff_info` that accepts a `Staged_ledger_diff.t` (not `With_valid_signatures_and_proofs.t`) and skips `validate_commands`, going directly to `get_impl`. The implementation mirrors `get_unchecked` but operates on the unchecked diff type. **Files to modify:** - `src/lib/staged_ledger/staged_ledger.ml:1234-1244` — branch on `skip_verification` - `src/lib/staged_ledger/pre_diff_info.ml` — add `get_trusted` (copy `get`, remove `validate_commands` call) - `src/lib/staged_ledger/pre_diff_info.mli` — expose `get_trusted` **Expected impact:** Blocks with transactions drop from ~4.1s to ~0.9s (the ledger application cost). Total loading time for whale-2 would drop from 61 min to ~21 min (build time) / ~30 min (wall clock). ### 2. Skip body reference computation during loading (eliminates ~23ms/block) When `skip_staged_ledger_verification = Some \`All`, skip the `read_all_proofs_from_disk` + `compute_reference` + hash comparison at `validation.ml:484-492`. The block was already validated before persistence; the body reference cannot have changed. **File:** `src/lib/mina_block/validation.ml:484-492` — wrap in a guard on `skip_staged_ledger_verification`. ### 3. Skip sparse ledger witness creation during loading (estimated ~100-200ms/block) `Sparse_ledger.of_ledger_subset_exn` (`staged_ledger.ml:497`) constructs a witness for each transaction. This is needed for SNARK statement generation but not for applying the transaction to the ledger. During loading, the result is used only to compute `source_ledger_hash` and `target_ledger_hash` for the `Pre_statement`, which feeds into `update_ledger_and_get_statements`. If the `Pre_statement` assertions can be skipped when `skip_verification = Some \`All`, the sparse ledger construction can be eliminated. ### Projected loading times after fix #1 | | whale-2 | whale-3 | whale-4 | |---|---|---|---| | Blocks | 2,153 | 1,304 | 360 | | Current build time | 61.2 min | 33.2 min | 6.8 min | | Projected build time | ~21 min | ~12 min | ~3 min | | Speedup | ~2.9x | ~2.8x | ~2.3x | The lower speedup for whale-4 reflects its smaller fraction of blocks with transactions (fewer blocks benefit from skipping verification). With all three fixes combined, blocks with transactions would drop from ~4.1s to ~0.6-0.7s (just core ledger mutation), and total build time would approach the empty-block rate (~12ms) as the lower bound.