High sustained MDBX COW writes at mainnet tip (110-140 MiB/block)

#### System information Erigon version: ```text erigon version 3.5.4-28bc52d6 ``` The same behavior was also observed on Erigon `3.4.2`. OS & Version: ```text Guest: Ubuntu 24.04.4 LTS, Linux 6.8.0-137-generic, x86_64 Host: Proxmox VE 9.2.0, Linux 7.0.14-6-pve, x86_64 VM: 8 vCPU, 40 GiB RAM Disk: WD_BLACK SN850X 4 TB NVMe Host filesystem: ext4, noatime Guest data mount: virtiofs ``` There is no btrfs, ZFS, or autodefrag layer in the data path. SMART reports no media or data integrity errors. The node has very low JSON-RPC traffic during the measurements. Commit hash: `28bc52d6` Erigon command (addresses shortened; all other relevant flags are unchanged): ```text /usr/bin/erigon \ --chain=mainnet \ --datadir=/var/lib/eth/erigon \ --authrpc.addr=127.0.0.1 \ --authrpc.port=8551 \ --authrpc.jwtsecret=/var/lib/eth/jwt/jwt.hex \ --http \ --http.addr=<private-ip> \ --http.port=8545 \ --http.api=eth,net,web3,erigon,trace,ots \ --port=30303 \ --torrent.port=42069 \ --nat=stun \ --caplin.nat=stun \ --metrics \ --metrics.addr=<private-ip> \ --metrics.port=6060 \ --sync.loop.block.limit=10000 \ --batchSize=1g \ --caplin.max-peer-count=64 ``` Consensus Layer: embedded Caplin. I also reproduced the execution-layer write rate with Caplin disabled (`--externalcl`) and Lighthouse BN 8.2.1 connected to the Engine API. Consensus Layer command for the external-CL control: ```text lighthouse bn \ --network=mainnet \ --datadir=/var/lib/eth/lighthouse \ --execution-endpoint=http://127.0.0.1:8551 \ --execution-jwt=/var/lib/eth/jwt/jwt.hex \ --checkpoint-sync-url=https://mainnet.checkpoint.sigp.io ``` Chain/Network: Ethereum mainnet #### Expected behaviour I expected steady-state writes at chain tip to be substantially lower than `110-140 MiB` of MDBX copy-on-write pages and `160-180 MiB` of physical NVMe writes per execution block, especially while the MDBX file and total database size remain constant. If this write volume is expected for Erigon 3 at mainnet tip, please confirm it and consider documenting the expected steady-state write rate or write amplification. At the observed rate, the host writes approximately `1.2-1.4 TB/day` if the workload remains constant. #### Actual behaviour Erigon stays fully synced and healthy, but continuously writes approximately `13-15 MiB/s` to the physical NVMe at chain tip. The write activity continues without meaningful RPC load and stops when execution stops processing new blocks. The strongest 30-minute baseline on Erigon 3.5.4 processed 150 blocks with zero head lag and zero restarts: | Metric | Result | |---|---:| | Physical NVMe writes | `13.522 MiB/s` | | Physical NVMe writes per block | `162.26 MiB/block` | | MDBX COW writes | `9.166 MiB/s` | | MDBX COW writes per block | `109.99 MiB/block` | | MDBX page splits | `361.59/block` | | MDBX file size delta | `0` | The following controls were run on the same node and data directory: | Test | Duration | Blocks | Physical NVMe | MDBX COW | Notes | |---|---:|---:|---:|---:|---| | `--externalcl`, no CL input | `300s` | `0` | `0.369 MiB/s` | `0` | Execution head intentionally static | | Embedded Caplin | `300s` | `25` | `12.30 MiB/s` | `10.57 MiB/s` | Chain tip, lag `0` | | Caplin, parallel execution | `1800s` | `150` | `13.522 MiB/s` | `9.166 MiB/s` | Chain tip, lag `0` | | Caplin, `--exec.serial` | `1800s` | `149` | `13.358 MiB/s` | `10.180 MiB/s` | Chain tip, lag `0` | | External Lighthouse | `1812s` | `151` | `14.955 MiB/s` | `11.565 MiB/s` | Chain tip, lag `0`; Lighthouse was backfilling | Normalizing the two 30-minute Caplin execution modes by block: | Metric | Parallel | Serial | Serial change | |---|---:|---:|---:| | Physical NVMe write/block | `162.26 MiB` | `161.37 MiB` | `-0.55%` | | MDBX COW/block | `109.99 MiB` | `122.97 MiB` | `+11.81%` | | MDBX page splits/block | `361.59` | `415.05` | `+14.78%` | Therefore `--exec.serial` does not reduce the write amplification. Replacing Caplin with Lighthouse also does not reduce Erigon MDBX COW. The external Lighthouse physical-disk result includes its checkpoint backfill, but the MDBX COW metric is emitted by Erigon and remains high independently of that CL I/O. During longer observation periods: - `mdbx.dat` remained approximately `6.26 GiB`; - total `db_size{db="chaindata"}` remained constant; - `ReclaimableSpace`, `CommitmentVals`, and `ChangeSets3` changed in both directions instead of growing without bound; - `db_rotx_overloaded_total` remained `0`; - the execution head remained at the public head; - running Erigon with `--externalcl` and no CL input reduced physical writes to the host background level while the execution head remained static. This looks like sustained MDBX COW/page-split churn caused by per-block execution state and commitment updates rather than database growth, Caplin, pruning backlog, or RPC traffic. #### Steps to reproduce the behaviour 1. Sync Erigon 3.5.4 on Ethereum mainnet and wait for `eth_syncing=false` and zero public-head lag. 2. Wait at least five minutes after catch-up so catch-up execution is excluded. 3. Enable Erigon Prometheus metrics and scrape them every 15 seconds. 4. Measure an uninterrupted 30-minute chain-tip window. 5. Record the start and end execution block numbers and normalize all counters by the number of processed blocks. 6. Calculate MDBX COW bytes from the 16 KiB MDBX page counter: ```promql increase(db_pgops{db="chaindata",phase="cow"}[30m]) * 16384 ``` 7. Measure physical device writes using node-exporter or Linux diskstats: ```promql increase(node_disk_written_bytes_total{device="nvme0n1"}[30m]) ``` 8. Confirm that the physical write rate falls to the host background level when execution is intentionally held static with `--externalcl` and no CL. For a control that isolates consensus input, start Erigon with `--externalcl` and no CL. MDBX COW remains zero while the execution head is static. Then attach a synced external CL: MDBX COW returns as execution resumes at chain tip. #### Related work This report may be an operator-visible measurement of the work already tracked in: - #19801, which states that commitment dominates at tip and proposes delayed trie writes to reduce MDBX write amplification; - #21801, which attributes random MDBX page writes, splits, and overflow churn to large randomly keyed Commitment values and proposes a sequential value layout. Could you confirm whether `110-140 MiB` of MDBX COW per mainnet block is expected on 3.5.4 and whether these two changes are intended to address this steady-state write volume? I can provide raw Prometheus series, a longer test window, selected Erigon logs, or test a specific branch/build if useful. #### Backtrace There is no crash or backtrace. Erigon remains synced, has no automatic restarts, and logs no errors during the measurement windows.