Bug: Repeated Database compactification failures

<!-- IMPORTANT: IF YOU DON'T FILL THIS TEMPLATE COMPLETELY IT WILL TAKE MORE TIME FOR US TO HELP YOU! SOME EXTERNAL ELECTRS GUIDES SUCH AS RASPIBOLT ARE OUTDATED AND DO NOT WORK SO SHOULD NOT BE FOLLOWED! Please try with OUR usage instructions first. If you use electrs integrated into other project report the bug to their project (unless you are the project author who found the bug is in electrs itself)! --> **Describe the bug** Compacitifcation of index DB fails. **Electrs version** v0.10.10 **To Reproduce** Steps to reproduce the behavior: 1. Build 2. Configure and start electrs 3. Wait 4. Repeated compactification failures **Expected behavior** Initial sync/indexing/compactification to complete without issues. **Configuration** <details> <summary>electrs.toml</summary> ```bash Oct 24 16:23:19 btc-apps electrs[3343]: Starting electrs 0.10.10 on x86_64 linux with Config { network: Bitcoin, db_path: "/home/electrs/.electrs/index_db/bitcoin", db_log_dir: None, db_parallelism: 2, daemon_auth: UserPass("electrs", "<sensitive>"), daemon_rpc_addr: 10.1.1.10:8332, daemon_p2p_addr: 10.1.1.10:8333, electrum_rpc_addr: 10.1.1.11:50001, monitoring_addr: 127.0.0.1:4224, wait_duration: 10s, jsonrpc_timeout: 15s, index_batch_size: 10, index_lookup_limit: None, reindex_last_blocks: 0, auto_reindex: true, ignore_mempool: false, sync_once: false, skip_block_download_wait: false, disable_electrum_rpc: false, server_banner: "Welcome to electrs 0.10.10 (Electrum Rust Server)!", signet_magic: f9beb4d9 } ``` </details> Environment variables: Default, unless modified by config Arguments: "--db-parallelism=2" --- See `systemd` script below **System running electrs** - Deployment method: Build from source, following instructions in docs `Building from source on an Ubuntu 25.04 / Debian 13` - Debian 13 - The system is a Proxmox VM with 2 vCPU and 4GB RAM. Host runs on EPYC 7551; VM inherits from host. - Bitcoin Core v30 running on another VM with same hw/config; no issues in connectivity encountered; Bitcoin Core running successfully **Electrum client** Electrum 4.5.6 **Additional context** `systemd` is used to manage the app. The scripts has been adapted from the one in the docs and there's three of them. The main one, `electrs.service`: <details> <summary>electrs.service</summary> ```bash electrs@btc-apps:~$ sudo cat /etc/systemd/system/electrs.service [Unit] Description=Electrs Electrum Server After=bitcoind-online.service [Service] WorkingDirectory=/home/electrs/.electrs ExecStart=/home/electrs/electrs/target/release/electrs --db-parallelism=2 KillMode=process User=electrs Group=electrs Type=simple Restart=on-failure TimeoutSec=300 RestartSec=15 Environment="RUST_BACKTRACE=1" # Hardening measures PrivateTmp=true ProtectSystem=full NoNewPrivileges=true MemoryDenyWriteExecute=true [Install] WantedBy=multi-user.target ``` </details> `bitcoind-online.service` is: <details> <summary>bitcoind.service</summary> ```bash electrs@btc-apps:~$ sudo cat /etc/systemd/system/bitcoind-online.service [Unit] Description=Check Bitcoin Core's daemon is online Requires=network-online.target After=network-online.target Before=electrs.service [Service] User=electrs WorkingDirectory=/home/electrs/.electrs ExecStart=/usr/bin/bash /home/electrs/.electrs/bitcoind-online.sh Restart=on-failure TimeoutSec=300 RestartSec=3 [Install] WantedBy=multi-user.target ``` </details> Finally, `bitcoind-online.sh`: ```bash electrs@btc-apps:~$ cat .electrs/bitcoind-online.sh #!/usr/bin/bash curl --user electrs:TheSecretPasswordNoOneCanGuess \ --data-binary '{"jsonrpc": "2.0", "id": "bitcoind.live", "method": "uptime"}' \ -H 'content-type: application/json' http://10.1.1.10:8332/ ``` `systemd` restarts `electrs` repeatedly and overnight I hit "restart counter is at 65". **Error** <details> <summary>systemd logs</summary> ``` Oct 24 16:23:19 btc-apps electrs[3343]: [2025-10-24T14:23:19.699Z DEBUG tiny_http] Server listening on 127.0.0.1:4224 Oct 24 16:23:19 btc-apps electrs[3343]: [2025-10-24T14:23:19.699Z INFO electrs::metrics::metrics_impl] serving Prometheus metrics on 127.0.0.1:4224 Oct 24 16:23:19 btc-apps electrs[3343]: [2025-10-24T14:23:19.699Z INFO electrs::server] serving Electrum RPC on 10.1.1.11:50001 Oct 24 16:23:19 btc-apps electrs[3343]: [2025-10-24T14:23:19.703Z DEBUG tiny_http] Running accept thread Oct 24 16:23:20 btc-apps electrs[3343]: [2025-10-24T14:23:20.329Z INFO electrs::db] "/home/electrs/.electrs/index_db/bitcoin": 2897 SST files, 222.522031304 GB, 19.44784542 Grows Oct 24 16:23:20 btc-apps electrs[3343]: [2025-10-24T14:23:20.329Z DEBUG electrs::db] DB Some(Config { compacted: false, format: 0 }) Oct 24 16:23:21 btc-apps electrs[3343]: [2025-10-24T14:23:21.861Z INFO electrs::chain] loading 920565 headers, tip=000000000000000000013b0d29be67c2d8118e7217e8cd1f1f675ea4baa0f89b Oct 24 16:23:23 btc-apps electrs[3343]: [2025-10-24T14:23:23.131Z INFO electrs::chain] chain updated: tip=000000000000000000013b0d29be67c2d8118e7217e8cd1f1f675ea4baa0f89b, height=920565 Oct 24 16:23:23 btc-apps electrs[3343]: [2025-10-24T14:23:23.133Z DEBUG bitcoincore_rpc] JSON-RPC request: getblockchaininfo [] Oct 24 16:23:23 btc-apps electrs[3343]: [2025-10-24T14:23:23.154Z DEBUG bitcoincore_rpc] JSON-RPC request: getnetworkinfo [] Oct 24 16:23:23 btc-apps electrs[3343]: [2025-10-24T14:23:23.157Z DEBUG bitcoincore_rpc] JSON-RPC request: getnetworkinfo [] Oct 24 16:23:23 btc-apps electrs[3343]: [2025-10-24T14:23:23.158Z DEBUG bitcoincore_rpc] JSON-RPC request: getblockchaininfo [] Oct 24 16:23:23 btc-apps electrs[3343]: [2025-10-24T14:23:23.158Z DEBUG bitcoincore_rpc] JSON-RPC request: getnetworkinfo [] Oct 24 16:23:23 btc-apps electrs[3343]: [2025-10-24T14:23:23.160Z DEBUG electrs::p2p] peer version: VersionMessage { version: 70016, services: ServiceFlags(3081), timestamp: 1761315803, receiver: Address {services: ServiceFlags(NONE), address: 0.0.0.0, port: 0}, sender: Address {services: ServiceFlags(NETWORK|WITNESS|NETWORK_LIMITED|P2P_V2), address: 0.0.0.0, port: 0}, nonce: 16415212804266093349, user_agent: "/Satoshi:30.0.0/", start_height: 920567, relay: true } Oct 24 16:23:23 btc-apps electrs[3343]: [2025-10-24T14:23:23.160Z DEBUG electrs::p2p] got 2 new headers Oct 24 16:23:23 btc-apps electrs[3343]: [2025-10-24T14:23:23.160Z INFO electrs::index] indexing 2 blocks: [920566..920567] Oct 24 16:23:23 btc-apps electrs[3343]: [2025-10-24T14:23:23.161Z DEBUG electrs::p2p] loading 2 blocks Oct 24 16:23:23 btc-apps electrs[3343]: [2025-10-24T14:23:23.355Z DEBUG electrs::index] writing 16495 funding and 16685 spending rows from 7134 transactions, 2 blocks Oct 24 16:23:23 btc-apps electrs[3343]: [2025-10-24T14:23:23.371Z INFO electrs::chain] chain updated: tip=00000000000000000000b0c20db59ac71bcb893d950bbaf0324063d71b590a5d, height=920567 Oct 24 16:23:23 btc-apps electrs[3343]: [2025-10-24T14:23:23.372Z DEBUG electrs::p2p] got 0 new headers Oct 24 16:23:23 btc-apps electrs[3343]: [2025-10-24T14:23:23.372Z DEBUG electrs::db] flushing DB column families Oct 24 16:23:23 btc-apps electrs[3343]: [2025-10-24T14:23:23.539Z INFO electrs::db] starting config compaction Oct 24 16:23:23 btc-apps electrs[3343]: [2025-10-24T14:23:23.567Z INFO electrs::db] starting headers compaction Oct 24 16:23:26 btc-apps electrs[3343]: [2025-10-24T14:23:26.629Z INFO electrs::db] starting txid compaction Oct 24 16:23:29 btc-apps electrs[3343]: [2025-10-24T14:23:29.058Z DEBUG electrs::server] 0: recv {"jsonrpc":"2.0","method":"server.version","id":0,"params":["electrum/4.5.6","1.4"]} Oct 24 16:33:29 btc-apps electrs[3343]: [2025-10-24T14:33:29.058Z DEBUG electrs::server] 0: disconnected Oct 24 16:33:29 btc-apps electrs[3343]: [2025-10-24T14:33:29.107Z DEBUG electrs::server] 1: recv {"jsonrpc":"2.0","method":"server.version","id":0,"params":["electrum/4.5.6","1.4"]} Oct 24 16:35:18 btc-apps electrs[3343]: [2025-10-24T14:35:18.945Z INFO electrs::db] starting funding compaction Oct 24 16:35:27 btc-apps electrs[3343]: [2025-10-24T14:35:27.620Z INFO electrs::db] starting spending compaction Oct 24 16:35:27 btc-apps electrs[3343]: thread 'main' panicked at src/db.rs:373:14: Oct 24 16:35:27 btc-apps electrs[3343]: DB::put failed: Error { message: "IO error: While open a file for random read: /home/electrs/.electrs/index_db/bitcoin/001109.sst: Too many open files" } Oct 24 16:35:27 btc-apps electrs[3343]: stack backtrace: Oct 24 16:35:27 btc-apps electrs[3343]: 0: rust_begin_unwind Oct 24 16:35:27 btc-apps electrs[3343]: 1: core::panicking::panic_fmt Oct 24 16:35:27 btc-apps electrs[3343]: 2: core::result::unwrap_failed Oct 24 16:35:27 btc-apps electrs[3343]: 3: electrs::db::DBStore::set_config Oct 24 16:35:27 btc-apps electrs[3343]: 4: electrs::index::Index::sync Oct 24 16:35:27 btc-apps electrs[3343]: 5: electrs::server::run Oct 24 16:35:27 btc-apps electrs[3343]: note: Some details are omitted, run with `RUST_BACKTRACE=full` for a verbose backtrace. Oct 24 16:35:27 btc-apps electrs[3343]: [2025-10-24T14:35:27.623Z INFO electrs::db] closing DB at /home/electrs/.electrs/index_db/bitcoin Oct 24 16:35:28 btc-apps electrs[3343]: [2025-10-24T14:35:28.640Z DEBUG electrs::p2p] closing p2p_loop thread: no more requests to handle Oct 24 16:35:28 btc-apps electrs[3343]: [2025-10-24T14:35:28.640Z DEBUG electrs::p2p] closing p2p_send thread: no more messages to send Oct 24 16:35:28 btc-apps electrs[3343]: [2025-10-24T14:35:28.640Z DEBUG electrs::p2p] closing p2p_recv thread: connection closed Oct 24 16:35:28 btc-apps systemd[1]: electrs.service: Main process exited, code=exited, status=101/n/a Oct 24 16:35:28 btc-apps systemd[1]: electrs.service: Failed with result 'exit-code'. Oct 24 16:35:28 btc-apps systemd[1]: electrs.service: Consumed 15min 19.235s CPU time, 3.1G memory peak. ``` </details> [DB LOG file attached.](https://github.com/user-attachments/files/23140358/LOG.txt) **End state** After more than one day and repeated restarts it seems to have finally finished. Current size of index on disk is ~211GB, which doesn't appear very compacted. Reporting in case this is of any relevance and off the expected behavior. Once the indexing/compactification finished, every attempt by the client to connect causes a crash of the server, which gets restarted by `systemd`. The error is the same as above ``` Oct 25 09:18:48 btc-apps electrs[12876]: DB scan failed: Error { message: "IO error: While open a file for random read: /home/electrs/.electrs/index_db/bitcoin/001274.sst: Too many open files" } ```