Severe I/O delays during zvol page cache writeback

### System information Type | Version/Name --- | --- Distribution Name | Proxmox VE Distribution Version | 9.2.11 Kernel Version | 7.0.14-14-pve Architecture | x86-64 OpenZFS Version | 2.4.4 ### Describe the problem you're observing Page cache writeback from data written to a ZVOL causes high I/O delays for all unrelated I/O, affecting even unrelated pools. Delays were observed to reach values also exceeding 10s. In particular, this was found to effect I/O on ZVOLs attached to QEMU VMs. eBPF tracing shows that writeback hands the data to zvol_write() [1] one `4k` page at a time, only being aggregated at the ZIO layer by vdev_queue_aggregate() [2]. Further, `zpool iostat -r <pool> 1` shows a huge number of `16k` sync read requests (default 16k volblocksize) in addition to the aggregated writes. Below shows an example output thereof for `fio` writeload with `direct=0` (for the exact fio command invocation see reproducer section): ``` proxmox sync_read sync_write async_read async_write scrub trim rebuild req_size ind agg ind agg ind agg ind agg ind agg ind agg ind agg ---------- ----- ----- ----- ----- ----- ----- ----- ----- ----- ----- ----- ----- ----- ----- 512 0 0 0 0 0 0 11 0 0 0 0 0 0 0 1K 0 0 3 0 0 0 20 0 0 0 0 0 0 0 2K 0 0 0 0 0 0 21 2 0 0 0 0 0 0 4K 0 0 0 0 0 0 10 2 0 0 0 0 0 0 8K 0 0 0 0 0 0 0 9 0 0 0 0 0 0 16K 41.3K 0 0 0 0 0 1 0 0 0 0 0 0 0 32K 0 0 0 0 0 0 0 3 0 0 0 0 0 0 64K 0 0 0 0 0 0 0 37 0 0 0 0 0 0 128K 0 0 0 0 0 0 0 6.33K 0 0 0 0 0 0 256K 0 0 0 0 0 0 0 0 0 0 0 0 0 0 512K 0 0 0 0 0 0 0 0 0 0 0 0 0 0 1M 0 0 0 0 0 0 0 0 0 0 0 0 0 0 2M 0 0 0 0 0 0 0 0 0 0 0 0 0 0 4M 0 0 0 0 0 0 0 0 0 0 0 0 0 0 8M 0 0 0 0 0 0 0 0 0 0 0 0 0 0 16M 0 0 0 0 0 0 0 0 0 0 0 0 0 0 ------------------------------------------------------------------------------------------------------------ ``` We'd interpret this as one DMU transaction being performed per 4k page, plus a read-modify-write of the surrounding volblock for every page that only partially covers a volblock, therefore leading to additional synchronous reads and causing the observed increase in latency. When bypassing the host page cache, e.g. via the `direct=1` flag to fio commands, no more latency spikes are observed with write load. Also, setting module parameter `zvol_use_blk_mq=1` (see #12483) was found to reduce latency spikes. Building kernels without `CONFIG_BUFFER_HEAD` set leads to bigger block I/O requests being submitted during writeback, but is not a viable option since many filesystems including ext4 depend on `CONFIG_BUFFER_HEAD` being set. Using `direct=1` results in no more I/O latency and only write requests being performed: ``` proxmox sync_read sync_write async_read async_write scrub trim rebuild req_size ind agg ind agg ind agg ind agg ind agg ind agg ind agg ---------- ----- ----- ----- ----- ----- ----- ----- ----- ----- ----- ----- ----- ----- ----- 512 0 0 0 0 0 0 15 0 0 0 0 0 0 0 1K 0 0 3 0 0 0 25 0 0 0 0 0 0 0 2K 0 0 0 0 0 0 25 2 0 0 0 0 0 0 4K 0 0 0 0 0 0 13 7 0 0 0 0 0 0 8K 0 0 0 0 0 0 0 10 0 0 0 0 0 0 16K 0 0 0 0 0 0 10 3 0 0 0 0 0 0 32K 0 0 0 0 0 0 4 6 0 0 0 0 0 0 64K 0 0 0 0 0 0 0 153 0 0 0 0 0 0 128K 0 0 0 0 0 0 0 6.05K 0 0 0 0 0 0 256K 0 0 0 0 0 0 0 0 0 0 0 0 0 0 512K 0 0 0 0 0 0 0 0 0 0 0 0 0 0 1M 0 0 0 0 0 0 0 0 0 0 0 0 0 0 2M 0 0 0 0 0 0 0 0 0 0 0 0 0 0 4M 0 0 0 0 0 0 0 0 0 0 0 0 0 0 8M 0 0 0 0 0 0 0 0 0 0 0 0 0 0 16M 0 0 0 0 0 0 0 0 0 0 0 0 0 0 ------------------------------------------------------------------------------------------------------------ ``` ### Describe how to reproduce the problem `fio` commands were used to reliable reproduce via write load on a ZVOL: ``` fio -iodepth=1 -rw=write -ioengine=libaio -size=60G -numjobs=1 -group_reporting -filename=/dev/zvol/<pool>/<zvol> -name=seqwr -bs=4M -direct=0 ``` Latency was measured via `ioping` on a ZVOL on the same as well as an unrelated pool via: ``` ioping /dev/zvol/<pool>/<zvol> ``` Related open issue: https://github.com/openzfs/zfs/issues/12483 [1] https://github.com/openzfs/zfs/blob/zfs-2.4.4/module/os/linux/zfs/zvol_os.c#L192 [2] https://github.com/openzfs/zfs/blob/zfs-2.4.4/module/zfs/vdev_queue.c#L588