Reproducible ZAP-leaf chunk-chain corruption causes soft lockup in `zfs_readdir` / `zap_lookup`

### System information | Type| Version| |---|---| | Distribution | Debian Trixie | | Kernel | `6.12.86+deb13-amd64` | | ZFS package | `zfs-linux 2.3.2-3` (stable-pu, not released yet) | | Architecture | x86_64 | ### Describe the problem you're observing `zap_leaf_lookup_closest()` (readdir / `fzap_cursor_retrieve`) and `zap_leaf_lookup()` (open / `zap_lookup`) both walk a per-bucket chunk chain by following `le->le_next` with no termination guard. In the field I have a dataset whose ZAP leaves contain `l_hash[]` entries that point at chunks now living on the leaf's **freelist** (`ZAP_CHUNK_FREE`). Following `le->le_next` from such a slot traverses the freelist instead of an entry chain and never reaches `CHAIN_END`, so the loop runs forever and the calling CPU appears as a kernel soft lockup. The `ASSERT3U(chunk, <, ZAP_LEAF_NUMCHUNKS(l))` / `ASSERT3U(le->le_type, ==, ZAP_CHUNK_ENTRY)` guards immediately inside both loops are debug-only and compile to no-ops in production builds. ## Symptom ``` watchdog: BUG: soft lockup - CPU#9 stuck for 159252s! [updatedb.plocat:35447] ... RIP: 0010:zap_leaf_lookup_closest+0xad/0x190 [zfs] Call Trace: fzap_cursor_retrieve+0x197/0x370 [zfs] zap_cursor_retrieve+0x216/0x370 [zfs] zfs_readdir+0x219/0x4b0 [zfs] zpl_iterate+0x53/0x80 [zfs] iterate_dir+0x10e/0x200 __x64_sys_getdents64+0x86/0x130 do_syscall_64+0x87/0x1b0 entry_SYSCALL_64_after_hwframe+0x76/0x7e ``` Two unrelated processes (`updatedb.plocate` and a userland directory walker) were wedged simultaneously on the **same leaf object** (matching kernel pointers in `R14` / `RDI`) at the **same** outer-loop state (`RBX = 0xa1`) across watchdog samples taken ~30 s apart. `RBX` is callee-saved; the fact that it does not change across many seconds of watchdog snapshots confirms the outer `for (uint16_t lh = ...; lh <= bestlh; lh++)` is not advancing — the inner chain walk is what is non-terminating. `zpool scrub` reports no errors. The encrypted block's MAC is valid for the corrupt cleartext, so the bad state was written, encrypted and authenticated as-is. ## Diagnostic patch A small defensive change bounds both reader-side chain walks by `ZAP_LEAF_NUMCHUNKS(l)`, validates the chunk index against the leaf size, validates `le_type == ZAP_CHUNK_ENTRY`, and on any violation logs `cmn_err(CE_WARN)` with the leaf `blkid` and offending chunk and returns `EIO`. Callers already propagate non-zero / non-`ENOENT` back to userspace, so `getdents64` and `openat` surface `EIO` instead of wedging the CPU. The patch is not proposed as a permanent fix, it just converts the deadlock into a diagnosable error, so the possible *write*-path defect can be hunted without losing the host. It is meaningful for ordinary users, but it might not be complete if there turns to have a real write-path defect. [5000-zap_leaf-bound-chain-walks.patch](https://github.com/user-attachments/files/28115871/5000-zap_leaf-bound-chain-walks.patch) With the patch installed the kernel emits, on every read of the affected directory: ``` WARNING: ZAP object=187992 leaf blkid=162: chunk 0 has type 253 (expected 252); pool may be corrupt WARNING: ZAP object=427452 leaf blkid=162: chunk 0 has type 253 (expected 252); pool may be corrupt ``` Type **253** is `ZAP_CHUNK_FREE`; type **252** is `ZAP_CHUNK_ENTRY` (`include/sys/zap_leaf.h:99-100`). So `l_hash[<lh>]` of leaf 162 in the affected object is pointing at chunk 0, but chunk 0 is on the freelist. ## Related issues I think this is the same on-disk defect described in the zfs-discuss thread *"kernel watchdog BUG: soft lockup CPU#6 stuck for 23s! [rm]"* (reporter `jjb2016`, August 2020): <https://zfsonlinux.topicbox.com/groups/zfs-discuss/T44de0b2672c30cd0-Md52dd2c779aa07c269612421/kernel-watchdog-bug-soft-lockup-cpu6-stuck-for-23s-rm-1121909> The thread is the same one referenced from a 2020 comment on **openzfs/zfs#6918** but appears to describe a different bug from itself, both are directory-walk hangs but openzfs/zfs#6918 is D and current one is R but soft lockup.