ITADN

bug: prefilter fragment universe off-by-one when an index has no fragment bitmap

#8249Openwjones127 创建于 25 天前
bug
W
wjones127commented
In \`DatasetPreFilter::new\` (rust/lance/src/index/prefilter.rs:70-72): ```rust let all_have_bitmaps = indices.iter().all(|idx| idx.fragment_bitmap.is_some()); if !all_have_bitmaps { fragments.insert_range(0..dataset.manifest.max_fragment_id.unwrap_or(0)); ``` \`max_fragment_id\` is an inclusive high-water mark, but \`insert_range\` is exclusive at the end, so the highest fragment id is always excluded from the constructed "all fragments" universe. And if the field is unset, \`unwrap_or(0)\` produces an empty set. Consequence: when any index lacks a \`fragment_bitmap\` (legacy indices), \`create_deletion_mask\` never sees the highest-id fragment, so - its deletion file is not folded into the block mask — deleted rows in that fragment leak through vector/scalar search prefilters - if that fragment was dropped from the dataset entirely, its stale index entries are not blocked either With an empty set (field unset) no deletion mask is built at all. Compare rust/lance/src/index/append.rs:2703, which computes the same universe correctly with \`0..=max_fragment_id\`. Fix: use \`0..=max\` via the \`Manifest::max_fragment_id()\` method (which falls back to computing from the fragment list), or derive the set from \`manifest.fragments\` directly. Affects non-stable-row-id datasets too; found during a stable row id path review.
0 条评论