<!-- Release notes generated using configuration in .github/release.yml at v11.0.0-beta.6 --> ## What's Changed ### New Features 🎉 * feat(mem-wal): derive index catch-up from the version a commit read by @xuanyu-z in https://github.com/lance-format/lance/pull/8481 **Full Changelog**: https://github.com/lance-format/lance/compare/v11.0.0-beta.5...v11.0.0-beta.6

<!-- Release notes generated using configuration in .github/release.yml at v10.0.0 --> ## What's Changed ### Breaking Changes 🛠 * perf(compaction)!: skip building row-address maps when index remapping is not needed by @everySympathy in https://github.com/lance-format/lance/pull/7778 * perf(cache)!: use fixed-size cache keys by @Ali2Arslan in https://github.com/lance-format/lance/pull/7878 * fix(blob)!: preserve null selections across blob APIs by @Xuanwo in https://github.com/lance-format/lance/pull/7903 ### Critical Fixes ‼️ * fix(exec): preserve order when coalescing a sorted multi-partition plan by @wjones127 in https://github.com/lance-format/lance/pull/7868 * fix(encoding): bounds-check the length prefix in `VariableFullZipDecoder::unzip` by @professor-moody in https://github.com/lance-format/lance/pull/8138 * fix(encoding): reject out-of-bounds variable-width offsets on decode by @Xuanwo in https://github.com/lance-format/lance/pull/8144 Both fixes were backported to `release/v10.0` in [#8146](https://github.com/lance-format/lance/pull/8146). The first removes an unchecked length-prefix read from release builds; the second rejects corrupt offsets before constructing Arrow arrays. ### New Features 🎉 * feat(index): support segmented ngram index by @zhangyue19921010 in https://github.com/lance-format/lance/pull/7244 * feat: accept TableProvider write inputs for merge_insert and insert by @wjones127 in https://github.com/lance-format/lance/pull/7368 * feat(python): add Pydantic model instance auto-conversion and from_pydantic_model() classmethod by @vinaysurtani in https://github.com/lance-format/lance/pull/7383 * feat(index): write zone map seeds into data file footers during append by @westonpace in https://github.com/lance-format/lance/pull/7427 * feat(python): expose DataOverlay commit operation by @wjones127 in https://github.com/lance-format/lance/pull/7540 * feat: support cross-store (cross-account) deep_clone by @sezruby in https://github.com/lance-format/lance/pull/7545 * feat(index): mask data overlay files in scalar and vector index queries by @wjones127 in https://github.com/lance-format/lance/pull/7549 * feat: support structured_query in the directory namespace query_table by @hamersaw in https://github.com/lance-format/lance/pull/7592 * feat: support merge_insert when source omits blob columns by @nyl3532016 in https://github.com/lance-format/lance/pull/7615 * feat: add code analyzer for FTS by @Xuanwo in https://github.com/lance-format/lance/pull/7681 * feat(compaction): compact fragments over an overlay-count limit by @wjones127 in https://github.com/lance-format/lance/pull/7772 * feat(fts): record scorer_build_ms for the exec-local BM25 scorer fallback by @LuQQiu in https://github.com/lance-format/lance/pull/7807 * feat(fts): read inverted index params without opening the segment by @LuQQiu in https://github.com/lance-format/lance/pull/7816 * feat: expose cached file metadata APIs on FileFragment by @jiaqizho in https://github.com/lance-format/lance/pull/7820 * feat(index): per-query index cache hit/miss stats by @yanghua in https://github.com/lance-format/lance/pull/7862 * feat: add batch blob range reads by @Xuanwo in https://github.com/lance-format/lance/pull/7864 * feat: introduce exact file format identity and persisted codecs by @Xuanwo in https://github.com/lance-format/lance/pull/7879 * feat(python): expose LABEL_LIST segment index build by @ddupg in https://github.com/lance-format/lance/pull/7884 * feat(java): expose session metadata cache stats to java api by @zhangyue19921010 in https://github.com/lance-format/lance/pull/7885 * feat(format): read and write sparse structural pages by @Xuanwo in https://github.com/lance-format/lance/pull/7889 * feat(mem_wal): drive the WAL append on a flush-interval ticker by @hamersaw in https://github.com/lance-format/lance/pull/7894 * feat(index): support segmented bloom filters by @jackye1995 in https://github.com/lance-format/lance/pull/7925 * feat(index): use ACORN-1 traversal for prefiltered HNSW search by @leohoare in https://github.com/lance-format/lance/pull/7927 * feat: support segmented RTree indexes by @jackye1995 in https://github.com/lance-format/lance/pull/7932 * feat(namespace): idempotent create_table_version with strict CAS and dotfile-safe reserved marker by @XuQianJin-Stars in https://github.com/lance-format/lance/pull/7940 * feat(fts): add native exact segment selection by @Xuanwo in https://github.com/lance-format/lance/pull/7976 ### Bug Fixes 🐛 * fix(index): normalize same-column scalar index predicates by @zouhuajian in https://github.com/lance-format/lance/pull/6782 * fix: avoid panic due to empty IO buffers on tokio runtime shutdown by @ilya-zlobintsev in https://github.com/lance-format/lance/pull/7478 * fix: don't quote column names in list_indices by @dantasse in https://github.com/lance-format/lance/pull/7503 * fix(commit): verify finalized external manifests by @BubbleCal in https://github.com/lance-format/lance/pull/7578 * fix: skip empty string docs in FTS stats by @Ecthlion in https://github.com/lance-format/lance/pull/7699 * fix: keep ids and addresses aligned in filter_deleted_ids under stabl… by @wkalt in https://github.com/lance-format/lance/pull/7704 * fix(mem-wal): propagate final flush failures from ShardWriter::close by @u70b3 in https://github.com/lance-format/lance/pull/7769 * fix: bracket match query display by @jackye1995 in https://github.com/lance-format/lance/pull/7796 * fix(sql): reject invalid LIKE/ILIKE escape strings by @wjones127 in https://github.com/lance-format/lance/pull/7810 * fix(table): route tos:// scheme to ConditionalPutCommitHandler by @zhangyue19921010 in https://github.com/lance-format/lance/pull/7824 * fix(python): allow blob writer cleanup across threads by @justinrmiller in https://github.com/lance-format/lance/pull/7827 * fix(index): reuse cached FTS document lengths by @Xuanwo in https://github.com/lance-format/lance/pull/7830 * fix(index): sort JSON-path values once after extraction, not the raw column by @wjones127 in https://github.com/lance-format/lance/pull/7835 * fix: make FTS metadata loading retry-safe by @Xuanwo in https://github.com/lance-format/lance/pull/7838 * fix: reenable flaky tests on test_torch_kmeans.py by @cswpy in https://github.com/lance-format/lance/pull/7849 * fix(fts): restore format version environment fallback by @Xuanwo in https://github.com/lance-format/lance/pull/7855 * fix(core): validate CPU/IO env vars with a diagnosable error by @LuciferYang in https://github.com/lance-format/lance/pull/7856 * fix: dispatch concurrent blob range reads by @Xuanwo in https://github.com/lance-format/lance/pull/7861 * fix(fts): use v3 as the capability gate by @Xuanwo in https://github.com/lance-format/lance/pull/7866 * fix(index): return error instead of panicking when hierarchical k-means can't reach target_k by @wjones127 in https://github.com/lance-format/lance/pull/7869 * fix: trim fragment-reuse index for non-covering indexes by @xuanyu-z in https://github.com/lance-format/lance/pull/7870 * fix(mem_wal): harden shard open and fail reads on a poisoned writer by @hamersaw in https://github.com/lance-format/lance/pull/7872 * fix(fts): preserve parameter compatibility by @Xuanwo in https://github.com/lance-format/lance/pull/7874 * fix(core): bound SlotBackoff retry sleeps and fix slot overflow by @LuciferYang in https://github.com/lance-format/lance/pull/7883 * fix(mem_wal): read-your-writes via split index-apply and dual visibility cursors by @hamersaw in https://github.com/lance-format/lance/pull/7888 * fix(mem-wal): propagate task shutdown failures by @u70b3 in https://github.com/lance-format/lance/pull/7915 * fix(mem-wal): refill scans after shadow filtering by @u70b3 in https://github.com/lance-format/lance/pull/7917 * fix(index): correct offset→row-id translation for overlay masking under deletions by @wjones127 in https://github.com/lance-format/lance/pull/7918 * fix(index): keep overlays masked when updating a non-indexed column by @wjones127 in https://github.com/lance-format/lance/pull/7926 * fix(namespace): propagate dir-namespace storage errors instead of mapping to TableNotFound by @brendanclement in https://github.com/lance-format/lance/pull/7931 * fix(index): rebind vector readers to current object store by @u70b3 in https://github.com/lance-format/lance/pull/7944 * fix(blob): unwrap Option in merge_insert blob tests after take_blobs null API by @XuQianJin-Stars in https://github.com/lance-format/lance/pull/7963 * fix: persist finalized external manifest metadata by @Xuanwo in https://github.com/lance-format/lance/pull/7964 * fix(blob): preserve null and empty values during compaction by @Xuanwo in https://github.com/lance-format/lance/pull/7965 * fix(ci): unblock cargo publish and dry-run packaging in CI by @wjones127 in https://github.com/lance-format/lance/pull/7989 * fix: preserve blob selection cardinality by @Xuanwo in https://github.com/lance-format/lance/pull/8003 * fix(index): prune stale index coverage during rebase by @ddupg in https://github.com/lance-format/lance/pull/8011 * fix(index): make vector append segment-set-native by @Xuanwo in https://github.com/lance-format/lance/pull/8047 * fix(arrow): preserve all-null struct validity during merge by @mmatczuk in https://github.com/lance-format/lance/pull/8049 * fix(io): stabilize wrapper identity in object store cache by @Xuanwo in https://github.com/lance-format/lance/pull/8068 * fix(blob): preserve null and empty values across reads and compaction by @Xuanwo in https://github.com/lance-format/lance/pull/8070 * fix(fts): propagate nested multimatch limits by @BubbleCal in https://github.com/lance-format/lance/pull/8073 ### Documentation 📚 * docs: implement the Lance Docs design as the mkdocs site theme by @Xuanwo in https://github.com/lance-format/lance/pull/7821 * docs: make the docs site theme usable on mobile and narrow screens by @Xuanwo in https://github.com/lance-format/lance/pull/7853 * docs: fix nested agent guide loading and trim root AGENTS.md by @Xuanwo in https://github.com/lance-format/lance/pull/7992 * docs: include lance-context in the full website build by @beinan in https://github.com/lance-format/lance/pull/8014 * docs: define legacy compatibility boundaries by @Xuanwo in https://github.com/lance-format/lance/pull/8039 ### Performance Improvements 🚀 * perf(encoding): cache complex-all-null levels in RLE run form and drain run-by-run by @Ali2Arslan in https://github.com/lance-format/lance/pull/7564 * perf(index): norm-addend cache and slim top-k heap for FTS scoring by @BubbleCal in https://github.com/lance-format/lance/pull/7629 * perf(encoding): compact mini-block chunk index for cached page state by @Ali2Arslan in https://github.com/lance-format/lance/pull/7651 * perf(index): list index segment files concurrently when committing segments by @xloya in https://github.com/lance-format/lance/pull/7657 * perf: plan take operations through FilteredReadExec's range-read path by @LuQQiu in https://github.com/lance-format/lance/pull/7672 * perf(encoding): avoid inline bitpacking input copy by @u70b3 in https://github.com/lance-format/lance/pull/7696 * perf(encoding): auto-select sparse structural pages by @Xuanwo in https://github.com/lance-format/lance/pull/7756 * perf(filtered-read): reuse plan-time fragment metadata for the read stream by @LuQQiu in https://github.com/lance-format/lance/pull/7792 * perf(dataset): read transactions by version without populating session caches by @xuanyu-z in https://github.com/lance-format/lance/pull/7817 * perf(fts): resolve deferred row_ids via the cached whole ROW_ID column by @LuQQiu in https://github.com/lance-format/lance/pull/7897 * perf(fts): search partitions in pipelined chunks by @LuQQiu in https://github.com/lance-format/lance/pull/7950 * perf(cache): opt-in quick_cache backend for contention-free reads by @LuQQiu in https://github.com/lance-format/lance/pull/7953 * perf: improve performance of CandidateBin.split_for_size by @fangbo in https://github.com/lance-format/lance/pull/8000 * perf(cache): use the quick_cache backend for the session metadata cache by @LuQQiu in https://github.com/lance-format/lance/pull/8013 * perf: avoid cloning every fragment descriptor in take and FTS planning by @LuQQiu in https://github.com/lance-format/lance/pull/8102 * perf(fts): pre-load doc lengths in parallel on cold deferred search path by @westonpace in https://github.com/lance-format/lance/pull/8119, backported in https://github.com/lance-format/lance/pull/8146 ### Other Changes * refactor(fts): separate document identity from row addresses by @Xuanwo in https://github.com/lance-format/lance/pull/7863 * refactor(mem-wal): rename flushed MemTable/generation to SSTable by @jackye1995 in https://github.com/lance-format/lance/pull/7943 * refactor: rename MemWAL compaction progress by @jackye1995 in https://github.com/lance-format/lance/pull/7957 * refactor(index): remove dead segment commit branch by @ddupg in https://github.com/lance-format/lance/pull/7971 * refactor: make v1 the canonical legacy file owner by @Xuanwo in https://github.com/lance-format/lance/pull/8020 * refactor: make encoding mechanisms version-free by @Xuanwo in https://github.com/lance-format/lance/pull/8021 * refactor: add exact current-format writers by @Xuanwo in https://github.com/lance-format/lance/pull/8022 * refactor(encoding): pass mini-block compression context by @Xuanwo in https://github.com/lance-format/lance/pull/8038 **Full Changelog**: https://github.com/lance-format/lance/compare/release-root/10.0.0-beta.N...v10.0.0

<!-- Release notes generated using configuration in .github/release.yml at v7.0.0-beta.14 --> ## What's Changed ### New Features 🎉 * feat(java): add Java bindings for MemWAL APIs by @touch-of-grey in https://github.com/lance-format/lance/pull/6833 **Full Changelog**: https://github.com/lance-format/lance/compare/v7.0.0-beta.13...v7.0.0-beta.14

<!-- Release notes generated using configuration in .github/release.yml at v7.0.0-beta.12 --> ## What's Changed ### New Features 🎉 * feat: builder-style MemWAL initialization API by @touch-of-grey in https://github.com/lance-format/lance/pull/6815 ### Bug Fixes 🐛 * fix(mem_wal): keep sealed memtables queryable until flush commits by @hamersaw in https://github.com/lance-format/lance/pull/6814 ### Documentation 📚 * docs: correct the suffix of wal file and unify the term "shard_id" by @jiengup in https://github.com/lance-format/lance/pull/6807 ### Other Changes * refactor: rename ShardSpec to ShardingSpec by @touch-of-grey in https://github.com/lance-format/lance/pull/6813 **Full Changelog**: https://github.com/lance-format/lance/compare/v7.0.0-beta.11...v7.0.0-beta.12

<!-- Release notes generated using configuration in .github/release.yml at v7.0.0-beta.10 --> ## What's Changed ### New Features 🎉 * feat: add Lance-native MemWAL HNSW for shard writer by @touch-of-grey in https://github.com/lance-format/lance/pull/6795 * feat(io): make shared-memory:// provider available outside tests by @hamersaw in https://github.com/lance-format/lance/pull/6805 ### Bug Fixes 🐛 * fix(index): fix range query bound inclusive-ness by @dentiny in https://github.com/lance-format/lance/pull/6796 **Full Changelog**: https://github.com/lance-format/lance/compare/v7.0.0-beta.9...v7.0.0-beta.10

<!-- Release notes generated using configuration in .github/release.yml at v7.0.0-beta.2 --> ## What's Changed ### Bug Fixes 🐛 * fix: always do wal tailer cursor update by @touch-of-grey in https://github.com/lance-format/lance/pull/6673 **Full Changelog**: https://github.com/lance-format/lance/compare/v7.0.0-beta.1...v7.0.0-beta.2

<!-- Release notes generated using configuration in .github/release.yml at v6.0.0-beta.3 --> ## What's Changed ### New Features 🎉 * feat(java): add Dataset.sample() API by @beinan in https://github.com/lance-format/lance/pull/6500 * feat: expose has_stable_row_ids property on LanceDataset by @pengw0048 in https://github.com/lance-format/lance/pull/6531 * feat: support zonemap index segments by @beinan in https://github.com/lance-format/lance/pull/6593 * feat: add ANNIvfPartitionExecProto by @LuQQiu in https://github.com/lance-format/lance/pull/6612 ### Bug Fixes 🐛 * fix: prevent arithmetic overflow in U64Segment encoding selection for sparse/extreme row id ranges by @ivscheianu in https://github.com/lance-format/lance/pull/6516 * fix: index type try_from miss RTree and BLOOMFILTER by @wojiaodoubao in https://github.com/lance-format/lance/pull/6568 ### Documentation 📚 * docs: correct repetition level example in encoding docs by @BubbleCal in https://github.com/lance-format/lance/pull/6585 **Full Changelog**: https://github.com/lance-format/lance/compare/v6.0.0-beta.2...v6.0.0-beta.3

<!-- Release notes generated using configuration in .github/release.yml at v5.1.0-beta.3 --> ## What's Changed ### Bug Fixes 🐛 * fix: include column_metadatas and column_infos in CachedFileMetadata::DeepSizeOf by @jiaoew1991 in https://github.com/lance-format/lance/pull/6480 * fix(index): preserve fts prewarm position codec by @BubbleCal in https://github.com/lance-format/lance/pull/6485 * fix: handle FlatBin quantization in optimize_vector_indices_v2 by @jackye1995 in https://github.com/lance-format/lance/pull/6488 **Full Changelog**: https://github.com/lance-format/lance/compare/v5.1.0-beta.2...v5.1.0-beta.3