ITADN

Add `getFragmentStatistics()` to the Java `blocking_dataset` API to expose per-fragment statistics

#8071Openzhangyue19921010 创建于 2026-07-29
### What's changed Adds a metadata-only bulk API that returns statistics for every fragment in the current dataset version as three parallel primitive arrays: ```java FragmentStatistics stats = dataset.getFragmentStatistics(); ``` ### Why these fields matter (Spark query planning as an example) These scalars are exactly what a distributed engine needs at plan time, and every one of them requires a full-table, per-fragment enumeration — including but not limited to `LanceSplit.planScan()`, `LanceScanBuilder`, and `pushOffset()`. ### Why iterating `getFragments()` can't meet this need (performance) `getFragments()` materializes the complete metadata object graph for **every** fragment across the JNI boundary: the `files` list (each `DataFile` with its path string and field-id arrays), the deletion file, `rowIdMeta`, boxed row counts, and so on — 5+ JNI calls and dozens er fragment. The planner then reads exactly two numbers out of it, `(id, numRows)`, and throws away the other 99% of what was materialized. Measured on a table with 800K fragments: a single `getFragments()` call takes **~8 seconds** (millions of JNI boundary crossings plus GC pressure), and one Spark planning pass triggers 2–3 such calls — **~24 seconds** of driver time for this alone. Note that the fragment liste decoded manifest: the cost isn't fetching the data, it's object conversion nobody asked for. The new API compresses the same information into one O(n) in-memory pass plus a single bulk JNI array copy (~19 MB for 800K fragments), measured in **milliseconds**. The larger the metadata, the bigger the win.
0 条评论