Add `getFragmentStatistics()` to the Java `blocking_dataset` API to expose per-fragment statistics
### What's changed
Adds a metadata-only bulk API that returns statistics for every fragment in the current dataset version as three parallel primitive arrays:
```java
FragmentStatistics stats = dataset.getFragmentStatistics();
```
### Why these fields matter (Spark query planning as an example)
These scalars are exactly what a distributed engine needs at plan time, and every one of them requires a full-table, per-fragment enumeration — including but not limited to `LanceSplit.planScan()`, `LanceScanBuilder`, and `pushOffset()`.
### Why iterating `getFragments()` can't meet this need (performance)
`getFragments()` materializes the complete metadata object graph for **every** fragment across the JNI boundary: the `files` list (each `DataFile` with its path string and field-id arrays), the deletion file, `rowIdMeta`, boxed row counts, and so on — 5+ JNI calls and dozens er fragment. The planner then reads exactly two numbers out of it, `(id, numRows)`, and throws away the other 99% of what was materialized.
Measured on a table with 800K fragments: a single `getFragments()` call takes **~8 seconds** (millions of JNI boundary crossings plus GC pressure), and one Spark planning pass triggers 2–3 such calls — **~24 seconds** of driver time for this alone. Note that the fragment liste decoded manifest: the cost isn't fetching the data, it's object conversion nobody asked for.
The new API compresses the same information into one O(n) in-memory pass plus a single bulk JNI array copy (~19 MB for 800K fragments), measured in **milliseconds**. The larger the metadata, the bigger the win.
0 条评论