[Python][Parquet] Expose SizeStatisticsLevel in the Parquet writer API
Type: enhancementComponent: ParquetComponent: Python
### Describe the enhancement requested
The Parquet C++ library already lets you control how much [`SizeStatistics`](https://github.com/apache/parquet-format/blob/master/src/main/thrift/parquet.thrift) metadata is written, via `WriterProperties::Builder::set_size_statistics_level()` (added in #40594). The available levels are:
- `None` — write no size statistics
- `ColumnChunk` — write size statistics only at the column-chunk level (`ColumnMetaData.size_statistics`)
- `PageAndColumnChunk` — also write page-level histograms into `ColumnIndex` / `OffsetIndex` (**this is the C++ default**)
However, **PyArrow does not currently expose this setting.** There is no way from `pyarrow.parquet.write_table`, `ParquetWriter`, or the dataset writer to change the size-statistics level, so every file written by PyArrow always includes `SizeStatistics` at the `PageAndColumnChunk` level.
We have hit one concrete problem with it being unconditionally written:
**Reader compatibility.** Some third-party / older Parquet readers do not handle the newer `SizeStatistics` thrift fields gracefully and fail to read files that PyArrow produces. Being able to write with `size_statistics_level="none"` produces files those readers can consume, without having to downgrade any other writer settings.
The C++ capability exists and is stable; this request is purely to surface it through the Python binding layer.
### Component(s)
Python, Parquet
0 条评论