ITADN

Add data-file byte totals to manifest_list entries, so table size can be read without scanning manifest bodies

#17743OpenHaoXuAI 创建于 9 天前
H
HaoXuAIcommented
### Feature Request / Improvement `manifest_list` entries summarize counts and rows, but not bytes: ``` added_files_count existing_files_count deleted_files_count added_rows_count existing_rows_count deleted_rows_count manifest_length <- the manifest file's own size, not the data it describes ``` Every `data_file` already carries `file_size_in_bytes`, so the number exists — it just isn't rolled up where it can be read cheaply. Getting a table's reachable bytes today means opening **every** manifest body and summing per entry, which is decompression-bound and almost entirely redundant: manifests are immutable and inherited largely unchanged between snapshots, so the same bytes are re-inflated every run. ## Proposal Three optional fields on `manifest_file`, mirroring the existing triplets: ``` added_files_size_in_bytes existing_files_size_in_bytes deleted_files_size_in_bytes ``` Zero write cost — the writer already holds every `file_size_in_bytes`. Backward compatible: absent means readers fall back to reading bodies. ## Why snapshot summary `total-files-size` doesn't cover it It gives one snapshot's total, and summing across live snapshots double counts files shared between them. Manifests are the natural dedup unit, since a manifest is inherited wholesale — dedup by `manifest_path`, then sum. That composes; per-snapshot totals don't. Useful well beyond one use case: cost attribution, compaction and retention planning, catalog-wide dashboards. #14803 and #14820 added delete-file size totals to `PartitionsTable`; this is the same idea in the manifest list, readable without a scan. I found no prior discussion (searched `added_files_size`, `manifest_list` + bytes, spec-labelled size proposals) — happy to be pointed at it. As a spec change this presumably wants a dev@ thread and spec PR; filing here to gauge support.
1 条评论