ITADN

[Bug]: Corruption in checkpoint parquet files

#4433OpenGauravWaghmare 创建于 2026-05-07
bugbinding/ruststorage/gcpawaiting-feedback
G
GauravWaghmarecommented
### What happened? Hi, we are seeing the following error frequently when reading checkpoint files written by delta-rs ``` Kernel error: Arrow error: Arrow: Parquet argument error: Parquet error: Required field type_ is missing ``` we checked the file and saw it was corrupted ``` ➜ parquet scan "test-checkpoint/deltafusion_TracingProjectId=76fde80e6a5a20839affad871_gateway_model_metrics__delta_log_00000000000000318199.checkpoint.parquet" Unknown error java.lang.RuntimeException: Failed on record 0 in test-checkpoint/deltafusion_TracingProjectId=76fde80e6a5a20839affad871_gateway_model_metrics__delta_log_00000000000000318199.checkpoint.parquet at org.apache.parquet.cli.commands.ScanCommand.run(ScanCommand.java:75) at org.apache.parquet.cli.Main.run(Main.java:169) at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:76) at org.apache.parquet.cli.Main.main(Main.java:197) Caused by: java.lang.RuntimeException: Failed while reading Parquet file: test-checkpoint/deltafusion_TracingProjectId=76fde80e6a5a20839affad871_gateway_model_metrics__delta_log_00000000000000318199.checkpoint.parquet at org.apache.parquet.cli.BaseCommand$1$1.advance(BaseCommand.java:360) at org.apache.parquet.cli.BaseCommand$1$1.<init>(BaseCommand.java:337) at org.apache.parquet.cli.BaseCommand$1.iterator(BaseCommand.java:335) at org.apache.parquet.cli.commands.ScanCommand.run(ScanCommand.java:70) ... 3 more Caused by: java.io.IOException: can not read class org.apache.parquet.format.PageHeader: Required field 'uncompressed_page_size' was not found in serialized data! Struct: org.apache.parquet.format.PageHeader$PageHeaderStandardScheme@17ae98d7 at org.apache.parquet.format.Util.read(Util.java:393) at org.apache.parquet.format.Util.readPageHeader(Util.java:133) at org.apache.parquet.hadoop.ParquetFileReader$Chunk.readPageHeader(ParquetFileReader.java:1897) at org.apache.parquet.hadoop.ParquetFileReader$Chunk.readAllPages(ParquetFileReader.java:1959) at org.apache.parquet.hadoop.ParquetFileReader$Chunk.readAllPages(ParquetFileReader.java:1920) at org.apache.parquet.hadoop.ParquetFileReader.readChunkPages(ParquetFileReader.java:1454) at org.apache.parquet.hadoop.ParquetFileReader.internalReadRowGroup(ParquetFileReader.java:1188) at org.apache.parquet.hadoop.ParquetFileReader.readNextRowGroup(ParquetFileReader.java:1135) at org.apache.parquet.hadoop.ParquetFileReader.readNextFilteredRowGroup(ParquetFileReader.java:1380) at org.apache.parquet.hadoop.InternalParquetRecordReader.checkRead(InternalParquetRecordReader.java:140) at org.apache.parquet.hadoop.InternalParquetRecordReader.nextKeyValue(InternalParquetRecordReader.java:245) at org.apache.parquet.hadoop.ParquetReader.read(ParquetReader.java:136) at org.apache.parquet.hadoop.ParquetReader.read(ParquetReader.java:140) at org.apache.parquet.cli.BaseCommand$1$1.advance(BaseCommand.java:356) ... 6 more Caused by: shaded.parquet.org.apache.thrift.protocol.TProtocolException: Required field 'uncompressed_page_size' was not found in serialized data! Struct: org.apache.parquet.format.PageHeader$PageHeaderStandardScheme@17ae98d7 at org.apache.parquet.format.PageHeader$PageHeaderStandardScheme.read(PageHeader.java:1085) at org.apache.parquet.format.PageHeader$PageHeaderStandardScheme.read(PageHeader.java:995) at org.apache.parquet.format.PageHeader.read(PageHeader.java:870) at org.apache.parquet.format.Util.read(Util.java:390) ... 19 more ``` Not sure how to debug this any further, would appreciate any help. Thanks. ### Expected behavior File should be readable ### Operating System Linux ### Binding Rust ### Bindings Version v0.31.1 ### Steps to reproduce _No response_ ### Relevant logs ```shell ```
10 条评论