[Bug]: Corruption in checkpoint parquet files
bugbinding/ruststorage/gcpawaiting-feedback
### What happened?
Hi, we are seeing the following error frequently when reading checkpoint files written by delta-rs
```
Kernel error: Arrow error: Arrow: Parquet argument error: Parquet error: Required field type_ is missing
```
we checked the file and saw it was corrupted
```
➜ parquet scan "test-checkpoint/deltafusion_TracingProjectId=76fde80e6a5a20839affad871_gateway_model_metrics__delta_log_00000000000000318199.checkpoint.parquet"
Unknown error
java.lang.RuntimeException: Failed on record 0 in test-checkpoint/deltafusion_TracingProjectId=76fde80e6a5a20839affad871_gateway_model_metrics__delta_log_00000000000000318199.checkpoint.parquet
at org.apache.parquet.cli.commands.ScanCommand.run(ScanCommand.java:75)
at org.apache.parquet.cli.Main.run(Main.java:169)
at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:76)
at org.apache.parquet.cli.Main.main(Main.java:197)
Caused by: java.lang.RuntimeException: Failed while reading Parquet file: test-checkpoint/deltafusion_TracingProjectId=76fde80e6a5a20839affad871_gateway_model_metrics__delta_log_00000000000000318199.checkpoint.parquet
at org.apache.parquet.cli.BaseCommand$1$1.advance(BaseCommand.java:360)
at org.apache.parquet.cli.BaseCommand$1$1.<init>(BaseCommand.java:337)
at org.apache.parquet.cli.BaseCommand$1.iterator(BaseCommand.java:335)
at org.apache.parquet.cli.commands.ScanCommand.run(ScanCommand.java:70)
... 3 more
Caused by: java.io.IOException: can not read class org.apache.parquet.format.PageHeader: Required field 'uncompressed_page_size' was not found in serialized data! Struct: org.apache.parquet.format.PageHeader$PageHeaderStandardScheme@17ae98d7
at org.apache.parquet.format.Util.read(Util.java:393)
at org.apache.parquet.format.Util.readPageHeader(Util.java:133)
at org.apache.parquet.hadoop.ParquetFileReader$Chunk.readPageHeader(ParquetFileReader.java:1897)
at org.apache.parquet.hadoop.ParquetFileReader$Chunk.readAllPages(ParquetFileReader.java:1959)
at org.apache.parquet.hadoop.ParquetFileReader$Chunk.readAllPages(ParquetFileReader.java:1920)
at org.apache.parquet.hadoop.ParquetFileReader.readChunkPages(ParquetFileReader.java:1454)
at org.apache.parquet.hadoop.ParquetFileReader.internalReadRowGroup(ParquetFileReader.java:1188)
at org.apache.parquet.hadoop.ParquetFileReader.readNextRowGroup(ParquetFileReader.java:1135)
at org.apache.parquet.hadoop.ParquetFileReader.readNextFilteredRowGroup(ParquetFileReader.java:1380)
at org.apache.parquet.hadoop.InternalParquetRecordReader.checkRead(InternalParquetRecordReader.java:140)
at org.apache.parquet.hadoop.InternalParquetRecordReader.nextKeyValue(InternalParquetRecordReader.java:245)
at org.apache.parquet.hadoop.ParquetReader.read(ParquetReader.java:136)
at org.apache.parquet.hadoop.ParquetReader.read(ParquetReader.java:140)
at org.apache.parquet.cli.BaseCommand$1$1.advance(BaseCommand.java:356)
... 6 more
Caused by: shaded.parquet.org.apache.thrift.protocol.TProtocolException: Required field 'uncompressed_page_size' was not found in serialized data! Struct: org.apache.parquet.format.PageHeader$PageHeaderStandardScheme@17ae98d7
at org.apache.parquet.format.PageHeader$PageHeaderStandardScheme.read(PageHeader.java:1085)
at org.apache.parquet.format.PageHeader$PageHeaderStandardScheme.read(PageHeader.java:995)
at org.apache.parquet.format.PageHeader.read(PageHeader.java:870)
at org.apache.parquet.format.Util.read(Util.java:390)
... 19 more
```
Not sure how to debug this any further, would appreciate any help. Thanks.
### Expected behavior
File should be readable
### Operating System
Linux
### Binding
Rust
### Bindings Version
v0.31.1
### Steps to reproduce
_No response_
### Relevant logs
```shell
```
10 条评论