ITADN

[Feature]: Support committing AddAction and RemoveAction together in Python

#4480Openlukoou3 创建于 2026-05-25
help wantedbinding/python
L
lukoou3commented
### Is your feature request related to a problem? ## Feature Request Expose a Python transaction API that can commit `AddAction` and `RemoveAction` together in a single Delta transaction. ## Use Case I am building a Python service that ingests CSV logs into Delta Lake tables. For append-only writes, `DeltaTable.create_write_transaction()` works well because I can write Parquet files myself and commit the generated `AddAction`s. However, for custom compaction / file rewrite workflows, I need to atomically commit both: - `AddAction`s for newly compacted Parquet files - `RemoveAction`s for old small files being replaced Currently the Python API appears to accept only `AddAction`s: ```python DeltaTable.create_write_transaction( actions: list[AddAction], ... ) ``` This is enough for append-only writes, but not for compaction. If old files cannot be removed in the same transaction, queries will read both old and new files and produce duplicate data. ### Describe the solution you'd like ## Proposed Solution Expose a more general Python transaction API, or extend the existing one, to allow committing both add and remove actions atomically: ```python table.create_transaction( actions=[ AddAction(...), RemoveAction(...), ], operation="OPTIMIZE", ) ``` The exact API name is not important. The key requirement is the ability to commit add and remove file actions atomically through delta-rs. ## Alternatives - `DeltaTable.optimize.compact()` works for built-in compaction, but does not support custom rewrite pipelines. - Manually writing `_delta_log` JSON files is unsafe because it bypasses delta-rs transaction handling, conflict checks, protocol validation, and commit hooks. ### Describe alternatives you've considered _No response_ ### Priority None ### Additional context _No response_ ### Contribution - [ ] I'm willing to submit a pull request for this feature - [ ] I can help with testing this feature - [ ] I can help with documentation for this feature
0 条评论