[Feature]: Support committing AddAction and RemoveAction together in Python
help wantedbinding/python
### Is your feature request related to a problem?
## Feature Request
Expose a Python transaction API that can commit `AddAction` and `RemoveAction` together in a single Delta transaction.
## Use Case
I am building a Python service that ingests CSV logs into Delta Lake tables.
For append-only writes, `DeltaTable.create_write_transaction()` works well because I can write Parquet files myself and commit the generated `AddAction`s.
However, for custom compaction / file rewrite workflows, I need to atomically commit both:
- `AddAction`s for newly compacted Parquet files
- `RemoveAction`s for old small files being replaced
Currently the Python API appears to accept only `AddAction`s:
```python
DeltaTable.create_write_transaction(
actions: list[AddAction],
...
)
```
This is enough for append-only writes, but not for compaction. If old files cannot be removed in the same transaction, queries will read both old and new files and produce duplicate data.
### Describe the solution you'd like
## Proposed Solution
Expose a more general Python transaction API, or extend the existing one, to allow committing both add and remove actions atomically:
```python
table.create_transaction(
actions=[
AddAction(...),
RemoveAction(...),
],
operation="OPTIMIZE",
)
```
The exact API name is not important. The key requirement is the ability to commit add and remove file actions atomically through delta-rs.
## Alternatives
- `DeltaTable.optimize.compact()` works for built-in compaction, but does not support custom rewrite pipelines.
- Manually writing `_delta_log` JSON files is unsafe because it bypasses delta-rs transaction handling, conflict checks, protocol validation, and commit hooks.
### Describe alternatives you've considered
_No response_
### Priority
None
### Additional context
_No response_
### Contribution
- [ ] I'm willing to submit a pull request for this feature
- [ ] I can help with testing this feature
- [ ] I can help with documentation for this feature
0 条评论