Core: ManifestFilterManager thread safety issue during parallel overwrite commit
bug
### Apache Iceberg version
1.10.2
Also appears present in 1.10.0, 1.10.1, 1.11.0, and main based on the current `ManifestFilterManager` implementation.
### Query engine
Spark
### Please describe the bug 🐞
A Spark `OverwriteByExpression` / full-table overwrite can crash during the Iceberg commit phase because `ManifestFilterManager` mutates a shared, non-thread-safe `deleteFiles` set from multiple manifest-filtering worker threads.
The crash observed was:
```text
java.lang.ClassCastException: class java.util.LinkedHashMap$Entry cannot be cast to
class java.util.HashMap$TreeNode
at java.base/java.util.HashMap$TreeNode.moveRootToFront(HashMap.java:1986)
at java.base/java.util.HashMap$TreeNode.putTreeVal(HashMap.java:2165)
at java.base/java.util.HashMap.putVal(HashMap.java:636)
at java.base/java.util.HashMap.put(HashMap.java:610)
at java.base/java.util.HashSet.add(HashSet.java:221)
at org.apache.iceberg.util.WrapperSet.add(WrapperSet.java:97)
at org.apache.iceberg.util.DataFileSet.add(DataFileSet.java:26)
at org.apache.iceberg.ManifestFilterManager.lambda$filterManifestWithDeletedFiles$3(ManifestFilterManager.java:518)
at java.base/java.lang.Iterable.forEach(Iterable.java:75)
at org.apache.iceberg.ManifestFilterManager.filterManifestWithDeletedFiles(ManifestFilterManager.java:490)
at org.apache.iceberg.ManifestFilterManager.filterManifest(ManifestFilterManager.java:370)
at org.apache.iceberg.ManifestFilterManager.lambda$filterManifests$0(ManifestFilterManager.java:224)
at org.apache.iceberg.util.Tasks$Builder.runTaskWithRetry(Tasks.java:413)
...
at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1136)
```
ManifestFilterManager.filterManifests processes manifests in parallel using the worker pool. For manifests that contain deleted files, filterManifestWithDeletedFiles adds copied files into the instance-level deleteFiles set:
```
deleteFiles.add(fileCopy);
```
That set is a DataFileSet, backed by WrapperSet, which uses a plain LinkedHashSet. Concurrent add calls can corrupt the underlying HashMap, which then surfaces as the LinkedHashMap$Entry cannot be cast to HashMap$TreeNode exception during bucket treeification.
This is especially likely for full-table overwrites because the delete expression is effectively alwaysTrue(): every live data file in every matching manifest is added to the same shared set while manifests are being filtered in parallel.
### Willingness to contribute
- [x] I can contribute a fix for this bug independently
- [x] I would be willing to contribute a fix for this bug with guidance from the Iceberg community
- [ ] I cannot contribute a fix for this bug at this time
关闭于 2026-07-04 1 条评论