ITADN

Core: ManifestFilterManager thread safety issue during parallel overwrite commit

#16978Closedliucao-dd 创建于 2026-06-27
bug
L
liucao-ddcommented
### Apache Iceberg version 1.10.2 Also appears present in 1.10.0, 1.10.1, 1.11.0, and main based on the current `ManifestFilterManager` implementation. ### Query engine Spark ### Please describe the bug 🐞 A Spark `OverwriteByExpression` / full-table overwrite can crash during the Iceberg commit phase because `ManifestFilterManager` mutates a shared, non-thread-safe `deleteFiles` set from multiple manifest-filtering worker threads. The crash observed was: ```text java.lang.ClassCastException: class java.util.LinkedHashMap$Entry cannot be cast to class java.util.HashMap$TreeNode at java.base/java.util.HashMap$TreeNode.moveRootToFront(HashMap.java:1986) at java.base/java.util.HashMap$TreeNode.putTreeVal(HashMap.java:2165) at java.base/java.util.HashMap.putVal(HashMap.java:636) at java.base/java.util.HashMap.put(HashMap.java:610) at java.base/java.util.HashSet.add(HashSet.java:221) at org.apache.iceberg.util.WrapperSet.add(WrapperSet.java:97) at org.apache.iceberg.util.DataFileSet.add(DataFileSet.java:26) at org.apache.iceberg.ManifestFilterManager.lambda$filterManifestWithDeletedFiles$3(ManifestFilterManager.java:518) at java.base/java.lang.Iterable.forEach(Iterable.java:75) at org.apache.iceberg.ManifestFilterManager.filterManifestWithDeletedFiles(ManifestFilterManager.java:490) at org.apache.iceberg.ManifestFilterManager.filterManifest(ManifestFilterManager.java:370) at org.apache.iceberg.ManifestFilterManager.lambda$filterManifests$0(ManifestFilterManager.java:224) at org.apache.iceberg.util.Tasks$Builder.runTaskWithRetry(Tasks.java:413) ... at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1136) ``` ManifestFilterManager.filterManifests processes manifests in parallel using the worker pool. For manifests that contain deleted files, filterManifestWithDeletedFiles adds copied files into the instance-level deleteFiles set: ``` deleteFiles.add(fileCopy); ``` That set is a DataFileSet, backed by WrapperSet, which uses a plain LinkedHashSet. Concurrent add calls can corrupt the underlying HashMap, which then surfaces as the LinkedHashMap$Entry cannot be cast to HashMap$TreeNode exception during bucket treeification. This is especially likely for full-table overwrites because the delete expression is effectively alwaysTrue(): every live data file in every matching manifest is added to the same shared set while manifests are being filtered in parallel. ### Willingness to contribute - [x] I can contribute a fix for this bug independently - [x] I would be willing to contribute a fix for this bug with guidance from the Iceberg community - [ ] I cannot contribute a fix for this bug at this time
关闭于 2026-07-04 1 条评论