ITADN

[source-mongodb-v2] _id field schema discovery is extremely expensive

#80160Opensamsternatmiter 创建于 2026-06-17
type/bugarea/connectorscommunityneeds-triageteam/extensibilityautoteam
S
samsternatmitercommented
### Connector Name source-mongodb-v2 ### Connector Version 2.0.7 ### What step the error happened? During the sync ### Relevant information We are syncing a collection with tens of millions of documents. We noticed in our MongoDB logs that we had some queries taking **30 minutes**. It seems that it's this query: https://github.com/airbytehq/airbyte/blob/a8e626e457d88014d2236624751e787f6040cb5d/airbyte-integrations/connectors/source-mongodb-v2/src/main/java/io/airbyte/integrations/source/mongodb/InitialSnapshotHandler.java#L125 Here's a screenshot from our MongoDB Atlas <img width="423" height="643" alt="Image" src="https://github.com/user-attachments/assets/3f72f5f3-9e0d-491e-bd79-6e1703b0b849" /> It seems that the purpose of this query is to discover if any `_id` fields have a type other that `ObjectId`. In our case we are willing to guarantee that they don't. Could you introduce an option to skip this discovery? Doing a full collection scan is very expensive for this operation, it's unindexable and so it will always be slow. ### Relevant log output ```shell N/A ``` ### Contribute - [x] Yes, I want to contribute --- **Internal Tracking:** https://github.com/airbytehq/oncall/issues/12891
2 条评论