[source-salesforce] Rotating refresh tokens: operation config snapshotted at creation goes stale when a concurrent or preceding operation rotates the token (Salesforce RTR)

### Connector Name source-salesforce ### Connector Version 2.8.0 (platform 0.64.7) ### What step the error happened? During the sync ### Relevant information Follow-up to #80783, which was resolved by #80892. That change works: source-salesforce now captures the rotated refresh token in `login()` and emits a `CONNECTOR_CONFIG` control message, and the platform persists it. With Salesforce Refresh Token Rotation (RTR) enabled we can confirm the persistence path itself is functioning. What we are still seeing is a timing problem around that persistence. Operations appear to be hydrated with a snapshot of the connector config taken when the operation is created, not when it executes. Under RTR every refresh token is single use, so any rotation that is persisted between snapshot time and execution time leaves the executing operation holding a token that Salesforce has already invalidated. The result is `400 invalid_grant: expired access/refresh token` on an operation whose sibling operation succeeded seconds earlier. This report covers two distinct defects. They compound each other, so we are filing them together. **Defect 1 (platform): operation inputs are hydrated with a config snapshot taken at creation time rather than at execution time.** **Defect 2 (source-salesforce): `invalid_grant` from the token endpoint is treated as retryable and is retried for over 38 minutes, when it is a non-retryable 400 that should fail fast as a config error.** #### Environment - Self-hosted Airbyte OSS, platform 0.64.7, running on GKE. - Official `airbyte/source-salesforce:2.8.0` image. - Salesforce connected app with Refresh Token Rotation enabled. Every redemption of a `refresh_token` grant issues a new refresh token and immediately invalidates the previous one. #### Preconditions to reproduce 1. A Salesforce connected app with Refresh Token Rotation enabled, so refresh tokens are single use. 2. A source-salesforce source on 2.8.0 using OAuth (client id, client secret, refresh token). 3. Two operations against the same source that overlap or run close together, for example a discoverSchema or a check running while or shortly before a sync job is created. Regular scheduled syncs on a busy instance produce this naturally. #### Observed behavior We saw two shapes of the same underlying problem. All timestamps below are UTC on 2026-07-27. **Shape A, within a single job (job id 3175628).** 1. At 16:50:48 the check pod `source-salesforce-check-3175628-0-ksnsb` succeeded, and the platform logged the config update: ``` ConnectorMessageProcessor(updateConfigFromControlMessage):235 - Optional control message present. Updating... ``` So the rotation was persisted correctly at this point. 2. The replication pod for the same job, `replication-job-3175628-attempt-0`, then ran with the pre-rotation refresh token and received `400 invalid_grant: expired access/refresh token` from `https://login.salesforce.com/services/oauth2/token`. 3. The connector then retried the token POST at roughly 43 requests per minute. We captured at least 1,662 rejected requests over at least 38 minutes before the job failed, with zero successes. That count is a floor, not a ceiling, because our log query truncated at 2,000 events. **Shape B, across operations (the majority of our failures: 40 or more sync jobs on 2026-07-27).** 1. A discoverSchema or check operation redeems the refresh token, rotating it, and the platform persists the new one. 2. A sync job created around the same time was hydrated with the pre-rotation config. Its check-before-sync step fails with `invalid_grant: expired access/refresh token`, surfaced as "Checking source connection failed". 3. The gap between the successful sibling operation and the failure was 15 to 80 seconds in the cases we measured. #### Recovery behavior The next scheduled job on the same connection typically succeeds, using the persisted rotated token. Connections therefore flap rather than break permanently: one job fails, the next succeeds. Each flap is still a failed sync and still fires failure notifications, so on a busy instance the noise is significant. #### Expected behavior 1. An operation should use the latest persisted connector config at the moment it executes. If a control message updates the config partway through a job, later steps of that job should pick up the updated config rather than continuing with the snapshot the job started with. 2. `invalid_grant` on the token endpoint should fail immediately as a configuration error. It is not a transient condition and retrying cannot succeed. Retrying it for 38 minutes at 43 requests per minute delays the failure signal, wastes a worker slot, and puts avoidable load on the identity provider, with a real risk of provider side rate limiting or throttling. #### Hypothesis We want to be clear that this is our reading of the observed behavior and not a claim about platform internals, which we have not read. The pattern that fits everything we saw is that config is snapshotted into the operation input when the operation is created, and is not re-read at execution time. With non-rotating refresh tokens this is harmless, because the snapshotted token stays valid. With RTR the snapshot has a lifetime measured in seconds: as soon as any other operation redeems the token, every outstanding snapshot holding the old token is dead. Shape A additionally suggests that even a control message update applied within a job does not propagate to the later steps of that same job. If the real mechanism is different, the observable outcome we care about is the same: with single use refresh tokens, an operation can be handed a token that was already invalidated before the operation started. #### Asks 1. Platform: hydrate replication and check inputs with the latest persisted config at execution time, or re-hydrate after a control message update lands mid-job. 2. source-salesforce: treat `invalid_grant` from the token endpoint as a non-retryable 400 and fail fast as a config error instead of retrying. We are filing both here since airbyte-platform does not accept issues. Happy to provide more log detail or to test a patch. ### Relevant log output ```shell # Platform, job 3175628, after check pod source-salesforce-check-3175628-0-ksnsb succeeded at 16:50:48 UTC ConnectorMessageProcessor(updateConfigFromControlMessage):235 - Optional control message present. Updating... # Same job, replication pod replication-job-3175628-attempt-0, using the pre-rotation refresh token POST https://login.salesforce.com/services/oauth2/token 400 {"error":"invalid_grant","error_description":"expired access/refresh token"} # repeated at ~43 requests/minute, at least 1,662 rejected requests over at least 38 minutes, zero successes # Shape B, check-before-sync on a job created near a successful sibling discoverSchema/check Checking source connection failed {"error":"invalid_grant","error_description":"expired access/refresh token"} ``` ### Contribute - [ ] Yes, I want to contribute --- **Internal Tracking:** https://github.com/airbytehq/oncall/issues/13169