ITADN

Socket-based syncs blocked by AKS ContainerNoPrivilegeEscalation Policy

#83782OpenJeremyEinfeld 创建于 13 天前
type/bugarea/platformcommunityneeds-triageteam/composeteam/platform-moveautoteam
J
JeremyEinfeldcommented
### Helm Chart Version 2.0.17 ### What step the error happened? During the Sync ### Relevant information Following an update of the Databricks destination to 4.0.1, connections from sources supporting `SOCKET` transport began failing with this error during the sync step: ``` ERROR Stage Pipeline Exception: io.airbyte.workload.launcher.pipeline.stages.model.StageError: io.airbyte.workers.exception.KubeClientException: Failed to create pod replication-job-15469-attempt-0. message: io.airbyte.workers.exception.KubeClientException: Failed to create pod replication-job-15469-attempt-0. stackTrace: [Ljava.lang.StackTraceElement;@5e61742d ``` Sources that only supported `STDOI` continued to work. ### Debugging The failure was traced to an AKS gatekeeper denial for a ContainerNoPrivilegeEscalation policy violation. I was able to repro this failure using a security context based on [the `securityContext` created](https://github.com/airbytehq/airbyte-platform/blob/2d96861799e85bec7ddc9a60cccd93866bd549e2/airbyte-workload-launcher/src/main/kotlin/pods/factories/ReplicationPodFactory.kt#L125-L126) when the `SOCKET` transport is selected: ``` securityContext: runAsUser: 1000 runAsGroup: 1000 fsGroup: 1000 fsGroupChangePolicy: OnRootMismatch runAsNonRoot: true seccompProfile: type: RuntimeDefault ``` When tested against the AKS gatekeeper: ``` PS > kubectl create --dry-run=server -f "socket-dryrun-pod.yaml" 2>&1 Error from server (Forbidden): error when creating "socket-dryrun-pod.yaml": admission webhook "validation.gatekeeper.sh" denied the request: [azurepolicy-k8sazurev3noprivilegeescalatio-f398463faa240def6d38] Privilege escalation container is not allowed: destination [azurepolicy-k8sazurev3noprivilegeescalatio-f398463faa240def6d38] Privilege escalation container is not allowed: init [azurepolicy-k8sazurev3noprivilegeescalatio-f398463faa240def6d38] Privilege escalation container is not allowed: orchestrator [azurepolicy-k8sazurev3noprivilegeescalatio-f398463faa240def6d38] Privilege escalation container is not allowed: source ``` The ContainerNoPrivilegeEscalation policy (https://github.com/Azure/azure-policy/blob/f600c952718e8d2fc1c4fe92c516100d2ad85999/built-in-policies/policyDefinitions/Kubernetes/ContainerNoPrivilegeEscalation.json#L169) is set to block `securityContext` configs that use any value other than `false` if they set a `runAsUser` value other than zero: ``` PS > kubectl get constrainttemplate k8sazurev3noprivilegeescalation -o jsonpath='{.spec.targets[0].rego}' package k8sazurev3noprivilegeescalation import data.lib.exclude_container_image.is_excluded # The following scenerios will be viewed as violations. Windows containers are not supported ## allowPrivilegeEscalation=true ## runAsUser != 0 + missing allowPrivilegeEscalation=false violation[{"msg": msg, "details": {}}] { not input.review.object.spec.nodeSelector["kubernetes.io/os"] == "windows" c := input_containers[_] not input_container_excluded(c.name) not is_excluded(c) input_allow_privilege_escalation(c) msg := sprintf("Privilege escalation container is not allowed: %v", [c.name]) } input_allow_privilege_escalation(c) { c.securityContext.allowPrivilegeEscalation == true } input_allow_privilege_escalation(c) { not c.securityContext.allowPrivilegeEscalation == false c.securityContext.runAsUser != 0 } input_allow_privilege_escalation(c) { not c.securityContext.allowPrivilegeEscalation == false not c.securityContext.runAsUser input.review.object.spec.securityContext.runAsUser != 0 } input_containers[c] { c := input.review.object.spec.containers[_] } input_containers[c] { c := input.review.object.spec.initContainers[_] } input_containers[c] { c := input.review.object.spec.ephemeralContainers[_] } input_container_excluded(field) { field == input.parameters.excludedContainers[_] } ``` ### Solutions Given an updated `securityContext` value with `allowPrivilegeEscalation: false`, the gatekeeper dry-run permitted the pod creation. This is consistent with the definition for [`WorkloadSecurityContextProvider.baseContainerSecurityContext()`](https://github.com/airbytehq/airbyte-platform/blob/v1.8.0/airbyte-workload-launcher/src/main/kotlin/context/WorkloadSecurityContextProvider.kt#L104-L114): ``` private fun baseContainerSecurityContext( user: Long, group: Long, ): SecurityContextBuilder = SecurityContextBuilder() .withRunAsUser(user) .withRunAsGroup(group) .withAllowPrivilegeEscalation(false) .withReadOnlyRootFilesystem(false) .withCapabilities(CapabilitiesBuilder().addAllToDrop(DEFAULT_CAPABILITIES).build()) } ``` The current workaround is to use an AKS cluster mutation policy to inject `allowPrivilegeEscalation: false`, but ideally Airbyte should work out of the box with the ContainerNoPrivilegeEscalation policy in effect? The issue reported at https://github.com/airbytehq/airbyte/issues/77613 is partially related, but in that case the `securityContext` is reported as an empty `{}` which is what we're seeing for `STDIO` sources without a gatekeeper failure. ### Relevant log output ```shell 2026-08-06 14:18:02 platform INFO APPLY Stage: BUILD — (workloadId=c20f60e7-c0e9-440f-8795-73d841838c28_15469_0_sync) 2026-08-06 14:18:02 platform INFO APPLY Stage: CLAIM — (workloadId=c20f60e7-c0e9-440f-8795-73d841838c28_15469_0_sync) 2026-08-06 14:18:02 platform INFO Claimed: true for workload c20f60e7-c0e9-440f-8795-73d841838c28_15469_0_sync via API in dataplane AUTO3fbd01c3-f8e0-4107-9f4a-dd14270550a1 (3cb4b628-3345-4590-bd66-ddf751089fd8) 2026-08-06 14:18:02 platform INFO APPLY Stage: LOAD_SHED — (workloadId=c20f60e7-c0e9-440f-8795-73d841838c28_15469_0_sync) 2026-08-06 14:18:02 platform INFO APPLY Stage: CHECK_STATUS — (workloadId=c20f60e7-c0e9-440f-8795-73d841838c28_15469_0_sync) 2026-08-06 14:18:02 platform INFO No pod found running for workload c20f60e7-c0e9-440f-8795-73d841838c28_15469_0_sync 2026-08-06 14:18:02 platform INFO APPLY Stage: MUTEX — (workloadId=c20f60e7-c0e9-440f-8795-73d841838c28_15469_0_sync) 2026-08-06 14:18:02 platform INFO Mutex key: c20f60e7-c0e9-440f-8795-73d841838c28 specified for workload: c20f60e7-c0e9-440f-8795-73d841838c28_15469_0_sync. Attempting to delete existing pods... 2026-08-06 14:18:02 platform INFO Mutex key: c20f60e7-c0e9-440f-8795-73d841838c28 specified for workload: c20f60e7-c0e9-440f-8795-73d841838c28_15469_0_sync found no existing pods. Continuing... 2026-08-06 14:18:02 platform INFO APPLY Stage: ARCHITECTURE — (workloadId=c20f60e7-c0e9-440f-8795-73d841838c28_15469_0_sync) 2026-08-06 14:18:02 platform INFO APPLY Stage: LAUNCH — (workloadId=c20f60e7-c0e9-440f-8795-73d841838c28_15469_0_sync) 2026-08-06 14:18:02 platform INFO [initContainer] image: airbyte/workload-init-container:1.8.5 resources: ResourceRequirements(claims=[], limits={memory=1024Mi, cpu=1}, requests={memory=1024Mi, cpu=1}, additionalProperties={}) 2026-08-06 14:18:02 platform INFO Launching replication pod: replication-job-15469-attempt-0 (selectors = {workload_type=jobs}) with containers: 2026-08-06 14:18:02 platform INFO [source] image: airbyte/source-mysql:3.51.1 resources: ResourceRequirements(claims=[], limits={memory=4Gi, cpu=1}, requests={memory=2Gi, cpu=0.5}, additionalProperties={}) 2026-08-06 14:18:02 platform INFO [destination] image: airbyte/destination-databricks:4.0.1 resources: ResourceRequirements(claims=[], limits={memory=4Gi, cpu=1}, requests={memory=2Gi, cpu=0.5}, additionalProperties={}) 2026-08-06 14:18:02 platform INFO [orchestrator] image: airbyte/container-orchestrator:1.8.5 resources: ResourceRequirements(claims=[], limits={memory=1024Mi, cpu=1}, requests={memory=1024Mi, cpu=1}, additionalProperties={}) 2026-08-06 14:18:02 platform ERROR Stage Pipeline Exception: io.airbyte.workload.launcher.pipeline.stages.model.StageError: io.airbyte.workers.exception.KubeClientException: Failed to create pod replication-job-15469-attempt-0. message: io.airbyte.workers.exception.KubeClientException: Failed to create pod replication-job-15469-attempt-0. stackTrace: [Ljava.lang.StackTraceElement;@5e61742d 2026-08-06 14:18:02 platform INFO Attempting to update workload: c20f60e7-c0e9-440f-8795-73d841838c28_15469_0_sync to FAILED. 2026-08-06 14:18:02 platform INFO Pipeline aborted after error for workload: c20f60e7-c0e9-440f-8795-73d841838c28_15469_0_sync. 2026-08-06 14:18:03 platform INFO ----- START POST REPLICATION OPERATIONS ----- 2026-08-06 14:18:03 platform INFO No post-replication operation(s) to perform. 2026-08-06 14:18:03 platform INFO ----- END POST REPLICATION OPERATIONS ----- ``` --- **Internal Tracking:** https://github.com/airbytehq/oncall/issues/12122
2 条评论