KubeVirt CRD deployment stuck in deploying state on AKS

<!-- Make sure that you visit our User Guide at https://kubevirt.io/user-guide. --> **What happened**: Steps to create deployment for KubeVirt custom resource definition sits in deploying phase with pods giving node affinity selector error with reason that seems to be incorrect. Looking to find the reason this pod deployment is failing even with affinity selector satisfied. There is only one selector , Node-Selectors kubernetes.io/os=linux, which is fulfilled on all nodes. Node used for userpool supports nested virtualization : https://docs.azure.cn/en-us/virtual-machines/sizes/general-purpose/dv3-series?tabs=sizebasic#dsv3-series kubectl get nodes NAME STATUS ROLES AGE VERSION aks-agentpool-10736312-vmss000000 Ready <none> 79m v1.32.9 aks-agentpool-10736312-vmss000001 Ready <none> 79m v1.32.9 **aks-userpool1-10736312-vmss000000 Ready <none> 79m v1.32.9** kubectl get pods -n kubevirt NAME READY STATUS RESTARTS AGE **kubevirt-bce890c14e61c30622a246d83327c952e1047e9e-jobzsx6zpj8v4 0/1 Pending 0 59m** virt-operator-7c8bdfb574-hmzqf 1/1 Running 0 60m virt-operator-7c8bdfb574-r6kbk 1/1 Running 0 60m kubectl describe pod kubevirt-bce890c14e61c30622a246d83327c952e1047e9e-jobzsx6zpj8v4 -n kubevirt Name: kubevirt-bce890c14e61c30622a246d83327c952e1047e9e-jobzsx6zpj8v4 Namespace: kubevirt Priority: 0 Service Account: kubevirt-operator Node: <none> Labels: batch.kubernetes.io/controller-uid=20e8f405-37eb-49a6-a3fb-b01fc8e12dd0 batch.kubernetes.io/job-name=kubevirt-bce890c14e61c30622a246d83327c952e1047e9e-jobzsx6z controller-uid=20e8f405-37eb-49a6-a3fb-b01fc8e12dd0 job-name=kubevirt-bce890c14e61c30622a246d83327c952e1047e9e-jobzsx6z kubevirt.io=virt-operator-strategy-dumper Annotations: <none> Status: Pending IP: IPs: <none> Controlled By: Job/kubevirt-bce890c14e61c30622a246d83327c952e1047e9e-jobzsx6z Containers: install-strategy-upload: Image: quay.io/kubevirt/virt-operator:v1.6.3 Port: <none> Host Port: <none> Command: virt-operator --dump-install-strategy Environment: VIRT_OPERATOR_IMAGE: quay.io/kubevirt/virt-operator:v1.6.3 TARGET_INSTALL_NAMESPACE: kubevirt TARGET_IMAGE_PULL_POLICY: IfNotPresent TARGET_DEPLOYMENT_CONFIG: {"id":"bce890c14e61c30622a246d83327c952e1047e9e","namespace":"kubevirt","registry":"quay.io/kubevirt","kubeVirtVersion":"v1.6.3","virtOperatorImage":"quay.io/kubevirt/virt-operator:v1.6.3","additionalProperties":{"CertificateRotationStrategy":"\u003cv1.KubeVirtCertificateRotateStrategy Value\u003e","Configuration":"\u003cv1.KubeVirtConfiguration Value\u003e","CustomizeComponents":"\u003cv1.CustomizeComponents Value\u003e","ImagePullPolicy":"IfNotPresent","ImagePullSecrets":"null","Infra":"\u003c*v1.ComponentConfig Value\u003e","MonitorAccount":"","MonitorNamespace":"","ProductComponent":"","ProductName":"","ProductVersion":"","ServiceMonitorNamespace":"","SynchronizationPort":"","UninstallStrategy":"","WorkloadUpdateStrategy":"\u003cv1.KubeVirtWorkloadUpdateStrategy Value\u003e","Workloads":"\u003c*v1.ComponentConfig Value\u003e"}} Mounts: /var/run/secrets/kubernetes.io/serviceaccount from kube-api-access-9lhrr (ro) Conditions: Type Status PodScheduled False Volumes: kube-api-access-9lhrr: Type: Projected (a volume that contains injected data from multiple sources) TokenExpirationSeconds: 3607 ConfigMapName: kube-root-ca.crt Optional: false DownwardAPI: true QoS Class: BestEffort **Node-Selectors: kubernetes.io/os=linux** Tolerations: node-role.kubernetes.io/control-plane:NoSchedule op=Exists node-role.kubernetes.io/master:NoSchedule op=Exists node.kubernetes.io/not-ready:NoExecute op=Exists for 300s node.kubernetes.io/unreachable:NoExecute op=Exists for 300s **Events: Type Reason Age From Message ---- ------ ---- ---- ------- Warning FailedScheduling 2m52s (x13 over 63m) default-scheduler 0/3 nodes are available: 3 node(s) didn't match Pod's node affinity/selector. preemption: 0/3 nodes are available: 3 Preemption is not helpful for scheduling. Normal NotTriggerScaleUp 2m36s (x361 over 62m) cluster-autoscaler pod didn't trigger scale-up: 2 node(s) didn't match Pod's node affinity/selector** kubectl describe node aks-userpool1-10736312-vmss000000 Name: aks-userpool1-10736312-vmss000000 Roles: <none> Labels: agentpool=userpool1 beta.kubernetes.io/arch=amd64 beta.kubernetes.io/instance-type=Standard_D4_v3 beta.kubernetes.io/os=linux failure-domain.beta.kubernetes.io/region=eastus2 failure-domain.beta.kubernetes.io/zone=eastus2-1 kubernetes.azure.com/agentpool=userpool1 kubernetes.azure.com/azure-cni-overlay=true kubernetes.azure.com/cluster=MC_test-rg_testaks_eastus2 kubernetes.azure.com/consolidated-additional-properties=627157f7-bff2-11f0-a13c-aef3cc7b6334 kubernetes.azure.com/kubelet-identity-client-id=a1065c3e-852a-4571-b309-9bbf938b9a5b kubernetes.azure.com/kubelet-serving-ca=cluster kubernetes.azure.com/localdns-state=disabled kubernetes.azure.com/mode=user kubernetes.azure.com/network-name=aks-vnet-31772138 kubernetes.azure.com/network-resourcegroup=test-rg kubernetes.azure.com/network-stateless-cni=false kubernetes.azure.com/network-subnet=aks-subnet kubernetes.azure.com/network-subscription=7865d617-4ed0-48fd-8a84-caba6d118afe kubernetes.azure.com/node-image-version=AKSUbuntu-2204containerd-202510.29.0 kubernetes.azure.com/nodenetwork-vnetguid=526ac3c5-cefb-4c23-9817-56803937c572 kubernetes.azure.com/nodepool-type=VirtualMachineScaleSets kubernetes.azure.com/os-sku=Ubuntu kubernetes.azure.com/os-sku-effective=Ubuntu2204 kubernetes.azure.com/os-sku-requested=Ubuntu kubernetes.azure.com/podnetwork-type=overlay kubernetes.azure.com/role=agent kubernetes.azure.com/sku-cpu=4 kubernetes.azure.com/sku-memory=16384 kubernetes.azure.com/storageprofile=managed kubernetes.azure.com/storagetier=Standard_LRS kubernetes.io/arch=amd64 kubernetes.io/hostname=aks-userpool1-10736312-vmss000000 **kubernetes.io/os=linux** node.kubernetes.io/instance-type=Standard_D4_v3 storageprofile=managed storagetier=Standard_LRS topology.disk.csi.azure.com/zone=eastus2-1 topology.kubernetes.io/region=eastus2 topology.kubernetes.io/zone=eastus2-1 Annotations: alpha.kubernetes.io/provided-node-ip: 10.224.0.6 csi.volume.kubernetes.io/nodeid: {"disk.csi.azure.com":"aks-userpool1-10736312-vmss000000","file.csi.azure.com":"aks-userpool1-10736312-vmss000000"} node.alpha.kubernetes.io/ttl: 0 volumes.kubernetes.io/controller-managed-attach-detach: true CreationTimestamp: Wed, 12 Nov 2025 18:09:54 +0000 Taints: <none> Unschedulable: false Lease: HolderIdentity: aks-userpool1-10736312-vmss000000 AcquireTime: <unset> RenewTime: Wed, 12 Nov 2025 19:25:47 +0000 Conditions: Type Status LastHeartbeatTime LastTransitionTime Reason Message ---- ------ ----------------- ------------------ ------ ------- KernelDeadlock False Wed, 12 Nov 2025 19:24:52 +0000 Wed, 12 Nov 2025 18:13:50 +0000 KernelHasNoDeadlock kernel has no deadlock FrequentKubeletRestart False Wed, 12 Nov 2025 19:24:52 +0000 Wed, 12 Nov 2025 18:13:51 +0000 NoFrequentKubeletRestart kubelet is functioning properly FrequentContainerdRestart False Wed, 12 Nov 2025 19:24:52 +0000 Wed, 12 Nov 2025 18:13:51 +0000 NoFrequentContainerdRestart containerd is functioning properly KubeletProblem False Wed, 12 Nov 2025 19:24:52 +0000 Wed, 12 Nov 2025 18:13:51 +0000 KubeletIsUp kubelet service is up ReadonlyFilesystem False Wed, 12 Nov 2025 19:24:52 +0000 Wed, 12 Nov 2025 18:13:50 +0000 FilesystemIsNotReadOnly Filesystem is not read-only FrequentDockerRestart False Wed, 12 Nov 2025 19:24:52 +0000 Wed, 12 Nov 2025 18:13:51 +0000 NoFrequentDockerRestart docker is functioning properly VMEventScheduled False Wed, 12 Nov 2025 19:24:52 +0000 Wed, 12 Nov 2025 18:14:44 +0000 NoVMEventScheduled VM has no scheduled event FrequentUnregisterNetDevice False Wed, 12 Nov 2025 19:24:52 +0000 Wed, 12 Nov 2025 18:13:51 +0000 NoFrequentUnregisterNetDevice node is functioning properly ContainerRuntimeProblem False Wed, 12 Nov 2025 19:24:52 +0000 Wed, 12 Nov 2025 18:13:51 +0000 ContainerRuntimeIsUp container runtime service is up FilesystemCorruptionProblem False Wed, 12 Nov 2025 19:24:52 +0000 Wed, 12 Nov 2025 18:13:51 +0000 FilesystemIsOK Filesystem is healthy MemoryPressure False Wed, 12 Nov 2025 19:25:10 +0000 Wed, 12 Nov 2025 18:09:54 +0000 KubeletHasSufficientMemory kubelet has sufficient memory available DiskPressure False Wed, 12 Nov 2025 19:25:10 +0000 Wed, 12 Nov 2025 18:09:54 +0000 KubeletHasNoDiskPressure kubelet has no disk pressure PIDPressure False Wed, 12 Nov 2025 19:25:10 +0000 Wed, 12 Nov 2025 18:09:54 +0000 KubeletHasSufficientPID kubelet has sufficient PID available Ready True Wed, 12 Nov 2025 19:25:10 +0000 Wed, 12 Nov 2025 18:10:35 +0000 KubeletReady kubelet is posting ready status Addresses: InternalIP: 10.224.0.6 Hostname: aks-userpool1-10736312-vmss000000 Capacity: cpu: 4 ephemeral-storage: 129886128Ki hugepages-1Gi: 0 hugepages-2Mi: 0 memory: 16375372Ki pods: 30 Allocatable: cpu: 3860m ephemeral-storage: 119703055367 hugepages-1Gi: 0 hugepages-2Mi: 0 memory: 15607372Ki pods: 30 System Info: Machine ID: 9c2da526fbf840c2b9ff0340423e92b4 System UUID: ebb2caf4-e776-a940-983f-041eaa7d2286 Boot ID: 8e13d3cd-95db-487e-841f-220c9417e7bb Kernel Version: 5.15.0-1098-azure OS Image: Ubuntu 22.04.5 LTS Operating System: linux Architecture: amd64 Container Runtime Version: containerd://1.7.28-1 Kubelet Version: v1.32.9 Kube-Proxy Version: v1.32.9 ProviderID: azure:///subscriptions/7865d617-4ed0-48fd-8a84-caba6d118afe/resourceGroups/mc_test-rg_testaks_eastus2/providers/Microsoft.Compute/virtualMachineScaleSets/aks-userpool1-10736312-vmss/virtualMachines/0 Non-terminated Pods: (8 in total) Namespace Name CPU Requests CPU Limits Memory Requests Memory Limits Age --------- ---- ------------ ---------- --------------- ------------- --- kube-system azure-cns-9xcnd 40m (1%) 40m (1%) 250Mi (1%) 250Mi (1%) 76m kube-system azure-ip-masq-agent-69k99 50m (1%) 500m (12%) 36Mi (0%) 250Mi (1%) 76m kube-system cloud-node-manager-ct9f2 50m (1%) 0 (0%) 50Mi (0%) 512Mi (3%) 76m kube-system csi-azuredisk-node-9l28t 30m (0%) 0 (0%) 60Mi (0%) 5800Mi (38%) 76m kube-system csi-azurefile-node-wngnk 30m (0%) 0 (0%) 60Mi (0%) 600Mi (3%) 76m kube-system konnectivity-agent-57c4545d69-s2vbz 20m (0%) 1 (25%) 20Mi (0%) 1Gi (6%) 62m kube-system kube-proxy-9jmmd 100m (2%) 0 (0%) 0 (0%) 0 (0%) 76m kubevirt virt-operator-7c8bdfb574-r6kbk 10m (0%) 0 (0%) 450Mi (2%) 0 (0%) 65m Allocated resources: (Total limits may be over 100 percent, i.e., overcommitted.) Resource Requests Limits -------- -------- ------ cpu 330m (8%) 1540m (39%) memory 926Mi (6%) 8436Mi (55%) ephemeral-storage 0 (0%) 0 (0%) hugepages-1Gi 0 (0%) 0 (0%) hugepages-2Mi 0 (0%) 0 (0%) Events: <none> **What you expected to happen**: kubecrt-cr deployment should succeed with all components deployed. **How to reproduce it (as minimally and precisely as possible)**: Deploy AKS cluster on Azure with a usernodepool having nested virtualization enabled for the SKU and follow https://kubevirt.io/quickstart_cloud/ . In this case **Additional context**: Ref Doc being followed for installation : (https://kubevirt.io/quickstart_cloud/) **Environment**: - KubeVirt version (use `virtctl version`): kubectlCopy export VERSION=$(curl -s https://storage.googleapis.com/kubevirt-prow/release/kubevirt/kubevirt/stable.txt) echo $VERSION bash: kubectlCopy: command not found v1.6.3 - Kubernetes version (use `kubectl version`): 1.32.9 - VM or VMI specifications: Standard_D4_v3 https://docs.azure.cn/en-us/virtual-machines/sizes/general-purpose/dv3-series?tabs=sizebasic#sizes-in-series - Cloud provider or hardware configuration: Azure - OS (e.g. from /etc/os-release): Ubuntu2204 - Kernel (e.g. `uname -a`): aks-userpool1-10736312-vmss000000:/# uname -a Linux aks-userpool1-10736312-vmss000000 5.15.0-1098-azure #107-Ubuntu SMP Thu Oct 2 22:41:20 UTC 2025 x86_64 x86_64 x86_64 GNU/Linux - Install tools: Only the steps mentioned on https://kubevirt.io/quickstart_cloud/ - Others: N/A