The Machine Config Operator (MCO) is stuck in a continuous loop, re-syncing ControllerConfig every 34-36 minutes

**OpenShift Version**: 4.18.2 **Bug Type**: MCO Template Controller False Positive Detection **Impact**: 121GB/hour excessive network traffic **Environment**: AWS IPI Multi-AZ NAME VERSION AVAILABLE PROGRESSING SINCE STATUS version 4.18.2 True False 3d4h Cluster version is 4.18.2 Problem Statement The Machine Config Operator (MCO) is stuck in a continuous loop, re-syncing ControllerConfig every 34-36 minutes due to false detection of pull secret changes. This generates excessive network traffic (121GB/hour) and significant AWS cross-AZ data transfer costs, making the cluster unsuitable for production evaluation. Business Impact Cost Impact: Excessive AWS cross-AZ data transfer charges ($1,200+/month projected) Performance Impact: Network components overwhelmed by unnecessary traffic Evaluation Blocker: Cannot properly assess OpenShift for production deployment Operational Impact: Difficult to distinguish legitimate vs. bug-related traffic Environment Details Cluster Configuration: - Platform: AWS IPI Installation - Topology: 3 masters + 3 workers across 3 availability zones - OpenShift Version: 4.18.2 - Cluster Size: 6 nodes total - Network: Multi-AZ with cross-zone communication - License: Red Hat Developer/Demo License AWS Details: - Region: eu-west-1 - Availability Zones: eu-west-1a, eu-west-1b, eu-west-1c - Instance Types: m6i.xlarge 4vCPU and 16 GiB x3 for master abd x3 for workers - Network Configuration: Default IPI networking Technical Details Root Cause Analysis The MCO template controller incorrectly detects pull secret changes when none have occurred, triggering unnecessary ControllerConfig regeneration and machine config distribution across all cluster nodes. Evidence of Bug 1. MCO Log Pattern bash# Command used to identify the issue: oc logs -n openshift-machine-config-operator deployment/machine-config-controller --tail=100 | grep -i "pull-secret\|re-sync" # Output showing the bug pattern: I0708 00:56:26.898860 1 kubelet_config_controller.go:222] Re-syncing all kubelet config controller generated MachineConfigs due to apiServer cluster change I0708 00:56:26.901006 1 template_controller.go:198] Re-syncing ControllerConfig due to apiServer cluster change I0708 00:56:26.901698 1 template_controller.go:146] Re-syncing ControllerConfig due to secret pull-secret change I0708 01:30:14.458997 1 template_controller.go:146] Re-syncing ControllerConfig due to secret pull-secret change Analysis: MCO restart at 00:56:26 immediately triggers pull secret re-sync Next cycle occurs at 01:30:14 (33 minutes 48 seconds later) Pattern indicates 34-36 minute continuous loop No actual pull secret modifications occurred (verified via oc get events . Network Traffic Evidence Traffic Query Used: promqlsum by (namespace) ( increase(container_network_transmit_bytes_total{container="POD"}[5m]) + increase(container_network_receive_bytes_total{container="POD"}[5m]) ) Results (5-minute windows): openshift-machine-config-operator: 35.95M (leading cluster traffic) openshift-multus: 30.27M (secondary impact) openshift-monitoring: 18.13M openshift-ovn-kubernetes: 17.5M Total cluster: ~200M/5min = 2.4TB/day Historical Pattern: Baseline traffic: 10-20GB/day expected for 6-node cluster Observed traffic: 500GB-2TB/day (25-100x higher than normal) Traffic spikes correlate exactly with MCO log timestamps 3. Cluster State Validation bash# Pull secret unchanged: oc get secret pull-secret -n openshift-config -o yaml # Resource version stable, no modification events # MCO status healthy: oc get co machine-config # Shows: True False False (Available, Progressing, Degraded) # Machine Config Pools updated: oc get mcp # All pools show: True False False (Updated, Updating, Degraded) Attempted Workarounds 1. MCO Controller Restart (Temporary Relief) bashoc delete pods -n openshift-machine-config-operator -l k8s-app=machine-config-controller Result: Provides temporary relief but bug resumes immediately after restart 2. Pull Secret Validation Verified pull secret content unchanged No external modifications detected Resource version changes appear to be metadata-only 3. Cluster Component Verification All cluster operators healthy No degraded machine config pools No scheduling or resource issues identified Prometheus Graphs Traffic Spike Correlation: Graph showing traffic spike at 03:11 AM correlating with MCO cycle Component Impact: Visual evidence of coordinated spikes across network components Timeline Analysis: Traffic patterns matching predicted 34-36 minute MCO cycles Log Excerpts Complete MCO logs showing pull secret loop pattern Event logs confirming no actual pull secret modifications cluster id : bd619fb0-2a28-4383-830a-8d807c9154c1 "Service level agreement (SLA) Self-Support 60-day evaluation" 4.18.2 was deployed because 4.19 currently has no availability of open shift-builds operator - was deployed to do some internal training and course boot camp pipelines were developed on 4.18.2 **Note**: Unable to file via Red Hat Support due to demo license restrictions. This is a significant product quality issue affecting latest release with measurable cost impact.