ITADN

[Performance] etcd storage exhaustion caused by frequent status updates of large Pods in high-density scheduling scenarios

#5236Openwangyang0616 创建于 2026-04-23
kind/bugarea/performance
W
wangyang0616commented
### Description #### 1. Background & Scenario In large-scale heterogeneous computing clusters (covering both GPU and NPU architectures), we encounter a critical stability issue where etcd storage is rapidly exhausted, leading to cluster-wide unavailability. This occurs in the following production scenarios: - Large Pod Objects: Pods for AI training tasks are often "heavyweight" (>100KB), carrying extensive environment variables, complex scheduling constraints, or large-scale distributed configuration data in annotations. - High-Density Pending Queues: During resource contention or large-job submissions, thousands of Pods remain in the Pending state for extended periods. - Unstable Cluster Topology: In large clusters, nodes are frequently added, removed, or updated (due to autoscaling, hardware failures, or maintenance). #### 2. Problem Description When the cluster is under heavy load, the Volcano scheduler re-evaluates the massive Pending queue in every cycle. Because the node list and resource availability are constantly shifting, the FitError or Unschedulable messages for these Pods change almost every scheduling session. Current Volcano implementation uses the UpdateStatus interface to synchronize these frequent, minor changes. This creates a severe "Write Storm" with massive data amplification: - Full Payload Transmission: Each UpdateStatus call transmits the entire Pod object (including the 100KB+ Spec) even if only a timestamp or a brief error message in the Status field has changed. - etcd Revision Explosion: etcd's MVCC mechanism stores a complete new copy of the 100KB+ object for every minor update, leading to a rapid spike in the database size. - Cascading Failure: The constant serialization of large objects consumes excessive CPU on the Scheduler, API Server, and etcd. Eventually, etcd reaches its storage quota (--quota-backend-bytes), causing the entire Kubernetes control plane to stop accepting writes. ### Steps to reproduce the issue _No response_ ### Describe the results you received and expected - From "Full" to "Incremental" (Efficiency): Replace the UpdateStatus interface with a Patch mechanism. Even when Pod objects are heavyweight (>100KB) due to complex AI configurations, status updates should only transmit byte-level delta data, eliminating redundant network I/O and etcd storage bloat. - Adaptive Back-pressure (Stability): Implement a pressure-aware throttling mechanism triggered by the Pending Queue length. In high-load scenarios, the scheduler should automatically reduce the update frequency for Pending Pods to prevent etcd "write storms" caused by transient node fluctuations. - Zero-Latency for Critical Transitions (Reliability): Maintain a "Fast Track" for essential state changes (e.g., Pending $\rightarrow$ Running). While rate-limiting repetitive failure messages, the system must ensure that successful scheduling and task startups are synchronized to the API Server without any delay. ### What version of Volcano are you using? master ### Any other relevant information _No response_
2 条评论