# v1.14.0 ## Summary Volcano v1.14.0 establishes Volcano as a unified scheduling platform for diverse workloads at scale. This release introduces a scalable multi-scheduler architecture with dynamic node scheduling shard, enabling multiple schedulers to coordinate efficiently across large clusters. A new Agent Scheduler provides fast scheduling for latency-sensitive AI Agent workloads while working seamlessly with the Volcano batch scheduler. Network topology aware scheduling gains significant enhancements including HyperNode-level binpacking, SubGroup policies, and two-level Gang scheduling. Volcano Global integration advances with HyperJob for multi-cluster training and data-aware scheduling. Colocation now support generic operating systems with CPU Throttling, Memory QoS, and Cgroup V2. Additionally, integrated Ascend vNPU scheduling enables efficient sharing of Ascend AI accelerators. ## What's New ### Key Features Overview - **Scalable Multi-Scheduler with Dynamic Node Scheduling Shard (Alpha)**: Dynamically compute candidate node pools for schedulers with extensible strategies - **Fast Scheduling for AI Agent Workloads (Alpha)**: A new Agent Scheduler for latency-sensitive AI Agent workloads is introduced, working in coordination with Volcano batch scheduler to establish a unified scheduling platform - **Network Topology Aware Scheduling Enhancements**: Support hyperNode-level binpacking, SubGroup level network topology aware scheduling, and two-level Gang scheduling for distributed workloads - **Volcano Global Enhancements**: HyperJob for multi-cluster training and data-aware scheduling for federated environments - **Colocation for Generic OS**: CPU Throttling, Memory QoS, CPU Burst with Cgroup V2 support on Ubuntu, CentOS, and other generic operating systems - **Ascend vNPU Scheduling**: Integrated support for Ascend 310P/910 series vNPU scheduling with MindCluster and HAMi modes ### Key Feature Details ### Scalable Multi-Scheduler with Dynamic Node Scheduling Shard (Alpha) **Background and Motivation**: As Volcano evolves to support diverse scheduling workloads at massive scale, the single scheduler architecture faces significant challenges. Different workload types (batch training, AI agents, microservices) have distinct scheduling requirements and resource utilization patterns. A single scheduler becomes a bottleneck, and static resource allocation leads to inefficient cluster utilization. The Sharding Controller introduces a scalable multi-scheduler architecture that dynamically computes candidate node pools for each scheduler. Unlike strict partitioning, the Sharding Controller calculates dynamic candidate node pools rather than enforcing hard isolation between schedulers. This flexible approach enables Volcano to serve as a unified scheduling platform for diverse workloads while maintaining high throughput and low latency. **Alpha Feature Notice**: This feature is currently in alpha stage. The NodeShard CRD (Node Scheduling Shard) API structure and the underlying scheduling shard concepts are actively evolving. **Key Capabilities**: - **Dynamic Node Scheduling Shard Strategies**: Compute dynamic candidate node pools based on various policies. Currently supports scheduling shard by CPU utilization, with an extensible design to support more policies in the future. - **NodeShard CRD**: Manages dynamic candidate node pools for specific schedulers. - **Large-scale Cluster Support**: Architecture designed to support large-scale clusters by distributing load across multiple schedulers - **Scheduler Coordination**: Enable seamless coordination among various scheduler combinations (e.g., multiple Batch Schedulers, or a mix of Agent and Batch Schedulers), establishing Volcano as a unified scheduling platform **Configuration**: ```bash # Sharding Controller startup flags --scheduler-configs="volcano:volcano:0.0:0.6:false:2:100,agent-scheduler:agent:0.7:1.0:true:2:100" --shard-sync-period=60s --enable-node-event-trigger=true # Config format: name:type:min_util:max_util:prefer_warmup:min_nodes:max_nodes ``` Related: - PR: [#4777](https://github.com/volcano-sh/volcano/pull/4777) - Design Doc: [Sharding Controller Design](https://github.com/volcano-sh/volcano/blob/v1.14.0/docs/design/sharding_controller.md) - Contributors: @ssfffss, @Haoran, @qi-min ### Fast Scheduling for AI Agent Workloads (Alpha) **Background and Motivation**: AI Agent workloads are latency-sensitive with frequent task creation, requiring ultra-fast scheduling with high throughput. The Volcano batch scheduler is optimized for batch workloads and processes pods at fixed intervals, which cannot guarantee low latency for Agent workloads. To establish Volcano as a unified scheduling platform for both batch and latency-sensitive workloads, we introduce a dedicated Agent Scheduler. The Agent Scheduler works in coordination with the Volcano batch scheduler through the Sharding Controller (which is introduced in "Scalable Multi-Scheduler with Dynamic Node Scheduling Shard" feature). This architecture positions Volcano as a unified scheduling platform capable of handling diverse workload types. **Alpha Feature Notice**: This feature is currently in alpha stage and under active development. The Agent Scheduler related APIs, configuration options, and scheduling algorithms may be refined in future releases. **Key Capabilities**: - **Fast-Path Scheduling**: Independent scheduler optimized for latency-sensitive workloads such as AI Agent workloads - **Multi-Worker Parallel Scheduling**: Multiple workers process pods concurrently from the scheduling queue, increasing throughput - **Optimistic Concurrency Control**: Conflict-Aware Binder resolves scheduling conflicts before executing real binding - **Optimized Scheduling Queue**: Enhanced queue mechanism with urgent retry support - **Unified Platform Integration**: Seamless coordination with Volcano batch scheduler via Sharding Controller Related: - Issue: [#4722](https://github.com/volcano-sh/volcano/issues/4722) - PRs: [#4804](https://github.com/volcano-sh/volcano/pull/4804), [#4801](https://github.com/volcano-sh/volcano/pull/4801), [#4805](https://github.com/volcano-sh/volcano/pull/4805) - Design Doc: [Agent Scheduler Design](https://github.com/volcano-sh/volcano/blob/v1.14.0/docs/design/agent-scheduler.md) - Contributors: @qi-min, @JesseStutler, @handan-yxh ### Network Topology Aware Scheduling Enhancements **Background and Motivation**: Volcano v1.14.0 brings significant enhancements to network topology aware scheduling, addressing the growing demands of distributed workloads including LLM training, HPC, and other network-intensive applications. **Key Enhancements**: - **SubGroup Level Topology Awareness**: Support fine-grained network topology constraints at the SubGroup/Partition level. - **Flexible Network Tier Configuration**: Support `highestTierName` for specifying maximum network tier constraints by name. - **Two-Level Gang Scheduling**: Improved gang scheduling to support both Job-level and SubGroup-level consistency. - **Volcano Job Partitioning**: Enable partitioning of Volcano Jobs for better resource management and fault isolation. - **HyperNode-Level Binpacking**: Optimization for resource utilization across network topology boundaries. **Configuration Example - Volcano Job**: ```yaml apiVersion: batch.volcano.sh/v1alpha1 kind: Job metadata: name: llm-training-job spec: # ...other fields networkTopology: mode: hard highestTierAllowed: 2 # Job can cross up to Tier 2 HyperNodes tasks: - name: trainer replicas: 8 partitionPolicy: totalPartitions: 2 # Split into 2 partitions partitionSize: 4 # 4 pods per partition minPartitions: 2 # Minimum 2 partitions required networkTopology: mode: hard highestTierAllowed: 1 # Each partition must stay within Tier 1 template: spec: containers: - name: trainer image: training-image:v1 resources: requests: nvidia.com/gpu: 8 ``` **Configuration Example - PodGroup SubGroupPolicy**: ```yaml apiVersion: scheduling.volcano.sh/v1beta1 kind: PodGroup metadata: name: llm-training-pg spec: minMember: 4 networkTopology: mode: hard highestTierAllowed: 2 subGroupPolicy: - name: "trainer" subGroupSize: 4 labelSelector: matchLabels: volcano.sh/task-spec: trainer matchLabelKeys: - volcano.sh/partition-id networkTopology: mode: hard highestTierAllowed: 1 ``` Related: - Issues: [#4188](https://github.com/volcano-sh/volcano/issues/4188), [#4368](https://github.com/volcano-sh/volcano/issues/4368), [#4869](https://github.com/volcano-sh/volcano/issues/4869) - PRs: [#4721](https://github.com/volcano-sh/volcano/pull/4721), [#4810](https://github.com/volcano-sh/volcano/pull/4810), [#4795](https://github.com/volcano-sh/volcano/pull/4795), [#4785](https://github.com/volcano-sh/volcano/pull/4785), [#4889](https://github.com/volcano-sh/volcano/pull/4889) - Design Doc: [Network Topology Aware Scheduling](https://github.com/volcano-sh/volcano/blob/v1.14.0/docs/design/Network%20Topology%20Aware%20Scheduling.md) - Contributors: @ouyangshengjia, @3sunny, @zhaoqi, @wangyang0616, @MondayCha, @Tau721 ### Colocation for Generic OS This release brings comprehensive improvements to Volcano's colocation capabilities, with a major milestone: **support for generic operating systems** (Ubuntu, CentOS, etc.) in addition to OpenEuler. This enables broader adoption of Volcano Agent for resource sharing between online and offline workloads. **New Features in v1.14.0**: 1. **CPU Throttling (CPU Suppression)** The CPU usage of online pods dynamically changes. To better isolate online and offline workloads, the CPU quota allocated to offline pods needs to change dynamically according to the actual usage of online pods. When offline pods consume more CPU than their quota, CPU suppression is triggered; if not exceeded, their quota can gradually recover, enabling adaptive resource allocation. Key design: - Dynamically adjusts BestEffort root cgroup CPU quota based on node allocatable CPU and real-time usage - Follows a "monitor-event-handler" architecture with conservative updates to avoid jitter ```yaml cpuThrottlingConfig: enable: true cpuThrottlingThreshold: 80 # Allow BE quota up to 80% of allocatable CPU cpuJitterLimitPercent: 1 # Emit updates when quota changes by >=1% cpuRecoverLimitPercent: 10 # Cap quota increases to 10% per update ``` 2. **Memory QoS (Cgroup V2)** Cgroup V2 based memory isolation for colocation environments. This feature introduces the `ColocationConfiguration` CRD, which allows users to define memory QoS policies for specific workloads. The system automatically applies these policies to matching pods, calculating Cgroup V2 settings (`memory.high`, `memory.low`, `memory.min`) based on the configured percentages and the pod's memory requests/limits. Key capabilities: - **New API**: `ColocationConfiguration` CRD for defining memory isolation policies via label selectors - **Dynamic Calculation**: - `memory.high` = `pod.limits.memory` * `highRatio` % - `memory.low` = `pod.requests.memory` * `lowRatio` % - `memory.min` = `pod.requests.memory` * `minRatio` % - **Unified Interface**: Robust detection and support for Cgroup V2 environment Usage Example: ```yaml apiVersion: config.volcano.sh/v1alpha1 kind: ColocationConfiguration metadata: name: colo-config1 spec: selector: matchLabels: app: offline-test memoryQos: # memory.high = memory.limits * 100% (No throttling if 100) highRatio: 100 # memory.low = memory.requests * 50% (Protect 50% of request) lowRatio: 50 # memory.min = memory.requests * 0% (No absolute guarantee) minRatio: 0 ``` 3. **CPU Burst for Generic OS** Extended CPU Burst support to generic operating systems (Ubuntu, CentOS, etc.), allowing latency-sensitive workloads to burst beyond their CPU limits when resources are available. ```yaml cpuBurstConfig: enable: true ``` 4. **Cgroup V2 Full Support** Volcano Agent now fully supports Cgroup V2 environments, ensuring seamless operation on modern Linux distributions. - **Automatic Detection**: Automatically detects cgroup version (v1/v2) and driver type (systemd/cgroupfs) without manual configuration. - **Unified Interface**: Adapts internal resource managers (CPU, Memory) to work with both v1 and v2 interfaces. Related: - Issues: [#4631](https://github.com/volcano-sh/volcano/issues/4631), [#4466](https://github.com/volcano-sh/volcano/issues/4466), [#4468](https://github.com/volcano-sh/volcano/issues/4468), [#4912](https://github.com/volcano-sh/volcano/issues/4912) - PRs: [#4632](https://github.com/volcano-sh/volcano/pull/4632), [#4945](https://github.com/volcano-sh/volcano/pull/4945), [#4913](https://github.com/volcano-sh/volcano/pull/4913), [#4984](https://github.com/volcano-sh/volcano/pull/4984) - Design Docs: [CPU Throttle Design](https://github.com/volcano-sh/volcano/blob/v1.14.0/docs/design/cpu-throttle-design.md), [Agent Cgroup V2 Adaptation](https://github.com/volcano-sh/volcano/blob/v1.14.0/docs/design/agent-cgroup-v2-adaptation.md) - Contributors: @Haibara-Ai97, @JesseStutler, @ouyangshengjia ### Ascend vNPU Scheduling **Background and Motivation**: Volcano v1.14.0 introduces integrated support for Ascend vNPU (virtual NPU) scheduling, enabling efficient sharing of Ascend AI processors across multiple workloads. This feature supports two modes to accommodate different deployment scenarios. **Supported Modes**: 1. **MindCluster Mode** - Integrated from the official Ascend cluster scheduling add-on - Supports Ascend 310P series with dynamic virtualization - Uses `huawei.com/npu-core` resource name - Supports vnpu-level (low/high) and vnpu-dvpp configurations 2. **HAMi Mode** - Developed by the HAMi community - Supports both Ascend 310 and 910 series - Supports heterogeneous Ascend clusters (910A, 910B2, 910B3, 310P) - Memory-based virtualization with automatic alignment **Scheduler Configuration**: ```yaml # MindCluster Mode - name: deviceshare arguments: deviceshare.AscendMindClusterVNPUEnable: true # HAMi Mode - name: deviceshare arguments: deviceshare.AscendHAMiVNPUEnable: true deviceshare.SchedulePolicy: binpack # or spread ``` **Usage Example (HAMi Mode)**: ```yaml apiVersion: v1 kind: Pod metadata: name: ascend-pod spec: schedulerName: volcano containers: - name: inference image: ascend-image:v1 resources: limits: huawei.com/Ascend310P: "1" huawei.com/Ascend310P-memory: "4096" ``` Related: - Issues: [#4718](https://github.com/volcano-sh/volcano/issues/4718) - PRs: [#4656](https://github.com/volcano-sh/volcano/pull/4656), [#4717](https://github.com/volcano-sh/volcano/pull/4717) - User Guide: [How to Use vNPU](https://github.com/volcano-sh/volcano/blob/v1.14.0/docs/user-guide/how_to_use_vnpu.md) - Contributors: @JackyTYang, @DSFans2014 ### Volcano Global Enhancements The new version of Volcano-global further enhances the ability of multi-cluster scheduling **Key Enhancements**: - **Introduce HyperJob for large-scale cross-cluster training**: Enables large-scale cross-cluster training by orchestrating jobs across multiple clusters. - **Data-Aware Scheduling**: Optimizes workload placement based on data locality. For detailed information, please refer to the [Volcano Global v0.3.0 Release Notes](https://github.com/volcano-sh/volcano-global/releases/tag/v0.3.0). Related: - Design Doc: [HyperJob Design](https://github.com/volcano-sh/volcano/blob/v1.14.0/docs/design/hyperjob-multi-cluster-job-splitting.md) - Contributors: @JesseStutler, @fx147 ### Scheduler Stability and Performance **Reclaim Refactoring and Enhancements** The Reclaim mechanism has been significantly improved through a comprehensive refactor of the Reclaim Action and critical logic fixes in the Capacity Plugin. These changes collectively enhance the accuracy, stability, and performance of resource reclamation in multi-tenant clusters. Key improvements: - **Reclaim Action Refactoring**: The reclaim workflow has been restructured to improve code readability, maintainability, and test coverage. - **Enhanced Capacity Plugin Logic**: - **Precise Victim Selection**: Fixed `reclaimableFn` to correctly handle scalar resources and ensure victims are selected based on all relevant resource dimensions, taking the reclaimer's requirements into account. - **Accurate Resource Comparison**: Corrected `preemptiveFn` resource comparison logic to prevent incorrect preemption decisions when specific resource dimensions are zero. - **Improved Stability**: Addressed edge cases in resource calculation to prevent scheduling loops and incorrect evictions. Related: - Issues: [#3738](https://github.com/volcano-sh/volcano/issues/3738), [#4658](https://github.com/volcano-sh/volcano/issues/4658), [#4918](https://github.com/volcano-sh/volcano/issues/4918) - PRs: [#4794](https://github.com/volcano-sh/volcano/pull/4794), [#4659](https://github.com/volcano-sh/volcano/pull/4659), [#4919](https://github.com/volcano-sh/volcano/pull/4919) - Contributors: @guoqinwill, @hajnalmt ## Other Notable Changes ### API Changes New CRDs and API enhancements in v1.14.0: **1. NodeShard CRD (Alpha)** (@ssfffss, @Haoran, @qi-min) Node Scheduling Shard CRD. Manages dynamic candidate node pools for specific schedulers. **2. HyperJob CRD** (@JesseStutler) Multi-cluster training job abstraction for the Volcano Global enhancements. **3. DataSource/DataSourceClaim CRDs** (@fx147) Data-aware scheduling for federated environments. **4. SubGroupPolicy API** (@wangyang0616) Defines fine-grained network topology constraints and gang scheduling policies at the SubGroup level. See Network Topology Aware Scheduling section for details. **5. DequeueStrategy** (@wangdongyang1) New configuration for queue dequeue strategies. **6. Kubebuilder Validation Markers** (@zhoujinyu) Added CEL validation markers to batch, scheduling, topology, and flow APIs. **7. ColocationConfiguration CRD** (@ouyangshengjia) Introduced `ColocationConfiguration` CRD to configure colocation policies (currently supporting Memory QoS). This API allows users to define percentage-based coefficients (range 0-100) for `highRatio`, `lowRatio`, and `minRatio`, which are multiplied with pod memory limits or requests to dynamically calculate Cgroup V2 interfaces (`memory.high`, `memory.low`, `memory.min`). ### Features and Enhancements - **ValidatingAdmissionPolicy/MutatingAdmissionPolicy**: Optional enhanced admission mechanisms ([#4596](https://github.com/volcano-sh/volcano/pull/4596), @suyiiyii) - **GitHub Container Registry support**: Added GHCR as an alternative to Docker Hub ([#4845](https://github.com/volcano-sh/volcano/issues/4845), [#4846](https://github.com/volcano-sh/volcano/pull/4846), @medyagh) - **In-place Pod Update**: Support for updating pods without recreation ([#3839](https://github.com/volcano-sh/volcano/pull/3839), @guoqinwill) - **Scheduler performance dashboards**: Added Grafana dashboards for scheduler metrics ([#4885](https://github.com/volcano-sh/volcano/pull/4885), @guoqinwill) - **Distributed Inference Support (Kthena)**: Enables role-based elasticity and Gang Scheduling for distributed inference workloads ([#4781](https://github.com/volcano-sh/volcano/issues/4781), @wangyang0616) - **E2E test improvements**: Enhanced context collection on test failures ([#4764](https://github.com/volcano-sh/volcano/issues/4764), @neeraj542) - **Staging directory for APIs**: Introduced staging directory structure for volcano.sh/apis ([#4823](https://github.com/volcano-sh/volcano/pull/4823), @PersistentJZH) ### Bug Fixes - **Scheduling main loop blocked**: Fixed timeout due to un-released PreBind lock ([#4692](https://github.com/volcano-sh/volcano/issues/4692), @guoqinwill) - **Scheduler panic on hierarchy queue**: Fixed panic when child queue capability check fails ([#4673](https://github.com/volcano-sh/volcano/issues/4673), @hajnalmt, @zhengjiajin) - **minAvailable mismatch**: Fixed job pending state caused by minAvailable/minPartitions mismatch ([#4867](https://github.com/volcano-sh/volcano/issues/4867), @zhengchenyu) - **Job aborting state transition**: Fixed early transition to Aborted when terminating pods are removed ([#4828](https://github.com/volcano-sh/volcano/issues/4828), @kingeasternsun) - **Memory leak from metrics**: Fixed memory leak caused by leftover metrics data due to race conditions ([#4821](https://github.com/volcano-sh/volcano/issues/4821), @halcyon-r) - **Queue hierarchy validation**: Fixed cluster-wide scheduling failure due to missing validation for child queue guarantee sum ([#4818](https://github.com/volcano-sh/volcano/issues/4818), @zjj2wry) - **RestartTask policy on PodEvicted**: Fixed unexpected behavior when pod is evicted ([#4814](https://github.com/volcano-sh/volcano/issues/4814), @ouyangshengjia) - **Task validity check failure**: Fixed scheduling failures caused by task member changes in PodGroup ([#4802](https://github.com/volcano-sh/volcano/issues/4802), @ouyangshengjia) - **Scheduler memory leak**: Fixed memory leak in volcano-scheduler deletedJobs queue ([#4745](https://github.com/volcano-sh/volcano/issues/4745), @fengruotj) - **VGPU multi-card memory bug**: Fixed VGPU request when multiple cards have different total video memory ([#4713](https://github.com/volcano-sh/volcano/issues/4713), @weapons97) - **GPU resource calculation error**: Fixed gpu resource error at allocate state in checkNodeGPUSharingPredicateAndScore ([#4884](https://github.com/volcano-sh/volcano/pull/4884), @chenw66) - **Capacity plugin preemptiveFn**: Fixed resource comparison preventing reclaim ([#4658](https://github.com/volcano-sh/volcano/issues/4658), @hajnalmt) - **Capacity plugin queue share**: Fixed queue share calculation for best-effort queues ([#4865](https://github.com/volcano-sh/volcano/pull/4865), @guoqinwill) - **Colocation cgroup configuration**: Fixed systemd driver with containerd runtime oversubscribed cgroups ([#4912](https://github.com/volcano-sh/volcano/issues/4912), @JesseStutler) - **NUMA topology snapshot panic**: Fixed panic when numatopology is added and updated in short time before snapshot ([#4891](https://github.com/volcano-sh/volcano/issues/4891), @qi-min) - **FailedScheduling message**: Fixed inaccurate message when podgroup is ready ([#4978](https://github.com/volcano-sh/volcano/pull/4978), @ruanwenjun) - **Nil pod handling**: Fixed nil pod when AlreadyExists error on create ([#4985](https://github.com/volcano-sh/volcano/pull/4985), @Aman-Kumar) - **vNPU multi-type cluster**: Fixed vNPU not working in multi-type NPU clusters ([#4779](https://github.com/volcano-sh/volcano/pull/4779), @kiritoxkiriko) - **Hami vNPU memory allocation**: Fixed memory allocation restrictions in Hami vNPU scene ([#4859](https://github.com/volcano-sh/volcano/pull/4859), @haorenhui) - **MindCluster bugs**: Fixed multiple bugs in MindCluster vNPU mode ([#4775](https://github.com/volcano-sh/volcano/pull/4775), @DSFans2014) - **PodGroup controller replicaset annotation**: Fixed annotation handling for replicaset ([#4792](https://github.com/volcano-sh/volcano/pull/4792), @hajnalmt) - **Reclaimed resource calculation**: Fixed to add node futureIdle resource first ([#4774](https://github.com/volcano-sh/volcano/pull/4774), @jiahuat) - **syncTask and retryResyncTask**: Fixed cache communication issues ([#4938](https://github.com/volcano-sh/volcano/pull/4938), @hajnalmt) ### Dependencies - Support Kubernetes 1.34 ([#4704](https://github.com/volcano-sh/volcano/pull/4704), @suyiiyii, @tunedev) ## Upgrade Instructions To upgrade to Volcano v1.14.0: ```bash # Using Helm helm repo update helm upgrade volcano volcano-sh/volcano --version 1.14.0 # Using kubectl kubectl apply -f https://raw.githubusercontent.com/volcano-sh/volcano/v1.14.0/installer/volcano-development.yaml ``` **Note**: Review the new CRDs (NodeShard, HyperJob) if you plan to use the Agent Scheduler, Sharding Controller, or cross-cluster training features. ## Contributors Thank you to all contributors who made this release possible: @3sunny, @Aman-Kumar, @DSFans2014, @GautamBytes, @Haibara-Ai97, @Haoran, @JackyTYang, @JesseStutler, @LiZhenCheng9527, @MondayCha, @PersistentJZH, @Rushabh-Mehta, @Tau721, @Wonki4, @Xavier-Chang, @Zhou-Zihang, @Tau721, @chenw66, @dafu-wu, @fengruotj, @freshwlnd, @fx147, @guoqinwill, @hairuiyang, @hajnalmt, @halcyon-r, @handan-yxh, @haorenhui, @jiahuat, @kingeasternsun, @kiritoxkiriko, @kube-gopher, @limengxuan, @medyagh, @neeraj542, @ouyangshengjia, @qi-min, @ruanwenjun, @ssfffss, @suyiiyii, @tanjie.master, @tharXdev, @tunedev, @wangdongyang1, @wangyang0616, @weapons97, @zhaoqi, @zhengchenyu, @zhengjiajin, @zhoujinyu, @zjh, @zjj2wry ## Full Changelog [https://github.com/volcano-sh/volcano/compare/v1.13.0...v1.14.0](https://github.com/volcano-sh/volcano/compare/v1.13.0...v1.14.0)

## What's Changed ### Bug fixes - [Cherry-pick v1.12] add hcclrank job plugin by @wangdongyang1 in https://github.com/volcano-sh/volcano/pull/4555 - Automated cherry pick of #4347: When some scalar resources are 0 in deserved, hierarychical queues validation can not pass by @wuxiaobao in https://github.com/volcano-sh/volcano/pull/4586 - Automated cherry pick of #4590: add permissions for managing namespaces in admission rules by @suyiiyii in https://github.com/volcano-sh/volcano/pull/4594 - [Cherry-pick v1.12] fix mpi job plugin panic when mpi job only has master task by @wangdongyang1 in https://github.com/volcano-sh/volcano/pull/4619 - [Cherry-pick v1.12]Sync kube-scheduler:Improve CSILimits plugin accuracy by using VolumeAttachments by @guoqinwill in https://github.com/volcano-sh/volcano/pull/4627 - Automated cherry pick of #4599: fix: report all scalar metrics for each queue by @hajnalmt in https://github.com/volcano-sh/volcano/pull/4651 - [Cherry-pick 1.12] fix: Initialize realCapability field in newQueueAttr by @dafu-wu in https://github.com/volcano-sh/volcano/pull/4695 - [cherry-pick 1.12]Scheduling main loop blocked and timeout due to un-released PreBind lock in Volcano by @guoqinwill in https://github.com/volcano-sh/volcano/pull/4699 - [release-1.12] Cherry-pick #4786 and #4792: fix replicaset KubeGroupNameAnnotation handling and replicaSet podgroup update synchronization by @hajnalmt in https://github.com/volcano-sh/volcano/pull/4843 - Automated cherry pick of #4829: keep terminating pod in job by @wangdongyang1 in https://github.com/volcano-sh/volcano/pull/4861 - [release-1.12] fix potential panic on numa resources info updating in snapshot by @qi-min in https://github.com/volcano-sh/volcano/pull/4898 - [release-1.12] Fix gpu resource error by @chenw66 in https://github.com/volcano-sh/volcano/pull/4915 - [release-1.12] Fix: Changes to task members in a PodGroup caused task validity checks to fail during scheduling by @ouyangshengjia in https://github.com/volcano-sh/volcano/pull/4920 - [release-1.12] Fix scheduler panic when metrics are disabled by @Copilot in https://github.com/volcano-sh/volcano/pull/4921 - [release-1.12] Update metrics_client_prometheus.go by @nitindhiman314e in https://github.com/volcano-sh/volcano/pull/4932 ## Maintenance - [release-1.12] Add Free Disk Space step to E2E workflows by @Copilot in https://github.com/volcano-sh/volcano/pull/4851 **Full Changelog**: https://github.com/volcano-sh/volcano/compare/v1.12.2...v1.12.3

## What's Changed ### Bug fixes * Automated cherry pick of #4670: fix: ci err caused bt ray e2e default image by @Wonki4 in https://github.com/volcano-sh/volcano/pull/4681 * [Cherry-pick 1.13] fix: Initialize realCapability field in newQueueAttr by @dafu-wu in https://github.com/volcano-sh/volcano/pull/4694 * [cherry-pick 1.13]Scheduling main loop blocked and timeout due to un-released PreBind lock in Volcano by @guoqinwill in https://github.com/volcano-sh/volcano/pull/4700 * [release-1.13] Fix scheduler panic when metrics are disabled by @Copilot in https://github.com/volcano-sh/volcano/pull/4770 * Cherry-pick PR #4786 to release-1.13: Fix replicaSet podgroup update synchronization by @jiahuat in https://github.com/volcano-sh/volcano/pull/4799 * [release-1.13] fix: replicaset KubeGroupNameAnnotation handling by @hajnalmt in https://github.com/volcano-sh/volcano/pull/4826 * [release-1.13] fix: constant cache warnings by @hajnalmt in https://github.com/volcano-sh/volcano/pull/4831 * [release-1.13] fix: capacity plugin's preemptivefn logic by @hajnalmt in https://github.com/volcano-sh/volcano/pull/4830 * [release-1.13] Fix: Changes to task members in a PodGroup caused task validity checks to fail during scheduling by @ouyangshengjia in https://github.com/volcano-sh/volcano/pull/4852 ### Maintenance * [release-1.13] Add Free Disk Space step to E2E workflows by @Copilot in https://github.com/volcano-sh/volcano/pull/4763 **Full Changelog**: https://github.com/volcano-sh/volcano/compare/v1.13.0...v1.13.1

## What's New Welcome to the v1.13.0 release of Volcano! 🚀 🎉 📣 In this release, we have brought a series of significant enhancements that have been long-awaited by community users: - **AI Training and Inference Enhancements** - [Support LeaderWorkerSet for Large Model Inference Scenarios](#support-leaderworkerset-for-large-model-inference-scenarios) - [Introduce Cron VolcanoJob](#introduce-cron-volcanojob) - [Support Label-based HyperNode Auto Discovery](#support-label-based-hypernode-auto-discovery) - [Add Native Ray Framework Support](#add-native-ray-framework-support) - [Introduce HCCL Plugin Support](#introduce-hccl-plugin-support) - **Resource Management and Scheduling Enhancements** - [Introduce ResourceStrategyFit Plugin](#introduce-resourcestrategyfit-plugin) - [Independent Scoring Strategy by Resource Type](#independent-scoring-strategy-by-resource-type) - [Scarce Resource Avoidance (SRA)](#scarce-resource-avoidance-sra) - [Enhance NodeGroup Functionality](#enhance-nodegroup-functionality) - **Colocation Enhancements** - [Decouple Colocation from OS](#decouple-colocation-from-os) - [Support Custom OverSubscription Resource Names](#support-custom-oversubscription-resource-names) ## Support LeaderWorkerSet for Large Model Inference Scenarios [LeaderWorkerSet (LWS)](https://github.com/kubernetes-sigs/lws) is an API for deploying a group of Pods on Kubernetes. It is primarily used to address multi-host inference in AI/ML inference workloads, especially scenarios that require sharding large language models (LLMs) and running them across multiple devices on multiple nodes. Since its open-source release, Volcano has actively integrated with upstream and downstream ecosystems, building a comprehensive community ecosystem for batch computing such as AI and big data. In the [v0.7](https://github.com/kubernetes-sigs/lws/releases/tag/v0.7.0) release of LWS, it natively integrated Volcano's AI scheduling capabilities. When used with the new version of Volcano, LWS automatically creates PodGroups, which are then scheduled and managed by Volcano, thereby implementing advanced capabilities like Gang scheduling for large model inference scenarios. Looking ahead, Volcano will continue to expand its ecosystem integration capabilities, providing robust scheduling and resource management support for more projects dedicated to enabling distributed inference on Kubernetes. Usage documentation: [LeaderWorkerSet With Gang](https://github.com/kubernetes-sigs/lws/tree/main/docs/examples/sample/gang-scheduling). Related PRs: https://github.com/kubernetes-sigs/lws/pull/496, https://github.com/kubernetes-sigs/lws/pull/498, @[JesseStutler](https://github.com/JesseStutler) ## Introduce Cron VolcanoJob This release introduces support for Cron Volcano Jobs. Users can now periodically create and run Volcano Jobs based on a predefined schedule, similar to native Kubernetes CronJobs, to achieve periodic execution of batch computing tasks like AI and big data. Detailed features are as follows: - **Scheduled Execution**: Define the execution cycle of jobs using standard Cron expressions (`spec.schedule`). - **Timezone Support**: Set the timezone in `spec.timeZone` to ensure jobs execute at the expected local time. - **Concurrency Policy**: Control concurrent behavior via `spec.concurrencyPolicy`: - `AllowConcurrent`: Allows concurrent execution of multiple jobs (default). - `ForbidConcurrent`: Skips the current scheduled execution if the previous job has not completed. - `ReplaceConcurrent`: Terminates the previous job if it is still running and starts a new one. - **History Management**: Configure the number of successful (`successfulJobsHistoryLimit`) and failed (`failedJobsHistoryLimit`) job history records to retain; old jobs are automatically cleaned up. - **Missed Schedule Handling**: The `startingDeadlineSeconds` field allows tolerating scheduling delays within a certain timeframe; timeouts are considered missed executions. - **Status Tracking**: The CronJob status (`status`) tracks currently active jobs, the last scheduled time, and the last successful completion time for easier monitoring and management. Related PRs: https://github.com/volcano-sh/apis/pull/192, https://github.com/volcano-sh/volcano/pull/4560, @[GoingCharlie](https://github.com/volcano-sh/volcano/commits?author=GoingCharlie), @[hwdef](https://github.com/hwdef), @[Monokaix](https://github.com/volcano-sh/volcano/commits?author=Monokaix) Usage example: [Cron Volcano Job Example](https://github.com/volcano-sh/volcano/blob/master/example/cronjob/cronjob.yaml). ## Support Label-based HyperNode Auto Discovery Volcano officially launched network topology-aware scheduling capability in v1.12 and pioneered the UFM auto-discovery mechanism based on InfiniBand (IB) networks. However, for hardware clusters that do not support IB networks or use other network architectures (such as Ethernet), manually maintaining the network topology remains cumbersome. To address this issue, the new version introduces a Label-based HyperNode auto-discovery mechanism. This feature provides users with a universal and flexible way to describe network topology, transforming complex topology management tasks into simple node label management. This mechanism allows users to define the correspondence between topology levels and node labels in the volcano-controller-configmap. The Volcano controller periodically scans all nodes in the cluster and automatically performs the following tasks based on their labels: - **Automatic Topology Construction**: Automatically builds multi-layer HyperNode topology structures from top to bottom (e.g., rack -> switch -> node) based on a set of labels on the nodes. - **Dynamic Maintenance**: When node labels change, or nodes are added or removed, the controller automatically updates the members and structure of the HyperNodes, ensuring the topology information remains consistent with the cluster state. - **Support for Multiple Topology Types**: Allows users to define multiple independent network topologies simultaneously to adapt to different hardware clusters (e.g., GPU clusters, NPU clusters) or different network partitions. Configuration example: ```yaml # volcano-controller-configmap.yaml apiVersion: v1 kind: ConfigMap metadata: name: volcano-controller-configmap namespace: volcano-system data: volcano-controller.conf: | networkTopologyDiscovery: - source: label enabled: true interval: 10m # Discovery interval config: networkTopologyTypes: # Define a topology type named topology-A topology-A: # Define topology levels, ordered from top to bottom - nodeLabel: "volcano.sh/hypercluster" # Top-level HyperNode - nodeLabel: "volcano.sh/hypernode" # Middle-level HyperNode - nodeLabel: "kubernetes.io/hostname" # Bottom-level physical node ``` This feature is enabled by adding the label source to the Volcano controller's ConfigMap. The above configuration defines a three-layer topology structure named `topology-A`: - Top Level (Tier 2): Defined by the `volcano.sh/hypercluster` label. - Middle Level (Tier 1): Defined by the `volcano.sh/hypernode` label. - Bottom Level: Physical nodes, identified by the Kubernetes built-in `kubernetes.io/hostname` label. When a node is labeled as follows, it will be automatically recognized and classified into the topology path `cluster-s4 -> node-group-s0`: ```yaml # Labels for node node-0 labels: kubernetes.io/hostname: node-0 volcano.sh/hypernode: node-group-s0 volcano.sh/hypercluster: cluster-s4 ``` The label-based network topology auto-discovery feature offers excellent generality and flexibility. It is not dependent on specific network hardware (like IB), making it suitable for various heterogeneous clusters, and allows users to flexibly define hierarchical structures of any depth through labels. It automates complex topology maintenance tasks into simple node label management, significantly reducing operational costs and the risk of errors. Furthermore, this mechanism dynamically adapts to changes in cluster nodes and labels, maintaining the accuracy of topology information in real-time without manual intervention. Related PR: https://github.com/volcano-sh/volcano/pull/4629, @[zhaoqi612](https://github.com/zhaoqi612) Usage documentation: [HyperNode Auto Discovery](https://github.com/volcano-sh/volcano/blob/master/docs/user-guide/how_to_use_hypernode_auto_discovery.md). ## Add Native Ray Framework Support [Ray](https://docs.ray.io/) is an open-source unified distributed computing framework whose core goal is to simplify parallel computing from single machines to large-scale clusters, especially suitable for scaling Python and AI applications. To manage and run Ray on Kubernetes, the community provides KubeRay—an operator specifically designed for Kubernetes. It acts as a bridge between Kubernetes and the Ray framework, greatly simplifying the deployment and management of Ray clusters and jobs. Historically, running Ray workloads on Kubernetes primarily relied on the KubeRay Operator. KubeRay integrated Volcano in its [v0.4.0 release (released in 2022)](https://docs.ray.io/en/master/cluster/kubernetes/k8s-ecosystem/volcano.html) for scheduling and resource management of Ray Clusters, addressing issues like resource deadlocks in distributed training scenarios. With this new version of Volcano, users can now directly create and manage Ray clusters and submit computational tasks through native Volcano Jobs. This provides Ray users with an alternative usage scheme, allowing them to more directly utilize Volcano's capabilities such as Gang Scheduling, queue management and fair scheduling, and job lifecycle management for running Ray workloads. Related PR: https://github.com/volcano-sh/volcano/pull/4581, @[Wonki4](https://github.com/Wonki4) Design documentation: [Ray Framework Plugin Design Doc](https://github.com/volcano-sh/volcano/blob/master/docs/design/distributed-framework-plugins.md). Usage documentation: [Ray Plugin User Guide](https://github.com/volcano-sh/volcano/blob/master/docs/user-guide/how_to_use_ray_plugin.md). ## Introduce HCCL Plugin Support The new version adds an HCCL Rank plugin (`hcclrank`) to Volcano Jobs, used for automatically assigning HCCL Ranks to Pods in distributed tasks. This includes: - New implementation of the `hcclrank` plugin for Volcano Jobs, supporting automatic calculation and injection of HCCL Rank into Pod annotations based on task type (master/worker) and index. - The plugin supports custom master/worker task names, allowing users to specify the master/worker roles in distributed tasks. This feature enhances Volcano's native support for HCCL communication scenarios, such as Huawei Ascend, facilitating automatic management and assignment of Ranks in AI training tasks. Related PR: https://github.com/volcano-sh/volcano/pull/4524, @[kingeasternsun](https://github.com/kingeasternsun) ## Enhance NodeGroup Functionality In hierarchical queue structures, repeatedly configuring the same node group affinity (`nodeGroupAffinity`) for each sub-queue as its parent queue leads to configuration redundancy and difficult maintenance. To solve this problem, the Nodegroup plugin adds support for inheriting affinity within hierarchical queues. Once enabled, the scheduler resolves the effective affinity for a queue according to the following rules: 1. **Prioritize Self-Configuration**: If the queue has defined `spec.affinity`, it uses this configuration directly. 2. **Upward Inheritance**: If the queue has not defined `spec.affinity`, it searches upward through its parents and inherits the affinity configuration defined by the nearest ancestor queue. 3. **Override Capability**: A child queue can override the inherited configuration by defining its own `spec.affinity`, ensuring flexibility. This feature allows administrators to set unified node group affinity at a parent queue (e.g., department level), and all child queues (e.g., team level) will automatically inherit this setting, simplifying management. For queues without NodeAffinity configuration, the "strict" parameter in the plugin controls scheduling behavior. When `strict` is set to `true` (the default value), tasks in these queues cannot be scheduled to any nodes. When `strict` is set to `false`, these tasks are allowed to be scheduled to regular nodes that do not have the `volcano.sh/nodegroup-name` label. In the nodegroup plugin parameters of the scheduler configuration file, setting `enableHierarchy: true` enables hierarchical queue mode, and setting `strict: false` configures non-strict mode. Example configuration is as follows: ```yaml actions: "allocate, backfill, preempt, reclaim" tiers: - plugins: - name: nodegroup arguments: enableHierarchy: true # Enable hierarchical support strict: false # Set to non-strict mode, allowing tasks in the queue to be scheduled to nodes without the "volcano.sh/nodegroup-name" label ``` Related PRs: https://github.com/volcano-sh/volcano/pull/4455, @[JesseStutler](https://github.com/JesseStutler), @[wuyueandrew](https://github.com/wuyueandrew) NodeGroup design documentation: [NodeGroup Design.](https://github.com/volcano-sh/volcano/blob/master/docs/design/node-group.md) NodeGroup usage documentation: [NodeGroup User Guide.](https://github.com/volcano-sh/volcano/blob/master/docs/user-guide/how_to_use_nodegroup_plugin.md) ## Introduce ResourceStrategyFit Plugin In the native Kubernetes `noderesources` fit strategy, only a single aggregated (`MostAllocated`) or dispersed (`LeastAllocated`) strategy can be applied to all resources. This has limitations in complex heterogeneous computing environments (like AI/ML clusters). To meet differentiated scheduling requirements, Volcano introduces the enhanced `ResourceStrategyFit` plugin. This plugin now integrates two core features: Independent scoring strategies by resource type and Scarce Resource Avoidance (SRA). ### Independent Scoring Strategy by Resource Type This feature allows users to specify `MostAllocated` (binpack) or `LeastAllocated` (spread) strategies for different resources (e.g., cpu, memory, nvidia.com/gpu) independently, and assign different weights to them. The scheduler calculates the node score meticulously based on the independent configuration for each resource. To simplify the management of resources within the same family (e.g., different model GPUs from the same vendor), this feature also supports suffix wildcard (`*`) matching for resource names. - **Syntax Rules**: Only suffix wildcards are supported, e.g., `nvidia.com/gpu/*`. Patterns like `*` or `vendor.*/gpu` are considered invalid. - **Matching Priority**: Uses the "longest prefix match" principle. Exact matches have the highest priority; when no exact match exists, the wildcard pattern with the longest prefix is selected. Configuration Example: The following configuration sets a high-priority binpack strategy for a specific V100 GPU model, a generic binpack strategy for all other NVIDIA GPUs, and a spread strategy for CPU resources. Pod-level resource scoring strategy configuration is also supported. ```yaml actions: "enqueue, allocate, backfill, reclaim, preempt" tiers: - plugins: - name: resource-strategy-fit arguments: resourceStrategyFitWeight: 10 resources: # Exact match, highest priority nvidia.com/gpu-v100: type: MostAllocated weight: 3 # Wildcard match, applies to all other NVIDIA GPUs nvidia.com/gpu/*: type: MostAllocated weight: 2 # Exact match for CPU resource cpu: type: LeastAllocated weight: 1 ``` ### Scarce Resource Avoidance (SRA) SRA is a "soft" strategy designed to improve the overall utilization of expensive or scarce resources (like GPUs). It influences node scoring to guide ordinary tasks that do not require specific scarce resources (e.g., CPU-only tasks) to avoid nodes containing those resources where possible. This helps "reserve" scarce resource nodes for tasks that truly need them, thereby reducing resource contention and task waiting time. Mechanism: 1. Users define a set of "scarce resources" (e.g., `nvidia.com/gpu`) in the configuration. 2. When scheduling a Pod that does *not* request any of the defined scarce resources, the SRA policy takes effect. 3. The scheduler reduces the score of nodes that possess these scarce resources. The more types of scarce resources a node has, the lower its score. 4. For Pods that *do* request scarce resources, the SRA policy does not negatively impact their scheduling decisions. Configuration Example: The following configuration defines `nvidia.com/gpu` as a scarce resource. When scheduling a CPU-only task, nodes with GPUs will have their scores reduced, making the task more likely to be scheduled onto nodes without GPUs. ```yaml actions: "enqueue, allocate, backfill, reclaim, preempt" tiers: - plugins: - name: resource-strategy-fit arguments: # ... binpack/spread strategy configuration for resourceStrategyFit ... resources: nvidia.com/gpu: type: MostAllocated weight: 2 cpu: type: LeastAllocated weight: 1 # SRA policy configuration sra: enable: true resources: "nvidia.com/gpu" # Define scarce resource list, comma-separated weight: 10 # Weight of the SRA policy in the total score resourceWeight: nvidia.com/gpu: 1 # Define nvidia.com/gpu as a scarce resource and its weight ``` By combining the binpack/spread strategies of ResourceStrategyFit with the avoidance strategy of SRA, users can achieve more refined and efficient scheduling of heterogeneous resources. Related PRs: https://github.com/volcano-sh/volcano/pull/4391, https://github.com/volcano-sh/volcano/pull/4454, https://github.com/volcano-sh/volcano/pull/4512, @[LY-today](https://github.com/LY-today), @[XbaoWu](https://github.com/XbaoWu), @[ditingdapeng](https://github.com/ditingdapeng), @[kingeasternsun](https://github.com/kingeasternsun) Design documentation: [ResourceStrategyFit Design](https://github.com/volcano-sh/volcano/blob/master/docs/design/resource-strategy-fit-scheduling.md) Usage documentation: [ResourceStrategyFit User Guide](https://github.com/volcano-sh/volcano/blob/master/docs/user-guide/how_to_use_resource_strategy_fit_plugin.md) ## Decouple Colocation from OS Volcano's co-location capability consists of two parts: application-level and kernel-level. Application-level co-location provides unified scheduling for online and offline workloads, dynamic resource overcommitment, node pressure eviction, etc. Kernel-level co-location involves QoS guarantees for resources like CPU, Memory, and Network at the kernel level, which typically requires support from a specific OS (like OpenEuler). In the new version, Volcano decouples the co-location capability from the OS. For users using an OS that does not support kernel-level co-location, they can choose to use Volcano's application-level co-location capabilities to achieve unified scheduling of online and offline tasks, dynamic resource overcommitment, and high-priority task guarantees. Specific usage: When installing the Volcano agent, specify the `--supported-features` parameter: ```shell helm install volcano . --create-namespace -n volcano-system --set custom.colocation_enable=true --set "custom.agent_supported_features=OverSubscription\,Eviction\,Resources" ``` Related PRs: https://github.com/volcano-sh/volcano/pull/4409, https://github.com/volcano-sh/volcano/pull/4630, @[ShuhanYan](https://github.com/ShuhanYan), @[Monokaix](https://github.com/Monokaix) Colocation documentation: https://volcano.sh/en/docs/colocation/ ## Support Custom OverSubscription Resource Names The Volcano co-location Agent adds parameters `--extend-resource-cpu-name` and `--extend-resource-memory-name`, allowing users to customize the names of overcommitted resources. This supports custom naming for CPU and memory resources (defaults are `kubernetes.io/batch-cpu` and `kubernetes.io/batch-memory` respectively), enhancing flexibility in setting overcommitted resource names. Specific usage: When installing Volcano, specify the `--extend-resource-cpu-name` and `--extend-resource-memory-name` parameters: ```shell helm install volcano . --create-namespace -n volcano-system --set custom.colocation_enable=true --set custom.agent_extend_resource_cpu_name=example.com/cpu --set custom.agent_extend_resource_memory_name=example.com/memory ``` Related PRs: https://github.com/volcano-sh/volcano/pull/4413, https://github.com/volcano-sh/volcano/pull/4630, @[ShuhanYan](https://github.com/ShuhanYan), @[Monokaix](https://github.com/Monokaix) Colocation documentation: https://volcano.sh/en/docs/colocation/ ## Add Kubernetes 1.33 Support The Volcano version keeps pace with the Kubernetes community releases. v1.13 supports the latest Kubernetes v1.33 release, ensuring functionality and reliability through comprehensive UT and E2E test cases. For participating in Volcano's adaptation work for new Kubernetes versions, refer to: [adapt-k8s-todo](https://github.com/volcano-sh/volcano/blob/v1.13.0/docs/design/adapt-k8s-todo.md). Related PR: https://github.com/volcano-sh/volcano/pull/4430, @[mahdikhashan](https://github.com/mahdikhashan) ## Overall Changes * Support topology aware in the preempt action by @bibibox in https://github.com/volcano-sh/volcano/pull/4279 * Move InitCycleState from openSession to OpenSession by @ElectricFish7 in https://github.com/volcano-sh/volcano/pull/4378 * fix: Node resource topology awareness, stop scheduling and notReady by @LY-today in https://github.com/volcano-sh/volcano/pull/4373 * Support the allocation callback function provided by the extender. by @zhifei92 in https://github.com/volcano-sh/volcano/pull/4377 * Delete secrets permission for volcano agent by @JesseStutler in https://github.com/volcano-sh/volcano/pull/4389 * [Automation Enhancement] Update cherry-pick shell and add cherry-pick guide doc by @JesseStutler in https://github.com/volcano-sh/volcano/pull/4366 * [Automation Enhancement]: Add bump version script; Make version release more automated by @JesseStutler in https://github.com/volcano-sh/volcano/pull/4372 * fix incorrect scheduler log by @archlitchi in https://github.com/volcano-sh/volcano/pull/4419 * fix: rolling restart admission webhooks after helm upgrade by @junzebao in https://github.com/volcano-sh/volcano/pull/4396 * Only refresh podgroup to running when pods are scheduled by @JesseStutler in https://github.com/volcano-sh/volcano/pull/4384 * Fix panic while `vcctl queue list` after podgroups' queue was deleted by @halcyon-r in https://github.com/volcano-sh/volcano/pull/4428 * When some scalar resources are 0 in deserved, hierarychical queues validation can not pass by @JesseStutler in https://github.com/volcano-sh/volcano/pull/4347 * Move kube-scheduler related metrics initilization to server.go to avoid panic by @JesseStutler in https://github.com/volcano-sh/volcano/pull/4422 * fix: add ResourceStrategyFit plugin by @LY-today in https://github.com/volcano-sh/volcano/pull/4391 * [feature] upgrade k8s to v1.33.2 by @mahdikhashan in https://github.com/volcano-sh/volcano/pull/4430 * Update volcano author copyright header by @JesseStutler in https://github.com/volcano-sh/volcano/pull/4379 * Add support for skipping handler registration and QoS manager initialization based on configuration in Volcano Agent by @ShuhanYan in https://github.com/volcano-sh/volcano/pull/4409 * [doc] Jobflow parameter overrides by @mahdikhashan in https://github.com/volcano-sh/volcano/pull/4411 * Support for configurable extended resource names in volcano agent by @ShuhanYan in https://github.com/volcano-sh/volcano/pull/4413 * Fix: Sync Jobflow status with vcjob Status Terminated by @dongjiang1989 in https://github.com/volcano-sh/volcano/pull/4443 * Correct step order to enable caching in E2E workflows by @GautamBytes in https://github.com/volcano-sh/volcano/pull/4464 * fix enable node device score plugin by @coldzerofear in https://github.com/volcano-sh/volcano/pull/4340 * Fix the issue where SelectBestNode returns nil when plugin scores are… by @guoqinwill in https://github.com/volcano-sh/volcano/pull/4445 * [Cherry-pick master] Check queue spec to ensure the rationality of resource size by @XbaoWu in https://github.com/volcano-sh/volcano/pull/4470 * fix: Volcano scheduler panic when scheduling Pods with delayed binding PVCs by @ouyangshengjia in https://github.com/volcano-sh/volcano/pull/4484 * fix node count reconcile by @Monokaix in https://github.com/volcano-sh/volcano/pull/4473 * Fix incorrect definition of `ReleaseNameEnvKey` by @ouyangshengjia in https://github.com/volcano-sh/volcano/pull/4486 * Add missing capacity metrics in hierarchical queues by @JesseStutler in https://github.com/volcano-sh/volcano/pull/4487 * Support configuring resources managed by the extender by @zhifei92 in https://github.com/volcano-sh/volcano/pull/4482 * Set root capability only when user not set it by @houyuting in https://github.com/volcano-sh/volcano/pull/4354 * Bump github.com/onsi/gomega from 1.35.1 to 1.38.0 by @dependabot[bot] in https://github.com/volcano-sh/volcano/pull/4495 * inherit non-existent scalar resources from the parent queue by @lhlxc in https://github.com/volcano-sh/volcano/pull/4504 * Fix volume binding e2e testing by @JesseStutler in https://github.com/volcano-sh/volcano/pull/4483 * fix #4499: scheduler panic caused by pod using gpu-number scheduled by volcano with old version (e.g v1.7) by @linuxfhy in https://github.com/volcano-sh/volcano/pull/4500 * test: add e2e test cases for hypernode by @Xu-Wentao in https://github.com/volcano-sh/volcano/pull/3983 * fix#4497: hypernode update bug by @cyf-2002 in https://github.com/volcano-sh/volcano/pull/4498 * Integrate with leaderworkerset: Volcano controller doesn't need to create podgroup for statefulset if statefulset pods have associated podgroup by @JesseStutler in https://github.com/volcano-sh/volcano/pull/4478 * fix: update podGroup when statefulSet update by @Poor12 in https://github.com/volcano-sh/volcano/pull/4517 * fix: deepcopy podgroup before update by @Poor12 in https://github.com/volcano-sh/volcano/pull/4526 * Fix dra flaky test: use `exec env` instead logs by @JesseStutler in https://github.com/volcano-sh/volcano/pull/4531 * ✨ add hcclrank job plugin by @kingeasternsun in https://github.com/volcano-sh/volcano/pull/4524 * add kingeasternsun as reviewer by @kingeasternsun in https://github.com/volcano-sh/volcano/pull/4561 * Fix panic in job controller's killPods action by @neo502721 in https://github.com/volcano-sh/volcano/pull/4569 * Add permissions for managing namespaces in admission rules by @suyiiyii in https://github.com/volcano-sh/volcano/pull/4590 * Revert "fix: Node resource topology awareness, stop scheduling and notReady" by @dafu-wu in https://github.com/volcano-sh/volcano/pull/4575 * Minor docs changes by @hajnalmt in https://github.com/volcano-sh/volcano/pull/4587 * Fix queue not counting volcano.sh/vgpu-memory and volcano.sh/vgpu-number properly by @archlitchi in https://github.com/volcano-sh/volcano/pull/4520 * feat: support wildcard syntax in resource-strategy-fit plugin by @ditingdapeng in https://github.com/volcano-sh/volcano/pull/4512 * use node.futureidle instead when pod has nominatedNodeName by @Monokaix in https://github.com/volcano-sh/volcano/pull/4588 * network topology aware hard mode support hypernode binpack by @kingeasternsun in https://github.com/volcano-sh/volcano/pull/4345 * Sync kube-scheduler: Improve CSILimits plugin accuracy by using VolumeAttachments by @guoqinwill in https://github.com/volcano-sh/volcano/pull/4608 * 🐛 fix mpi job plugin panic when mpi job only has master task by @kingeasternsun in https://github.com/volcano-sh/volcano/pull/4610 * Fix an issue in vc-scheduler about vgpu device memory allocated by @archlitchi in https://github.com/volcano-sh/volcano/pull/4615 * feat: add detail msg for pg event by @Poor12 in https://github.com/volcano-sh/volcano/pull/4544 * Cleanup: update MAINTAINERS.md with latest content location by @kevin-wangzefeng in https://github.com/volcano-sh/volcano/pull/4622 * Expose more helm config for agent by @Monokaix in https://github.com/volcano-sh/volcano/pull/4630 * Add hierarchical queue support for nodegroup plugin by @JesseStutler in https://github.com/volcano-sh/volcano/pull/4455 * fix: address remaining review comments from PR #4457 for ResourceStrategyFit plugin by @ditingdapeng in https://github.com/volcano-sh/volcano/pull/4635 * fix: fix ResourceStrategyFit plugin by @LY-today in https://github.com/volcano-sh/volcano/pull/4457 * feat: add cron volcano job by @GoingCharlie in https://github.com/volcano-sh/volcano/pull/4560 * feat: add ray plugin for job by @Wonki4 in https://github.com/volcano-sh/volcano/pull/4581 * Add sra policy for ResourceStrategyFit Plugin by @XbaoWu in https://github.com/volcano-sh/volcano/pull/4454 * Support identifying network topology from node labels and converted i… by @zhaoqi612 in https://github.com/volcano-sh/volcano/pull/4629 * Enhance README formatting with note callouts by @huntersman in https://github.com/volcano-sh/volcano/pull/4623 * Free up disk space by @Monokaix in https://github.com/volcano-sh/volcano/pull/4642 * fix: report all scalar metrics for each queue by @hajnalmt in https://github.com/volcano-sh/volcano/pull/4599 * Support configuring network-topology via pod annotations by @zhifei92 in https://github.com/volcano-sh/volcano/pull/4583 * Fix panic on volcano-vgpu when allocating multiple containers in a pod by @archlitchi in https://github.com/volcano-sh/volcano/pull/4633 * feat(resource-strategy-fit): add per-Pod scoring strategy by @kingeasternsun in https://github.com/volcano-sh/volcano/pull/4641 * Fix bump version script by @JesseStutler in https://github.com/volcano-sh/volcano/pull/4519 * Add nodegroup nonstrict mode by @JesseStutler in https://github.com/volcano-sh/volcano/pull/4652 * add nodegroup nonstrict to fit 0nodes queue by @wuyueandrew in https://github.com/volcano-sh/volcano/pull/4602 * Revise the user documentation for hypernode auto discovery by @zhaoqi612 in https://github.com/volcano-sh/volcano/pull/4654 * Automated: Bump version to v1.13.0 by @JesseStutler in https://github.com/volcano-sh/volcano/pull/4655 ## New Contributors * @ElectricFish7 made their first contribution in https://github.com/volcano-sh/volcano/pull/4378 * @zhifei92 made their first contribution in https://github.com/volcano-sh/volcano/pull/4377 * @junzebao made their first contribution in https://github.com/volcano-sh/volcano/pull/4396 * @ShuhanYan made their first contribution in https://github.com/volcano-sh/volcano/pull/4409 * @GautamBytes made their first contribution in https://github.com/volcano-sh/volcano/pull/4464 * @coldzerofear made their first contribution in https://github.com/volcano-sh/volcano/pull/4340 * @houyuting made their first contribution in https://github.com/volcano-sh/volcano/pull/4354 * @lhlxc made their first contribution in https://github.com/volcano-sh/volcano/pull/4504 * @cyf-2002 made their first contribution in https://github.com/volcano-sh/volcano/pull/4498 * @neo502721 made their first contribution in https://github.com/volcano-sh/volcano/pull/4569 * @suyiiyii made their first contribution in https://github.com/volcano-sh/volcano/pull/4590 * @dafu-wu made their first contribution in https://github.com/volcano-sh/volcano/pull/4575 * @ditingdapeng made their first contribution in https://github.com/volcano-sh/volcano/pull/4512 * @GoingCharlie made their first contribution in https://github.com/volcano-sh/volcano/pull/4560 * @Wonki4 made their first contribution in https://github.com/volcano-sh/volcano/pull/4581 * @zhaoqi612 made their first contribution in https://github.com/volcano-sh/volcano/pull/4629 * @huntersman made their first contribution in https://github.com/volcano-sh/volcano/pull/4623 **Full Changelog**: https://github.com/volcano-sh/volcano/compare/v1.12.2...v1.13.0

## What's Changed * Automated cherry pick of #4422: Move kube-scheduler related metrics initilization to server.go to avoid panic by @JesseStutler in https://github.com/volcano-sh/volcano/pull/4461 * Automated cherry pick of #4473: fix node count reconcile by @Monokaix in https://github.com/volcano-sh/volcano/pull/4488 * [cherry-pick for 1.12]Fix incorrect definition of ReleaseNameEnvKey by @ouyangshengjia in https://github.com/volcano-sh/volcano/pull/4490 * [cherry-pick for 1.12]Fix the issue where SelectBestNode returns nil when plugin scores are negative by @guoqinwill in https://github.com/volcano-sh/volcano/pull/4472 * Automated cherry pick of #4487: Add missing capacity metrics in hierarchical queues by @JesseStutler in https://github.com/volcano-sh/volcano/pull/4494 * [Cherry-pick] Add bump version script; Make version release more automated by @JesseStutler in https://github.com/volcano-sh/volcano/pull/4521 * [Cherry-pick] fix: update podGroup when statefulSet update by @Poor12 in https://github.com/volcano-sh/volcano/pull/4522 * Automated: Bump version to v1.12.2 by @JesseStutler in https://github.com/volcano-sh/volcano/pull/4518 **Full Changelog**: https://github.com/volcano-sh/volcano/compare/v1.12.1...v1.12.2

## What's Changed * Fix queue update conflicts when upgrading to new version by @Monokaix in https://github.com/volcano-sh/volcano/pull/4336 * Bump image to v1.12.1 by @Monokaix in https://github.com/volcano-sh/volcano/pull/4337 **Full Changelog**: https://github.com/volcano-sh/volcano/compare/v1.12.0...v1.12.1

**Important:** This release addresses multiple critical security vulnerabilities. We strongly advise all users to upgrade to immediately to protect your systems and data. ## Security Fixes - Remove the execute permission for some files, chmod to 644 ([#4171](https://github.com/volcano-sh/volcano/pull/4171) **@JesseStutler**) - add a switch to control whether enable pprof in scheduler ([#4173](https://github.com/volcano-sh/volcano/pull/4173) **@JesseStutler**) - Add warning msg when TLS verification disabled([#4211](https://github.com/volcano-sh/volcano/pull/4211) **@Monokaix**) - Add http server timeout([#4208](https://github.com/volcano-sh/volcano/pull/4208) **@Monokaix**) ## Other Improvements - Bump image to v1.11.2 ([#4232](https://github.com/volcano-sh/volcano/pull/4232) **@JesseStutler**) - Fix: remove controller-manager metrics that should not be introduced ([#4202](https://github.com/volcano-sh/volcano/pull/4202) **@dongjiang1989**) - Filter useless logs in binpack ([#4240](https://github.com/volcano-sh/volcano/pull/4240) **@XbaoWu**) ## Important Notes Before Upgrading **Change: Volcano Scheduler pprof Endpoint Disabled by Default** For security enhancement, the pprof endpoint for the Volcano Scheduler is now disabled by default in this release. If you require this endpoint for debugging or monitoring, you will need to explicitly enable it post-upgrade. This can be achieved by: - If you are using helm, specifying `custom.scheduler_pprof_enable=true` during Helm installation or upgrade. - OR, manually setting the command-line argument `--enable-pprof=true` when starting the Volcano Scheduler. Please be aware of the security implications before enabling this endpoint in production environments.

**Important** This release addresses multiple critical security vulnerabilities. We strongly advise all users to upgrade immediately to protect your systems and data. ## Security Fixes - Remove the execute permission for some files, chmod to 644 ([#4171](https://github.com/volcano-sh/volcano/pull/4171) **@JesseStutler**) - add a switch to control whether enable pprof in scheduler ([#4173](https://github.com/volcano-sh/volcano/pull/4173) **@JesseStutler**) - Add warning msg when TLS verification disabled([#4211](https://github.com/volcano-sh/volcano/pull/4211) **@Monokaix**) - Add http server timeout([#4208](https://github.com/volcano-sh/volcano/pull/4208) **@Monokaix**) ## Other Improvements - Bump image to 1.11.0-network-topology-preview.3 ([#4237](https://github.com/volcano-sh/volcano/pull/4237) **@JesseStutler**) - Add NetworkTopology plugin score doc ([#4213](https://github.com/volcano-sh/volcano/pull/4213) **@ecosysbin**) - HyperNode supports select Nodes By labels ([#4068](https://github.com/volcano-sh/volcano/pull/4068) **@ecosysbin**) - Update ubuntu base image ([#4197](https://github.com/volcano-sh/volcano/pull/4197) **@Monokaix**) ## Important Notes Before Upgrading **Change: Volcano Scheduler pprof Endpoint Disabled by Default** For security enhancement, the pprof endpoint for the Volcano Scheduler is now disabled by default in this release. If you require this endpoint for debugging or monitoring, you will need to explicitly enable it post-upgrade. This can be achieved by: - If you are using helm, specifying `custom.scheduler_pprof_enable=true` during Helm installation or upgrade. - OR, manually setting the command-line argument `--enable-pprof=true` when starting the Volcano Scheduler. Please be aware of the security implications before enabling this endpoint in production environments.