# Gate-Controlled Scheduling for Cluster Autoscalers Compatibility

## Table of Contents

- [Motivation](#motivation)
  - [How Volcano Currently Sets Unschedulable](#how-volcano-currently-sets-unschedulable)
- [Proposal](#proposal)
  - [Goals](#goals)
  - [Non-Goals](#non-goals)
  - [High-Level Implementation](#high-level-implementation)
    - [API Constant Definition](#api-constant-definition)
    - [Changes to the Volcano MutatingAdmissionWebhook](#changes-to-the-volcano-mutatingadmissionwebhook)
    - [Changes to the Volcano Scheduler](#changes-to-the-volcano-scheduler)
      - [Queue Admission Checks](#queue-admission-checks)
      - [Queue Allocation and Gate Removal](#queue-allocation-and-gate-removal)
      - [Queue Capacity Accounting for Ungated Pods](#queue-capacity-accounting-for-ungated-pods)
        - [Dynamic Reserved Calculation](#dynamic-reserved-calculation)
        - [Capacity Check with Reserved Resources](#capacity-check-with-reserved-resources)
        - [Cache Initialization](#cache-initialization)
        - [Cache Updates During Allocation](#cache-updates-during-allocation)
- [Limitations](#limitations)
- [Feature gate](#feature-gate)
- [Conclusions](#conclusions)
- [Related Issues](#related-issues)

## Motivation

Cluster autoscalers such as [Cluster Autoscaler (CA)](https://github.com/kubernetes/autoscaler/tree/master) or
[Karpenter](https://github.com/kubernetes-sigs/karpenter) are present in almost all Kubernetes environments to
dynamically adjust node capacity based on scheduler signals. Both systems monitor pod scheduling conditions,
specifically looking for pods that match:

```yaml
type: PodScheduled
status: "False"
reason: Unschedulable
```

These pods, usually marked by the kube-scheduler, are interpreted as evidence of insufficient cluster capacity,
triggering scale-up simulations to eventually add new nodes.

Please refer to the links below regarding detecting `Unschedulable` Pods in Cluster Autoscaler and Karpenter:

> - [CA (v1.34.1): listers.go – func isUnschedulable(pod \*apiv1.Pod) bool](https://github.com/kubernetes/autoscaler/blob/cluster-autoscaler-1.34.1/cluster-autoscaler/utils/kubernetes/listers.go#L161-L170)
>   (also check
>   [FAQ.md](https://github.com/kubernetes/autoscaler/blob/cluster-autoscaler-release-1.34/cluster-autoscaler/FAQ.md#how-does-scale-up-work))
> - [Karpenter (v1.11.1): scheduling.go – func FailedToSchedule(pod \*corev1.Pod) bool](https://github.com/kubernetes-sigs/karpenter/blob/v1.11.1/pkg/utils/pod/scheduling.go#L122)

This mechanism works as intended with the default `kube-scheduler`, but can cause unintended behavior when used with
Volcano. Volcano's current implementation marks pods as `Unschedulable` for any allocation failure, regardless of
whether the failure is due to insufficient cluster capacity (where autoscaling is appropriate) or queue capacity limits
(where autoscaling is not needed). This causes autoscalers to incorrectly trigger scale-up operations when pods are
simply waiting for queue admission (refer to
[Volcano's issue \#4710](https://github.com/volcano-sh/volcano/issues/4710)).

### How Volcano Currently Sets Unschedulable

> Note: Every mention of Volcano in this document refers to the
> [v1.14.0 release](https://github.com/volcano-sh/volcano/releases/tag/v1.14.0).

Volcano sets the `Unschedulable` condition on pods through its cache event recording mechanism in
[`pkg/scheduler/cache/cache.go`](https://github.com/volcano-sh/volcano/blob/v1.14.0/pkg/scheduler/cache/cache.go). After
each scheduling cycle, the
[`RecordJobStatusEvent`](https://github.com/volcano-sh/volcano/blob/v1.14.0/pkg/scheduler/cache/cache.go#L1582) function
examines tasks (pods) that have not been allocated and updates their `PodScheduled` condition with `status=False` and
`reason=Unschedulable`.

This happens for **any** task that fails allocation, regardless of whether the failure is due to insufficient cluster
resources (legitimately triggering a scale-up) or queue capacity limits (but also triggering a scale-up). Autoscalers
cannot distinguish between these scenarios, as they only see the `Unschedulable` condition.

## Proposal

This proposal leverages the use of Kubernetes `schedulingGates` (as defined in
[KEP-3521](https://github.com/kubernetes/enhancements/tree/master/keps/sig-scheduling/3521-pod-scheduling-readiness#summary))
to control when a Pod is considered ready for scheduling. The core idea is to delay setting the `Unschedulable`
condition until the Queue has enough capacity to accommodate the Pod.

### Goals

- Prevent downstream integrators (like CA, Karpenter) from triggering unnecessary scale-ups caused by Pods marked as
  `Unschedulable`.
- Provide an opt-in mechanism, controlled via Pod annotation.
- Preserve existing Volcano scheduling semantics.
- Non-blocking implementation (avoid scheduler performance degradation).

### Non-Goals

- Altering the behavior or internal logic of Cluster Autoscaler or Karpenter.
- Rejecting or blocking Pod creation at admission time.
- Introducing custom autoscaler integrations or external controllers.

### High-Level Implementation

In this proposal, changes will be required to both the **admission process** and also the **scheduler routines**, as the
latter is responsible for applying the logic for actions and plugins.

#### API Constant Definition

The following annotation key should be defined as a constant in the
[Scheduling v1beta1 API package](https://github.com/volcano-sh/volcano/blob/v1.14.0/staging/src/volcano.sh/apis/pkg/apis/scheduling/v1beta1/labels.go)
to be used throughout the implementation:

```go
const QueueAllocationGateKey = GroupName + "/queue-allocation-gate"
```

Besides, helper functions `api.HasOnlyVolcanoSchedulingGate()` and `api.HasQueueAllocationGateAnnotation()`, mentioned
throughout the document, are used to identify pods that participate in this gate management mechanism. These functions
can be defined in
[`pkg/scheduler/api/helpers.go`](https://github.com/volcano-sh/volcano/blob/v1.14.0/pkg/scheduler/api/helpers.go) to
avoid code duplication and maintain consistency across the codebase.

#### Changes to the Volcano `MutatingAdmissionWebhook`

Volcano's `MutatingAdmissionWebhook` must be extended to detect Pods annotated with
`schedulingv1beta1.QueueAllocationGateKey="true"`. At **creation time**, the webhook should patch the Pod's
`spec.schedulingGates` (through the
[`createPatch`](https://github.com/volcano-sh/volcano/blob/v1.14.0/pkg/webhooks/admission/pods/mutate/mutate_pod.go#L103)
function) by adding a new `schedulingGate` entry using the same constant:

```go
// Existing createPatch function needs to include the new optional patch
func createPatch(pod *v1.Pod) ([]byte, error) {

    // ...

    // Add scheduling gates if opted-in
    patchGates := patchSchedulingGates(pod)
    if patchGates != nil {
        patch = append(patch, *patchGates)
    }

    // ...
}

// patchSchedulingGates adds a scheduling gate for Volcano-managed pods.
// The gate prevents cluster autoscalers from seeing the pod until Volcano
// determines it's ready (queue admission + gang scheduling satisfied).
func patchSchedulingGates(pod *v1.Pod) *patchOperation {
	// Check if opt-in annotation is present
	if !api.HasQueueAllocationGateAnnotation(pod) {
		klog.V(4).Infof("Pod %s/%s does not have opt-in annotation, skipping gate",
			pod.Namespace, pod.Name)
		return nil
	}

	gate := v1.PodSchedulingGate{Name: schedulingv1beta1.QueueAllocationGateKey}

	// Idempotent: do not add a duplicate Volcano gate.
	// This prevents appending the same gate multiple times if the mutation is retried.
	for _, g := range pod.Spec.SchedulingGates {
		if g.Name == gate.Name {
			return nil
		}
	}

	// Parent missing: The schedulingGates slice hasn't been initialized yet.
	// We must use "add" on the base path with an array containing our gate.
	if pod.Spec.SchedulingGates == nil {
		return &patchOperation{
			Op:    "add",
			Path:  "/spec/schedulingGates",
			Value: []v1.PodSchedulingGate{gate},
		}
	}

	// Parent exists: We can safely append to the existing array.
	// Using the "-" path operator tells JSON Patch to append to the end of the array,
	// preventing us from overwriting gates added by parallel webhooks.
	return &patchOperation{
		Op:    "add",
		Path:  "/spec/schedulingGates/-",
		Value: gate,
	}
}
```

**`schedulingGates` Field Immutability**

Kubernetes `schedulingGates` can only be removed and **not added after pod creation**
([see PodSpec documentation](https://kubernetes.io/docs/reference/kubernetes-api/workload-resources/pod-v1/#scheduling)). As such, removing the gate can result in setting the `Unschedulable` condition on the pod if no node can fit it or actually binding it to a node.

#### Changes to the Volcano Scheduler

Volcano already adopted the concept of `schedulingGates` through the **Pod Scheduling Readiness** design document
([pod-scheduling-readiness.md](https://github.com/volcano-sh/volcano/blob/v1.14.0/docs/design/pod-scheduling-readiness.md)),
which skips tasks of a job whose Pods are scheduling gated, ensuring that a scheduling gated pod will not be bound to a
node. In this case, the
[allocate](https://github.com/volcano-sh/volcano/blob/v1.14.0/pkg/scheduler/actions/allocate/allocate.go#L255),
[backfill](https://github.com/volcano-sh/volcano/blob/v1.14.0/pkg/scheduler/actions/backfill/backfill.go#L145),
[reclaim](https://github.com/volcano-sh/volcano/blob/v1.14.0/pkg/scheduler/actions/reclaim/reclaim.go#L97), and
[preempt](https://github.com/volcano-sh/volcano/blob/v1.14.0/pkg/scheduler/actions/preempt/preempt.go#L234) actions
already skip Pods that
[have at least one gate](https://github.com/volcano-sh/volcano/blob/v1.14.0/pkg/scheduler/api/job_info.go#L250).

##### Queue Admission Checks

The [`GetSchGatedPodResources()`](https://github.com/volcano-sh/volcano/blob/v1.14.0/pkg/scheduler/api/job_info.go#L578)
method should be enhanced to exclude pods that only have the new proposed Volcano scheduling gate from the resource
deduction. This ensures these pods are counted in inqueue resources, making them **eligible for queue admission
checks**:

```go
func (ji *JobInfo) GetSchGatedPodResources() *Resource {
    res := EmptyResource()
    for _, task := range ji.Tasks {
        if task.SchGated {
            // Exclude tasks that are only Volcano scheduling gated
            // These should be counted in inqueue resources, not deducted
            if api.HasOnlyVolcanoSchedulingGate(task.Pod) {
                continue
            }
            res.Add(task.Resreq)
        }
    }
    return res
}
```

This change allows
[`DeductSchGatedResources()`](https://github.com/volcano-sh/volcano/blob/v1.14.0/pkg/scheduler/api/job_info.go#L592) to
treat Volcano-gated pods differently from other scheduling-gated pods, ensuring they remain visible to the enqueue
action while still being prevented from allocation until the gate is removed.

##### Queue Allocation and Gate Removal

In the current Volcano implementation, tasks with external (non-Volcano) scheduling gates are skipped when building the job worksheet, so they never enter the allocation loop (please refer to [`organizeJobWorksheet()`](https://github.com/volcano-sh/volcano/blob/v1.14.0/pkg/scheduler/actions/allocate/allocate.go#L255)). However, in this proposal, we must ensure that Pods with the new proposed Volcano scheduling gate are also included in the job worksheet, so they can be accounted for in the queue capacity check:

```go
// In organizeJobWorksheet(), when building the worksheet for each subJob:
for _, task := range subJob.TaskStatusIndex[api.Pending] {
    // Skip tasks with external (non-Volcano) scheduling gates
    if task.SchGated && !api.HasOnlyVolcanoSchedulingGate(task.Pod) {
        klog.V(4).Infof("Task <%v/%v> has external scheduling gate, skip it.",
            task.Namespace, task.Name)
        continue
    }
    // ... push task into worksheet ...
}
```

Afterwards, each time the `Allocate` action is executed,
[`allocateResourcesForTasks(...)`](https://github.com/volcano-sh/volcano/blob/v1.14.0/pkg/scheduler/actions/allocate/allocate.go#L551)
tries to allocate resources for each Task in a given Queue and
[`ssn.Allocatable(queue, task)`](https://github.com/volcano-sh/volcano/blob/v1.14.0/pkg/scheduler/actions/allocate/allocate.go#L569)
gives us the needed signal (by running every plugin check) for eventually removing the gate added previously by the
**MutatingAdmissionWebhook**. 

The implementation removes gates asynchronously after the capacity check passes, handled by background workers to avoid
blocking the scheduler. Gated tasks are skipped in the current cycle and only processed after the gate is actually
removed by the async worker. The following code snippet showcases the high-level changes to the function
[`allocateResourcesForTasks(...)`](https://github.com/volcano-sh/volcano/blob/v1.14.0/pkg/scheduler/actions/allocate/allocate.go#L551):

```go
// Enhance allocateResourcesForTasks with gate management
func (alloc *Action) allocateResourcesForTasks(...) {

    // ...

    for !tasks.Empty() {
        task := tasks.Pop().(*api.TaskInfo)

        // Check queue capacity (enhanced to account for reserved ungated pods)
        if !ssn.Allocatable(queue, task) {
            // ...
            continue
        }

		// If task passed allocation check and has the QueueAllocationGate, initiate async gate removal.
		// Gate will be removed by the background worker (best effort).
        if task.SchGated && api.HasQueueAllocationGateAnnotation(task.Pod) {
            klog.V(3).Infof("Task %s/%s has the QueueAllocationGate, queue async gate removal", task.Namespace, task.Name)
            alloc.schGateManager.enqueue(task)
        }

		// Skip gated tasks. If someone added the Volcano gate without the opt-in annotation,
		// warn them since the gate will never be removed automatically.
		if task.SchGated {
			if api.HasOnlyVolcanoSchedulingGate(task.Pod) && !api.HasQueueAllocationGateAnnotation(task.Pod) {
				klog.Warningf("Task %s/%s has Volcano scheduling gate but missing annotation %q; gate will not be removed automatically",
					task.Namespace, task.Name, schedulingv1beta1.QueueAllocationGateKey)
			}
			continue
		}

        // ... predicate checks ...
        // PrePredicate, node filtering, prioritization:
        // If any fail, pod is marked Unschedulable after the gate is removed in the next cycle

        // Allocate task to best node if found
        if err := alloc.allocateResourcesForTask(stmt, task, bestNode, job); err == nil {
            // ... allocation successful, will proceed to bind ...
        }
    }

    // ...
}
```

Note that gated tasks are **not** allocated in the same cycle, they are skipped after enqueuing the gate removal. The
task will be picked up in the next scheduling cycle once the async worker has removed the gate and the informer cache
reflects the updated pod state.

To support asynchronous gate removal, the gate management infrastructure could live in a dedicated
`SchGateManager` struct in `pkg/scheduler/gate/schedulinggate.go`. The manager is owned by the
`Scheduler` and its lifecycle is tied to the scheduler process. It should then be passed to each scheduling session via `OpenSession()` and accessed by the allocate
action through `ssn.SchGateManager()`:

```go
// SchGateManager handles asynchronous removal of Volcano scheduling gates from pods.
type SchGateManager struct {
    kubeClient kubernetes.Interface
    opCh       chan gateRemovalOp
    workersWg  sync.WaitGroup
    stopCh     chan struct{}
    workerNum  int
    // ...
}

// Background worker processes gate removal operations
func (m *SchGateManager) worker() {
    defer m.workersWg.Done()
    for {
        select {
        case <-m.stopCh:
            return
        case op := <-m.opCh:
            if err := cache.RemoveVolcanoSchGate(m.kubeClient, op.namespace, op.name); err != nil {
                klog.Errorf("Failed to remove gate from %s/%s: %v", op.namespace, op.name, err)
            }
        }
    }
}
```

The `Scheduler` creates and owns the manager:

```go
// In Scheduler.Run():
pc.schGateManager = gate.NewSchGateManager(pc.cache.Client(), options.ServerOpts.GateRemovalWorkerNum)
pc.schGateManager.Start()
go func() {
    <-stopCh
    pc.schGateManager.Stop()
}()

// Passed to each session:
ssn := framework.OpenSession(pc.cache, plugins, configurations, pc.schGateManager)
```

##### Queue Capacity Accounting for Ungated Pods

When a pod's gate is removed due to lack of cluster capacity, it becomes visible to autoscalers but may still remain
unallocated (_e.g._, waiting for matching nodes). This creates a potential race condition: between gate removal and
actual allocation to a node, other pods might consume the available queue capacity, leaving the ungated pod unable to
allocate despite being `Unschedulable`. To illustrate this race condition, consider the following scenario:

1. Three Pods (`pod-1`, `pod-2`, `pod-3`) with `schedulerName: volcano`, each requesting `1 CPU` and `1 GiB`.
2. `pod-2` has a `nodeSelector` for a node that does not exist yet (will be created by the cluster autoscaler).
3. A Volcano Queue with `capability: 1 CPU, 1 GiB`.

For each of these Pods, there should be **only one allocation at a time** in the Queue. The allocation can progress as
follows:

- Initially, the webhook **adds gates for every pod**:

  ```
  NAME    PHASE    CONDITION          GATES
  pod-1   Pending  SchedulingGated    scheduling.volcano.sh/queue-allocation-gate
  pod-2   Pending  SchedulingGated    scheduling.volcano.sh/queue-allocation-gate
  pod-3   Pending  SchedulingGated    scheduling.volcano.sh/queue-allocation-gate
  ```

- **After the 1st scheduling cycle**, `pod-1` passes the capacity check
([`ssn.Allocatable(queue, task)`](https://github.com/volcano-sh/volcano/blob/v1.14.0/pkg/scheduler/actions/allocate/allocate.go#L569)
returns `true`) and gets its gate removed. Since it does not have any node selector and there are nodes available, it
gets scheduled and eventually moves to the `Running` phase:

  ```
  NAME    PHASE    CONDITION          GATES
  pod-1   Running
  pod-2   Pending  SchedulingGated    scheduling.volcano.sh/queue-allocation-gate
  pod-3   Pending  SchedulingGated    scheduling.volcano.sh/queue-allocation-gate
  ```

- **After `pod-1` is deleted or terminates**, in the next cycle `pod-2` gets its gate removed (passes queue capacity check) but fails
  to find matching nodes due to its `nodeSelector`. It transitions to `Unschedulable`, in order to trigger the cluster
  autoscaler. However, **without capacity reservation** for `pod-2`, the queue will appear empty. If `pod-3` is created,
  it will pass the capacity check and get scheduled:

  ```
  NAME    PHASE    CONDITION          GATES
  pod-2   Pending  Unschedulable
  pod-3   Running
  ```

This creates the race condition since the queue can only handle one pod at a time (1 CPU capacity), but now we have
`pod-3` running **and** `pod-2` triggering autoscaling. When the autoscaler provisions a new node for `pod-2`, it **will
never be scheduled** because `pod-3` is already consuming the queue's capacity.

To prevent this, the queue capacity accounting logic must be enhanced to treat **ungated pods as "reservations" that
count toward the queue's capacity checks**. Specifically, the
[`ssn.Allocatable(queue, task)`](https://github.com/volcano-sh/volcano/blob/v1.14.0/pkg/scheduler/framework/session_plugins.go#L351)
function (which invokes the capacity plugin) needs to account for:

1. Pods with `AllocatedStatus` (bound, binding, running) toward queue capacity _(current behavior)_.
2. **`Pending` pods that have had their gates removed** as "reserved" resources _(new behavior)_.

###### Dynamic Reserved Calculation

Rather than modifying the `allocated` attribute (which semantically represents bound resources), we can add a new helper
method `queueAllocatableWithReserved` to the
[capacity plugin](https://github.com/volcano-sh/volcano/blob/v1.14.0/pkg/scheduler/plugins/capacity/capacity.go). This
new method extends the existing
[`queueAllocatable`](https://github.com/volcano-sh/volcano/blob/v1.14.0/pkg/scheduler/plugins/capacity/capacity.go#L885)
function by tracking reserved tasks in a per-task cache that is rebuilt at the start of each scheduling cycle and
updated incrementally during allocation.

The capacity plugin struct is extended with the reserved task cache:

```go
type capacityPlugin struct {
    // ... existing fields ...

    // queueGateReservedTasks tracks tasks that passed capacity checks but cannot be scheduled
    // These tasks reserve queue capacity to prevent other tasks from consuming it
    // Rebuilt fresh at the start of each scheduling cycle in OnSessionOpen
    queueGateReservedTasks map[api.QueueID]map[api.TaskID]*api.TaskInfo
}
```

###### Capacity Check with Reserved Resources

The `queueAllocatableWithReserved` method performs capacity checks including reserved resources:

```go
func (cp *capacityPlugin) queueAllocatableWithReserved(attr *queueAttr, candidate *api.TaskInfo, queue *api.QueueInfo) bool {
    // Calculate total reserved resources directly from cache
    reserved := api.EmptyResource()
    if queueGateReserved := cp.queueGateReservedTasks[queue.UID]; queueGateReserved != nil {
        for _, task := range queueGateReserved {
            if task.UID != candidate.UID {
                // Skip candidate to avoid double-counting (it will be added in futureUsed below)
                reserved.Add(task.Resreq)
            }
        }
    }

    // Include reserved resources in capacity check
    futureUsed := attr.allocated.Clone().Add(reserved).Add(candidate.Resreq)
    allocatable, _ := futureUsed.LessEqualWithDimensionAndResourcesName(attr.realCapability, candidate.Resreq)

    return allocatable
}
```

The existing
[`queueAllocatable`](https://github.com/volcano-sh/volcano/blob/v1.14.0/pkg/scheduler/plugins/capacity/capacity.go#L890)
method is modified to call this new helper, enhancing the checks performed during the allocation process by including
reserved resources:

```go
func (cp *capacityPlugin) queueAllocatable(queue *api.QueueInfo, candidate *api.TaskInfo) bool {
    attr := cp.queueOpts[queue.UID]
    return cp.queueAllocatableWithReserved(attr, candidate, queue)
}
```

###### Cache Initialization

The cache is rebuilt at the start of each scheduling cycle in
[`OnSessionOpen`](https://github.com/volcano-sh/volcano/blob/v1.14.0/pkg/scheduler/plugins/capacity/capacity.go#L91):

```go
func (cp *capacityPlugin) OnSessionOpen(ssn *framework.Session) {
    // Rebuild reserved cache for this scheduling cycle
    cp.buildQueueReservedTasksCache(ssn)

    // ... rest of capacity plugin initialization ...
}

func (cp *capacityPlugin) buildQueueReservedTasksCache(ssn *framework.Session) {
    // Initialize the cache for this session
    cp.queueGateReservedTasks = make(map[api.QueueID]map[api.TaskID]*api.TaskInfo)

    // Scan all pending tasks and rebuild cache
    for _, job := range ssn.Jobs {
        for _, task := range job.TaskStatusIndex[api.Pending] {
            // Tasks that passed capacity have: NO gate + HAS annotation + Pending status
            if !task.SchGated && api.HasQueueAllocationGateAnnotation(task.Pod) {
                if cp.queueGateReservedTasks[job.Queue] == nil {
                    cp.queueGateReservedTasks[job.Queue] = make(map[api.TaskID]*api.TaskInfo)
                }
                cp.queueGateReservedTasks[job.Queue][task.UID] = task
            }
        }
    }
}
```

###### Cache Updates During Allocation

The cache is updated incrementally during the scheduling session through the existing `AllocateFunc` and
`DeallocateFunc` event handlers registered by the capacity plugin.

**Adding Tasks to Reserved Cache**

When a task passes the capacity check, the capacity plugin adds it to the reserved cache directly within its
[`AddAllocatableFn`](https://github.com/volcano-sh/volcano/blob/v1.14.0/pkg/scheduler/plugins/capacity/capacity.go#L202)
callback registered in
[`OnSessionOpen()`](https://github.com/volcano-sh/volcano/blob/v1.14.0/pkg/scheduler/plugins/capacity/capacity.go#L91):

```go
// In capacity plugin's OnSessionOpen
ssn.AddAllocatableFn(cp.Name(), func(queue *api.QueueInfo, candidate *api.TaskInfo) bool {
    // ... queue state checks ...

    allocatable := cp.checkQueueAllocatableHierarchically(ssn, queue, candidate)

    // If queue has capacity and task has the QueueAllocationGate annotation.
    if allocatable && api.HasQueueAllocationGateAnnotation(candidate.Pod) {
        cp.addTaskToReservedCache(queue.UID, candidate)
    }

    return allocatable
})
```

**Removing Tasks from Reserved Cache on Allocation**

When a task is successfully allocated (tentatively assigned to a node), the capacity plugin's existing `AllocateFunc`
event handler removes it from the reserved cache. At this point the task's resources are counted in `attr.allocated`,
so they must no longer be in the reserved cache to avoid double-counting:

```go
// In capacity plugin's AllocateFunc event handler
AllocateFunc: func(event *framework.Event) {
    // ... existing allocated resource accounting ...

    // Remove task from reserved cache when it gets allocated
    cp.removeTaskFromReservedCache(event.Task.UID)
},
```

**Restoring Tasks to Reserved Cache on Rollback**

If an allocation is rolled back (*e.g.*, gang scheduling where not all tasks could be placed), the `DeallocateFunc`
event handler restores the task to the reserved cache so capacity remains accounted for:

```go
// In capacity plugin's DeallocateFunc event handler
DeallocateFunc: func(event *framework.Event) {
    // ... existing deallocated resource accounting ...

    // Restore task to reserved cache on rollback
    if api.HasQueueAllocationGateAnnotation(event.Task.Pod) {
        cp.addTaskToReservedCache(job.Queue, event.Task)
    }
},
```

The capacity plugin owns all its reservation logic internally through private methods (`addTaskToReservedCache` and `removeTaskFromReservedCache`):

```go
func (cp *capacityPlugin) addTaskToReservedCache(queueID api.QueueID, task *api.TaskInfo) {
    if cp.queueGateReservedTasks[queueID] == nil {
        cp.queueGateReservedTasks[queueID] = make(map[api.TaskID]*api.TaskInfo)
    }
    cp.queueGateReservedTasks[queueID][task.UID] = task
}

func (cp *capacityPlugin) removeTaskFromReservedCache(taskID api.TaskID) {
    for queueID, tasks := range cp.queueGateReservedTasks {
        if _, exists := tasks[taskID]; exists {
            delete(tasks, taskID)
            if len(tasks) == 0 {
                delete(cp.queueGateReservedTasks, queueID)
            }
            return
        }
    }
}
```

## Feature gate

The entire scheduling-gates queue-admission flow is controlled by a [feature gate](https://github.com/volcano-sh/volcano/blob/v1.14.0/pkg/features/volcano_features.go), for example, `SchedulingGatesQueueAdmission` (Alpha and off by default). When the gate is enabled, all of the behavior described in this document applies. This keeps existing clusters and configurations unchanged unless the feature is explicitly enabled (*e.g.* `--feature-gates=SchedulingGatesQueueAdmission=true`).

## Limitations

Once a pod's gate is removed (*i.e.*, because it has passed the queue capacity check), the pod is accounted for in the queue's reserved capacity until it is scheduled or the reservation is cleaned up. If that pod remains unschedulable (*e.g.*, no node matches, or the cluster is waiting for the autoscaler to add nodes), it continues to hold queue capacity. The scheduler cannot distinguish an ungated pod that will never become schedulable from one that will become schedulable after the autoscaler expands the cluster. As a result, other pods that could schedule may be blocked by this reserved capacity indefinitely. In this version of the design, reservations are not released for long-unschedulable pods (*e.g.*, via a timeout) to avoid over-committing the queue when those pods eventually get nodes. Operators should be aware that in such scenarios, queue capacity can be held by ungated-but-unschedulable pods and may delay scheduling of other workloads.

## Conclusions

Scheduling Gates are a Kubernetes feature that allows external controllers to delay pod scheduling until specific
conditions are met. The proposed design leverages this mechanism to defer scheduling until the queue has sufficient
capacity, preventing pods from appearing as `Unschedulable` when they're simply waiting for queue admission and falsely
trigger cluster scale-ups.

This implementation requires pods to opt in via the `schedulingv1beta1.QueueAllocationGateKey: "true"` annotation
(defined as `scheduling.volcano.sh/queue-allocation-gate`), making it a conservative approach ensuring backward compatibility
whilst allowing users to adopt the feature gradually. Future iterations **could enable this behavior by default** once
the feature maturity is validated in production environments.

## Related Issues

The following issues are related to the matters discussed in this proposal:

- [PodGroup isn't triggering scaling up in Kubernetes, when using Cluster Autoscaler \#2558](https://github.com/volcano-sh/volcano/issues/2558)
- [Remove Undetermined reason to fix cluster autoscaler compatibility \#2602](https://github.com/volcano-sh/volcano/pull/2602)
- [Support Pod Scheduling Readiness \#3555](https://github.com/volcano-sh/volcano/issues/3555)
- [Cluster Autoscaler node scale-up for Pods that exceed Queue's capability \#4710](https://github.com/volcano-sh/volcano/issues/4710)
