DRA: workload requeued after backoff loses its preprocessed DRA resources (queue Info overwritten)
kind/bug
<!-- Not security-related: alpha KueueDRAIntegration, self-limited, no demonstrated over-admission. Filing publicly per that judgment. -->
/kind bug
**What happened**:
With `KueueDRAIntegration` enabled, `handleDRA` pre-computes a workload's DRA-backed resources and hands them to the queue through `workload.WithPreprocessedDRAResources(...)`, so the queued `Info` reflects the workload's real device/quota footprint. The backoff-requeue path in `Reconcile` re-adds the workload to the queue *without* that option, so the queued `Info` is rebuilt from the raw pod-spec requests: the DRA logical resources are dropped, and the extended resources that DRA was meant to replace are left in.
`handleDRA` builds the options and, for an admissible workload, adds it to the queue with them ([workload_controller.go#L209-L215](https://github.com/kubernetes-sigs/kueue/blob/0db50952edb58b8ae948ae9e12a818aabfc5418c/pkg/controller/core/workload_controller.go#L209-L215)):
```go
var queueOptions []workload.InfoOption
if len(draResources) > 0 || len(replacedExtendedResources) > 0 {
queueOptions = append(queueOptions, workload.WithPreprocessedDRAResources(draResources, replacedExtendedResources))
}
if workload.IsAdmissible(wl) {
if err := r.queues.AddOrUpdateWorkload(log, wl.DeepCopy(), queueOptions...); err != nil { ... }
}
```
The backoff-requeue path calls the same method with no options ([#L556-L561](https://github.com/kubernetes-sigs/kueue/blob/0db50952edb58b8ae948ae9e12a818aabfc5418c/pkg/controller/core/workload_controller.go#L556-L561)):
```go
if !workload.IsAdmissible(&wl) {
return ctrl.Result{}, nil
}
if err := r.queues.AddOrUpdateWorkload(log, wl.DeepCopy()); err != nil { // no queueOptions
...
}
```
That option is what makes the accounting correct. `Info`'s per-podset requests only fold in the DRA logical resources (and remove the replaced extended resources) when `preprocessedDRAResources != nil` ([workload.go#L649-L664](https://github.com/kubernetes-sigs/kueue/blob/0db50952edb58b8ae948ae9e12a818aabfc5418c/pkg/workload/workload.go#L649-L664)):
```go
if features.Enabled(features.KueueDRAIntegration) && info.preprocessedDRAResources != nil {
// first, remove extended resources converted to DRA logical resources
// then, add the DRA logical resources
}
```
Both calls run in the same reconcile for the same workload. `handleDRA` is invoked at [#L523-L527](https://github.com/kubernetes-sigs/kueue/blob/0db50952edb58b8ae948ae9e12a818aabfc5418c/pkg/controller/core/workload_controller.go#L523-L527) and returns `done == false` on the admissible path, so `Reconcile` falls through to the requeue block at L536. `IsAdmissible` is true for a pending workload that holds no quota reservation ([workload.go#L1234](https://github.com/kubernetes-sigs/kueue/blob/0db50952edb58b8ae948ae9e12a818aabfc5418c/pkg/workload/workload.go#L1234)) — exactly a workload whose backoff has just elapsed. So `handleDRA` first stores the correct, DRA-adjusted `Info` at L215, and then L561 overwrites it with the option-less one. The good footprint is discarded within the same reconcile.
**What you expected to happen**:
The requeue-after-backoff path should queue the workload with the same DRA-adjusted footprint that `handleDRA` produces: either by passing the preprocessed DRA resources to `AddOrUpdateWorkload` at L561, or by not overwriting the entry `handleDRA` already added with them.
**How to reproduce it (as minimally and precisely as possible)**:
Read path (no cluster needed): on `main` (0db5095), compare the two `r.queues.AddOrUpdateWorkload` calls in `pkg/controller/core/workload_controller.go` — L215 passes `queueOptions`, L561 does not — against the consumer in `pkg/workload/workload.go` L649, which only adjusts the requests when `preprocessedDRAResources != nil`.
Behavior (with `KueueDRAIntegration` on): submit a workload that requests DRA-backed resources (ResourceClaimTemplate / DRA-backed extended resources) and is not immediately admitted, so it goes through a backoff. After the backoff elapses and the workload is requeued, its queue `Info` no longer includes the DRA resources it needs, and any extended resources that should have been replaced are counted instead.
**Anything else we need to know?**:
Behind the alpha `KueueDRAIntegration` gate. I traced the controller and the `Info` builder but not the scheduler's admission path, so I can't say whether the scheduler admits or orders the workload from the option-less `Info` before a later reconcile re-adds it correctly. I'm reporting this as an accounting inconsistency on the requeue path, not a demonstrated over-admission. The fix likely needs the preprocessed resources available at L561 (they're currently local to `handleDRA`); happy to send a PR if the approach sounds right.
Verified by reading `main` at `0db50952edb58b8ae948ae9e12a818aabfc5418c`. I only looked at `main`.
**Environment**:
- Kubernetes version (use `kubectl version`): n/a (control-plane logic, read on `main`)
- Kueue version (use `git describe --tags --dirty --always`): `main` at 0db5095
- Cloud provider or hardware configuration: n/a
- OS: n/a
- Others: `KueueDRAIntegration` feature gate enabled
2 条评论