A ClusterQueue custom metric label makes pod_scheduling_gate_removal_seconds panic, exiting the manager from the ungaters

**What happened**: `RecordPodSchedulingGateRemovalSeconds` passes three label values to a vector that was registered with three fixed labels plus the configured ClusterQueue custom labels, so with `CustomMetricLabels` enabled and at least one ClusterQueue-sourced label configured, the first scheduling gate removal panics the controller. The vector is registered at `pkg/metrics/metrics.go:664-670`: ```go PodSchedulingGateRemovalSeconds = prometheus.NewHistogramVec( prometheus.HistogramOpts{...}, append([]string{"name", "cluster_queue", "is_group"}, clusterQueueMetricsLabels...), ) ``` and recorded at `pkg/metrics/metrics.go:1041-1043`: ```go func RecordPodSchedulingGateRemovalSeconds(name string, clusterQueue kueue.ClusterQueueReference, isGroup bool, latency time.Duration) { PodSchedulingGateRemovalSeconds.WithLabelValues(name, string(clusterQueue), strconv.FormatBool(isGroup)).Observe(latency.Seconds()) } ``` This is the only recording helper in that file that takes `clusterQueueMetricsLabels` in its registration and doesn't append `customLabelValues` when it records. `RecordWorkloadCreationLatency` (`:1036`) and `QuotaReservedWorkload` (`:1045`) both build `labels := append([]string{...}, customLabelValues...)` first. **What you expected to happen**: The observation is recorded, as it is for every other metric carrying custom labels. **How to reproduce it (as minimally and precisely as possible)**: Configure at least one custom metric label and enable the gate. The source kind can be left out, since `DefaultCustomMetricLabelSourceKind` (`apis/config/v1beta2/defaults.go:55`) is already `ClusterQueue`: ```yaml featureGates: CustomMetricLabels: true controllerMetrics: customLabels: - name: team ``` Then run any workload whose Pods carry a Kueue scheduling gate. The callers are `pkg/controller/jobs/pod/pod_controller.go:298` and `:337`, `pkg/controller/tas/topology_ungater.go:328`, and `pkg/controller/elasticjobs/elastic_job_ungater.go:202`, all through `pkg/util/pod/pod.go:189`. As a unit test against `v0.20.0-devel-296-g0e28ec3c0`: ```go func TestProbeGateRemovalCardinality(t *testing.T) { features.SetFeatureGateDuringTest(t, features.CustomMetricLabels, true) cl := NewCustomLabels([]configapi.ControllerMetricsCustomLabel{{Name: "team"}}) InitMetricVectors(cl) QuotaReservedWorkload("cq", "pc", time.Second, []string{"t"}, nil) // control RecordPodSchedulingGateRemovalSeconds("wl", "cq", false, time.Second) } ``` ``` extra ClusterQueue labels: [custom_team] CONTROL OK: sibling with custom values recorded fine SUBJECT PANIC: inconsistent label cardinality: expected 4 label values but got 3 in []string{"wl", "cq", "false"} ``` The control line matters: the same configuration records fine through a helper that appends the custom values, so the cardinality itself isn't the problem. **Anything else we need to know?**: Two things worth stating so the severity is not read as higher than it is. A default installation isn't affected. `NewCustomLabels` returns nil unless the `CustomMetricLabels` gate is on and the config lists at least one label (`pkg/metrics/custom_labels.go:66-69`), and `LabelNames` then returns nil, so the three recorded values match the three registered ones. The default source kind being `ClusterQueue` decides where a configured label reads its value from; it doesn't add a label on its own. The same registration is also missing `replica_role`, which every other ClusterQueue-scoped vector in that file carries. I haven't checked whether that's deliberate, so I am mentioning it rather than reporting it. The metric was added in #12137. I did not find an existing issue or open PR for this. I am happy to send the fix. **Environment**: - Kubernetes version: n/a, reproduced as a unit test - Kueue version (use `git describe --tags --dirty --always`): `v0.20.0-devel-296-g0e28ec3c0` - Cloud provider or hardware configuration: n/a - OS: Linux - Kernel: n/a - Install tools: n/a - Others: go1.26.5 --- This issue was written in part with the assistance of generative AI. I ran the reproducer above and read each call site myself.