<!--
**Note:** When your KEP is complete, all of these comment blocks should be removed.

To get started with this template:

- [ ] **Fill out this file as best you can.**
  At minimum, you should fill in the "Summary" and "Motivation" sections.
  These should be easy if you've preflighted the idea of the KEP with the
  appropriate SIG(s).
- [ ] **Create a PR for this KEP.**
  Assign it to people in the SIG who are sponsoring this process.
- [ ] **Merge early and iterate.**
  Avoid getting hung up on specific details and instead aim to get the goals of
  the KEP clarified and merged quickly. The best way to do this is to just
  start with the high-level sections and fill out details incrementally in
  subsequent PRs.

Just because a KEP is merged does not mean it is complete or approved. Any KEP
marked as `provisional` is a working document and subject to change. You can
denote sections that are under active debate as follows:

```
<<[UNRESOLVED optional short context or usernames ]>>
Stuff that is being argued.
<<[/UNRESOLVED]>>
```

When editing KEPS, aim for tightly-scoped, single-topic PRs to keep discussions
focused. If you disagree with what is already in a document, open a new PR
with suggested changes.

One KEP corresponds to one "feature" or "enhancement" for its whole lifecycle.
You do not need a new KEP to move from beta to GA, for example. If
new details emerge that belong in the KEP, edit the KEP. Once a feature has become
"implemented", major changes should get new KEPs.

The canonical place for the latest set of instructions (and the likely source
of this file) is [here](/keps/NNNN-kep-template/README.md).

**Note:** Any PRs to move a KEP to `implementable`, or significant changes once
it is marked `implementable`, must be approved by each of the KEP approvers.
If none of those approvers are still appropriate, then changes to that list
should be approved by the remaining approvers and/or the owning SIG (or
SIG Architecture for cross-cutting KEPs).
-->
# KEP-4951: Configurable tolerance for HPA

<!--
Ensure the TOC is wrapped with
  <code>&lt;!-- toc --&rt;&lt;!-- /toc --&rt;</code>
tags, and then generate with `hack/update-toc.sh`.
-->

<!-- toc -->
- [Release Signoff Checklist](#release-signoff-checklist)
- [Summary](#summary)
- [Motivation](#motivation)
  - [Goals](#goals)
  - [Non-Goals](#non-goals)
- [Proposal](#proposal)
  - [Risks and Mitigations](#risks-and-mitigations)
- [Design Details](#design-details)
  - [Test Plan](#test-plan)
      - [Prerequisite testing updates](#prerequisite-testing-updates)
      - [Unit tests](#unit-tests)
      - [Integration tests](#integration-tests)
      - [e2e tests](#e2e-tests)
  - [Graduation Criteria](#graduation-criteria)
    - [Alpha](#alpha)
    - [Beta](#beta)
    - [Stable](#stable)
  - [Upgrade / Downgrade Strategy](#upgrade--downgrade-strategy)
    - [Upgrade](#upgrade)
    - [Downgrade](#downgrade)
  - [Version Skew Strategy](#version-skew-strategy)
- [Production Readiness Review Questionnaire](#production-readiness-review-questionnaire)
  - [Feature Enablement and Rollback](#feature-enablement-and-rollback)
  - [Rollout, Upgrade and Rollback Planning](#rollout-upgrade-and-rollback-planning)
  - [Monitoring Requirements](#monitoring-requirements)
  - [Dependencies](#dependencies)
  - [Scalability](#scalability)
  - [Troubleshooting](#troubleshooting)
- [Implementation History](#implementation-history)
- [Drawbacks](#drawbacks)
- [Alternatives](#alternatives)
- [Infrastructure Needed (Optional)](#infrastructure-needed-optional)
<!-- /toc -->

## Release Signoff Checklist

<!--
**ACTION REQUIRED:** In order to merge code into a release, there must be an
issue in [kubernetes/enhancements] referencing this KEP and targeting a release
milestone **before the [Enhancement Freeze](https://git.k8s.io/sig-release/releases)
of the targeted release**.

For enhancements that make changes to code or processes/procedures in core
Kubernetes—i.e., [kubernetes/kubernetes], we require the following Release
Signoff checklist to be completed.

Check these off as they are completed for the Release Team to track. These
checklist items _must_ be updated for the enhancement to be released.
-->

Items marked with (R) are required *prior to targeting to a milestone / release*.

- [x] (R) Enhancement issue in release milestone, which links to KEP dir in [kubernetes/enhancements] (not the initial KEP PR)
- [x] (R) KEP approvers have approved the KEP status as `implementable`
- [x] (R) Design details are appropriately documented
- [x] (R) Test plan is in place, giving consideration to SIG Architecture and SIG Testing input (including test refactors)
  - [x] e2e Tests for all Beta API Operations (endpoints)
  - [x] (R) Ensure GA e2e tests meet requirements for [Conformance Tests](https://github.com/kubernetes/community/blob/master/contributors/devel/sig-architecture/conformance-tests.md)
  - [x] (R) Minimum Two Week Window for GA e2e tests to prove flake free
- [x] (R) Graduation criteria is in place
  - [x] (R) [all GA Endpoints](https://github.com/kubernetes/community/pull/1806) must be hit by [Conformance Tests](https://github.com/kubernetes/community/blob/master/contributors/devel/sig-architecture/conformance-tests.md)
- [x] (R) Production readiness review completed
- [x] (R) Production readiness review approved
- [x] "Implementation History" section is up-to-date for milestone
- [x] User-facing documentation has been created in [kubernetes/website], for publication to [kubernetes.io]
- [x] Supporting documentation—e.g., additional design documents, links to mailing list discussions/SIG meetings, relevant PRs/issues, release notes

[kubernetes.io]: https://kubernetes.io/
[kubernetes/enhancements]: https://git.k8s.io/enhancements
[kubernetes/kubernetes]: https://git.k8s.io/kubernetes
[kubernetes/website]: https://git.k8s.io/website

## Summary

<!--
This section is incredibly important for producing high-quality, user-focused
documentation such as release notes or a development roadmap. It should be
possible to collect this information before implementation begins, in order to
avoid requiring implementors to split their attention between writing release
notes and implementing the feature itself. KEP editors and SIG Docs
should help to ensure that the tone and content of the `Summary` section is
useful for a wide audience.

A good summary is probably at least a paragraph in length.

Both in this section and below, follow the guidelines of the [documentation
style guide]. In particular, wrap lines to a reasonable length, to make it
easier for reviewers to cite specific portions, and to minimize diff churn on
updates.

[documentation style guide]: https://github.com/kubernetes/community/blob/master/contributors/guide/style-guide.md
-->

[Horizontal Pod Autoscaler][] (HPA) regularly estimates how many replicas a given Deployment (or other resource with a `/scale` subresource) should instantiate.
HPAs define one (or more) metrics (e.g. CPU utilization) on which autoscaling is based. The number of replicas is derived from the ratio between the *expected* and *current* value of this metric ([Algorithm details][]).

For example, for a workload with 100 `currentReplicas` and a usage ratio
(`currentMetricValue`/`desiredMetricValue`) of 1.07, the calculated `desiredReplicas`
would be 107  (100 * 1.07).

However, to avoid flapping, scaling actions are skipped if the usage ratio is approximately 1, within a
globally-configurable *tolerance*,  set to 10% by default. In the example above, no scaling action would
take place, since the ratio is within this tolerance.

This proposal adds a parameter to HPAs allowing users to configure this tolerance per HPA resource.
For the example above, we could configure the tolerance in the workload's HPA to 5%, which would
allow the scale-up to 107 replicas to proceed.

[Horizontal Pod Autoscaler]: https://kubernetes.io/docs/tasks/run-application/horizontal-pod-autoscale/
[Algorithm details]: https://kubernetes.io/docs/tasks/run-application/horizontal-pod-autoscale/#algorithm-details

## Motivation

Today the horizontal autoscaling tolerance is a cluster-wide parameter set using the [Kube Control Manager][] `--horizontal-pod-autoscaler-tolerance` parameter. It is by default set to 10%. While this value is often appropriate, it is considered too coarse grained in a number of scenario.

This issue has been raised multiple times ([#116984][], [#125987][], [#62013][],
[#aks-3068][], [#keda-1100][]), with users commenting that:

1. For large deployments, a 10% tolerance translates into very significant resources (i.e. hundreds of pods).
2. This tolerance can slow down scaling operations, hindering responsiveness in
case of surges.
3. Scale-ups are more a problem than scale-downs since typically pods are slower to initialize than to shut down, and since responding to load increase is typically more critical than freeing resources.

Since appropriate tolerance values are workload-dependent, this KEP proposes to let users add custom tolerance values to `HorizontalPodAutoscaler` resources, overriding the existing default value when present.

This solution integrates seamlessly with the existing HPA API since it already allows users to [fine-tune the autoscaler behavior](https://kubernetes.io/docs/tasks/run-application/horizontal-pod-autoscale/#configurable-scaling-behavior).
The exact API recommended here has been previously proposed in [kep-853][] (see [here](https://github.com/kubernetes/enhancements/pull/1234#discussion_r333036990)), but it was then decided to implement it separately.

[#116984]: https://github.com/kubernetes/kubernetes/issues/116984
[#125987]: https://github.com/kubernetes/kubernetes/issues/125987
[#62013]: https://github.com/kubernetes/kubernetes/issues/62013
[#aks-3068]: https://github.com/Azure/AKS/issues/3068
[#keda-1100]: https://github.com/kedacore/keda-docs/issues/1100
[Kube Control Manager]: https://kubernetes.io/docs/reference/command-line-tools-reference/kube-controller-manager/
[kep-853]: https://github.com/kubernetes/enhancements/blob/master/keps/sig-autoscaling/853-configurable-hpa-scale-velocity/README.md

### Goals

- Allow users to optionally override the default workload autoscaling tolerance on a per-HPA bases.

### Non-Goals

- Allow to customize the cluster-wise tolerance given by Kube Control Manager `--horizontal-pod-autoscaler-tolerance` parameter.


## Proposal

We propose to add a new field to the existing [`HPAScalingRules`][] object:

- `tolerance`: (float) the minimum change (from 1.0) in the desired-to-actual metrics ratio for the horizontal pod autoscaler to consider scaling. Must be greater than or equal to 0.

The `tolerance` field is optional, and when not specified the HPA will continue to use the
value of the global `--horizontal-pod-autoscaler-tolerance` as the tolerance for scaling
calculations.

Since there are separate `HPAScalingRules` objects defined for an HPA's
`spec.behavior.scaleUp` and `spec.behavior.scaleDown`, it is possible to specify different
`tolerance` values for scaling up vs. scaling down.

[HPAScalingRules]: https://kubernetes.io/docs/reference/generated/kubernetes-api/v1.31/#hpascalingrules-v2-autoscaling

### Risks and Mitigations

There should be minimal risk introduced by the proposed changes:
- The new field is optional, and its absence results in no changes to the current autoscaling behavior
- When specified, the new value doesn't change the autoscaling algorithm used, but just overrides a single value used during the calculation. This value can already be changed via the `--horizontal-pod-autoscaler-tolerance` option of the `kube-controller-manager`.
- If a change to the new field results in undesirable behavior, the change can be reverted by deploying the previous version of the HPA resource, or removing the `tolerance` field entirely.

## Design Details

The `HorizontalPodAutoscaler` API is updated to add a new `tolerance` field to the `HPAScalingRules` object:

```golang
type HPAScalingRules struct {
  // tolerance is the tolerance on the ratio between the current and desired
  // metric value under which no updates are made to the desired number of
  // replicas.
  // +optional
  Tolerance *resource.Quantity

  // Existing fields.
  StabilizationWindowSeconds *int32
  SelectPolicy *ScalingPolicySelect
  Policies []HPAScalingPolicy
}
```

This new tolerance will be used in the autoscaling controller
[replica_calculator.go][]. The current logic is:

```golang
if math.Abs(1.0-usageRatio) <= c.tolerance { /* ... */ }
```

It will be replaced by:

```diff
- if math.Abs(1.0-usageRatio) <= c.tolerance { /* ... */ }
+ // Down and Up scaling tolerances default to c.tolerance if unset.
+ downTolerance, upTolerance := c.tolerance, c.tolerance
+ if scaleDown.tolerance != nil {
+   downTolerance = scaleDown.tolerance.AsApproximateFloat64()
+ }
+ if scaleUp.tolerance != nil {
+   upTolerance = scaleUp.tolerance.AsApproximateFloat64()
+ }
+
+ if (1.0-downTolerance) <= usageRatio && usageRatio <= (1.0+upTolerance) { /* ... */ }
```

Since the added field is optional and its omission does not change the existing
autoscaling behavior, this feature will only be added to the latest stable API
version `pkg/apis/autoscaling/v2`. Older versions (i.e. `v1`, `v2beta1`,
`v2beta2`) will not include the new field, but converters will be updated where
needed to comply with [round-trip requirements][].

The feature presented in this KEP only allows users to tune an existing parameter, and
as such doesn't require any new HPA Events or modify any Status. The validation logic
will be updated to ensure that the `tolerance` field cannot be set to a negative value.

[replica_calculator.go]: https://github.com/kubernetes/kubernetes/blob/master/pkg/controller/podautoscaler/replica_calculator.go
[round-trip requirements]: https://kubernetes.io/docs/reference/using-api/deprecation-policy/#deprecating-parts-of-the-api

### Test Plan

<!--
**Note:** *Not required until targeted at a release.*
The goal is to ensure that we don't accept enhancements with inadequate testing.

All code is expected to have adequate tests (eventually with coverage
expectations). Please adhere to the [Kubernetes testing guidelines][testing-guidelines]
when drafting this test plan.

[testing-guidelines]: https://git.k8s.io/community/contributors/devel/sig-testing/testing.md
-->

[x] I/we understand the owners of the involved components may require updates to
existing tests to make this code solid enough prior to committing the changes necessary
to implement this enhancement.

##### Prerequisite testing updates

<!--
Based on reviewers feedback describe what additional tests need to be added prior
implementing this enhancement to ensure the enhancements have also solid foundations.
-->

##### Unit tests

<!--
In principle every added code should have complete unit test coverage, so providing
the exact set of tests will not bring additional value.
However, if complete unit test coverage is not possible, explain the reason of it
together with explanation why this is acceptable.
-->

<!--
Additionally, for Alpha try to enumerate the core package you will be touching
to implement this enhancement and provide the current unit coverage for those
in the form of:
- <package>: <date> - <current test coverage>
The data can be easily read from:
https://testgrid.k8s.io/sig-testing-canaries#ci-kubernetes-coverage-unit

This can inform certain test coverage improvements that we want to do before
extending the production code to implement this enhancement.
-->

- `/apis/autoscaling/validation`: `2024-11-13` - `95.6`
- `/pkg/controller/podautoscaler`: `2024-11-13` - `96.4`

##### Integration tests

<!--
Integration tests are contained in k8s.io/kubernetes/test/integration.
Integration tests allow control of the configuration parameters used to start the binaries under test.
This is different from e2e tests which do not allow configuration of parameters.
Doing this allows testing non-default options and multiple different and potentially conflicting command line options.
-->

<!--
This question should be filled when targeting a release.
For Alpha, describe what tests will be added to ensure proper quality of the enhancement.

For Beta and GA, add links to added tests together with links to k8s-triage for those tests:
https://storage.googleapis.com/k8s-triage/index.html
-->

A test will be added to the [podautoscaler integration tests] to ensure HPA
configurable tolerances are correctly taken into account both when scaling up
and down.

- [Test HPA with tolerance](https://github.com/kubernetes/kubernetes/tree/master/test/integration/podautoscaler) ([Testgrid](https://testgrid.k8s.io/sig-release-master-blocking#integration-master&width=20&include-filter-by-regex=sig-autoscaling) [triage search](https://storage.googleapis.com/k8s-triage/index.html?job=ci-kubernetes-integration&test=sig-autoscaling))

[podautoscaler integration tests]: https://github.com/kubernetes/kubernetes/tree/master/test/integration/podautoscaler

##### e2e tests

<!--
This question should be filled when targeting a release.
For Alpha, describe what tests will be added to ensure proper quality of the enhancement.

For Beta and GA, add links to added tests together with links to k8s-triage for those tests:
https://storage.googleapis.com/k8s-triage/index.html

We expect no non-infra related flakes in the last month as a GA graduation criteria.
-->

Existing e2e tests ensure the autoscaling behavior uses the default tolerance when no
configurable tolerance is specified.

The new [e2e autoscaling tests] covering this feature are:

- [Test with large configurable tolerance](https://github.com/kubernetes/kubernetes/blob/07142400ecd02126602ffaa6f91712cd3f1e170c/test/e2e/autoscaling/horizontal_pod_autoscaling_behavior.go#L509): [SIG autoscaling](https://testgrid.k8s.io/sig-autoscaling-hpa-presubmits#gci-gce-autoscaling-hpa-cpu-alpha-beta-pull&include-filter-by-regex=HPAConfigurableTolerance.*large%20configurable%20tolerance), [triage search](https://storage.googleapis.com/k8s-triage/index.html?test=HPAConfigurableTolerance.*large%20configurable%20tolerance)
- [Test with both larger and smaller than default tolerances](https://github.com/kubernetes/kubernetes/blob/96914184bc3fe6d3c0195b5828c07a821ca6e093/test/e2e/autoscaling/horizontal_pod_autoscaling_behavior.go#L549): [SIG autoscaling](https://testgrid.k8s.io/sig-autoscaling-hpa-presubmits#gci-gce-autoscaling-hpa-cpu-alpha-beta-pull&include-filter-by-regex=HPAConfigurableTolerance.*small.*tolerance), [triage search](https://storage.googleapis.com/k8s-triage/index.html?test=HPAConfigurableTolerance.*small.*tolerance)

[e2e autoscaling tests]: https://github.com/kubernetes/kubernetes/tree/master/test/e2e/autoscaling

### Graduation Criteria

<!--
**Note:** *Not required until targeted at a release.*

Define graduation milestones.

These may be defined in terms of API maturity, [feature gate] graduations, or as
something else. The KEP should keep this high-level with a focus on what
signals will be looked at to determine graduation.

Consider the following in developing the graduation criteria for this enhancement:
- [Maturity levels (`alpha`, `beta`, `stable`)][maturity-levels]
- [Feature gate][feature gate] lifecycle
- [Deprecation policy][deprecation-policy]

Clearly define what graduation means by either linking to the [API doc
definition](https://kubernetes.io/docs/concepts/overview/kubernetes-api/#api-versioning)
or by redefining what graduation means.

In general we try to use the same stages (alpha, beta, GA), regardless of how the
functionality is accessed.

[feature gate]: https://git.k8s.io/community/contributors/devel/sig-architecture/feature-gates.md
[maturity-levels]: https://git.k8s.io/community/contributors/devel/sig-architecture/api_changes.md#alpha-beta-and-stable-versions
[deprecation-policy]: https://kubernetes.io/docs/reference/using-api/deprecation-policy/

Below are some examples to consider, in addition to the aforementioned [maturity levels][maturity-levels].

#### Alpha

- Feature implemented behind a feature flag
- Initial e2e tests completed and enabled

#### Beta

- Gather feedback from developers and surveys
- Complete features A, B, C
- Additional tests are in Testgrid and linked in KEP

#### GA

- N examples of real-world usage
- N installs
- More rigorous forms of testing—e.g., downgrade tests and scalability tests
- Allowing time for feedback

**Note:** Generally we also wait at least two releases between beta and
GA/stable, because there's no opportunity for user feedback, or even bug reports,
in back-to-back releases.

**For non-optional features moving to GA, the graduation criteria must include
[conformance tests].**

[conformance tests]: https://git.k8s.io/community/contributors/devel/sig-architecture/conformance-tests.md

#### Deprecation

- Announce deprecation and support policy of the existing flag
- Two versions passed since introducing the functionality that deprecates the flag (to address version skew)
- Address feedback on usage/changed behavior, provided on GitHub issues
- Deprecate the flag
-->

#### Alpha

- Feature implemented behind a `HPAConfigurableTolerance` feature flag
- Initial e2e tests completed and enabled

#### Beta

- All tests described in the [`e2e tests` section](#e2e-tests) are implemented
  and linked in this KEP.
- We have monitored for negative user feedback and addressed relevant concerns.

#### Stable

- Observe real-world usage.
- We have monitored for negative user feedback and addressed relevant concerns.

### Upgrade / Downgrade Strategy

#### Upgrade
Existing HPAs will continue to work as they do today, using the global `horizontal-pod-autoscaler-tolerance`
value from the `kube-controller-manager`. Users can use the new feature by enabling the Feature
Gate (alpha only) and setting the new `tolerance` field on an HPA.

#### Downgrade
On downgrade, all HPAs will revert to using the global `horizontal-pod-autoscaler-tolerance`
value from the `kube-controller-manager`, regardless of any configured `tolerance` value on the HPA
itself.

### Version Skew Strategy

1. `kube-apiserver`: More recent instances will accept the new 'tolerance'
   field, while older will ignore it.
2. `kube-controller-manager`: An older version could receive an HPA containing
   the new `tolerance` field from a more recent API server, in which case it
   would ignore it (i.e. scale as if it was not present).

## Production Readiness Review Questionnaire

<!--

Production readiness reviews are intended to ensure that features merging into
Kubernetes are observable, scalable and supportable; can be safely operated in
production environments, and can be disabled or rolled back in the event they
cause increased failures in production. See more in the PRR KEP at
https://git.k8s.io/enhancements/keps/sig-architecture/1194-prod-readiness.

The production readiness review questionnaire must be completed and approved
for the KEP to move to `implementable` status and be included in the release.

In some cases, the questions below should also have answers in `kep.yaml`. This
is to enable automation to verify the presence of the review, and to reduce review
burden and latency.

The KEP must have a approver from the
[`prod-readiness-approvers`](http://git.k8s.io/enhancements/OWNERS_ALIASES)
team. Please reach out on the
[#prod-readiness](https://kubernetes.slack.com/archives/CPNHUMN74) channel if
you need any help or guidance.
-->

### Feature Enablement and Rollback

<!--
This section must be completed when targeting alpha to a release.
-->

###### How can this feature be enabled / disabled in a live cluster?

<!--
Pick one of these and delete the rest.

Documentation is available on [feature gate lifecycle] and expectations, as
well as the [existing list] of feature gates.

[feature gate lifecycle]: https://git.k8s.io/community/contributors/devel/sig-architecture/feature-gates.md
[existing list]: https://kubernetes.io/docs/reference/command-line-tools-reference/feature-gates/
-->

- [x] Feature gate (also fill in values in `kep.yaml`)
  - Feature gate name: HPAConfigurableTolerance
  - Components depending on the feature gate: `kube-controller-manager` and
    `kube-apiserver`.

###### Does enabling the feature change any default behavior?

<!--
Any change of default behavior may be surprising to users or break existing
automations, so be extremely careful here.
-->

No.

###### Can the feature be disabled once it has been enabled (i.e. can we roll back the enablement)?

<!--
Describe the consequences on existing workloads (e.g., if this is a runtime
feature, can it break the existing applications?).

Feature gates are typically disabled by setting the flag to `false` and
restarting the component. No other changes should be necessary to disable the
feature.

NOTE: Also set `disable-supported` to `true` or `false` in `kep.yaml`.
-->

The feature cannot be disabled: `tolerance` values set on HPAs are taken into
account.

Users can remove `tolerance` fields to ensure the feature is not used.

###### What happens if we reenable the feature if it was previously rolled back?

When the feature is re-enabled, any HPAs with configured `tolerance` values will use those when calculating replica counts, rather than the global tolerance from the `kube-controller-manager`.

###### Are there any tests for feature enablement/disablement?

<!--
The e2e framework does not currently support enabling or disabling feature
gates. However, unit tests in each component dealing with managing data, created
with and without the feature, are necessary. At the very least, think about
conversion tests if API types are being modified.

Additionally, for features that are introducing a new API field, unit tests that
are exercising the `switch` of feature gate itself (what happens if I disable a
feature gate after having objects written with the new field) are also critical.
You can take a look at one potential example of such test in:
https://github.com/kubernetes/kubernetes/pull/97058/files#diff-7826f7adbc1996a05ab52e3f5f02429e94b68ce6bce0dc534d1be636154fded3R246-R282
-->

[Unit tests have been added](https://github.com/kubernetes/kubernetes/blob/07142400ecd02126602ffaa6f91712cd3f1e170c/pkg/apis/autoscaling/validation/validation_test.go#L1648) to verify that HPAs with and without the new fields are
properly validated, both when the feature gate is enabled or not.

### Rollout, Upgrade and Rollback Planning

<!--
This section must be completed when targeting beta to a release.
-->

###### How can a rollout or rollback fail? Can it impact already running workloads?

<!--
Try to be as paranoid as possible - e.g., what if some components will restart
mid-rollout?

Be sure to consider highly-available clusters, where, for example,
feature flags will be enabled on some API servers and not others during the
rollout. Similarly, consider large clusters and how enablement/disablement
will rollout across nodes.
-->

This feature does not introduce new failure modes: during rollout/rollback, some
API servers will allow or disallow setting the new 'tolerance' field. The new
field is possibly ignored until the controller manager is fully updated.

###### What specific metrics should inform a rollback?

<!--
What signals should users be paying attention to when the feature is young
that might indicate a serious problem?
-->

A high `horizontal_pod_autoscaler_controller_metric_computation_duration_seconds`
metric can indicate a problem related to this feature.

###### Were upgrade and rollback tested? Was the upgrade->downgrade->upgrade path tested?

<!--
Describe manual testing that was done and the outcomes.
Longer term, we may want to require automated upgrade/rollback tests, but we
are missing a bunch of machinery and tooling and can't do that now.
-->

The upgrade→downgrade→upgrade testing was done manually using a 1.33 cluster with the following steps:

1.  Start the cluster with the HPA enabled:

    ```sh
    kind create cluster --name configurable-tolerance --image kindest/node:v1.33.0 --config config.yaml
    ```
    with the following `config.yaml` file content:
    ```yaml
    kind: Cluster
    apiVersion: kind.x-k8s.io/v1alpha4
    featureGates:
      "HPAConfigurableTolerance": true
    nodes:
    - role: control-plane
    - role: worker
    ```

    Install metrics-server:
    ```sh
    kubectl apply -f https://github.com/kubernetes-sigs/metrics-server/releases/download/v0.7.2/components.yaml
    kubectl patch -n kube-system deployment metrics-server --type=json   -p '[{"op":"add","path":"/spec/template/spec/containers/0/args/-","value":"--kubelet-insecure-tls"}]'
    ```
    
    Create a deployment starting Pods that consume a 50% CPU utilization, and an associated HPA with a very large tolerance:
    ```sh
    kubectl apply -f configurable-tolerance-test.yaml
    ```
    with the following `configurable-tolerance-test.yaml` file content:
    ```yaml
    apiVersion: apps/v1
    kind: Deployment
    metadata:
      name: cpu-stress-deployment
      labels:
        app: cpu-stressor
    spec:
      replicas: 1
      selector:
        matchLabels:
          app: cpu-stressor
      template:
        metadata:
          labels:
            app: cpu-stressor
        spec:
          containers:
            - name: cpu-stressor
              image: alpine:latest
              command: ["/bin/sh"]
              args:  # Load: 1% (10 milliCPU)
                - "-c"
                - "apk add --no-cache stress-ng && stress-ng --cpu 1 --cpu-load 1 --cpu-method=crc16 --timeout 3600s"
              resources:
                requests:
                  cpu: "20m"
    ---
    apiVersion: autoscaling/v2
    kind: HorizontalPodAutoscaler
    metadata:
      name: cpu-stress-hpa
    spec:
      scaleTargetRef:
        apiVersion: apps/v1
        kind: Deployment
        name: cpu-stress-deployment
      minReplicas: 1
      maxReplicas: 5
      metrics:
        - type: Resource
          resource:
            name: cpu
            target:
              type: Utilization
              averageUtilization: 10
      behavior:
        scaleUp:
          tolerance: 20.  # 2000%
    ```

    Check that, after a 5 minutes, `kubectl describe hpa cpu-stress-hpa` displays `ScalingLimited: False` (i.e.
    the HPA doesn't recommend to scale up because of the large tolerance).

2.  Simulate downgrade by disabling the feature for api server and control-plane (update the `config.yaml` file
    to set it to false). Follow the procedure described in step 1, and observe that this time
    `kubectl describe hpa cpu-stress-hpa` displays `ScalingLimited: True`.

4. Simulate downgrade by re-enabling the feature for api server and control-plane. Follow the procedure described
   in step 1, and observe that the HPA description mentions `ScalingLimited: False`, demonstrates that the feature
   is working again.

###### Is the rollout accompanied by any deprecations and/or removals of features, APIs, fields of API types, flags, etc.?

<!--
Even if applying deprecation policies, they may still surprise some users.
-->

No.

### Monitoring Requirements

<!--
This section must be completed when targeting beta to a release.

For GA, this section is required: approvers should be able to confirm the
previous answers based on experience in the field.
-->

###### How can an operator determine if the feature is in use by workloads?

<!--
Ideally, this should be a metric. Operations against the Kubernetes API (e.g.,
checking if there are objects with field X set) may be a last resort. Avoid
logs or events for this purpose.
-->

The presence of the new `tolerance` HPA field indicates that the feature is
used.

###### How can someone using this feature know that it is working for their instance?

<!--
For instance, if this is a pod-related feature, it should be possible to determine if the feature is functioning properly
for each individual pod.
Pick one more of these and delete the rest.
Please describe all items visible to end users below with sufficient detail so that they can verify correct enablement
and operation of this feature.
Recall that end users cannot usually observe component logs or access metrics.
-->

- [X] Events
  - Event Reason: `SuccessfulRescale`

The tolerance is applied on the ratio between the _current_ and _desired_ metric
values. Users can get both values using
[`kubectl describe`](https://github.com/kubernetes/kubernetes/blob/1b7a0591871772fbbc0fda430b3b73bc24c0e738/staging/src/k8s.io/kubectl/pkg/describe/describe.go#L4109)
and use them to verify that scaling events are triggered when their ratio is out
of tolerance.

The [controller-manager logs have been updated](https://github.com/kubernetes/kubernetes/blob/07142400ecd02126602ffaa6f91712cd3f1e170c/pkg/controller/podautoscaler/horizontal.go#L846) 
to help users understand the behavior of the autoscaler. The data added to the
logs includes the tolerance used for each scaling decision.

###### What are the reasonable SLOs (Service Level Objectives) for the enhancement?

<!--
This is your opportunity to define what "normal" quality of service looks like
for a feature.

It's impossible to provide comprehensive guidance, but at the very
high level (needs more precise definitions) those may be things like:
  - per-day percentage of API calls finishing with 5XX errors <= 1%
  - 99% percentile over day of absolute value from (job creation time minus expected
    job creation time) for cron job <= 10%
  - 99.9% of /health requests per day finish with 200 code

These goals will help you determine what you need to measure (SLIs) in the next
question.
-->

Although the absolute value of the `horizontal_pod_autoscaler_controller_metric_computation_duration_seconds`
metric depends on HPAs configuration, it should be unimpacted by this feature. This metric should not vary
by more than 5%.

###### What are the SLIs (Service Level Indicators) an operator can use to determine the health of the service?

<!--
Pick one more of these and delete the rest.
-->

This KEP is not expected to have any impact on SLIs/SLOs as it doesn't introduce
a new HPA behavior, but merely allows users to easily change the value of a
parameter that's otherwise difficult to update.

The standard HPA metric `horizontal_pod_autoscaler_controller_metric_computation_duration_seconds` can
be used to verify the HPA controller health.

###### Are there any missing metrics that would be useful to have to improve observability of this feature?

<!--
Describe the metrics themselves and the reasons why they weren't added (e.g., cost,
implementation difficulties, etc.).
-->

Users may want to see a signal that autoscaling isn't happening because of the
tolerance, but this is not directly related to this KEP (this problem already
exists today with the hard-coded 10% tolerance), and taking this KEP as an
opportunity to improve the situation is difficult (see
[this thread](https://github.com/kubernetes/enhancements/pull/4954#discussion_r1857098884)).

### Dependencies

<!--
This section must be completed when targeting beta to a release.
-->

###### Does this feature depend on any specific services running in the cluster?

<!--
Think about both cluster-level services (e.g. metrics-server) as well
as node-level agents (e.g. specific version of CRI). Focus on external or
optional services that are needed. For example, if this feature depends on
a cloud provider API, or upon an external software-defined storage or network
control plane.

For each of these, fill in the following—thinking about running existing user workloads
and creating new ones, as well as about cluster-level services (e.g. DNS):
  - [Dependency name]
    - Usage description:
      - Impact of its outage on the feature:
      - Impact of its degraded performance or high-error rates on the feature:
-->

No, this feature does not depend on any specific service.

### Scalability

<!--
For alpha, this section is encouraged: reviewers should consider these questions
and attempt to answer them.

For beta, this section is required: reviewers must answer these questions.

For GA, this section is required: approvers should be able to confirm the
previous answers based on experience in the field.
-->

###### Will enabling / using this feature result in any new API calls?

<!--
Describe them, providing:
  - API call type (e.g. PATCH pods)
  - estimated throughput
  - originating component(s) (e.g. Kubelet, Feature-X-controller)
Focusing mostly on:
  - components listing and/or watching resources they didn't before
  - API calls that may be triggered by changes of some Kubernetes resources
    (e.g. update of object X triggers new updates of object Y)
  - periodic API calls to reconcile state (e.g. periodic fetching state,
    heartbeats, leader election, etc.)
-->

No.

###### Will enabling / using this feature result in introducing new API types?

<!--
Describe them, providing:
  - API type
  - Supported number of objects per cluster
  - Supported number of objects per namespace (for namespace-scoped objects)
-->

No.

###### Will enabling / using this feature result in any new calls to the cloud provider?

<!--
Describe them, providing:
  - Which API(s):
  - Estimated increase:
-->

No.

###### Will enabling / using this feature result in increasing size or count of the existing API objects?

<!--
Describe them, providing:
  - API type(s):
  - Estimated increase in size: (e.g., new annotation of size 32B)
  - Estimated amount of new objects: (e.g., new Object X for every existing Pod)
-->

- This feature adds two new optional integer fields to `HorizontalPodAutoscaler`
  `v2` objects. Users should expect this object to increase in size (5 bytes)
  each time they set this new field.

###### Will enabling / using this feature result in increasing time taken by any operations covered by existing SLIs/SLOs?

<!--
Look at the [existing SLIs/SLOs].

Think about adding additional work or introducing new steps in between
(e.g. need to do X to start a container), etc. Please describe the details.

[existing SLIs/SLOs]: https://git.k8s.io/community/sig-scalability/slos/slos.md#kubernetes-slisslos
-->

No.

###### Will enabling / using this feature result in non-negligible increase of resource usage (CPU, RAM, disk, IO, ...) in any components?

<!--
Things to keep in mind include: additional in-memory state, additional
non-trivial computations, excessive access to disks (including increased log
volume), significant amount of data sent and/or received over network, etc.
This through this both in small and large cases, again with respect to the
[supported limits].

[supported limits]: https://git.k8s.io/community//sig-scalability/configs-and-limits/thresholds.md
-->

No.

###### Can enabling / using this feature result in resource exhaustion of some node resources (PIDs, sockets, inodes, etc.)?

<!--
Focus not just on happy cases, but primarily on more pathological cases
(e.g. probes taking a minute instead of milliseconds, failed pods consuming resources, etc.).
If any of the resources can be exhausted, how this is mitigated with the existing limits
(e.g. pods per node) or new limits added by this KEP?

Are there any tests that were run/should be run to understand performance characteristics better
and validate the declared limits?
-->

No.

### Troubleshooting

<!--
This section must be completed when targeting beta to a release.

For GA, this section is required: approvers should be able to confirm the
previous answers based on experience in the field.

The Troubleshooting section currently serves the `Playbook` role. We may consider
splitting it into a dedicated `Playbook` document (potentially with some monitoring
details). For now, we leave it here.
-->

###### How does this feature react if the API server and/or etcd is unavailable?

API server or etcd issues do not impact this feature.

###### What are other known failure modes?

<!--
For each of them, fill in the following information by copying the below template:
  - [Failure mode brief description]
    - Detection: How can it be detected via metrics? Stated another way:
      how can an operator troubleshoot without logging into a master or worker node?
    - Mitigations: What can be done to stop the bleeding, especially for already
      running user workloads?
    - Diagnostics: What are the useful log messages and their required logging
      levels that could help debug the issue?
      Not required until feature graduated to beta.
    - Testing: Are there any tests for failure mode? If not, describe why.
-->

We do not expect any new failure mode. (While setting `tolerance` below 10% can cause HPAs
to scale up and down as frequently as every 30s, and higher values might stop scaling altogether
if the metric remains within the tolerance band, the feature is still working as intended.
To make HPAs respond faster, decrease the tolerance value. Conversely, to make them respond
slower, increase the tolerance value.)

###### What steps should be taken if SLOs are not being met to determine the problem?

If possible increase the log level for kube-controller-manager and check controller logs:
1. Search for "Proposing desired replicas", verify that the tolerance is set as expected,
   and check (using `kubectl describe hpa`) if the ratio between the _current_ and _desired_
   metric values is in tolerance.
3. Look for warnings and errors which might point where the problem lies.

## Implementation History

<!--
Major milestones in the lifecycle of a KEP should be tracked in this section.
Major milestones might include:
- the `Summary` and `Motivation` sections being merged, signaling SIG acceptance
- the `Proposal` section being merged, signaling agreement on a proposed design
- the date implementation started
- the first Kubernetes release where an initial version of the KEP was available
- the version of Kubernetes where the KEP graduated to general availability
- when the KEP was retired or superseded
-->

2025-01-21: KEP PR merged.
2025-03-24: [Implementation PR](https://github.com/kubernetes/kubernetes/pull/130797) merged.
2025-05-15: Kubernetes v1.33 released (includes this feature).
2025-05-16: This KEP updated for beta graduation.
2026-05-21: This KEP updated for stable graduation.

## Drawbacks

<!--
Why should this KEP _not_ be implemented?
-->

No major drawbacks have been identified.

## Alternatives

<!--
What other approaches did you consider, and why did you rule them out? These do
not need to be as detailed as the proposal, but should include enough
information to express the idea and why it was not acceptable.
-->

On non-managed Kubernetes instances, users can update the cluster-wide
`--horizontal-pod-autoscaler-tolerance` tolerance parameter,

## Infrastructure Needed (Optional)

<!--
Use this section if you need things from the project/SIG. Examples include a
new subproject, repos requested, or GitHub details. Listing these here allows a
SIG to get the process for these resources started right away.
-->

N/A.
