## Release notes for kOps 1.36 series

kOps 1.36 adds Kubernetes 1.36 support, completes the move away from the in-tree cloud providers, refreshes the bundled components (containerd v2.2, etcd-manager, AWS Load Balancer Controller, EBS CSI driver, Cilium options, CoreDNS, etc.), reworks how the `kops-channels` addon manager is built and deployed, refreshes Karpenter support with kOps-managed `EC2NodeClass` and `NodePool` objects, introduces a hybrid bootstrap path for gossip clusters, expands Azure and Hetzner support, and lays the groundwork for Linode (Akamai) as a new cloud provider.

# Significant changes

## Kubernetes
* Kubernetes 1.36 support, including integration tests ([#18267](https://github.com/kubernetes/kops/pull/18267), [#18202](https://github.com/kubernetes/kops/pull/18202))
* `kube-apiserver`: align `deleteCollectionWorkers`, `MaxRequestInflight` and `compactionInterval` with kube-up defaults for large clusters ([#18081](https://github.com/kubernetes/kops/pull/18081), [#18086](https://github.com/kubernetes/kops/pull/18086))
* `kube-apiserver`: add `storageInitializationTimeout` field to allow etcd more time to become ready before kube-apiserver gives up ([#18556](https://github.com/kubernetes/kops/pull/18556))
* `kubelet`: migrate deprecated CLI flags into the kubelet configuration file passed via `--config` ([#18280](https://github.com/kubernetes/kops/pull/18280))
* `kubelet`: add new tunables (`kubeAPIQPS`/`kubeAPIBurst`, `nodeAllocatableUpdatePeriodSeconds` and friends) and only set `nodeAllocatableUpdatePeriodSeconds` on Kubernetes 1.35+ ([#18085](https://github.com/kubernetes/kops/pull/18085), [#18153](https://github.com/kubernetes/kops/pull/18153), [#18305](https://github.com/kubernetes/kops/pull/18305))
* `kube-proxy`: bind-mount the kubeconfig directory instead of the file so kubeconfig rotations survive ([#18344](https://github.com/kubernetes/kops/pull/18344))
* `kube-scheduler`: documentation clarification for `KubeSchedulerConfiguration` ([#18151](https://github.com/kubernetes/kops/pull/18151))
* `kops-controller`: validate instance group names ([#18391](https://github.com/kubernetes/kops/pull/18391))
* `kops-controller`: delete the legacy node reconciler and route DigitalOcean through the non-legacy Identifier ([#18227](https://github.com/kubernetes/kops/pull/18227))
* Remove the unused in-tree cloud config plumbing ([#18347](https://github.com/kubernetes/kops/pull/18347))

## Container Runtime
* Default containerd to v2.2.4 (with runc 1.3.5) for Kubernetes 1.32+, and v1.7.32 for older clusters; kOps stays on containerd 2.2.x to avoid a sandbox-image regression in containerd 2.3 ([#18364](https://github.com/kubernetes/kops/pull/18364), [#18447](https://github.com/kubernetes/kops/pull/18447))
* Support containerd config TOML v3, with clearer config-file version behavior and dead-code cleanup from 1.6 ([#18291](https://github.com/kubernetes/kops/pull/18291)). Users who set `spec.containerd.configAdditions` on a cluster or instance group must update those entries to match the TOML v3 schema before upgrading.
* Map the containerd registry mirror wildcard to the `_default` directory ([#18291](https://github.com/kubernetes/kops/pull/18291))
* Stream verified container image bytes directly into `ctr import` from nodeup and drop the containerized-mounter Archive task ([#18278](https://github.com/kubernetes/kops/pull/18278), [#18277](https://github.com/kubernetes/kops/pull/18277))
* Dump containerd config files in `kops toolbox dump` for troubleshooting ([#18313](https://github.com/kubernetes/kops/pull/18313))

## Networking
* Update Cilium with several new tunables:
  - Schedule `cilium-operator` on control-plane nodes ([#18375](https://github.com/kubernetes/kops/pull/18375))
  - Allow setting arbitrary `cilium-config` entries via `Cilium.ExtraConfig` ([#18285](https://github.com/kubernetes/kops/pull/18285))
  - Add `EnableHostFirewall` field to `CiliumNetworkingSpec` ([#18152](https://github.com/kubernetes/kops/pull/18152))
  - Add `bpf-lb-sock` and `bpf-lb-sock-hostns-only` flags ([#18375](https://github.com/kubernetes/kops/pull/18375))
  - Require `k8s-connectivity` in the liveness probe ([#18237](https://github.com/kubernetes/kops/pull/18237))
  - Allow disabling `masquerade` in ENI IPAM mode ([#18432](https://github.com/kubernetes/kops/pull/18432))
* Calico: disable kube-proxy when running in eBPF mode ([#18334](https://github.com/kubernetes/kops/pull/18334))
* Calico: add `NFTablesMode` field to configure nftables support in Felix ([#18452](https://github.com/kubernetes/kops/pull/18452))
* Update kube-router to v2.10.0 and disable its strict external IP validation ([#18421](https://github.com/kubernetes/kops/pull/18421))
* Update AWS VPC CNI to v1.21.2 ([#18410](https://github.com/kubernetes/kops/pull/18410))
* CoreDNS: update to v1.14.3 then pin to v1.14.2 to stay on the supported branch, and bump CoreDNS memory on large clusters ([#18368](https://github.com/kubernetes/kops/pull/18368), [#18369](https://github.com/kubernetes/kops/pull/18369), [#18361](https://github.com/kubernetes/kops/pull/18361))
* CoreDNS: honor node taints in the `hostname` topology spread constraint ([#18535](https://github.com/kubernetes/kops/pull/18535))
* dns-controller: make `priorityClassName` configurable and default `Provider` when `ExternalDNS` is partially set ([#18298](https://github.com/kubernetes/kops/pull/18298), [#18302](https://github.com/kubernetes/kops/pull/18302))
* dns-controller: always apply the addon (rendered empty when unused) so a leftover dns-controller Deployment is removed when no longer needed ([#18567](https://github.com/kubernetes/kops/pull/18567))
* Drop deprecated GCS-based CNI plugin mirrors ([#17987](https://github.com/kubernetes/kops/pull/17987), [#17976](https://github.com/kubernetes/kops/pull/17976))

## gVisor RuntimeClass support

kOps now supports running workloads under [gVisor](https://gvisor.dev/) (runsc) as a containerd runtime handler. When enabled, kOps installs the `runsc` runtime via apt, configures containerd with a `runsc` runtime handler, and creates a `gvisor` RuntimeClass. Pods targeting the RuntimeClass are automatically scheduled to nodes with gVisor enabled via the `kops.k8s.io/gvisor` node label.

gVisor is configured per worker instance group. Cluster-level gVisor configuration is rejected, and instance group-level configuration is accepted only for workers:

```yaml
spec:
  containerd:
    gvisor:
      enabled: true
      platform: systrap  # default; also supports kvm
```

This feature is currently supported on Debian-family distributions only (Debian, Ubuntu).

## AWS
* Drop the in-tree `cloud-provider-aws` dependency from kOps ([#18336](https://github.com/kubernetes/kops/pull/18336))
* AWS Load Balancer Controller refresh:
  - Upgrade to v3.3.0, switching the manifest to a Helm + Kustomize pipeline ([#18221](https://github.com/kubernetes/kops/pull/18221), [#18276](https://github.com/kubernetes/kops/pull/18276))
  - Prune the bundled Deployment and drop the `ALBTargetControlConfig` CRD ([#18222](https://github.com/kubernetes/kops/pull/18222), [#18233](https://github.com/kubernetes/kops/pull/18233))
  - Bypass the LBC webhook for cert-manager so circular bootstrap issues are avoided ([#17999](https://github.com/kubernetes/kops/pull/17999))
  - Add `elasticloadbalancing:SetRulePriorities` and `ec2:DescribeSubnets` permissions ([#17999](https://github.com/kubernetes/kops/pull/17999))
  - Pin the Deployment label selectors so in-place upgrades from earlier kOps versions succeed ([#18426](https://github.com/kubernetes/kops/pull/18426))
* Karpenter refresh:
  - Upgrade to v1.13.0, replace the deprecated `karpenter.sh/provisioner-name` label with `karpenter.sh/nodepool`, and generate the addon manifest with a Helm + Kustomize pipeline ([#18479](https://github.com/kubernetes/kops/pull/18479))
  - Generate a managed `EC2NodeClass` and `NodePool` for each instance group with `spec.manager: Karpenter`, pruned when the instance group is removed ([#18497](https://github.com/kubernetes/kops/pull/18497))
  - Register Karpenter nodes with the `karpenter.sh/unregistered` taint ([#18486](https://github.com/kubernetes/kops/pull/18486))
  - Add missing EC2 read permissions to the Karpenter IAM policy ([#18487](https://github.com/kubernetes/kops/pull/18487))
* Update EBS CSI driver to v1.58.0, refresh the upstream policy, and gate `MutableCSINodeAllocatableCount` for Kubernetes 1.35+ ([#18220](https://github.com/kubernetes/kops/pull/18220))
* `kops-controller` is now served over HTTPS with a `/healthz` target on the API NLB target group ([#18236](https://github.com/kubernetes/kops/pull/18236), [#18174](https://github.com/kubernetes/kops/pull/18174))
* Abort rolling updates when load-balancer deregister fails instead of marching on with degraded targets ([#18338](https://github.com/kubernetes/kops/pull/18338))
* Reconcile target-group health check changes on existing target groups ([#18467](https://github.com/kubernetes/kops/pull/18467))
* `NodeTerminationHandler`:
  - Add `EnableOutOfServiceTaint` field ([#18140](https://github.com/kubernetes/kops/pull/18140))
  - Allow disabling `enableScheduledEventDraining` in Queue Processor mode ([#18339](https://github.com/kubernetes/kops/pull/18339))
* Mixed instances policy: apply `onDemandAllocationStrategy` to the ASG and propagate taints without value as ASG tags ([#18342](https://github.com/kubernetes/kops/pull/18342), [#18343](https://github.com/kubernetes/kops/pull/18343))
* `instanceRequirements`: add `excludedInstanceTypes` and fix a memory-assignment bug ([#18113](https://github.com/kubernetes/kops/pull/18113), [#18123](https://github.com/kubernetes/kops/pull/18123))
* NLB: add a security-group mode option ([#18211](https://github.com/kubernetes/kops/pull/18211))
* Also consider private subnets that already have an IPv6 CIDR as having a CIDR assigned ([#18089](https://github.com/kubernetes/kops/pull/18089))
* Disable `nm-cloud-setup` on RHEL 9 for AWS VPC CNI ([#18264](https://github.com/kubernetes/kops/pull/18264))
* Tighten and trim instance-role IAM permissions to match what each component actually uses ([#18251](https://github.com/kubernetes/kops/pull/18251), [#18355](https://github.com/kubernetes/kops/pull/18355), [#18362](https://github.com/kubernetes/kops/pull/18362), [#18363](https://github.com/kubernetes/kops/pull/18363), [#18372](https://github.com/kubernetes/kops/pull/18372))
* Use `HeadBucket` to resolve the S3 bucket region, pass the VFS scheme and provider-specific options to the S3 client, and silence warnings when the S3 provider has no supported checksum ([#18335](https://github.com/kubernetes/kops/pull/18335), [#18129](https://github.com/kubernetes/kops/pull/18129), [#18132](https://github.com/kubernetes/kops/pull/18132), [#18128](https://github.com/kubernetes/kops/pull/18128))

## Azure
* Grant the control-plane VM Scale Set identity the built-in **Contributor** role instead of **Owner** on the cluster resource group. Existing clusters keep the old Owner role assignment until it is removed with `az role assignment delete` or the cluster is deleted. ([#18533](https://github.com/kubernetes/kops/pull/18533))
* Deploy `cloud-controller-manager` (v1.36.2) for node lifecycle and load-balancer support ([#18197](https://github.com/kubernetes/kops/pull/18197), [#18529](https://github.com/kubernetes/kops/pull/18529))
* Add experimental Terraform target support ([#18149](https://github.com/kubernetes/kops/pull/18149))
* Add support for the Azure Disk CSI Driver (v1.34.4) ([#18141](https://github.com/kubernetes/kops/pull/18141), [#18527](https://github.com/kubernetes/kops/pull/18527))
* Scope the nodes-to-API NSG rules to the NAT gateway public IP ([#18522](https://github.com/kubernetes/kops/pull/18522))
* Use HTTPS for the kops-controller probe and load-balancer health check, and move probe / rule configuration into the model using SDK types ([#18182](https://github.com/kubernetes/kops/pull/18182), [#18183](https://github.com/kubernetes/kops/pull/18183), [#18190](https://github.com/kubernetes/kops/pull/18190))
* Use `/etc/kubernetes/azure.json` for the cloud config, load CCM/CSI config from a Secret via a new `azure-cloud-config` addon and stop writing the cloud-config file on nodes ([#18345](https://github.com/kubernetes/kops/pull/18345))
* Set the provider ID when starting kubelet ([#18155](https://github.com/kubernetes/kops/pull/18155))
* Enable CCM cloud routes for kubenet and Kindnet ([#18262](https://github.com/kubernetes/kops/pull/18262))
* Encode storage account in `azureblob://` URLs ([#18260](https://github.com/kubernetes/kops/pull/18260))
* List VMSS NICs in protokube gossip seed discovery and match VMSS VM/NIC ARM IDs case-insensitively in the dumper ([#18319](https://github.com/kubernetes/kops/pull/18319), [#18315](https://github.com/kubernetes/kops/pull/18315))
* Cluster deletion robustness: block disk deletion on the parent VMSS, delete `RoleAssignment` after the VM scale set, handle missing resource groups in disk listing, fix nil pointer dereferences and ordering issues ([#18196](https://github.com/kubernetes/kops/pull/18196), [#18186](https://github.com/kubernetes/kops/pull/18186), [#18191](https://github.com/kubernetes/kops/pull/18191), [#18184](https://github.com/kubernetes/kops/pull/18184), [#18185](https://github.com/kubernetes/kops/pull/18185))
* Retry tasks with failed provisioning state, fix Terraform `LoadBalancer` task dependencies on `PublicIP`, and use a larger default VM SKU (`Standard_D4ls_v6`) in tests ([#18157](https://github.com/kubernetes/kops/pull/18157), [#18154](https://github.com/kubernetes/kops/pull/18154), [#18194](https://github.com/kubernetes/kops/pull/18194))
* Restrict VMSS role assignments to the control plane and fix a control-plane role tag spelling ([#18357](https://github.com/kubernetes/kops/pull/18357), [#18353](https://github.com/kubernetes/kops/pull/18353))
* Add `regenerate.sh` for addons ([#18225](https://github.com/kubernetes/kops/pull/18225))

## GCP
* Add `kops-controller` to the GCE internal load balancer and expose it on the internal LB for gossip clusters ([#18169](https://github.com/kubernetes/kops/pull/18169), [#18307](https://github.com/kubernetes/kops/pull/18307))
* Use SSL health check for `kops-controller` on GCE ([#18171](https://github.com/kubernetes/kops/pull/18171))
* Support `role=apiserver` on GCE with `dns=none` and fix instance tags for `role=apiserver` ([#18159](https://github.com/kubernetes/kops/pull/18159), [#18175](https://github.com/kubernetes/kops/pull/18175))
* Allow BGP from nodes to the control plane for Calico ([#18351](https://github.com/kubernetes/kops/pull/18351))
* Don't request live migration on instance types that don't support it ([#18004](https://github.com/kubernetes/kops/pull/18004))
* Wait for `InstanceManagers`/`InstanceTemplates` deletion to complete and include MIG scaling errors when instances are not found ([#18013](https://github.com/kubernetes/kops/pull/18013), [#18247](https://github.com/kubernetes/kops/pull/18247))
* Allow `N4A` instance type ([#18330](https://github.com/kubernetes/kops/pull/18330))
* Register all zonal MIGs of a multi-zone instance group with cluster-autoscaler ([#18543](https://github.com/kubernetes/kops/pull/18543))
* Emit the `kops.k8s.io/instancegroup` node label ([#18484](https://github.com/kubernetes/kops/pull/18484))
* Drop the `cloud-provider-gcp` dependency and switch `clouddns` to a forked `gcetokensource` ([#18274](https://github.com/kubernetes/kops/pull/18274))
* Allow pods to reach metric ports running on control-plane nodes when using GCE alias IPs ([#18052](https://github.com/kubernetes/kops/pull/18052))
* Use Kubernetes 1.36 for the `apiserver` role e2e template, move it to `dns=none`, add e2e templates that combine internal load balancer + Cilium etcd, and reject `role=apiserver` + `dns=none` only on GCE when not supported ([#18147](https://github.com/kubernetes/kops/pull/18147), [#18146](https://github.com/kubernetes/kops/pull/18146), [#18162](https://github.com/kubernetes/kops/pull/18162), [#18214](https://github.com/kubernetes/kops/pull/18214))
* GCE: shrink the etcd-cluster disk label to fit the 63-character limit ([#18292](https://github.com/kubernetes/kops/pull/18292))
* GCE: support control-plane volume type configuration ([#17955](https://github.com/kubernetes/kops/pull/17955))
* GCE: fix instance-group deletion ([#18148](https://github.com/kubernetes/kops/pull/18148)) and nil-panic during deletion ([#18195](https://github.com/kubernetes/kops/pull/18195))
* Migrate the GCS bucket discovery store handling and reject GCS VFS paths without buckets ([#18360](https://github.com/kubernetes/kops/pull/18360))

## Hetzner
* Enable Cluster Autoscaler ([#18226](https://github.com/kubernetes/kops/pull/18226), [#18135](https://github.com/kubernetes/kops/pull/18135))
* Node-role servers now carry the `hcloud/node-group` label, which the Cluster Autoscaler Hetzner provider uses for node group membership. Servers created by earlier kOps versions receive the label when they are replaced. ([#18538](https://github.com/kubernetes/kops/pull/18538))
* Lowering the `minSize` of an instance group no longer removes surplus servers; `minSize` now acts as a floor. Remove specific servers with `kops delete instance`, or rely on Cluster Autoscaler scale-down. ([#18538](https://github.com/kubernetes/kops/pull/18538))
* Upgrade `hcloud-cloud-controller-manager` to v1.31.0 ([#18281](https://github.com/kubernetes/kops/pull/18281), [#18317](https://github.com/kubernetes/kops/pull/18317))
* Upgrade `hcloud-csi-driver` to v2.20.2 and reorder the CSI driver Deployment before the DaemonSet ([#18318](https://github.com/kubernetes/kops/pull/18318))
* Split the hcloud Secret into its own addon and let the CSI driver consume the CCM-provided secret ([#18317](https://github.com/kubernetes/kops/pull/18317), [#18318](https://github.com/kubernetes/kops/pull/18318))

## DigitalOcean
* Default machine type to `s-2vcpu-4gb-amd` ([#18227](https://github.com/kubernetes/kops/pull/18227))
* Migrate node identifier to the non-legacy `Identifier` interface and tag droplets with the instance group role ([#18227](https://github.com/kubernetes/kops/pull/18227))
* Remove the invalid `namespace` field from the CSI node-driver-registrar ClusterRole ([#18498](https://github.com/kubernetes/kops/pull/18498))

## OpenStack
* Resolve the `floatingip` TODO from kOps 1.21 ([#18314](https://github.com/kubernetes/kops/pull/18314))
* Enable hybrid bootstrap mode for gossip clusters ([#18245](https://github.com/kubernetes/kops/pull/18245))

## Linode (experimental)
* Initial Linode (Akamai) cloud provider support, including:
  - Cloud provider API registration ([#18166](https://github.com/kubernetes/kops/pull/18166))
  - VFS object storage schema ([#18138](https://github.com/kubernetes/kops/pull/18138))
  - VPC cloudup task ([#18316](https://github.com/kubernetes/kops/pull/18316))
  - nodeup configuration and node identity ([#18177](https://github.com/kubernetes/kops/pull/18177))
* The Linode provider is not yet ready for production use.

## Bare metal
* Don't try to use protobuf for bare-metal tooling ([#18068](https://github.com/kubernetes/kops/pull/18068))
* Unpin Kubernetes version for the metal provider ([#17944](https://github.com/kubernetes/kops/pull/17944))

## Etcd
* Update etcd-manager to v3.0.20260707 ([#18323](https://github.com/kubernetes/kops/pull/18323), [#18436](https://github.com/kubernetes/kops/pull/18436), [#18450](https://github.com/kubernetes/kops/pull/18450), [#18560](https://github.com/kubernetes/kops/pull/18560)), which brings:
  - Recovery of a cluster member whose data disk was replaced with an empty one ([etcd-manager#54](https://github.com/kubernetes-sigs/etcd-manager/pull/54), [etcd-manager#70](https://github.com/kubernetes-sigs/etcd-manager/pull/70))
  - etcd membership changes that are resilient to an unreachable member ([etcd-manager#71](https://github.com/kubernetes-sigs/etcd-manager/pull/71))
  - A new `--env-file` flag to load credentials from a file ([etcd-manager#67](https://github.com/kubernetes-sigs/etcd-manager/pull/67))
  - A go-runner-based distroless image ([etcd-manager#73](https://github.com/kubernetes-sigs/etcd-manager/pull/73))
  - Faster startup for single-member clusters by seeding the member itself into the gossip peers and skipping the post-election wait ([etcd-manager#75](https://github.com/kubernetes-sigs/etcd-manager/pull/75), [etcd-manager#77](https://github.com/kubernetes-sigs/etcd-manager/pull/77))
* Bump etcd to latest patches (3.5.31, 3.6.12) and drop support for etcd 3.4 ([#18290](https://github.com/kubernetes/kops/pull/18290), [#18436](https://github.com/kubernetes/kops/pull/18436))
* Add `ListenClientHTTPURLs` field to `EtcdManagerSpec` for HTTP-only etcd client listeners ([#18416](https://github.com/kubernetes/kops/pull/18416))
* Generate etcd-manager patch symlinks dynamically ([#18290](https://github.com/kubernetes/kops/pull/18290))
* Decouple `EtcdEvents` HTTP from main etcd cluster traffic in scalability tests ([#18370](https://github.com/kubernetes/kops/pull/18370))

## Channels and addons
* The `kops-channels` addon manager is now built as a kops-managed image and rendered as a static pod on control-plane nodes; protokube no longer applies channels or labels control-plane nodes ([#18215](https://github.com/kubernetes/kops/pull/18215), [#18328](https://github.com/kubernetes/kops/pull/18328), [#18373](https://github.com/kubernetes/kops/pull/18373), [#18374](https://github.com/kubernetes/kops/pull/18374))
* `kops-channels` gains `--node-name`, `--interval`, multi-URL apply support, and quicker retries until the first reconcile succeeds ([#18328](https://github.com/kubernetes/kops/pull/18328))
* `kops-channels`: move the node labeler from protokube ([#18215](https://github.com/kubernetes/kops/pull/18215))
* `kops-channels`: surface addon apply failures via a readiness probe ([#18433](https://github.com/kubernetes/kops/pull/18433))
* `kops-channels`: fix region detection and the discovery cache permission noise ([#18390](https://github.com/kubernetes/kops/pull/18390))
* `spec.addons` entries can now point at plain Kubernetes manifests, which are applied as-is, in addition to kOps `Addons` indexes ([#18511](https://github.com/kubernetes/kops/pull/18511))
* Drop the standalone `channels` binary from the kOps release artifacts ([#18374](https://github.com/kubernetes/kops/pull/18374))
* `addons`: render addons as fi-tasks so addon templates can reference the finalized task graph, and drop the legacy `9.99.0` version shim and the deprecated master `nodeAffinity` term ([#18215](https://github.com/kubernetes/kops/pull/18215), [#18257](https://github.com/kubernetes/kops/pull/18257))

## Gossip
* Introduce hybrid worker bootstrap for gossip clusters on AWS, Azure, GCE, and OpenStack: control-plane nodes keep using gossip while workers bootstrap directly against the API NLB / internal LB, so workers no longer need protokube ([#18245](https://github.com/kubernetes/kops/pull/18245), [#18307](https://github.com/kubernetes/kops/pull/18307))
* Restrict gossip seed discovery to control-plane nodes and stop exporting unused cloud credentials to worker nodes ([#18352](https://github.com/kubernetes/kops/pull/18352), [#18354](https://github.com/kubernetes/kops/pull/18354))
* New cluster creation now defaults to `dns=none` and logs a deprecation warning if gossip is requested ([#18245](https://github.com/kubernetes/kops/pull/18245))
* The `dns-controller` addon is now applied with an empty manifest on clusters that do not use dns-controller, so its leftover resources are cleaned up automatically when a cluster migrates to `dns=none`
* Validation: enforce supported DNS topology per cloud provider ([#18255](https://github.com/kubernetes/kops/pull/18255))
* Migrate protokube mesh gossip protobuf from gogo ([#18230](https://github.com/kubernetes/kops/pull/18230))
* Add minimal gossip create/update integration tests and an upgrade e2e test ([#18256](https://github.com/kubernetes/kops/pull/18256), [#18296](https://github.com/kubernetes/kops/pull/18296))

## Operating System Support
* Drop support for Amazon Linux 2 ([#17943](https://github.com/kubernetes/kops/pull/17943))
* Drop support for Ubuntu 20.04 ([#18235](https://github.com/kubernetes/kops/pull/18235))
* Drop support for Debian 10 ([#18235](https://github.com/kubernetes/kops/pull/18235))
* Add experimental support for Ubuntu 26.04 ([#18232](https://github.com/kubernetes/kops/pull/18232))
* Load the `nf_tables` module and install `iptables-nft` on RHEL 10+ ([#18179](https://github.com/kubernetes/kops/pull/18179))
* Load the `ip_set` module and disable `firewalld` on RHEL 10+ and Rocky Linux 10+ ([#18478](https://github.com/kubernetes/kops/pull/18478))
* Enable the `nf_conntrack` kernel module on Rocky 9 ([#17968](https://github.com/kubernetes/kops/pull/17968))
* Skip `ImageVolume` tests on Debian 11 and prevent `cloud-ifupdown-helper` from hijacking CNI veths on Debian 11 ([#18261](https://github.com/kubernetes/kops/pull/18261))
* Set E2E `--node-os-arch=arm64` for Rocky 10 ([#18192](https://github.com/kubernetes/kops/pull/18192))

## Other components
* Update cluster-autoscaler to v1.36.0, with per-minor versions for older Kubernetes releases ([#18110](https://github.com/kubernetes/kops/pull/18110), [#18541](https://github.com/kubernetes/kops/pull/18541))
* Update cert-manager to v1.19.5 and set `AWS_REGION` on the controller for Route53 dns-01 solvers ([#18428](https://github.com/kubernetes/kops/pull/18428))
* Run metrics-server in insecure mode for AI Conformance tests ([#18067](https://github.com/kubernetes/kops/pull/18067))
* Update Go to v1.25.7 / v1.25.8 / v1.25.9 / v1.26.2 / v1.26.3 / v1.26.5 ([#17956](https://github.com/kubernetes/kops/pull/17956), [#18058](https://github.com/kubernetes/kops/pull/18058), [#18173](https://github.com/kubernetes/kops/pull/18173), [#18267](https://github.com/kubernetes/kops/pull/18267), [#18395](https://github.com/kubernetes/kops/pull/18395), [#18546](https://github.com/kubernetes/kops/pull/18546))
* Build kOps binaries with `gcr.io/distroless/static` as the base image and strip release binaries by default ([#18403](https://github.com/kubernetes/kops/pull/18403), [#18263](https://github.com/kubernetes/kops/pull/18263))
* Drop the in-tree helm dependency from `kops toolbox`; switch to a forked `helmstrvals` ([#18272](https://github.com/kubernetes/kops/pull/18272))
* Switch `structured-merge-diff` from v4 to v6 ([#18273](https://github.com/kubernetes/kops/pull/18273))
* Upgrade `hashicorp/memberlist` to v0.5.4 ([#18230](https://github.com/kubernetes/kops/pull/18230))

# Other changes of note

* `kops reconcile cluster` accepts `--use-kubeconfig` to reuse an existing kubeconfig instead of regenerating it ([#18126](https://github.com/kubernetes/kops/pull/18126))
* `kops` accepts `--node-volume-type` flags on `create cluster` ([#18145](https://github.com/kubernetes/kops/pull/18145))
* `kops create instancegroup` aligns the node label across `create cluster` and `create instancegroup` ([#18341](https://github.com/kubernetes/kops/pull/18341))
* `kops get assets`: fix lookup when `spec.dnsZone` is a DNS name ([#18384](https://github.com/kubernetes/kops/pull/18384))
* `kops update cluster`: reject non-`http(s)` URLs for `assets.fileRepository` ([#18340](https://github.com/kubernetes/kops/pull/18340))
* `kops update cluster`: validate `assets.fileRepository` ([#18340](https://github.com/kubernetes/kops/pull/18340))
* `kops upgrade-ab`: allow kOps downgrades for upgrade-AB scenarios ([#18219](https://github.com/kubernetes/kops/pull/18219))
* `kops toolbox dump`: time out per-node log dumping (configurable with the new `--node-dump-timeout` flag), improve reliability and skip not-found nodes on GCE ([#18349](https://github.com/kubernetes/kops/pull/18349), [#18443](https://github.com/kubernetes/kops/pull/18443), [#18311](https://github.com/kubernetes/kops/pull/18311), [#18049](https://github.com/kubernetes/kops/pull/18049))
* `kops set` and the `--set` flag can now append new slice elements instead of failing on a missing index ([#18511](https://github.com/kubernetes/kops/pull/18511))
* `nodeup`: add experimental hybrid-bootstrap workers, skip protokube/channels assets on workers, populate `DefaultMachineType` for Cilium-ENI clusters, and use shared system-component env vars for `kops-channels` ([#18245](https://github.com/kubernetes/kops/pull/18245), [#18358](https://github.com/kubernetes/kops/pull/18358), [#18365](https://github.com/kubernetes/kops/pull/18365))
* `nodeup`: fix protokube skip on hybrid-bootstrap workers ([#18378](https://github.com/kubernetes/kops/pull/18378))
* `kops-controller`/`nodeup`: ensure files have the desired permissions before close and rename, and fix `PrivateKey.WriteTo` returning zero length ([#18379](https://github.com/kubernetes/kops/pull/18379))
* `dns-controller`: honor `klog -stderrthreshold` even when `-logtostderr` is true ([#18231](https://github.com/kubernetes/kops/pull/18231))
* Fix `HasHighlyAvailableControlPlane` to use `AllInstanceGroups` when an instance-group filter is in use ([#17740](https://github.com/kubernetes/kops/pull/17740))
* Fix `kops` panic on send to a closed results channel ([#18326](https://github.com/kubernetes/kops/pull/18326))
* `AssetBuilder` is now concurrency-safe ([#18181](https://github.com/kubernetes/kops/pull/18181))
* Side-loading uses the `KOPS_BASE_URL` image version ([#18200](https://github.com/kubernetes/kops/pull/18200))
* Verify the config server IPs with a DNS name ([#18241](https://github.com/kubernetes/kops/pull/18241))
* Remove the explicit `fs.inotify.max_user_watches` sysctl setting ([#17556](https://github.com/kubernetes/kops/pull/17556))
* Pull `actions/upload-artifact`, `actions/setup-go` and `actions/dependency-review-action` to their latest releases, pinned by commit SHA ([#18114](https://github.com/kubernetes/kops/pull/18114))
* Replace `shipbot` with a `gh`-based script for promoting binaries ([#18095](https://github.com/kubernetes/kops/pull/18095))
* Build: add a `kops-channels` image build and CI push, and run `make apimachinery` updates as needed ([#18328](https://github.com/kubernetes/kops/pull/18328))
* `gomod`: tidy and verify all modules dynamically ([#18401](https://github.com/kubernetes/kops/pull/18401))

## AI Conformance (experimental)
* New experimental test suite under `tests/ai-conformance` covering accelerator metrics, gang scheduling with Kueue, observability and AI-service metrics, pod and cluster autoscaling, GPU operator integration with Cilium gateway API, and more. The suite is intended for kOps CI/E2E and is not yet exposed as a user-facing feature ([#18056](https://github.com/kubernetes/kops/pull/18056), [#18075](https://github.com/kubernetes/kops/pull/18075), [#18055](https://github.com/kubernetes/kops/pull/18055), [#18053](https://github.com/kubernetes/kops/pull/18053), [#18100](https://github.com/kubernetes/kops/pull/18100), [#18112](https://github.com/kubernetes/kops/pull/18112), [#18063](https://github.com/kubernetes/kops/pull/18063), [#18064](https://github.com/kubernetes/kops/pull/18064), [#18071](https://github.com/kubernetes/kops/pull/18071), [#18074](https://github.com/kubernetes/kops/pull/18074), [#18073](https://github.com/kubernetes/kops/pull/18073), [#17992](https://github.com/kubernetes/kops/pull/17992), [#18001](https://github.com/kubernetes/kops/pull/18001), [#18005](https://github.com/kubernetes/kops/pull/18005), [#18006](https://github.com/kubernetes/kops/pull/18006), [#18008](https://github.com/kubernetes/kops/pull/18008), [#18009](https://github.com/kubernetes/kops/pull/18009), [#18018](https://github.com/kubernetes/kops/pull/18018), [#18066](https://github.com/kubernetes/kops/pull/18066), [#18067](https://github.com/kubernetes/kops/pull/18067), [#18076](https://github.com/kubernetes/kops/pull/18076), [#18077](https://github.com/kubernetes/kops/pull/18077), [#18084](https://github.com/kubernetes/kops/pull/18084), [#18087](https://github.com/kubernetes/kops/pull/18087), [#18101](https://github.com/kubernetes/kops/pull/18101), [#18108](https://github.com/kubernetes/kops/pull/18108))

# Breaking changes

* Support for Amazon Linux 2, Ubuntu 20.04 and Debian 10 has been removed; existing clusters running those distributions must be migrated to a supported OS before upgrading to kOps 1.36 ([#17943](https://github.com/kubernetes/kops/pull/17943), [#18235](https://github.com/kubernetes/kops/pull/18235))
* Support for Kubernetes 1.30 has been removed.
* Support for etcd 3.4 has been removed; clusters must be running etcd 3.5 or 3.6 ([#18290](https://github.com/kubernetes/kops/pull/18290))
* The standalone `channels` binary is no longer distributed; the `kops-channels` addon manager now runs as a static pod on control-plane nodes ([#18374](https://github.com/kubernetes/kops/pull/18374))
* The legacy `9.99.0` addons-version shim has been removed; addons set up by kOps versions prior to 1.22 must be re-applied before upgrading ([#18257](https://github.com/kubernetes/kops/pull/18257))
* The in-tree `cloud-provider-aws` and `cloud-provider-gcp` dependencies have been dropped from kOps; external cloud providers are required (already mandatory for Kubernetes 1.33+) ([#18336](https://github.com/kubernetes/kops/pull/18336), [#18274](https://github.com/kubernetes/kops/pull/18274))
* The legacy `azureblob://{container}/{key}` URL form (paired with the `AZURE_STORAGE_ACCOUNT` environment variable) is no longer accepted; state-store paths must use the new `azureblob://{account}/{container}/{key}` form ([#18260](https://github.com/kubernetes/kops/pull/18260))
* Users who set `spec.containerd.configAdditions` must update those entries to the containerd config TOML v3 schema before upgrading to kOps 1.36 ([#18291](https://github.com/kubernetes/kops/pull/18291))
* A cluster spec applied with the top-level `authorization` field omitted (`kops create -f` / `kops replace -f`) now defaults to RBAC instead of `AlwaysAllow`, matching `kops create cluster`. This takes effect when such a manifest is (re-)applied; clusters whose stored spec already sets `authorization` are unchanged. Set `spec.authorization.alwaysAllow: {}` to keep `AlwaysAllow`. ([#18431](https://github.com/kubernetes/kops/pull/18431))

# Known Issues

* None at this time

# Deprecations

* Support for Kubernetes version 1.30 is removed in kOps 1.36.

* Support for Kubernetes version 1.31 is deprecated and will be removed in kOps 1.37.

* Support for Amazon Linux 2, Ubuntu 20.04 and Debian 10 is removed in kOps 1.36.

* Support for etcd 3.4 is removed in kOps 1.36.

* The standalone `channels` binary is no longer distributed in kOps 1.36; `kops-channels` runs as a static pod on the control plane.

* Support for AWS Classic Load Balancer (CLB) for the API, deprecated since kOps 1.26, will be removed in kOps 1.37. New clusters using CLB will be rejected, and existing clusters will need to migrate to NLB before they can be upgraded. See the [CLB to NLB migration guide](https://github.com/kubernetes/kops/blob/master/permalinks/acm_nlb.md) for the procedure.

* Support for gossip DNS, deprecated since kOps 1.29, will be removed in kOps 1.37. New gossip clusters will be rejected, and existing clusters will need to migrate before they can be upgraded. This affects clusters whose name ends in `.k8s.local` and that were not created with `--dns=none`; clusters using `--dns=none`, even with a `.k8s.local` name, are not affected. Migrate to `--dns=none` or a hosted DNS zone; kOps 1.36 introduces hybrid bootstrap to make that migration easier. See the [gossip migration guide](https://kops.sigs.k8s.io/gossip/) for the migration procedure.
