// Module included in the following assemblies:
//
// * post_installation_configuration/day_2_core_cnf_clusters/updating/update-rhacm-talm-preparing-policies.adoc

:_mod-docs-content-type: REFERENCE
[id="core-cluster-upgrade-cgu-configuration_{context}"]
= ClusterGroupUpgrade custom resource configuration

[role="_abstract"]
You can configure `ClusterGroupUpgrade` custom resources (CRs) to orchestrate cluster updates with {cgu-operator-full} ({cgu-operator}).
A `ClusterGroupUpgrade` CR defines which clusters to update, which policies to apply, and how to manage batching and concurrency.

[IMPORTANT]
====
{rh-rhacm} policies used for updates must be set to `inform` mode. The `ClusterGroupUpgrade` CR temporarily changes these policies to `enforce` mode at the scheduled update time, applying them to the target clusters. When the `ClusterGroupUpgrade` CR completes or times out, the policies automatically revert to `inform` mode. This ensures that update policies are only enforced at the specific time you intend, rather than immediately when the policy is created.
====

== Basic ClusterGroupUpgrade CR

The following example shows a basic `ClusterGroupUpgrade` CR:

[source,yaml]
----
apiVersion: ran.openshift.io/v1alpha1
kind: ClusterGroupUpgrade
metadata:
  name: <cgu_name>
  namespace: <namespace>
spec:
  clusters:
  - <cluster_name_1>
  - <cluster_name_2>
  managedPolicies:
  - <policy_name>
  enable: false
  remediationStrategy:
    maxConcurrency: <max_concurrency>
    timeout: <timeout_minutes>
----

.ClusterGroupUpgrade CR fields
[cols="2,4",options="header"]
|===
|Field
|Description

|`name`
|Specifies a name for your `ClusterGroupUpgrade` CR, for example `core-zstream-upgrade-4.20.1`.

|`namespace`
|Specifies the namespace for the resource, for example `openshift-upgrade-policies`.

|`clusters`
|Specifies the names of your target clusters.

|`managedPolicies`
|Specifies the names of your update policies.

|`enable`
|Set to `false` to create the resource without starting the update. Set to `true` to start the update immediately.

|`maxConcurrency`
|Specifies the number of clusters to update simultaneously. Use `1` for canary testing, `2-5` for production rollouts, or higher for large fleets with proven procedures.

|`timeout`
|Specifies the maximum time in minutes to wait for each cluster update. See the timeout guidelines that follow.
|===

== Timeout guidelines

Calculate appropriate timeout values based on your environment:

* Base timeout: 60 minutes for z-stream updates, 120 minutes for y-stream updates.
* Add 30 minutes for each additional worker node beyond 3.
* Add 60 minutes for disconnected environments.
* Add 30 minutes for clusters with large image registries.

== Optional configuration fields

You can add the following optional fields to the `spec` section to customize the update behavior:

* Canary cluster testing
* Pre-update blocking checks
* Post-update label management

== Canary cluster testing

To test the update on a canary cluster before rolling it out to the full fleet, add a `canaries` list under `remediationStrategy` and use `clusterSelector` instead of `clusters` to target clusters by label:

[source,yaml]
----
  clusterSelector:
  - <label_selector>
  remediationStrategy:
    canaries:
    - <canary_cluster_name>
    maxConcurrency: 1
----

where:

* `<label_selector>`: Specifies a label selector for the target clusters, for example `upgrade-wave=1`.
* `<canary_cluster_name>`: Specifies the name of the canary cluster to update first. {cgu-operator} updates the canary cluster first, waits for success, and then proceeds with other clusters.

== Pre-update blocking checks

To require pre-update checks to pass before the update begins, add `blockingCRs` and `beforeEnable` fields:

[source,yaml]
----
  beforeEnable:
    addClusterLabels:
      pre-upgrade-check: pending
  blockingCRs:
  - name: <blocking_cr_name>
    namespace: <blocking_cr_namespace>
    kind: ConfigMap
    conditions:
    - type: <condition_type>
      status: "True"
----

where:

* `<blocking_cr_name>`: Specifies the name of the blocking custom resource, for example `etcd-backup`.
* `<condition_type>`: Specifies the condition that must be met before the update can proceed, for example `BackupComplete`.

== Post-update label management

To update cluster labels after the update completes, add an `afterCompletion` field:

[source,yaml]
----
  afterCompletion:
    addClusterLabels:
      upgraded-to: "<target_version>"
      upgrade-status: complete
    removeClusterLabels:
    - pre-upgrade-check
----

where `<target_version>` is the target version, for example `4.20`.

== Chained ClusterGroupUpgrade CRs for EUS-to-EUS updates

For Extended Update Support (EUS) to EUS updates, you can create chained `ClusterGroupUpgrade` CRs that use `blockingCRs` and `actions` to orchestrate a multi-step update:

[source,yaml]
----
---
apiVersion: ran.openshift.io/v1alpha1
kind: ClusterGroupUpgrade
metadata:
  name: upgrade-20to21
  namespace: openshift-upgrade-policies
spec:
  actions:
    afterCompletion:
      deleteClusterLabels:
        upgrade-version-to-4-21: ""
      deleteObjects: true
      addClusterLabels:
        unpause-workers-4-22: ""
  backup: false
  clusters:
  - <cluster_name>
  enable: true
  managedPolicies:
    - prep-20to22
    - upgrade-ocp-20to21
    - upgrade-olm-20to21
    - upgrade-validate-21
  preCaching: false
  preCachingConfigRef: {}
  remediationStrategy:
    maxConcurrency: 1
    timeout: 480
---
apiVersion: ran.openshift.io/v1alpha1
kind: ClusterGroupUpgrade
metadata:
  name: upgrade-21to22
  namespace: openshift-upgrade-policies
spec:
  actions:
    afterCompletion:
      deleteClusterLabels:
        upgrade-version-4-22: ""
        unpause-workers-4-22: ""
      deleteObjects: true
  backup: false
  clusters:
  - <cluster_name>
  enable: true
  managedPolicies:
    - upgrade-ocp-21to22
    - upgrade-olm-21to22
    - upgrade-validate-22
    - upgrade-worker-1
    - upgrade-worker-2
  preCaching: false
  preCachingConfigRef: {}
  remediationStrategy:
    maxConcurrency: 1
    timeout: 480
  blockingCRs:
  - name: upgrade-20to21
    namespace: openshift-upgrade-policies
----

where:

* `<cluster_name>`: Specifies the name of the target cluster.
* `actions`: Changes cluster labels as each step completes. This removes temporary update labels and unbinds completed update policies from the cluster.
* `blockingCRs`: Requires the first `ClusterGroupUpgrade` CR to complete before the second `ClusterGroupUpgrade` CR begins enforcing its policies.
* `upgrade-worker-1`, `upgrade-worker-2`: Separate unpause policies for each `MachineConfigPool` resource. Your cluster might have more machine config pools, requiring additional policies.

== Applying and monitoring a ClusterGroupUpgrade CR

After creating a `ClusterGroupUpgrade` CR, apply and enable it by running the following commands:

[source,terminal]
----
$ oc apply -f <cgu_cr_filename>.yaml
$ oc patch cgu <cgu_name> \
  -n <namespace> \
  --type merge \
  -p '{"spec":{"enable":true}}'
----

Monitor the `ClusterGroupUpgrade` status by running the following commands:

[source,terminal]
----
$ oc get cgu <cgu_name> -n <namespace> -o yaml
$ oc get cgu <cgu_name> -n <namespace> -w
----

Check the `status` section for update progress:

The following example shows the output:
+
[source,yaml]
----
status:
  conditions:
  - message: Upgrade is progressing
    reason: InProgress
    status: "True"
    type: Progressing
  managedPoliciesForUpgrade:
  - name: core-upgrade-4.20.1-policy
    namespace: openshift-upgrade-policies
  remediationPlan:
  - - spoke1
    - spoke2
  status:
    spoke1: complete
    spoke2: inprogress
----

Verify the `ClusterGroupUpgrade` CR entered the expected state by running the following commands:

[source,terminal]
----
$ oc get cgu -n <namespace>
$ oc get cgu <cgu_name> -n <namespace> -o jsonpath='{.status.clusters}'
----

The following example shows the output:
+
[source,terminal]
----
NAME                             AGE   STATE        DETAILS
core-zstream-upgrade-4.20.1     5m    InProgress   Remediating non-compliant policies
----

When the cluster is fully updated and the state of all `ClusterGroupUpgrade` CRs shows `Completed`, verify that all policies show `Compliant` by running the following command:

[source,terminal]
----
$ oc get policies -n <cluster_name>
----

The following example shows the output:
+
[source,terminal]
----
NAME                        REMEDIATION ACTION   COMPLIANCE STATE   AGE
upgrade.upgrade-ocp-21to22  inform               Compliant          1h
upgrade.upgrade-olm-21to22  inform               Compliant          10m
----

Delete the completed `ClusterGroupUpgrade` CRs from the update namespace by running the following command:

[source,terminal]
----
$ oc delete cgu -n <namespace> <cgu_name>
----

== Troubleshooting

* If the `ClusterGroupUpgrade` CR is stuck in `Preparing` state, verify that all managed policies exist and are bound to target clusters.
* If the `ClusterGroupUpgrade` CR fails with a timeout error, increase the `timeout` value in the `remediationStrategy` section.
