# ARO HCP Deployment Pipelines

ARO HCP deployment pipelines define the rollout process for infrastructure and services across different [Azure environments](environments.md). These pipelines follow a custom YAML format that allows defining deployment steps targeting specific Azure subscriptions, resource groups, and AKS clusters.

This document is intended for ARO HCP developers and SREs. It provides guidance on writing pipeline files that cover the deployment of different aspects of ARO HCP. Readers will learn how to structure pipeline files, define steps, and combine them effectively to deploy infrastructure and services.

## Pipeline File Structure

ARO HCP pipelines are defined in a custom YAML format that describes the deployment context, specifying which Azure subscriptions and resource groups are targeted. The pipeline file consists of high-level attributes that set the scope for deployments.

The following example illustrates the top-level structure of a pipeline file:

```yaml
serviceGroup: Microsoft.Azure.ARO.HCP.Region     (1)
rolloutName: Region Rollout                      (2)
resourceGroups:                                  (3)
- name: {{ .global.rg }}                         (4)
  subscription: {{ .global.subscription.key }}   (5)
  aksCluster: {{ .global.aksCluster }}           (6)
  steps: []                                      (7)
```

1. Defines the logical grouping of services being deployed. This value can be freely chosen as long as it is unique and prefixed with `Microsoft.Azure.ARO.HCP`.
2. `rolloutName`: Specifies a human-readable name for the rollout process.
3. `resourceGroups`: A list of deployment contexts targeted by the pipeline.
4. `resourceGroups.name`: The name of the resource group. Can be configured statically or dynamically using [configuration lookups](configuration.md) via Go template syntax.
5. `resourceGroups.subscription`: The Azure subscription ID associated with the resource group. Can be configured statically or dynamically using [configuration lookups](configuration.md) via Go template syntax.
6. `resourceGroups.aksCluster`: Optional. Specifies the name of the AKS cluster that deployment steps should target. Can be configured statically or dynamically using [configuration lookups](configuration.md) via Go template syntax.

### Stamped Resource Groups

Resource groups can be marked as [stamp](terminology.md#ev2-stamp)-scoped by setting `stamped: true`. Stamped resource groups are executed once per stamp (e.g. management cluster) in a region, with each stamp receiving its own isolated configuration context (`ctx.stamp`). Unstamped resource groups run exactly once per region.

```yaml
resourceGroups:
- name: region
  resourceGroup: '{{ .region.rg }}'
  subscription: '{{ .region.subscription.key }}'
  steps: []
- name: management
  stamped: true
  resourceGroup: '{{ .mgmt.rg }}'
  subscription: '{{ .mgmt.subscription.key }}'
  steps: []
```

In this example, `region` runs once per region while `management` runs once per stamp. See the [configuration documentation](configuration.md) for details on the `ctx.stamp` variable.

> [!IMPORTANT]
> Resource groups with the same `name` must be consistently stamped across all pipelines in the topology. If one pipeline marks a resource group as `stamped: true`, every other pipeline using that same resource group name must do the same. Inconsistent stamping will be rejected by validation.

## Pipeline Steps

A subscription/resourcegroup/AKS execution context needs to define a sequence of deployment steps. These steps define what is to deployed, how, and in what order.

### Step Structure

A step is defined using a set of properties that determine its function and execution behavior.

```yaml
...
resourceGroups:
- ...
  steps:
  - name: step-name                      (1)
    action: step-type                    (2)
    dependsOn:                           (3)
    - other-step-name
    <step type specific properties>      (4)
```

1. `name`: An identifier for the step that is unique within the entire pipeline file.
2. `action`: The type of step being executed. This can be an ARM step, a shell step, or any other supported step type (see [Step Types](#step-types)).
3. `dependsOn`: An optional list of other step names that must complete successfully before this step is executed. Dependencies must be cycle-free to ensure a valid execution order.
4. Step type specific properties that define the behavior of the step.

### Step Types

ARO HCP pipelines support multiple step types, each designed for a specific deployment scenario. The most commonly used types are:

#### ARM / Bicep Step

Used for deploying Azure infrastructure using [Bicep](bicep.md) templates. These steps allow defining and updating Azure resources declaratively.

```yaml
  ...
  steps:
  - name: region-infra
    action: ARM                                          (1)
    template: templates/region.bicep                     (2)
    parameters: configurations/region.tmpl.bicepparam    (3)
    deploymentLevel: ResourceGroup                       (4)
```

1. `action: ARM` marks the step as an ARM step.
2. `template`: The path to the Bicep template file that defines the infrastructure to be deployed.
3. `parameters`: The path to the Bicep parameters file that provides input values for the template.
4. `deploymentLevel`: The scope at which the deployment should occur. Valid values are `ResourceGroup` and `Subscription`.

This step type supports dry-run testing via [what-if](https://learn.microsoft.com/en-us/azure/azure-resource-manager/templates/deploy-what-if?tabs=azure-powershell).

Detailed information about Bicep templates in general and how to use the ARM step type, can be found in the [Bicep documentation](bicep.md#deploying-bicep-templates).

#### Shell Step

Execute shell commands or scripts within the pipeline environment. Shell steps are commonly used for deploying services via Helm charts, or performing other automation tasks.

```yaml
  ...
  steps:
  - name: upgrade-istio
    action: Shell                                       (1)
    command: make deploy                                (2)
    workingDir: ./component                             (3)
    variables:                                          (4)
    - name: TARGET_VERSION                              (5)
      configRef: svc.istio.targetVersion                (6)
    - name: RETRIES
      value: 5                                          (7)
```

1. `action: Shell` marks the step as a shell step.
2. `command`: The shell command to be executed. This can be a single command or a script file.
3. `workingDir`: The directory that the shell step will have access to at runtime, optional. Highly recommended. If left unset, the step will be able to access the entire repository, but will run in the working directory of the `pipeline.yaml` file. Such steps will always re-run in incremental mode.
4. `variables`: A list of environment variables that are set before executing the command.
5. `variables.name`: The name of the environment variable.
6. `variables.configRef`: The [configuration reference](configuration.md) to look up the value for the environment variable.
7. `variables.value`: Alternatively static values can be provided for the environment variable.

Currently, the following list of tools can be used within shell scripts:

- `az`
- `kubectl`
- `jq`
- `make`
- ...

See the [Shell extension](https://ev2docs.azure.net/features/service-artifacts/actions/shell-extensions/overview.html) documentation for more details.

>[!WARNING]
> TODO: we need to align and document the tool versions between the EV2 execution context and the Red Hat pipeline runner.

#### Helm Step

Used for deploying Kubernetes services onto AKS clusters using [Helm](https://helm.sh/) charts. Prefer this over Shell steps that invoke `helm` directly, since the shell execution environment may not include a Helm CLI. Helm steps manage the full lifecycle of a Helm release: the first deployment installs the chart, and subsequent deployments to the same release name upgrade it.

```yaml
  ...
  steps:
  - name: deploy-frontend
    action: Helm                                        (1)
    aksCluster: '{{ .svc.aks.name }}'                   (2)
    subnetName: 'aks-subnet'                            (3)
    releaseName: 'aro-hcp-frontend-dev'                 (4)
    releaseNamespace: '{{ .frontend.k8s.namespace }}'   (5)
    namespaceFiles:                                     (6)
    - namespace.yaml
    chartDir: ./deploy                                  (7)
    valuesFile: ./values.yaml                           (8)
    kustoDatabase: '{{ .kusto.serviceLogsDatabase }}'   (9)
    kustoTable: 'frontendLogs'                          (10)
    kustoEndpoint:                                      (11)
      resourceGroup: kusto
      step: kusto-lookup
      name: kustoUri
    inputVariables:                                     (12)
      frontendMsiClientId:                              (13)
        resourceGroup: service
        step: output
        name: frontendMsiClientId
    identityFrom:                                       (14)
      resourceGroup: global
      step: output
      name: globalMSIId
    timeout: 10m                                        (15)
    rollbackOnFailure: true                             (16)
```

1. `action: Helm` marks the step as a Helm step.
2. `aksCluster`: The name of the AKS cluster to deploy to. Can be configured dynamically using [configuration lookups](configuration.md). A kubeconfig is provisioned automatically (see [AKS cluster context](#aks-cluster)).
3. `subnetName`: Optional. For private AKS clusters, the name of the subnet the deployer connects to before accessing the cluster.
4. `releaseName`: The Helm release name. First deployment installs the chart; subsequent deployments to the same name upgrade it.
5. `releaseNamespace`: The Kubernetes namespace for the release, analogous to `helm --namespace`. Created automatically if it does not exist.
6. `namespaceFiles`: Optional list of Kubernetes namespace manifest files to apply before the Helm release. The release namespace itself does not need to be listed here — it is always created. Use these when a namespace requires specific labels, annotations, or other configuration. These files are pre-processed as Go templates.
7. `chartDir`: Path to the Helm chart directory, relative to the `pipeline.yaml` file.
8. `valuesFile`: Path to the Helm values file, relative to the `pipeline.yaml` file. Pre-processed as a Go template before being passed to Helm.

   > [!IMPORTANT]
   > Avoid Go template expressions (`{{ }}`) in the chart’s own `Chart.yaml` / `values.yaml` under `chartDir`, since Helm loads those before templatize preprocessing. Put templated content in the Helm step’s `valuesFile` and `namespaceFiles` instead (these are pre-processed by templatize and passed to Helm from a temporary location) - and place them outside of the charts directory.

9. `kustoDatabase`: Kusto database name for logging this deployment.
10. `kustoTable`: Kusto table for log ingestion.
11. `kustoEndpoint`: Input reference resolving to the Kusto cluster URI, used for deployment logging.
12. `inputVariables`: Optional. A map from placeholder names to step output references. Resolved values must be strings (non-string outputs will fail) and are injected into `valuesFile` and `namespaceFiles` by replacing `__<name>__` tokens.
    * Example:
      ```
      inputVariables:
        someImportantThing:
          resourceGroup: regional
          step: output
          name: outputVariableName
      ```
13. An `inputVariables` entry: the key (`frontendMsiClientId`) is the placeholder name; `resourceGroup`, `step`, and `name` identify which step output to read.
14. `identityFrom`: The managed identity used when running this deployment in EV2.
15. `timeout`: Optional. Maximum time to wait for the Helm release to reach ready state. Uses Go duration syntax (e.g., `30s`, `5m`, `1h`). Defaults to `5m`.
16. `rollbackOnFailure`: Optional. If `true`, rolls back to the previous release on upgrade failure, or uninstalls on install failure. Defaults to `false`.

The Helm step uses a **content-addressed cache** to skip execution when the chart directory, values file, namespace files, release name, release namespace, and target cluster are all unchanged since the last successful run, enabling efficient incremental deployments.

> [!NOTE]
> See also [service-deployment-concept.md#helm-chart](service-deployment-concept.md#helm-chart)

### Automated Retry

Steps can be configured to automatically retry on failure by adding an `automatedRetry` section. This is useful for handling transient failures such as network timeouts, temporary service unavailability, or rate limiting.

```yaml
  ...
  steps:
  - name: upgrade-aks-cluster
    action: Shell
    command: ./upgrade-cluster.sh
    workingDir: ./scripts
    automatedRetry:                                  (1)
      errorContainsAny:                              (2)
      - "context deadline exceeded"
      - "500 Internal Server Error"
      - "TooManyRequests"
      maximumRetryCount: 3                           (3)
      durationBetweenRetries: 2m                     (4)
```

1. `automatedRetry`: Optional configuration block for automatic retry behavior.
2. `errorContainsAny`: A list of strings to match against the error message (case-sensitive). If the error contains any of these strings, a retry will be triggered. Maximum of 16 items; total encoded length must not exceed 1KB.
3. `maximumRetryCount`: The maximum total number of step executions, including the initial attempt. Must be between 1 and 10. Defaults to 1 if not specified. For example, `3` means up to 3 total attempts, not 1 initial attempt plus 3 retries.
4. `durationBetweenRetries`: The time to wait between retry attempts. Should be between 1 minute (`1m`) and 3 hours (`3h`). Uses Go's time.Duration syntax (e.g., `30s`, `5m`, `1h`).

**Important notes:**

- The retry mechanism only triggers if the error message contains one of the specified strings
- Each retry attempt counts toward the total step execution time
- If the configured maximum number of attempts is exhausted and the step still fails, the entire pipeline will fail
- The `automatedRetry` configuration is available for all step types (ARM, Shell, Helm, etc.)

### Step execution context

All steps share a common execution context:

#### Azure session

When executing a shell step, an Azure session is provided for the defined subscription (`resourceGroups.subscription`) and resourcegroup (`resourceGroups.name`), allowing authenticated Azure operations. The identity used for this session is provided in the [configuration](configuration.md) as `global.globalMSIName` hosted in the `global.subscription` subscription. and `global.rg` Azure resourcegroup.

> [!IMPORTANT]
> Please note that this identity does not have access to all Azure resources in our subscriptions and resourcegroups by default. The necessary permissions need to be granted explicitly. You can observe this in various Bicep templates, where this identity is granted specific permissions, e.g. on Key Vaults or storage accounts.

#### Azure resource groups

The resourcegroup (`resourceGroups.name`) is pre-created before step execution starts.

#### AKS cluster

If an `resourceGroups.aksCluster` is specified, the `KUBECONFIG` environment variable is set and allows cluster admin interaction with the AKS cluster. This is mostly relevant for `Shell` steps.

#### Environment variables

All the variables defined in the `variables` section of a step are set as environment variables before executing the step. This allows steps to access configuration values or other necessary parameters.

In addition, an environment variable `zz_injected_EV2=1` is set when the step is executed within EV2. This environment variable is missing otherwise.

## Pipeline Deployment Scope

A pipeline is the smallest unit of deployment in ARO HCP. This means that all steps within a pipeline are executed from start to finish—there is no concept of executing a single step in only.

A pipeline typically corresponds to one architectural scope, such as global, regional, service or management. More details about these scopes can be found in the [architecture documentation](high-level-architecture.md). However, pipelines are not strictly limited to a single architectural scope. In some cases, cross-scope deployments are necessary, e.g. the regional DNS zone needs delegation from its parent, which resides in the global scope. You can find elaborate details about this in the [Bicep documentation](bicep.md#cross-subscription-deployments).

In most cases, each service has its own dedicated pipeline. This allows services to be rolled out individually.

By structuring pipelines appropriately, deployments can be managed efficiently while ensuring the right level of isolation and access across scopes.

> [!CAUTION]
> A pipelines [ARM step `deploymentLevel`](bicep.md#deploying-bicep-templates) should not be confused with the pipeline deployment scope. While the `deploymentLevel` is a technical aspect that controls if a Bicep module gets applied on a Resource Group vs Subscription level, the deployment scope defined in this section is a conceptual term that describes the alignment of a pipeline with a specific [architectural scope](high-level-architecture.md).

## Pipeline Execution in Different Environments

The various target environments for ARO HCP infrastructure and service deployment have very unique requirements on the rollout process, involving different tools and policies. Microsoft environments require deployments to be driven by ADO and their [EV2](https://ev2docs.azure.net) deployment service, which cannot be used for non-Microsoft environments. The Red Hat development environment on the other hand does not have such strictly regulated rollout processes and we chose to use GitHub actions as the overall driver for rollouts.

The custom pipeline format supports multiple deployment environments and tools to accommodate these different execution requirements and tooling.

> [!TIP]
> You can find out more about the different environments and ARO HCP instances in the [environments documentation](environments.md).

### Microsoft ADO and EV2

For Microsoft tenant deployments, we run ADO pipelines that use a dedicated EV2 generator. This generator translates pipeline files into EV2 manifests, uploads them to EV2, and performs an SDP rollout.

The pipeline format, with its resource group and subscription references and the list of steps, translates very well to the EV2 format. For example, the ARM and Shell step types are built-in step types in EV2. More details about the generation process will be provided in a dedicated document (tbd).

### Red Hat Pipeline Runner

Within the Red Hat development tenant for ARO HCP, we have two primary use cases for deployments.

Developers and SREs deploy their personal development ARO HCP instances using Makefile targets, such as `make entrypoint/Global` or `make pipeline/ACM`. Use your shell's tab-completion to view the available options for these targets. Behind the scenes, this executes the steps in a pipeline or under an entrypoint using a custom pipeline runner that interprets the pipeline files and takes actions accordingly, adhering to the defined expectations described in the [Step execution context](#step-execution-context) section.

In addition, some shared ARO HCP instances are continuously reconciled on each change in the ARO-HCP repository using [CI execution](ci/execution.md). These jobs leverage the same Makefile targets and the same pipeline executor as developers and SREs.

The custom pipeline runner can be found in [tooling/templatize](tooling/templatize).

To manually run a pipeline you can use the `make pipeline/` targets, e.g. to deploy `Microsoft.Azure.ARO.HCP.Maestro.Agent` with the `pers` environment architetype in the `dev` cloud (which is basically `public` cloud - see [configuration documentation](configuration.md)), run

```sh
make pipeline/Maestro.Agent DEPLOY_ENV=pers
```

The pipeline runner supports a dry-run mode that allows you to simulate the execution of a pipeline without actually deploying any resources. This is useful for verifying the correctness of the pipeline file and the expected behavior of the steps. Add the `DRY_RUN=true` option to the `make` command to enable dry-run mode:

```sh
make pipeline/Maestro.Agent DEPLOY_ENV=pers DRY_RUN=true
```

> [!IMPORTANT]
Please consult the step specific documentation for more details on the behavior of each step type during dry-run.
