# CI Cleanup

ARO HCP has three cleanup modes, and they are intentionally different:

1. strict per-test cleanup in the E2E framework
2. targeted environment teardown for personal dev and DEV CI
3. background hygiene from periodic cleanup jobs

The differences are deliberate. A cleanup path that is correct for a test is not always the right one for a background janitor job, and a path that is good for DEV teardown is not always the one we want in public cloud.

For the surrounding execution model, see [CI Execution](execution.md). For the operator view of the jobs that run these paths, see [CI Operations](operations.md).

## The Three Cleanup Modes

### Strict per-test cleanup

```mermaid
---
title: "STRICT - failure is signal"
---
flowchart LR
    testEnd(("e2e test<br/>finishes")) --> e2eCleanup["Per-test cleanup<br/><i>e2e framework</i>"]
    e2eCleanup --> rpDelete["Delete cluster via RP"]
    rpDelete --> waitRg["Wait for managed RGs<br/>to disappear"]
    waitRg --> leftovers["Clean test-owned leftovers<br/>(customer RG, app regs,<br/>leased identities)"]
```

This runs after each E2E test.

Its job is to clean up what the test created and to fail loudly if cleanup is broken. That is why this path is strict: leaking resources after a test usually means either the product did not delete something correctly or the test created something it did not own properly.

In the normal public-cloud flow, test cleanup:

1. deletes HCP clusters through the RP
2. waits for managed resource groups to disappear
3. finishes cleaning up test-owned leftovers such as the customer resource group, app registrations, and leased identities

This path is not just background hygiene. Cleanup failures are treated as test signal.

### Targeted environment teardown

```mermaid
---
title: "TOLERANT - topology-driven"
---
flowchart LR
    devDelete(("make clean /<br/>CI deprovision")) --> topoCleanup["Environment teardown<br/><i>templatize cleanup</i>"]
    topoCleanup --> deriveTargets["Derive target RGs<br/>from topology"]
    deriveTargets --> asyncDelete["Delete RGs<br/>(async, best-effort)"]
```

This is the cleanup path used when we explicitly tear down a known environment, most commonly:

- personal-dev cleanup
- DEV E2E CI deprovision

Its job is not to validate cluster deletion correctness. Its job is to remove a known set of environment resource groups derived from topology.

Because of that, it is more tolerant than per-test cleanup:

- it works from a known target set instead of scanning broadly
- it is comfortable with async deletion
- it favors getting the environment moving toward deletion over proving that every step completed synchronously

### Background hygiene

```mermaid
---
title: "BEST-EFFORT - background hygiene"
---
flowchart LR
    cronSchedule(("Prow cron<br/>schedule")) --> periodicCleanup["Periodic cleanup jobs<br/><i>Prow periodics</i>"]
    periodicCleanup --> expiredRg["Expired test RGs"]
    periodicCleanup --> expiredApp["Expired app registrations"]
    periodicCleanup --> kustoCleanup["Kusto role assignments"]
    periodicCleanup --> sweeperRg["cleanup-sweeper<br/>rg-ordered"]
    periodicCleanup --> sweeperShared["cleanup-sweeper<br/>shared-leftovers"]
```

This is the periodic cleanup layer defined in:

- `openshift/release: ci-operator/config/Azure/ARO-HCP/Azure-ARO-HCP-main__periodic-cleanup.yaml`

Its job is to keep subscriptions healthy over time, not to prove that a single test or delete operation behaved perfectly.

Today that periodic layer includes:

- expired test resource-group cleanup
- expired app-registration cleanup
- Kusto role-assignment cleanup
- `cleanup-sweeper` jobs for policy-driven resource-group cleanup and shared leftovers

The `cleanup-sweeper` `shared-leftovers` workflow runs per environment. Alongside the DEV shared subscriptions it now runs hourly against the INT, STG, and PROD e2e/customer subscriptions (`sweeper-shared-leftovers-{dev,int,stg,prod}` in the periodic-cleanup config), each reporting failures to its own `#aro-hcp-failures-<env>` channel. Each per-environment job selects its subscriptions through `VAULT_SECRET_PROFILE` (`int-rh`, `stg-rh`, `prod-rh`; DEV uses the default `dev` profile). This workflow deletes orphaned role assignments and purges orphaned soft-deleted Key Vaults; it does not delete resource groups, so it does not depend on the `createdAt` tag.

To resolve the principals behind orphaned role assignments, `shared-leftovers` reads the Microsoft Graph directory, which requires `Directory.Read.All`. The per-environment ARM identity (`VAULT_SECRET_PROFILE`) usually lacks that tenant-wide grant, so the step exports a dedicated Graph identity from `GRAPH_SECRET_PROFILE` (the dev bot, which holds `Directory.Read.All`) whenever its mounted profile differs from the ARM profile. When the two resolve to the same profile, no separate Graph credential is used and the ARM identity serves both. The sweeper binary reads that dedicated identity from `GRAPH_AZURE_CLIENT_ID` / `GRAPH_AZURE_TENANT_ID` / `GRAPH_AZURE_CLIENT_SECRET`, which the `aro-hcp-deprovision-cleanup-sweeper` step in `openshift/release` exports from the mounted Graph profile.

For `cleanup-sweeper` `rg-ordered`, candidate resource groups are chosen using `tooling/cleanup-sweeper/resourcegroups.policy.yaml`. Discovery treats the `createdAt` tag (RFC3339 timestamp on the resource group) as required for any `action: delete` rule: groups without a parseable tag are not candidates.

Azure does not set that tag by default. Subscriptions where `rg-ordered` should run must apply an Azure Policy (or equivalent) that stamps `tags['createdAt']` when a resource group is created, using `[utcNow()]` in the policy rule. The rule body lives in `tooling/cleanup-sweeper/scripts/rg-createdat-policy-rule.json`.

Because the `createdAt` tag is required for every `action: delete` rule, this policy is a hard per-subscription prerequisite: `rg-ordered` must not be enabled in a subscription (including the INT, STG, and PROD e2e subscriptions) until the `createdAt` policy is assigned there. The sweeper only requires a parseable `createdAt` tag, so groups created before the policy was assigned are not candidates until the tag is present (either stamped by the policy on new groups or backfilled onto existing ones, for example via policy remediation). Until the policy is in place, only `shared-leftovers` is safe to run in that subscription, since it does not delete resource groups.

For the DEV e2e/customer subscriptions (`.ci.dev.e2eSubscriptions`), this policy is assigned declaratively by the dev-ci pipeline: `dev-infrastructure/templates/createdat-rg-tag-policy.bicep` fans the subscription-scoped definition + assignment out to each e2e subscription, wired as the `createdat-rg-tag-policy` step in `dev-infrastructure/dev-ci/e2e-subscription-rbac-grants/pipeline.yaml`. That step lives in the privileged dev-ci tree because writing policy definitions/assignments needs `Microsoft.Authorization/policy*/write`, which the unattended Contributor bot lacks; run it with `make dev-ci-privileged-local-run` as an OWNERS member. For one-off or non-dev subscriptions, `tooling/cleanup-sweeper/scripts/create-createdat-policy-assignment.sh` creates or updates the same definition and assignment imperatively.

> **Keep the rule body in sync.** The policy rule is expressed in two places: `tooling/cleanup-sweeper/scripts/rg-createdat-policy-rule.json` (used by the imperative script) and, mirrored inline, in `dev-infrastructure/templates/createdat-rg-tag-policy-subscription.bicep` (used by the declarative dev-ci path). Both use the same definition/assignment resource names so they converge on one policy, but the rule body itself is duplicated: any change to the `if`/`then`/`[utcNow()]` logic must be applied to both files.

This path is intentionally best-effort. If one run leaves something behind, the next run can pick it up.

## Why They Behave Differently

### Why periodic cleanup is best-effort

Periodic cleanup runs in the background across shared subscriptions. If it is too aggressive, it becomes its own source of instability and Azure throttling.

That is why the background cleanup jobs are deliberately conservative. In particular, the `cleanup-sweeper` jobs currently run with a single worker, and `rg-ordered` runs with `wait=false`, to be gentler to the Network provider.

The goal is steady pressure in the right direction, not maximum deletion throughput in one run.

### Why E2E cleanup is strict

Per-test cleanup is part of test correctness.

If a test cannot clean up what it created, that usually means:

- cluster deletion did not complete correctly
- test-owned Azure resources were left behind
- the test itself created resources outside the framework's ownership model

That is why the framework treats cleanup failure as meaningful signal instead of just logging and moving on.

### Why DEV cleanup actively deletes managed resource groups but public-cloud cleanup does not

This is one of the most common sources of confusion.

In public cloud, managed resource groups are part of the cluster lifecycle owned by the RP and downstream platform components. The correct behavior is to delete the cluster through the RP and wait for the managed resource groups to disappear as a consequence of that owned flow.

Additionally, in public cloud the managed resource groups carry deny assignments that prevent customer principals from modifying them. Any attempt by cleanup jobs to directly delete those resource groups would simply fail with a permission error.

In DEV and other `no-rp` situations, that owner may already be gone or unavailable. If cleanup waits for someone else to delete managed resource groups, they can remain forever. In those cases cleanup has to be more direct: find the managed resource groups itself, clean them up, and then clean the parent resource group.

So the difference is not "DEV is special" for its own sake. The difference is whether there is still a live owner for managed resource-group deletion.

### Why dev cleanup actively deletes service association links but public-cloud cleanup does not

Service association links (SALs) are created by Azure when a subnet is delegated to a service. When a cluster uses a delegated customer subnet, Azure creates a SAL linking the subnet to the cluster resource.

In public cloud, SALs are part of the cluster lifecycle owned by the RP. The correct behavior is to delete the cluster through the RP and expect Azure to remove the SAL as a consequence of that deletion.

If SALs are left behind in public cloud, that is a signal that the RP did not complete deletion correctly. The cleanup framework treats that as meaningful signal, not something to silently clean up.

In dev and other `no-rp` situations, the RP may already be torn down. In dev e2e CI, each job spins up a fresh RP and tears it down after the run. If a SAL is orphaned and there is no RP left to clean it up, it blocks resource-group deletion with `InUseSubnetCannotBeDeleted` errors.

That is why the `no-rp` cleanup mode includes an optional FPA-authenticated SAL deletion step. When the RP is unavailable, cleanup deletes orphaned SALs directly before deleting the resource group. This requires FPA credentials because standard service principals lack permission to delete SALs.

## How Each Path Works

### E2E test teardown

This is implemented in:

- `test/util/framework/per_test_framework.go`

At the end of a test, the framework cleans:

- resource groups created by the test context
- app registrations created by the test
- leased identity leftovers

There are two cleanup modes in the framework:

- `standard`: used in normal public-cloud teardown
- `no-rp`: used when the RP is unavailable or the infrastructure has already been torn down

`standard` is strict and cluster-aware. `no-rp` is more direct and more best-effort.

### Templatize cleanup

This is implemented in:

- `tooling/templatize/cmd/entrypoint/cleanup/`

This path starts from topology, selects the resource groups that belong to the chosen entrypoint or pipeline, and tears those down.

That makes it a good fit for:

- personal-dev cleanup
- DEV E2E CI deprovision

It is targeted cleanup of known resources, not subscription-wide hygiene.

### Periodic cleanup

The consolidated periodic job definition lives in:

- `openshift/release: ci-operator/config/Azure/ARO-HCP/Azure-ARO-HCP-main__periodic-cleanup.yaml`

There are two broad styles of periodic cleanup:

- test-oriented cleanup jobs, such as expired resource groups and old test identities
- `cleanup-sweeper` jobs, which are meant for policy-driven resource-group cleanup and shared leftovers

The important point is that all of these are background hygiene jobs. They are there to keep shared environments healthy and reduce accumulation over time.

## Where To Look

If you need to change behavior, start here:

- consolidated periodic jobs: `openshift/release: ci-operator/config/Azure/ARO-HCP/Azure-ARO-HCP-main__periodic-cleanup.yaml`
- `cleanup-sweeper` CLI: `tooling/cleanup-sweeper/cmd/root/options.go`
- `rg-ordered` resource group `createdAt` tag policy: `tooling/cleanup-sweeper/scripts/create-createdat-policy-assignment.sh`
- ordered cleanup engine: `tooling/cleanup-sweeper/pkg/engine/resourcegroup_ordered_cleanup.go`
- templatize cleanup: `tooling/templatize/cmd/entrypoint/cleanup/`
- E2E teardown: `test/util/framework/per_test_framework.go`

For the CI execution view, also see [CI Execution](execution.md). For job-level operations, see [CI Operations](operations.md).
