# Cleanup Procedure for HCP Clusters Stuck on Deletion

## Overview

This document provides a systematic procedure for manually cleaning up ARO HCP clusters that become stuck during deletion. Stuck deletions occur when Kubernetes controllers fail to complete cleanup operations and remove resource finalizers, causing the deletion chain to halt indefinitely.

## Prerequisites

- Access to the Management Cluster where the HCP cluster is hosted (via `hcpctl mc breakglass <mc-name>`)
- Access to the Service Cluster (via `hcpctl sc breakglass <svc-name>`)
- `kubectl` access configured with appropriate permissions
- `jq` installed locally (and `curl` if you use the provided API examples)
- Cluster ID or cluster name of the stuck cluster

## Understanding the Resource Hierarchy

ARO HCP clusters create resources across multiple layers. Understanding this hierarchy is critical for systematic troubleshooting.

### Service Cluster Layer

```
Service Cluster (SVC)
├── Clusters Service (ns: clusters-service)
│   └── Cluster record (REST API, not a CRD)
│       └── State: installing / ready / uninstalling
└── Maestro Server (ns: maestro)
    └── ResourceBundle (REST API, not a CRD)
        ├── Namespace bundle (containsNamespaces: true)
        │   └── Creates *-00-hcp-namespaces ManifestWork
        └── Main cluster bundle
            └── Creates main ManifestWork with HostedCluster, secrets, etc.
```

### Management Cluster Layer

```
Management Cluster (MGMT)
│
├── Namespace: local-cluster
│   ├── ManifestWork (created by Maestro Agent from ResourceBundle)
│   │   ├── Contains all manifests for the HCP cluster
│   │   ├── Finalizer: cluster.open-cluster-management.io/manifest-work-cleanup
│   │   └── Status feedback from applied resources
│   ├── AppliedManifestWork (tracks applied resources)
│   │   └── Created by work agent as anchor for ManifestWork resources
│   └── ManagedCluster (ACM/MCE cluster registration)
│       ├── Finalizer: cluster.open-cluster-management.io/api-resource-cleanup
│       └── References the HostedCluster
│
├── Namespace: ocm-${CLUSTER_PREFIX}-${CLUSTER_ID} (HostedCluster namespace)
│   ├── HostedCluster (Primary HCP resource)
│   │   └── Finalizer: hypershift.openshift.io/finalizer
│   ├── NodePool(s) (one per worker pool)
│   │   └── Finalizer: hypershift.openshift.io/finalizer
│   ├── Secrets (credentials, certificates, pull secrets)
│   ├── ConfigMaps (configuration data)
│   └── Other Hypershift-managed resources
│
└── Namespace: ocm-${CLUSTER_PREFIX}-${CLUSTER_ID}-${CLUSTER_NAME} (Control Plane namespace)
    ├── Deployments (capi-provider, cluster-api, control-plane-operator)
    │   └── Finalizer: hypershift.openshift.io/component-finalizer
    ├── Cluster (cluster.x-k8s.io)
    │   └── Finalizer: cluster.cluster.x-k8s.io
    ├── HostedControlPlane
    │   └── Finalizer: hypershift.openshift.io/finalizer
    ├── MachineDeployment / MachineSet / Machine (cluster.x-k8s.io)
    │   └── Finalizers: cluster.x-k8s.io/machinedeployment,
    │                   cluster.x-k8s.io/machineset,
    │                   machine.cluster.x-k8s.io
    ├── AzureMachine (infrastructure.cluster.x-k8s.io)
    │   └── Finalizer: azuremachine.infrastructure.cluster.x-k8s.io
    ├── StatefulSets (etcd, control plane components)
    ├── Services, Pods, Secrets, ConfigMaps
    └── PersistentVolumeClaims (etcd storage)
```

> **Note**: `HostedControlPlane` lives in the **Control Plane namespace** (the `ocm-${CLUSTER_PREFIX}-${CLUSTER_ID}-${CLUSTER_NAME}` namespace), not in the `HostedCluster` namespace.

### Deletion Chain Dependencies

The deletion process follows this dependency order. Blocking at any level prevents cleanup of resources above it:

```
Clusters Service (sets state to "uninstalling")
  → Maestro Server (deletes ResourceBundle)
    → Maestro Agent (deletes ManifestWork on MGMT)
      → ManifestWork cleanup (deletes HostedCluster, ManagedCluster, etc.)
        → ManagedCluster destructor (hypershift-managed-cluster-destructor)
          → ManagedClusterAddon pre-delete hooks (finalizer: hosting-addon-pre-delete)
          → ManagedClusterAddon manifests cleanup (finalizer: hosting-manifests-cleanup)
        → HostedCluster deletion (finalizer: hypershift.openshift.io/finalizer)
          → NodePool deletion (finalizer: hypershift.openshift.io/finalizer)
          → Control Plane namespace cleanup
            → Deployments (finalizer: hypershift.openshift.io/component-finalizer)
            → Cluster CRD (finalizer: cluster.cluster.x-k8s.io)
            → HostedControlPlane (finalizer: hypershift.openshift.io/finalizer)
            → MachineDeployment / MachineSet / Machine (CAPI finalizers)
            → AzureMachine (finalizer: azuremachine.infrastructure.cluster.x-k8s.io)
```

**Key insight**: Deleting resources on the management cluster without first removing the source (ResourceBundle in Maestro / cluster record in CS) will result in the resources being **recreated** by the Maestro Agent.

## Systematic Troubleshooting Procedure

### Phase 1: Identify Stuck Resources (Management Cluster)

Connect to the management cluster:

```bash
hcpctl mc breakglass <mc-name>
export KUBECONFIG=<path-from-output>
```

#### Step 1: Find the Cluster Namespaces

```bash
# Find the HostedCluster prefix
kubectl get namespaces | grep ocm-
export CLUSTER_PREFIX="arohcpint"  # arohcpprod, arohcpstg,...

# Find namespaces for a specific cluster ID
export CLUSTER_ID="your-cluster-id"
kubectl get namespaces | grep ${CLUSTER_ID}
```

#### Step 2: Check Resource State

```bash
# Check HostedClusters with deletionTimestamp (stuck deleting)
kubectl get hostedcluster -A -o json | \
  jq -r '.items[] | select(.metadata.deletionTimestamp != null) |
  "\(.metadata.namespace)\t\(.metadata.name)\t\(.metadata.deletionTimestamp)"'

# Check ManifestWork stuck deleting
kubectl get manifestwork -n local-cluster -o json | \
  jq -r '.items[] | select(.metadata.deletionTimestamp != null) |
  "\(.metadata.name)\t\(.metadata.deletionTimestamp)"'

# Check for namespaces stuck in Terminating
kubectl get namespaces -o json | \
  jq -r '.items[] | select(.status.phase == "Terminating") |
  "\(.metadata.name)\t\(.metadata.deletionTimestamp)"'

# Check ManagedClusters (look for status JOINED vs Detaching)
kubectl get managedcluster

# Check ManagedClusterAddons in the cluster namespace
# (klusterlet auth loss during Detaching can leave addons with held finalizers)
kubectl get managedclusteraddon -n ${CLUSTER_ID}

# Check all HostedClusters and their status
kubectl get hostedcluster -A
```

#### Step 3: Identify Orphaned Resources

Look for mismatches indicating partial deletion:

```bash
# Namespaces without a HostedCluster (orphaned)
# Compare namespace list with HostedCluster list
kubectl get namespaces | grep ocm-${CLUSTER_PREFIX}
kubectl get hostedcluster -A

# ManifestWork without a corresponding cluster
kubectl get manifestwork -n local-cluster
```

#### Step 4: Investigate Why Resources Are Stuck

```bash
# Check HostedCluster conditions (look for errors)
kubectl get hostedcluster <name> -n <namespace> \
  -o jsonpath='{.status.conditions}' | jq .

# Check what's blocking a Terminating namespace
kubectl get namespace <namespace> -o json | jq '.status'

# Check Hypershift operator logs
kubectl logs -n hypershift deployment/operator --tail=50 | grep <cluster-name-or-id>

# Check resources with finalizers in a namespace (core/built-in kinds only)
kubectl get all -n <namespace> -o json | \
  jq '[.items[] | select(.metadata.finalizers != null) |
  {kind, name: .metadata.name, finalizers: .metadata.finalizers}]'

# NOTE: `kubectl get all` does NOT return many namespaced CRDs (including common blockers
# like HostedControlPlane and CAPI Cluster). Also check key Hypershift/CAPI CRDs:
kubectl get hostedclusters.hypershift.openshift.io,hostedcontrolplanes.hypershift.openshift.io,clusters.cluster.x-k8s.io \
  -n <namespace> -o json | \
  jq '[.items[] | select(.metadata.finalizers != null) |
  {kind, name: .metadata.name, finalizers: .metadata.finalizers}]'

# Check ManagedCluster conditions (look for Detaching state, klusterlet auth errors)
kubectl get managedcluster ${CLUSTER_ID} \
  -o jsonpath='{.status.conditions}' | jq .

# Check ManagedClusterAddon finalizers (hosting-manifests-cleanup, hosting-addon-pre-delete
# can be held when klusterlet user lost Azure AD authorization during Detaching)
kubectl get managedclusteraddon -n ${CLUSTER_ID} -o json | \
  jq '[.items[] | select(.metadata.finalizers != null) |
  {name: .metadata.name, finalizers: .metadata.finalizers}]'
```

### Phase 2: Check the Service Cluster

Before removing resources on the management cluster, check if the source (CS/Maestro) will recreate them.

Connect to the service cluster:

```bash
hcpctl sc breakglass <svc-name>
export KUBECONFIG=<path-from-output>
```

#### Step 5: Check Clusters Service

```bash
# List all clusters known to CS
kubectl exec -n clusters-service deployment/clusters-service -- \
  curl -s 'http://localhost:8000/api/clusters_mgmt/v1/clusters' | \
  jq '[.items[] | {id, name, state}]'

# Get details for a specific cluster
kubectl exec -n clusters-service deployment/clusters-service -- \
  curl -s "http://localhost:8000/api/clusters_mgmt/v1/clusters/${CLUSTER_ID}" | \
  jq '{id, name, state, azure: {subscription_id: .azure.subscription_id, resource_group_name: .azure.resource_group_name, resource_name: .azure.resource_name}}'
```

#### Step 6: Check Maestro Resource Bundles

```bash
# Search for resource bundles by cluster ID (search in manifest content)
#
# TIP: Use port-forward instead of kubectl exec for more reliable Maestro API
# inspection -- kubectl exec can truncate responses or drop fields:
#   kubectl port-forward -n maestro svc/maestro 8002:8000
#   curl -s 'http://localhost:8002/api/maestro/v1/resource-bundles?size=2900' | ...
#
kubectl exec -n maestro deployment/maestro -c maestro-server -- sh -c \
  "curl -s 'http://localhost:8000/api/maestro/v1/resource-bundles?size=2900'" | \
  jq --arg cid "${CLUSTER_ID}" '[.items[] | select(.manifests | tostring | test($cid)) |
  {id, name, created_at, consumer_name, deleted_at,
   labels: .metadata.labels}]'
```

### Phase 3: Resolution

#### Strategy 1: Delete via Clusters Service API (Preferred)

If the cluster still exists in CS, trigger deletion through the proper API:

```bash
# Delete cluster via the ARO-HCP API endpoint
kubectl exec -n clusters-service deployment/clusters-service -- \
  curl -s -X DELETE \
  "http://localhost:8000/api/aro_hcp/v1alpha1/clusters/${CLUSTER_ID}"
```

> **Note**: The endpoint is `/api/aro_hcp/v1alpha1/clusters/<id>`, NOT `/api/clusters_mgmt/v1/clusters/<id>`. The latter will reject the request with "ARO-HCP related clusters operation can only be performed using ARO-HCP endpoint".

Monitor the cluster state:

```bash
kubectl exec -n clusters-service deployment/clusters-service -- \
  curl -s "http://localhost:8000/api/clusters_mgmt/v1/clusters/${CLUSTER_ID}" | \
  jq '{id, name, state}'
# Expected: state should change to "uninstalling"
```

#### Strategy 2: Delete Orphaned Maestro Resource Bundles

If the cluster is gone from CS but resource bundles remain in Maestro (causing ManifestWork recreation):

```bash
# Delete the resource bundle by ID
kubectl exec -n maestro deployment/maestro -c maestro-server -- sh -c \
  "curl -s -X DELETE 'http://localhost:8000/api/maestro/v1/resource-bundles/<bundle-id>'"

# Verify deletion
kubectl exec -n maestro deployment/maestro -c maestro-server -- sh -c \
  "curl -s 'http://localhost:8000/api/maestro/v1/resource-bundles/<bundle-id>'"
# Expected: {"code":"maestro-7","reason":"Resource with id='...' not found"}
```

> **Note**: Maestro bundles behave differently on `DELETE`:
> - **Readonly bundles** (the `*-readonly` HostedCluster / NodePool reflections) are hard-deleted immediately.
> - **Bundles backing a `ManifestWork`** are only soft-deleted — the `DELETE` returns `204`, but the bundle remains listed with `deleted_at` set. The record is only removed from the Maestro store after the corresponding `ManifestWork` on the management cluster is also removed.
>
> If you see bundles still listed with a non-null `deleted_at` after `DELETE`, that is expected — proceed to Strategy 3 or Strategy 4 to clear the `ManifestWork` so Maestro can finish its own cleanup.

> **GOTCHA: `deleted_at` field location**
>
> The Maestro ResourceBundle API returns `deleted_at` at the **root level** of the JSON object, NOT inside `.metadata`. A query like `.metadata.deleted_at` will always return `null`, making bundles appear "clean" when they are actually soft-deleted. Always check the root-level `deleted_at` field:
> ```bash
> # WRONG — always returns null:
> jq '.items[] | .metadata.deleted_at'
>
> # CORRECT:
> jq '.items[] | {id, name, deleted_at}'
> ```

#### Strategy 3: Scale Down Maestro Agent and Clean ManifestWorks

When ResourceBundles are already soft-deleted (`deleted_at` is set) but the Maestro Agent on the management cluster continues recreating ManifestWorks (see Scenario 7), the safest approach is to stop the agent, clean up, then restart it.

> **⚠️ Prerequisite: Verify no customer-facing traffic on this management cluster.**
>
> Scaling the Maestro Agent to 0 stops **all** ManifestWork delivery on the management cluster, not just for the stuck clusters. While the agent is down, any in-flight or incoming ARM create, update, or delete operations targeting this MC will fail or time out.
>
> Before proceeding, confirm:
> 1. The MC has **no customer clusters** currently running on it, **and**
> 2. The region is **not actively serving new requests** that would be routed to this MC.
>
> If the region is live with customers, this procedure **requires a maintenance window**.
>
> If a maintenance window is not available, use [Strategy 4 (Manual Finalizer Removal)](#strategy-4-manual-finalizer-removal-on-management-cluster-last-resort) instead. Strategy 4 patches finalizers on individual resources without stopping the agent, so it does not block other operations on the MC.

Connect to the management cluster:

```bash
hcpctl mc breakglass <mc-name>
export KUBECONFIG=<path-from-output>
```

```bash
# 1. Scale down the Maestro Agent to stop ManifestWork recreation
kubectl scale deployment -n maestro maestro-agent --replicas=0

# 2. Wait for the agent pod to terminate
kubectl wait --for=delete pod -l app=maestro-agent -n maestro --timeout=60s

# 3. Patch finalizers and delete all ManifestWorks for the stuck clusters
for mw in $(kubectl get manifestwork -n local-cluster -o jsonpath='{.items[*].metadata.name}' | tr ' ' '\n' | grep -F "${CLUSTER_ID}"); do
  kubectl patch manifestwork $mw -n local-cluster \
    --type=merge -p='{"metadata":{"finalizers":null}}'
  kubectl delete manifestwork $mw -n local-cluster
done

# 4. Also clean up AppliedManifestWorks (cluster-scoped)
for amw in $(kubectl get appliedmanifestwork -o jsonpath='{.items[*].metadata.name}' | tr ' ' '\n' | grep -F "${CLUSTER_ID}"); do
  kubectl patch appliedmanifestwork $amw \
    --type=merge -p='{"metadata":{"finalizers":null}}'
  kubectl delete appliedmanifestwork $amw
done

# 5. Scale the Maestro Agent back up
kubectl scale deployment -n maestro maestro-agent --replicas=1

# 6. Verify the agent does NOT recreate the ManifestWorks
#    (it won't, because the bundles are soft-deleted)
kubectl get manifestwork -n local-cluster | grep -F "${CLUSTER_ID}"
```

> **Why scale down instead of restart?** Restarting the agent creates a race condition: you delete ManifestWorks while the agent is still running, and it recreates them before you finish. Scaling to 0 guarantees no recreation during cleanup.

> **Note**: After ManifestWorks are cleaned and the agent confirms no recreation, the CS destructor can complete. Verify on the service cluster that the cluster record is fully removed.

#### Strategy 4: Manual Finalizer Removal on Management Cluster (Last Resort)

Only after ensuring the source (CS/Maestro) won't recreate resources, remove finalizers **bottom-up** on the management cluster.

> **Note**: If Phase 1 Step 4 shows the `ManagedCluster` in `Detaching` state or `ManagedClusterAddon` resources with held finalizers, clear those **first** (step 0 below). The destruct chain stops at the `hypershift-managed-cluster-destructor` step when addon pre-delete hooks fail (commonly due to klusterlet auth loss during Detaching), and the CP namespace teardown is never even attempted.

```bash
# 0. (Conditional) Clear ManagedClusterAddon finalizers when the destruct chain is stuck
#    at the ManagedCluster layer (Detaching + addon pre-delete hooks failing).
#    Uses a merge patch (idempotent when .metadata.finalizers is absent).
for name in $(kubectl get managedclusteraddon -n ${CLUSTER_ID} -o jsonpath='{.items[*].metadata.name}'); do
  kubectl patch managedclusteraddon $name -n ${CLUSTER_ID} \
    --type=merge -p='{"metadata":{"finalizers":null}}'
done

# 1. Check what's blocking the CP namespace (if Terminating):
kubectl get namespace <cp-namespace> -o json | jq '.status.conditions[] | select(.type == "NamespaceContentRemaining")'

# 2. Remove finalizers from CP namespace resources (bottom-up, leaf resources first):
# Deployments with hypershift.openshift.io/component-finalizer
kubectl patch deployment <name> -n <cp-namespace> \
  --type=json -p='[{"op": "replace", "path": "/metadata/finalizers", "value": []}]'

# AzureMachines (infrastructure.cluster.x-k8s.io)
# Uses a merge patch so the command is idempotent even if .metadata.finalizers is absent
# on some items (JSONPatch `replace` would fail on those).
for name in $(kubectl get azuremachines.infrastructure.cluster.x-k8s.io -n <cp-namespace> -o jsonpath='{.items[*].metadata.name}'); do
  kubectl patch azuremachines.infrastructure.cluster.x-k8s.io $name -n <cp-namespace> \
    --type=merge -p='{"metadata":{"finalizers":null}}'
done

# Machines / MachineSets / MachineDeployments (cluster.x-k8s.io)
for kind in machines.cluster.x-k8s.io machinesets.cluster.x-k8s.io machinedeployments.cluster.x-k8s.io; do
  for name in $(kubectl get $kind -n <cp-namespace> -o jsonpath='{.items[*].metadata.name}'); do
    kubectl patch $kind $name -n <cp-namespace> \
      --type=merge -p='{"metadata":{"finalizers":null}}'
  done
done

# Cluster CRD with cluster.cluster.x-k8s.io finalizer
kubectl patch clusters.cluster.x-k8s.io <name> -n <cp-namespace> \
  --type=json -p='[{"op": "replace", "path": "/metadata/finalizers", "value": []}]'

# HostedControlPlane with hypershift.openshift.io/finalizer
kubectl patch hostedcontrolplanes.hypershift.openshift.io <name> -n <cp-namespace> \
  --type=json -p='[{"op": "replace", "path": "/metadata/finalizers", "value": []}]'

# 3. Remove NodePool finalizers in the HostedCluster namespace
# Uses a merge patch (idempotent when .metadata.finalizers is absent).
for name in $(kubectl get nodepool -n ocm-${CLUSTER_PREFIX}-${CLUSTER_ID} -o jsonpath='{.items[*].metadata.name}'); do
  kubectl patch nodepool $name -n ocm-${CLUSTER_PREFIX}-${CLUSTER_ID} \
    --type=merge -p='{"metadata":{"finalizers":null}}'
done

# 4. Remove HostedCluster finalizer (only after CP namespace resources and NodePools are cleared)
kubectl patch hostedcluster <name> -n ocm-${CLUSTER_PREFIX}-${CLUSTER_ID} \
  --type=json -p='[{"op": "replace", "path": "/metadata/finalizers", "value": []}]'

# 5. Remove ManifestWork finalizers
kubectl patch manifestwork <name> -n local-cluster \
  --type=json -p='[{"op": "replace", "path": "/metadata/finalizers", "value": []}]'

# 6. Delete orphaned namespaces (only after all resources are cleared)
kubectl delete namespace <namespace>
```

### Phase 4: Verify Cleanup

```bash
# On the Management Cluster
kubectl get namespaces | grep ${CLUSTER_ID}
kubectl get manifestwork -n local-cluster | grep ${CLUSTER_ID}
kubectl get managedcluster | grep ${CLUSTER_ID}
kubectl get hostedcluster -A | grep ${CLUSTER_ID}
kubectl get appliedmanifestwork | grep ${CLUSTER_ID}

# On the Service Cluster - verify CS no longer has the cluster
kubectl exec -n clusters-service deployment/clusters-service -- \
  curl -s 'http://localhost:8000/api/clusters_mgmt/v1/clusters' | \
  jq '[.items[] | {id, name, state}]'

# On the Service Cluster - verify no orphaned Maestro bundles
kubectl exec -n maestro deployment/maestro -c maestro-server -- sh -c \
  "curl -s 'http://localhost:8000/api/maestro/v1/resource-bundles?size=2900'" | \
  jq --arg cid "${CLUSTER_ID}" '[.items[] | select(.manifests | tostring | test($cid)) | {id, name}]'
```

> **Note**: If CS still has the cluster record in `uninstalling` state after management cluster cleanup, re-issue the ARO-HCP `DELETE` (Strategy 1). With Maestro and the management cluster cleared, the CS controller can now complete the deletion and subsequent `GET`s will return `404`.

## Common Stuck Scenarios

### Scenario 1: HostedCluster Stuck Deleting - Azure Resources Already Gone

**Symptoms**: Hypershift operator logs show "hostedcluster is still deleting" in a tight loop every ~5 seconds. HostedCluster conditions show `ResourceGroupNotFound` or similar Azure errors.

**Cause**: The Azure resource group was deleted externally, so the Hypershift operator can't clean up cloud resources and won't remove its finalizer.

**Fix**: Remove the `hypershift.openshift.io/finalizer` from the HostedCluster. This is safe because the Azure resources are already gone.

### Scenario 2: ManifestWork/Namespaces Keep Being Recreated

**Symptoms**: You delete ManifestWork or namespaces on the management cluster, but they reappear within seconds.

**Cause**: The Maestro ResourceBundle still exists on the service cluster. The Maestro Agent continuously reconciles and recreates the ManifestWork.

**Fix**: Delete the source first — either trigger deletion via CS API (Strategy 1) or delete the ResourceBundle from Maestro directly (Strategy 2). Only then clean up management cluster resources.

### Scenario 3: Orphaned Namespace Resource Bundles

**Symptoms**: Cluster is gone from CS, main ManifestWork is gone, but `*-00-hcp-namespaces` ManifestWork keeps being recreated along with the CP namespace.

**Cause**: CS deleted the main cluster resource bundles from Maestro but left behind the namespace-only bundle (`containsNamespaces: true` label). This is a known issue in the CS deletion flow.

**Fix**: Find and delete the orphaned resource bundle from Maestro (Strategy 2). The bundle can be identified by its `api.openshift.com/id` label matching the cluster ID.

### Scenario 4: CP Namespace Stuck Terminating

**Symptoms**: `kubectl get namespace <cp-namespace>` shows `Terminating` but never completes.

**Cause**: Resources with finalizers remain in the namespace. Common blockers:
- `clusters.cluster.x-k8s.io` with `cluster.cluster.x-k8s.io` finalizer
- `hostedcontrolplanes.hypershift.openshift.io` with `hypershift.openshift.io/finalizer`
- `deployments` with `hypershift.openshift.io/component-finalizer`

**Fix**: Check `kubectl get namespace <ns> -o json | jq '.status.conditions'` to identify blocking resources, then remove their finalizers (Strategy 4).

### Scenario 5: CP Namespace Stuck After Cluster and HostedControlPlane Are Cleared

**Symptoms**: After force-removing finalizers on the CAPI `Cluster`, the `HostedControlPlane`, and the Hypershift Deployments, the CP namespace is still `Terminating`. `kubectl get namespace <cp-ns> -o json | jq '.status.conditions'` references infrastructure or machine kinds.

**Cause**: CAPI machine resources hold their own finalizers and are not removed when the parent `Cluster` is force-deleted (finalizer cleared). Common blockers in the CP namespace:

- `azuremachines.infrastructure.cluster.x-k8s.io` (`azuremachine.infrastructure.cluster.x-k8s.io`)
- `machines.cluster.x-k8s.io` (`machine.cluster.x-k8s.io`)
- `machinesets.cluster.x-k8s.io` (`cluster.x-k8s.io/machineset`)
- `machinedeployments.cluster.x-k8s.io` (`cluster.x-k8s.io/machinedeployment`)

**Fix**: Patch the finalizers on all four kinds in the CP namespace (see Strategy 4, step 2).

### Scenario 6: ManagedCluster Stuck Detaching - Addon Finalizers Held

**Symptoms**: HCP deletion never advances past the ACM layer. `kubectl get managedcluster ${CLUSTER_ID}` shows the cluster in `Detaching` state for an extended period. `kubectl get managedclusteraddon -n ${CLUSTER_ID}` lists addons (typically `config-policy-controller`, `governance-policy-framework`) with `hosting-manifests-cleanup` and/or `hosting-addon-pre-delete` finalizers held. The CP namespace cleanup is never attempted because the destruct chain is blocked upstream at `hypershift-managed-cluster-destructor`.

**Cause**: The hosting addon pre-delete hook executes against the hosted cluster using the klusterlet user, but that identity has already lost its Azure AD authorization as part of the Detaching flow. The pre-delete hook fails, the addon's finalizers are never removed, and the ManagedCluster destructor cannot complete.

**Fix**: Clear the held `ManagedClusterAddon` finalizers manually (see Strategy 4, step 0). After this, the `ManagedCluster` destructor completes, the `ManifestWork` cleanup proceeds, and the rest of the deletion chain advances normally.

### Scenario 7: Maestro Agent Ignores Soft-Deleted ResourceBundles (CloudEvents Desync)

**Symptoms**: You delete ManifestWorks on the management cluster, but they are immediately recreated with a fresh `creationTimestamp`. This looks like Scenario 2, but checking Maestro shows the ResourceBundles are already soft-deleted (`deleted_at` is set at root level). AppliedManifestWorks carry Maestro labels (`work.open-cluster-management.io/applied-by: maestro-agent-*`).

**Cause**: A bug in the Maestro Agent's CloudEvents sync (tracked as ARO-28432). The Maestro Server accepted the DELETE and soft-deleted the ResourceBundle, but the deletion event was never propagated to the agent via the MQTT CloudEvents channel. The agent's local state still considers the bundles active, so it keeps recreating ManifestWorks. A Maestro restart may self-heal some clusters (by replaying state from the server), but not all — in AROSLSRE-1520, a production rollout reduced stuck clusters from 50 to ~16, but the remainder required manual intervention.

**Distinguishing from Scenario 2**: In Scenario 2 the bundles are active (no `deleted_at`). In Scenario 7 the bundles ARE soft-deleted but the agent ignores this. Check the root-level `deleted_at` field (see the GOTCHA note under Strategy 2) — if it is set but ManifestWorks are still being recreated, you are in Scenario 7.

**Fix**: Use Strategy 3 — scale down the Maestro Agent on the management cluster, clean all ManifestWorks and AppliedManifestWorks for the affected clusters, then scale the agent back up. The agent will re-sync from the server on startup and will NOT recreate the ManifestWorks because the bundles are soft-deleted. After the ManifestWorks are gone, the CS destructor can complete.

## Quick Reference: Key API Endpoints

| Component | Endpoint | Notes |
|---|---|---|
| CS - List clusters | `http://localhost:8000/api/clusters_mgmt/v1/clusters` | Via exec into CS pod |
| CS - Delete cluster | `http://localhost:8000/api/aro_hcp/v1alpha1/clusters/<id>` | Must use `aro_hcp` endpoint |
| Maestro - List bundles | `http://localhost:8000/api/maestro/v1/resource-bundles` | Via exec into maestro-server container |
| Maestro - Delete bundle | `http://localhost:8000/api/maestro/v1/resource-bundles/<id>` | DELETE method |

## Quick Reference: Maestro API Inspection via Port-Forward

Port-forward is more reliable than `kubectl exec` for Maestro API inspection. It avoids response truncation and ensures all JSON fields (including root-level `deleted_at`) are visible.

```bash
# On the Service Cluster
kubectl port-forward -n maestro svc/maestro 8002:8000

# Then in another terminal:
# List all resource bundles
curl -s 'http://localhost:8002/api/maestro/v1/resource-bundles?size=2900' | \
  jq '[.items[] | {id, name, deleted_at}]'

# Search for a specific cluster's bundles
curl -s 'http://localhost:8002/api/maestro/v1/resource-bundles?size=2900' | \
  jq --arg cid "${CLUSTER_ID}" '[.items[] | select(.manifests | tostring | test($cid)) |
  {id, name, deleted_at, labels: .metadata.labels}]'
```

## Quick Reference: Common Finalizers

| Finalizer | Resource | Controller |
|---|---|---|
| `hypershift.openshift.io/finalizer` | HostedCluster, HostedControlPlane | Hypershift Operator |
| `hypershift.openshift.io/component-finalizer` | Deployments in CP namespace | Hypershift Operator |
| `cluster.cluster.x-k8s.io` | Cluster (CAPI) | Cluster API |
| `machine.cluster.x-k8s.io` | Machine (CAPI) | Cluster API |
| `cluster.x-k8s.io/machineset` | MachineSet (CAPI) | Cluster API |
| `cluster.x-k8s.io/machinedeployment` | MachineDeployment (CAPI) | Cluster API |
| `azuremachine.infrastructure.cluster.x-k8s.io` | AzureMachine | CAPZ (Cluster API Provider Azure) |
| `cluster.open-cluster-management.io/manifest-work-cleanup` | ManifestWork | ACM Work Agent |
| `cluster.open-cluster-management.io/api-resource-cleanup` | ManagedCluster | ACM |
| `hosting-manifests-cleanup` | ManagedClusterAddon | Hosting Addon Controller |
| `hosting-addon-pre-delete` | ManagedClusterAddon | Hosting Addon Controller |

## Quick Reference: Key Controller Logs

```bash
# Hypershift operator (manages HostedCluster deletion)
kubectl logs -n hypershift deployment/operator --tail=100 | grep <cluster-name>

# Maestro Agent (manages ManifestWork lifecycle)
kubectl logs -n maestro deployment/maestro-agent -c maestro-agent | grep <cluster-id>

# Maestro Agent replica count (check if scaled down)
kubectl get deployment -n maestro maestro-agent -o jsonpath='{.spec.replicas}'

# Work agent (manages ManifestWork application)
kubectl logs -n open-cluster-management-agent deployment/klusterlet-work-agent
```
