# Dev-CI Topology

The `dev-ci` topology is the standalone, persistent CI-support environment for ARO HCP. It is separate from both the main `Global` / `Region` rollout graph and the on-demand DEV RP footprint that `e2e-parallel` creates inside a job run.

Use this document to understand what the `dev-ci` rollout owns today, what it deliberately does not own, and where the remaining mixed-management boundary still exists.

## Why Dev-CI Is Separate

`dev-ci` exists because some CI dependencies need to be long-lived and shared across many jobs, but they are not part of the normal product environment topology:

- `config/config-dev-ci.yaml` carries standalone configuration for these CI-supporting resources.
- `topology-dev-ci.yaml` defines a separate rollout graph with two entrypoints: `Microsoft.Azure.ARO.HCP.DevCI.Unprivileged` (the unattended postsubmit surface) and `Microsoft.Azure.ARO.HCP.DevCI.Privileged` (the on-demand, Owner-only grants surface).
- The long-term goal is for `openshift/release` jobs to consume this shared foundation without defining or deploying the `dev-ci` topology themselves, but that handoff is not complete yet.

This keeps persistent CI-support infrastructure separate from the per-job RP footprint that DEV local E2E provisions on demand.

## What The Topology Manages Today

Today `topology-dev-ci.yaml`'s `Microsoft.Azure.ARO.HCP.DevCI.Unprivileged` entrypoint contains one root service group plus four child service groups:

- `Microsoft.Azure.ARO.HCP.DevCI.Unprivileged`
  - Deploys shared dev/CI network resources, the `opstool` AKS cluster, and the shared Prometheus monitoring stack.
- `Microsoft.Azure.ARO.HCP.DevCI.TenantQuota`
  - Deploys the historically named `tenant-quota-collector`, the extensible DEV CI telemetry exporter.
- `Microsoft.Azure.ARO.HCP.DevCI.Gateway`
  - Deploys the shared Istio gateway and DNS wiring for `opstool`.
- `Microsoft.Azure.ARO.HCP.DevCI.CertManager`
  - Deploys `cert-manager` and the shared Azure DNS DNS-01 `ClusterIssuer`.
- `Microsoft.Azure.ARO.HCP.DevCI.CIHealth`
  - Deploys the CIHealth runtime at `cihealth.tools.hcpsvc.osadev.cloud`.

In other words, `dev-ci` owns the persistent CI support layer, not the full runtime behavior of every CI job.

## The Privileged Entrypoint

`topology-dev-ci.yaml` also declares a **standalone, on-demand** privileged tree rooted at the `Microsoft.Azure.ARO.HCP.DevCI.Privileged` entrypoint, deliberately kept **separate** from the unattended `Microsoft.Azure.ARO.HCP.DevCI.Unprivileged` entrypoint. It owns two privileged concerns:

1. **The CI bot Entra identity lifecycle** (`E2ESubscriptionRBAC` service group): creating/reconciling the bot Entra applications + service principals and minting their Key Vault client secrets. These are Entra **directory-object writes** (create app/SP, `addPassword`) — an unattended CI identity is deliberately not given app/SP-management or credential-minting rights over directory objects, so this runs on demand instead.
2. **The subscription-scoped RBAC grants** (`E2ESubscriptionRBACGrants` service group): the DEV E2E customer-subscription RBAC (custom role definitions + shared-principal role assignments) and the CI bot subscription-scoped RBAC. Every step creates subscription-scoped custom role definitions and/or role assignments, which require **Owner** (or an *unconstrained* User Access Administrator) on the target subscriptions — specifically the rights to write custom role definitions and to assign the privileged `Role Based Access Control Administrator` role to the mock arm-helper principal.

> Implementation detail: because the topology framework requires every service group to reference a pipeline, the `DevCI.Privileged` root is a **no-op grouping pipeline** (`dev-infrastructure/dev-ci/privileged/pipeline.yaml`, empty step list). The identity service group (`E2ESubscriptionRBAC`) hangs off that root, and the grants service group (`E2ESubscriptionRBACGrants`) is nested as its child. That nesting matters: the topology runs a parent's steps before its child's, so the identities are created before the grants that look them up via `existing`. Operators never target these service groups directly — always use the entrypoint via the make target below.

The identity that runs the unattended `dev-ci` postsubmit (the `OpenShift Release Bot` service principal, app `38335e22-716a-4a21-bf20-15ab141823f0`) is deliberately **not** an Owner. It holds `Contributor` plus a *condition-constrained* `Role Based Access Control Administrator` / `User Access Administrator` whose Azure ABAC condition forbids assigning the `Owner`, `User Access Administrator`, and `Role Based Access Control Administrator` roles. That is exactly one of the assignments this entrypoint makes, and on some target subscriptions the bot also lacks `Microsoft.Authorization/roleDefinitions/write` — so the grants would fail if the postsubmit tried to apply them. It is likewise not meant to hold the Entra directory-object rights needed to create the bot apps/SPs or mint their secrets. The `DevCI.Privileged` entrypoint is therefore never wired into the unattended `DevCI.Unprivileged` graph and is run manually by a member of the OWNERS group (who has real Owner and the directory privileges to manage these identities) whenever a change touches the bot identities or these grants:

```bash
make dev-ci-privileged-local-run
```

Because the identity service group is the parent of the grants service group within this same entrypoint, a single privileged run creates/reconciles the CI bot identities first and then applies the grants (which look up the service principals via `existing`) — no separate bootstrap run of another entrypoint is required.

This split keeps the blast radius of the standing CI automation small: the postsubmit reconciles only the CI support layer that needs no Owner and no directory-object management, and the rare identity/credential and Owner-only changes are applied on demand.

## What It Does Not Manage

The current `dev-ci` topology intentionally does not own several adjacent pieces of CI:

- Prow jobs, ci-operator configuration, step-registry workflows, and Boskos inventory remain in `openshift/release`.
- The on-demand DEV RP footprint created during local E2E jobs is still provisioned by the release-side workflow, not by `topology-dev-ci.yaml`.
- Static consumer artifacts such as `dev-infrastructure/openshift-ci/msi-mock-pool.yaml`
  and `arm-helper-pool.yaml` are still generated separately.

For the runtime lease model itself, see [CI Identity Leasing](identity-leasing.md).

## The Current Mixed-Management Boundary

The DEV MSI mock and ARM helper service-principal pools used by local E2E jobs
are managed declaratively on the Azure side, with only the release-side consumer
artifacts left outside the rollout.

- The `Microsoft.Azure.ARO.HCP.DevCI.Privileged` entrypoint owns the pool end to end on the Azure side:
  - `dev-infrastructure/templates/mock-identity-apps.bicep` creates/updates both
    pooled Entra app families from `.ci.dev.mockIdentities.pool` and
    `.ci.dev.mockIdentities.armHelperPool`. It configures no auth on the apps.
  - The DEV `pin-mock-certs` and INT `pin-mock-certs-int` Shell steps run
    `tooling/entra-app-credentials` immediately after the corresponding app
    deployment. They create missing certificates and register each current
    Key Vault certificate as a pinned `keyCredential`; existing certificates and
    policies remain untouched. The DEV step uses separate CLI invocations for
    the named identities, MSI mock pool, and ARM helper pool.
    Templatize runs these steps with the invoking OWNERS member's Azure CLI
    credentials.
  - `dev-infrastructure/templates/mock-identity-rbac.bicep` resolves each
    principal's object ID via Microsoft Graph and applies the home- and
    E2E-subscription grants. Both pools receive DEV home-subscription grants so
    their members can be tested in personal development environments. Principal
    IDs are no longer stored in `config/config-dev-ci.yaml`. Because those grants
    require subscription Owner, they are applied on demand rather than by the
    postsubmit (see [The Privileged Entrypoint](#the-privileged-entrypoint)).
- What remains outside the rollout:
  - `make populate-msi-mock-pool` and `make populate-arm-helper-pool` perform live
    Entra lookups and write the static catalogs consumed by release-side jobs.
  - `openshift/release` still owns the Boskos inventories and lease contracts.

So the Entra objects, certificates, certificate pinning, and RBAC are
single-sourced in the topology; the remaining spread is limited to
release-side Boskos/catalog wiring.

## Long-Term Direction

The Azure-side intended state is now in place: a single privileged rollout owns
the app/SP, certificate, pinning, and RBAC lifecycle, driven by one source of
truth in `config/config-dev-ci.yaml`. The remaining work to close the loop:

- generate downstream consumer artifacts (the static pool catalog and the release-side Boskos inventory) from that same source instead of updating them separately

## Operator Entry Points

Useful local entry points for the current topology:

```bash
make dev-ci-local-run
make dev-ci-privileged-local-run
```

Use the first command for the standalone `dev-ci` postsubmit surface (shared network, `opstool` AKS, monitoring, gateway, cert-manager, CIHealth, quota) — it no longer touches the CI bot identities. Use the second — **an Owner-only, on-demand run performed by an OWNERS-group member** — when mock Entra applications, certificates, pinned credentials, or subscription-scoped RBAC need to be created or reconciled. Certificate rotation is a separate, explicitly disruptive CLI procedure.

## Where To Look

When you need to change or debug the standalone `dev-ci` topology, start here:

- `config/config-dev-ci.yaml`
- `topology-dev-ci.yaml`
- `dev-infrastructure/dev-ci/cluster/opstool-aks-pipeline.yaml`
- `dev-infrastructure/dev-ci/e2e-subscription-rbac/pipeline.yaml`
- `dev-infrastructure/dev-ci/e2e-subscription-rbac-grants/pipeline.yaml`
- `dev-infrastructure/configurations/mock-identity-apps.tmpl.bicepparam`
- `dev-infrastructure/configurations/mock-identity-rbac.tmpl.bicepparam`
- `dev-infrastructure/Makefile`
- `dev-infrastructure/openshift-ci/populate-mock-identity-pool.sh`
- [CI Identity Leasing](identity-leasing.md)
- [DEV CI Monitoring and Alert Response](dev-ci-monitoring.md)
- [Opstool CI Platform](opstool.md)

## See Also

- [CI Overview](README.md)
- [CI Execution](execution.md)
- [E2E Subscription Onboarding](e2e-subscription-onboarding.md)
- [CI Identity Leasing](identity-leasing.md)
- [CI Operations](operations.md)
