# kubernetes-manifests — agent guide

Production GitOps repo managing three Kubernetes clusters — Ottawa (primary:
media, databases, Rook-Ceph), Robbinsdale (home automation, Rook-Ceph), and
St. Petersburg (AI/ML: GPUs, KServe, Ray) — via Flux CD, with deep Tailscale
integration for cross-cluster networking and access.

## Ground rules

- **GitOps-first.** Never `helm install` / `kubectl apply` manifests directly;
  every durable Kubernetes object or configuration change must be committed in
  this repository and delivered by Flux. Do not use `kubectl create`, `apply`,
  `replace`, `edit`, or a write-capable API/MCP resource tool, even for a quick
  experiment. Read-only inspection is allowed. A narrowly scoped live
  `kubectl patch` is reserved for explicit incident mitigation or testing that
  the user requested; record the durable GitOps change immediately and remove
  any temporary object when the test ends. `kubectl apply` also cannot resolve
  Flux `${VARIABLE}` substitutions.
- **Verify with `tools/check.sh`** (or `tools/check.sh <cluster>`) — runs the
  CI render gate (`make test`), prints exactly one `✓ render OK: …` line on
  success and ~50 lines on failure. Do NOT run raw `make test` or `kustomize
  build`; they dump thousands of lines. The gate bootstraps the CI-pinned Flate
  release and renders all three clusters concurrently. If only one cluster
  changed, scope it (`tools/check.sh talos-ottawa`) to keep feedback fast.
  Flate intermittently wedges in a CPU spin instead of returning, so the gate
  kills the render's whole process group after `KMAN_CHECK_TIMEOUT` seconds
  (default 120) and exits 124. Exit 124 means retry, not a broken manifest — but
  only when the render was actually killed; a render that exits on its own is
  reported as a failure with its real diagnostics, never as a timeout.
  A full three-cluster render takes ~2 seconds. If it instead crawls and reports
  a scatter of unexplained failures, the cause is registry auth, not manifests:
  the gate exports a throwaway `DOCKER_CONFIG` precisely so chart fetches never
  shell out to `docker-credential-osxkeychain` and stall on macOS keychain
  prompts. It needs no registry credentials. Don't set `DOCKER_CONFIG` yourself
  unless you specifically want authenticated pulls.
- **Find before you read.** Use `tools/where.sh <pattern> <file>` (grep -n)
  to locate sections, then read narrow windows. Don't re-read large files.
  All `tools/*.sh` are anchored to the repo root, so they work from any cwd —
  no `cd` prefix needed. One-call lookups:
  - `tools/app.sh <name>` — where an app is defined + which clusters deploy it;
    `tools/app.sh --list` — full `app | base path | clusters` inventory.
  - `tools/refs.sh <name>` — every tracked file that references a name (for a
    decommission sweep); exits non-zero when nothing references it.
  - `tools/orphans.sh` — kustomization↔disk drift (listed-but-missing targets,
    unlisted/dead YAML). Exit 1 = discrepancies.
  - `tools/kc.sh <ot|rb|sp> …` — kubectl for a cluster (see Live cluster access).
- **Secrets are SOPS-encrypted** (`*.sops.yaml`, PGP key
  FAC8E7C3A2BC7DEE58A01C5928E1AB8AF0CF07A5). Never commit plaintext secrets,
  kubeconfigs, `talsecret.yaml`, or decrypted `*.dec` files.
- **Commit attribution must match the authenticated human user.** Before every
  commit, resolve the authenticated GitHub user and use an email already present
  in that user's commits in this repository (for `kbpersonal`, use
  `kbpersonal <kbpersonal@killinit.cc>`). Never copy an identity from repository
  history, documentation, another contributor, or a previous workspace. If the
  authenticated user cannot be determined unambiguously, ask the user to confirm
  their name and commit email, then stop before committing until they answer.
  No AI signatures, co-author trailers, or generated-with lines.
- You have local network access to pod/service/LB CIDRs over Tailscale —
  curl services directly, no port-forwarding needed.

## Layout

```
kubernetes/apps/
├── base/<ns>/<app>/         # real manifests, exactly once
│                            # (split <app>-<location>/ when config differs)
├── ottawa/<ns>/             # per-location overlays = thin pointer files
├── robbinsdale/<ns>/
└── stpetersburg/<ns>/
    ├── kustomization.yaml   # lists pointer files (+ namespace.yaml if owned)
    └── <app>.yaml           # Flux Kustomization CR, spec.path → base
clusters/                    # bootstrap + shared config ONLY (no workloads)
├── common/flux/repositories # Helm/OCI/Git sources
├── common/flux/vars         # common-settings / common-secrets (SOPS)
└── talos-<location>/        # per-cluster flux config + Talos bootstrap
tailscale/                   # policy.hujson (ACLs) + automation scripts
tools/                       # agent helper scripts (check.sh, where.sh, ...)
```

Reconcile chain: `clusters/talos-<location>/flux/config/cluster.yaml` defines
the `kubernetes-apps` Kustomization → `./kubernetes/apps/<location>`, which
injects SOPS decryption and the `substituteFrom` stack (common-settings/
common-secrets + cluster-settings/cluster-secrets) into every pointer.

**Deploy-to-some-clusters** = the pointer file exists only in those location
trees. An app in `base/` with no pointer anywhere is not deployed.

## Adding a new app

**Copy the worked example in `docs/reference/app-template.md`** — complete
base manifests, the pointer Kustomization, and a tailnet HTTPRoute with the
per-cluster `${CLUSTER_DOMAIN}` values already filled in. Reading it once beats
grepping the tree for the pointer shape, the `substituteFrom` stack, the domain
values, and an HTTPRoute example. The checklist below is the summary.

1. New Helm chart source? Add a HelmRepository under
   `clusters/common/flux/repositories/helm/<name>.yaml` and list it in that
   dir's `kustomization.yaml`.
2. Create `kubernetes/apps/base/<ns>/<app>/`:
   - `helmrelease.yaml` (or raw manifests)
   - `httproute.yaml` if exposing via Gateway API
   - `kustomization.yaml` — sets namespace, lists resources
   - `namespace.yaml` only if this app owns the namespace (keep
     `kustomize.toolkit.fluxcd.io/prune: disabled`)
3. Per target location, add the pointer
   `kubernetes/apps/<location>/<ns>/<app>.yaml` — a
   `kustomize.toolkit.fluxcd.io/v1` Kustomization with
   `spec.path: ./kubernetes/apps/base/<ns>/<app>[-<location>]`,
   `sourceRef.name: kubernetes-manifests`, plus any `dependsOn` /
   `postBuild.substitute`. The CR `metadata.name` is the app's identity.
4. List the pointer (and `namespace.yaml` if owned) in
   `kubernetes/apps/<location>/<ns>/kustomization.yaml`.
5. Verify with `tools/check.sh` before committing.

### Variable substitution

Every pointer inherits:

```yaml
postBuild:
  substituteFrom:
    - { kind: ConfigMap, name: common-settings }
    - { kind: Secret,    name: common-secrets }
    - { kind: ConfigMap, name: cluster-settings, optional: true }
    - { kind: Secret,    name: cluster-secrets,  optional: true }
```

## Critical gotchas

- **Talos control-plane patches are rolling operations.** A machine-config
  patch can restart static control-plane pods even when `talosctl` reports
  `Applied configuration without a reboot`. Never patch multiple control-plane
  nodes back-to-back. Apply to exactly one node, then verify that node's etcd
  member is healthy, its kube-apiserver is serving `/readyz`, and every
  Kubernetes node is `Ready` before touching the next control-plane node. Treat
  the node currently holding the control-plane VIP as the final node and repeat
  the readiness checks after it returns.
- **Rook-Ceph upgrades are per cluster, Rook before Ceph.** Renovate groups
  storage the same way it groups Talos: `rook-<location>` and `ceph-<location>`,
  one PR each (`.renovate/groups.json`). Merge a cluster's Rook operator PR
  first and let it settle — a new operator drives the older Ceph release, not
  the reverse — then its Ceph image PR, and never leave two clusters' storage
  rollouts in flight at once. `CephCluster.spec.cephVersion.image` and the
  `rook-ceph-tools` image must move together; `tools/check-versions.sh` fails
  the build when they drift, with `# version-sync: ignore` as the escape hatch.
- **`$` mangling:** Flux envsubst (drone/envsubst) eats bare `$` (bcrypt
  hashes, regex, shell). Any value containing `$` must live in a
  Secret/ConfigMap and be injected via `${VAR}`. Substitution is single-pass:
  `$` inside a replacement value is NOT re-processed. In raw manifests that
  Flux renders, write `$${...}` to emit a literal `${...}`.
- **Gateway hostnames:** check listener hostnames before adding an HTTPRoute:
  `kubectl get gateway <name> -n home -o jsonpath='{.spec.listeners[*].hostname}'`.
  Common gateways (`ts`, `private`, `public`) live in namespace `home` and
  accept `*.${CLUSTER_DOMAIN}` (e.g. `*.killinit.cc` for Ottawa); `public`
  (and now `private`) also accept `*.${COMMON_DOMAIN}` (`*.keiretsu.top`).
  `NoMatchingListenerHostname` in HTTPRoute status = no listener matches.
- **DNS:** `${CLUSTER_DOMAIN}` hostnames get DNS automatically (external-dns
  watches HTTPRoutes). `${COMMON_DOMAIN}` hostnames do NOT — you MUST add a
  CNAME entry in `kubernetes/apps/base/k8gb/k8gb-common/config/cnames.yaml`
  (Ottawa-only: target `"ottawa.${COMMON_DOMAIN}"`; multi-cluster/k8gb:
  `"<name>.cdn.${COMMON_DOMAIN}"`; both `cloudflare-proxied: "false"`).
  Without it the route shows Accepted=True but never resolves.
- **Tailnet DNS from pods:** do not publish Tailscale CGNAT (`100.64.0.0/10`)
  in public DNS and do not assume a new `*.keiretsu.ts.net` device name is
  automatically present in pod DNS. For every tailnet target, define an
  `ExternalName` Service annotated with `tailscale.com/tailnet-fqdn` and
  `tailscale.com/proxy-group: common-egress`. The Tailscale `DNSConfig`
  nameserver then publishes that original `.ts.net` name as the egress proxy's
  cluster IP. Verify the Service condition `TailscaleEgressSvcReady=True` and
  resolve the `.ts.net` name from a pod before configuring clients. See
  `docs/reference/tailscale-integration.md`.
- **Embedded workspace Tailscale:** maintainer workspaces run Tailscale in
  kernel TUN mode inside their single Kata container. Do not reintroduce an
  external Tailscale gateway sidecar or site-CIDR policy carving as a
  workaround for asymmetric Pod replies. The workspace entrypoint marks
  connections that arrive through the CNI and routes their replies through the
  main table while leaving new accepted-route traffic on `tailscale0`. Keep
  HuJSON `grants` and `via` rules focused on authorization and router
  eligibility; they do not
  control route injection.
- **Bhaiya sandbox runtimes:** all user-executed workspaces and shared-browser
  sessions use `kata-workspace`. Embedded-Tailscale workspaces use the separate
  `kata-tailscale` RuntimeClass because it alone receives the Pod Security
  exemption and TUN capabilities. Do not move ordinary sandboxes back to
  gVisor or grant them the Tailscale exemption. Bhaiya's destination-side
  Cilium policy must keep its Remote-Email-trusting UI listener inaccessible
  from workspace and subnet-router traffic; source workspace policy cannot see
  final destinations carried inside accepted Tailscale routes.
- **Adding Kata workers:** give each new Talos machine the
  `siderolabs/kata-containers` system extension and the node label
  `runtime.keiretsu.top/kata: "true"` in `talconfig.yaml`. Both
  `kata-workspace` and `kata-tailscale` select that label, so capacity expands
  without changing Bhaiya or carving per-node scheduling rules. For an
  existing node, install and boot the extension image before adding the label;
  a staged machine config can make labels effective on a plain reboot without
  installing the new OS image. Remove the RuntimeClass selectors and labels
  only after `talosctl get extensions` confirms Kata on every eligible node.
- **SOPS:** run `sops` from the directory whose `.sops.yaml` carries the
  creation rules. Edit with `sops <file>.sops.yaml`.

## Live cluster access

- **Use `tools/kc.sh <cluster> <kubectl args…>`** — it sets the one canonical
  `KUBECONFIG` and the right `--context`, execs kubectl (args + exit code pass
  through). Do not pass the display aliases `ottawa`, `robbinsdale`, or
  `stpetersburg` directly to `kubectl --context`; they are helper aliases, not
  necessarily kubeconfig context names. Don't re-export `KUBECONFIG`, guess a
  context from another tool's display output, or `cd` first. Invoke the helper
  by a path that resolves from the current workspace: use `tools/kc.sh` while
  in this repo, or `../kubernetes-manifests/tools/kc.sh` from the sibling
  `bhaiya` repo.

  ```bash
  tools/kc.sh ot -n media get pods    # ottawa
  tools/kc.sh rb get ns               # robbinsdale
  tools/kc.sh sp get nodes            # stpetersburg
  ../kubernetes-manifests/tools/kc.sh ot get vpa -A  # from ../bhaiya
  ```

  | alias            | kubeconfig            | context                                   |
  |------------------|-----------------------|-------------------------------------------|
  | `ot`/`ottawa`      | `<repo>/.kube/config` | `ottawa-k8s-operator.keiretsu.ts.net`       |
  | `rb`/`robbinsdale` | `<repo>/.kube/config` | `robbinsdale-k8s-operator.keiretsu.ts.net`  |
  | `sp`/`stpetersburg`| `<repo>/.kube/config` | `stpetersburg-k8s-operator.keiretsu.ts.net` |

- **Compact pod triage:** `tools/ktriage.sh <ot|rb|sp> <ns> <pod>` prints one
  pod's bounded read-only summary — phase/node/IP, container + init states,
  not-True conditions, latest 10 events, a 20-line log tail (plus previous logs
  for restarted containers) — instead of a 200–800-line `get pod -o yaml`. It
  routes every call through `kc.sh` and issues only `get`/`logs`. Exit 4 = pod
  read OK but a later section was unavailable (marked inline).
- The one canonical kubeconfig is `.kube/config` in the repo root (all three
  clusters). It is tracked on purpose and holds **no credentials** — three
  operator `server:` URLs and a `token: unused` placeholder, because the
  Tailscale auth proxy authorizes by tailnet identity. Never add a client cert,
  key, or real token to it, and never track anything else under `.kube/` (the
  discovery cache stays ignored). Container environments symlink/copy it to
  `/workspace/kubernetes-manifests/.kube/config`; do not guess that path on the
  host — prefer `tools/kc.sh`.
- Pod/service CIDRs per site: Robbinsdale 10.1/10.0/10.50, Ottawa
  10.3/10.2/10.169, St. Petersburg 10.5/10.4/10.73 (pods/services/LB, /16s).

## Commands

```bash
tools/check.sh [cluster]   # CI render gate — the sole verify command
                           # (one ✓ line on success; accepts ot/rb/sp aliases;
                           #  scope by cluster if only one changed)
tools/check-versions.sh    # talconfig↔tuppr and CephCluster↔toolbox version sync
tools/flate.sh <args…>     # flate at the CI-pinned version (bootstraps if needed)
tools/tests/run.sh         # offline self-tests for the tools/ helpers
tools/app.sh <name> | --list   # locate an app / full deploy inventory
tools/refs.sh <name>       # every tracked file referencing <name>
tools/orphans.sh           # kustomization ↔ disk drift
tools/kc.sh <ot|rb|sp> …   # kubectl for a cluster (root kubeconfig + context)
tools/ktriage.sh <ot|rb|sp> <ns> <pod>   # bounded read-only pod triage (vs -o yaml)
make diff                  # rendered diff vs origin/main
flux get all -A            # Flux status (live cluster)
flux reconcile kustomization <app> -n flux-system
```

## Deeper reference

- `kubernetes/README.md` — authoritative layout description
- `docs/reference/talos.md` — Talos/talhelper bootstrap structure
- `docs/reference/tailscale-integration.md` — operator resources, ACL policy,
  scripts, CI workflows
- `docs/reference/tsdb.md` — historical tsdb connector notes
- `docs/prompt-notes.md` — prompt patterns that worked/failed for build agents

## Command output hygiene

- Never use `--watch`, `watch`, or sleep-based polling loops for CI, PRs, or long-running jobs. Check once with minimal output (e.g. `gh pr checks <n> | tail -3`) and do other work between checks.
- Redirect noisy output (test suites, CI failure logs, benchmarks) to a file and inspect selectively: `cargo test ... > /tmp/out.log 2>&1; tail -30 /tmp/out.log`, then grep the file for details.
- Batch related shell steps into one command with `&&` instead of separate invocations.
- Delegate inherently noisy multi-step work (benchmark matrices, CI-log triage, retry loops) to a subagent and only bring back the summary.
