<div align="center">

<img src="https://raw.githubusercontent.com/kubernetes/kubernetes/master/logo/logo.png" width="144px" height="144px"/>

### Keiretsu — Multi-Cluster Kubernetes Infrastructure

_Managed with Flux, Tailscale, and GitHub Actions_

</div>

<div align="center">

#### Ottawa

[![Talos](https://img.shields.io/endpoint?url=https%3A%2F%2Fkromgo.killinit.cc%2Fbadges%2Ftalos_version%3Fformat%3Dshields&style=for-the-badge&logo=talos&logoColor=white&color=blue&label=%20)](https://talos.dev)&nbsp;&nbsp;
[![Kubernetes](https://img.shields.io/endpoint?url=https%3A%2F%2Fkromgo.killinit.cc%2Fbadges%2Fkubernetes_version%3Fformat%3Dshields&style=for-the-badge&logo=kubernetes&logoColor=white&color=blue&label=%20)](https://kubernetes.io)&nbsp;&nbsp;
[![Flux Version](https://img.shields.io/endpoint?url=https%3A%2F%2Fkromgo.killinit.cc%2Fbadges%2Fflux_version%3Fformat%3Dshields&style=for-the-badge&logo=flux&logoColor=white&color=blue&label=%20)](https://fluxcd.io)&nbsp;&nbsp;
[![Tailscale](https://img.shields.io/endpoint?url=https%3A%2F%2Fkromgo.killinit.cc%2Fbadges%2Ftailscale_operator_version%3Fformat%3Dshields&style=for-the-badge&logo=tailscale&logoColor=white&color=black&label=%20)](https://tailscale.com/kb/1236/kubernetes-operator)

[![Kubernetes API](https://img.shields.io/endpoint?url=https%3A%2F%2Fstatus.killinit.cc%2Fapi%2Fv1%2Fendpoints%2Fkubernetes_kubernetes-api%2Fhealth%2Fbadge.shields&style=for-the-badge&logo=kubernetes&logoColor=white&label=Kubernetes+API)](https://status.killinit.cc)&nbsp;&nbsp;
[![Garage](https://img.shields.io/endpoint?url=https%3A%2F%2Fstatus.killinit.cc%2Fapi%2Fv1%2Fendpoints%2Fstorage_garage%2Fhealth%2Fbadge.shields&style=for-the-badge&logo=amazons3&logoColor=white&label=Garage)](https://status.killinit.cc)&nbsp;&nbsp;

[![Age](https://img.shields.io/endpoint?url=https%3A%2F%2Fkromgo.killinit.cc%2Fbadges%2Fcluster_age_days%3Fformat%3Dshields&style=flat-square&label=Age)](https://github.com/home-operations/kromgo)&nbsp;&nbsp;
[![Uptime](https://img.shields.io/endpoint?url=https%3A%2F%2Fkromgo.killinit.cc%2Fbadges%2Fcluster_uptime_days%3Fformat%3Dshields&style=flat-square&label=Uptime)](https://github.com/home-operations/kromgo)&nbsp;&nbsp;
[![Nodes](https://img.shields.io/endpoint?url=https%3A%2F%2Fkromgo.killinit.cc%2Fbadges%2Fcluster_node_count%3Fformat%3Dshields&style=flat-square&label=Nodes)](https://github.com/home-operations/kromgo)&nbsp;&nbsp;
[![Pods](https://img.shields.io/endpoint?url=https%3A%2F%2Fkromgo.killinit.cc%2Fbadges%2Fcluster_pod_count%3Fformat%3Dshields&style=flat-square&label=Pods)](https://github.com/home-operations/kromgo)&nbsp;&nbsp;
[![CPU](https://img.shields.io/endpoint?url=https%3A%2F%2Fkromgo.killinit.cc%2Fbadges%2Fcluster_cpu_usage%3Fformat%3Dshields&style=flat-square&label=CPU)](https://github.com/home-operations/kromgo)&nbsp;&nbsp;
[![Memory](https://img.shields.io/endpoint?url=https%3A%2F%2Fkromgo.killinit.cc%2Fbadges%2Fcluster_memory_usage%3Fformat%3Dshields&style=flat-square&label=Memory)](https://github.com/home-operations/kromgo)&nbsp;&nbsp;
[![Alerts](https://img.shields.io/endpoint?url=https%3A%2F%2Fkromgo.killinit.cc%2Fbadges%2Fcluster_alert_count%3Fformat%3Dshields&style=flat-square&label=Alerts)](https://github.com/home-operations/kromgo)

</div>

<div align="center">

#### Robbinsdale

[![Talos](https://img.shields.io/endpoint?url=https%3A%2F%2Fkromgo.lukehouge.com%2Fbadges%2Ftalos_version%3Fformat%3Dshields&style=for-the-badge&logo=talos&logoColor=white&color=blue&label=%20)](https://talos.dev)&nbsp;&nbsp;
[![Kubernetes](https://img.shields.io/endpoint?url=https%3A%2F%2Fkromgo.lukehouge.com%2Fbadges%2Fkubernetes_version%3Fformat%3Dshields&style=for-the-badge&logo=kubernetes&logoColor=white&color=blue&label=%20)](https://kubernetes.io)&nbsp;&nbsp;
[![Flux Version](https://img.shields.io/endpoint?url=https%3A%2F%2Fkromgo.lukehouge.com%2Fbadges%2Fflux_version%3Fformat%3Dshields&style=for-the-badge&logo=flux&logoColor=white&color=blue&label=%20)](https://fluxcd.io)&nbsp;&nbsp;
[![Tailscale](https://img.shields.io/endpoint?url=https%3A%2F%2Fkromgo.lukehouge.com%2Fbadges%2Ftailscale_operator_version%3Fformat%3Dshields&style=for-the-badge&logo=tailscale&logoColor=white&color=black&label=%20)](https://tailscale.com/kb/1236/kubernetes-operator)

[![Kubernetes API](https://img.shields.io/endpoint?url=https%3A%2F%2Fstatus.lukehouge.com%2Fapi%2Fv1%2Fendpoints%2Fkubernetes_kubernetes-api%2Fhealth%2Fbadge.shields&style=for-the-badge&logo=kubernetes&logoColor=white&label=Kubernetes+API)](https://status.lukehouge.com)&nbsp;&nbsp;
[![Garage](https://img.shields.io/endpoint?url=https%3A%2F%2Fstatus.lukehouge.com%2Fapi%2Fv1%2Fendpoints%2Fstorage_garage-local-%28robbinsdale%29%2Fhealth%2Fbadge.shields&style=for-the-badge&logo=amazons3&logoColor=white&label=Garage)](https://status.lukehouge.com)&nbsp;&nbsp;

[![Age](https://img.shields.io/endpoint?url=https%3A%2F%2Fkromgo.lukehouge.com%2Fbadges%2Fcluster_age_days%3Fformat%3Dshields&style=flat-square&label=Age)](https://github.com/home-operations/kromgo)&nbsp;&nbsp;
[![Uptime](https://img.shields.io/endpoint?url=https%3A%2F%2Fkromgo.lukehouge.com%2Fbadges%2Fcluster_uptime_days%3Fformat%3Dshields&style=flat-square&label=Uptime)](https://github.com/home-operations/kromgo)&nbsp;&nbsp;
[![Nodes](https://img.shields.io/endpoint?url=https%3A%2F%2Fkromgo.lukehouge.com%2Fbadges%2Fcluster_node_count%3Fformat%3Dshields&style=flat-square&label=Nodes)](https://github.com/home-operations/kromgo)&nbsp;&nbsp;
[![Pods](https://img.shields.io/endpoint?url=https%3A%2F%2Fkromgo.lukehouge.com%2Fbadges%2Fcluster_pod_count%3Fformat%3Dshields&style=flat-square&label=Pods)](https://github.com/home-operations/kromgo)&nbsp;&nbsp;
[![CPU](https://img.shields.io/endpoint?url=https%3A%2F%2Fkromgo.lukehouge.com%2Fbadges%2Fcluster_cpu_usage%3Fformat%3Dshields&style=flat-square&label=CPU)](https://github.com/home-operations/kromgo)&nbsp;&nbsp;
[![Memory](https://img.shields.io/endpoint?url=https%3A%2F%2Fkromgo.lukehouge.com%2Fbadges%2Fcluster_memory_usage%3Fformat%3Dshields&style=flat-square&label=Memory)](https://github.com/home-operations/kromgo)&nbsp;&nbsp;
[![Alerts](https://img.shields.io/endpoint?url=https%3A%2F%2Fkromgo.lukehouge.com%2Fbadges%2Fcluster_alert_count%3Fformat%3Dshields&style=flat-square&label=Alerts)](https://github.com/home-operations/kromgo)

</div>

<div align="center">

#### St. Petersburg

[![Talos](https://img.shields.io/endpoint?url=https%3A%2F%2Fkromgo.rajsingh.info%2Fbadges%2Ftalos_version%3Fformat%3Dshields&style=for-the-badge&logo=talos&logoColor=white&color=blue&label=%20)](https://talos.dev)&nbsp;&nbsp;
[![Kubernetes](https://img.shields.io/endpoint?url=https%3A%2F%2Fkromgo.rajsingh.info%2Fbadges%2Fkubernetes_version%3Fformat%3Dshields&style=for-the-badge&logo=kubernetes&logoColor=white&color=blue&label=%20)](https://kubernetes.io)&nbsp;&nbsp;
[![Flux Version](https://img.shields.io/endpoint?url=https%3A%2F%2Fkromgo.rajsingh.info%2Fbadges%2Fflux_version%3Fformat%3Dshields&style=for-the-badge&logo=flux&logoColor=white&color=blue&label=%20)](https://fluxcd.io)&nbsp;&nbsp;
[![Tailscale](https://img.shields.io/endpoint?url=https%3A%2F%2Fkromgo.rajsingh.info%2Fbadges%2Ftailscale_operator_version%3Fformat%3Dshields&style=for-the-badge&logo=tailscale&logoColor=white&color=black&label=%20)](https://tailscale.com/kb/1236/kubernetes-operator)

[![Kubernetes API](https://img.shields.io/endpoint?url=https%3A%2F%2Fstatus.rajsingh.info%2Fapi%2Fv1%2Fendpoints%2Fkubernetes_kubernetes-api%2Fhealth%2Fbadge.shields&style=for-the-badge&logo=kubernetes&logoColor=white&label=Kubernetes+API)](https://status.rajsingh.info)&nbsp;&nbsp;
[![Garage](https://img.shields.io/endpoint?url=https%3A%2F%2Fstatus.rajsingh.info%2Fapi%2Fv1%2Fendpoints%2Fstorage_garage-local-%28stpetersburg%29%2Fhealth%2Fbadge.shields&style=for-the-badge&logo=amazons3&logoColor=white&label=Garage)](https://status.rajsingh.info)&nbsp;&nbsp;

[![Age](https://img.shields.io/endpoint?url=https%3A%2F%2Fkromgo.rajsingh.info%2Fbadges%2Fcluster_age_days%3Fformat%3Dshields&style=flat-square&label=Age)](https://github.com/home-operations/kromgo)&nbsp;&nbsp;
[![Uptime](https://img.shields.io/endpoint?url=https%3A%2F%2Fkromgo.rajsingh.info%2Fbadges%2Fcluster_uptime_days%3Fformat%3Dshields&style=flat-square&label=Uptime)](https://github.com/home-operations/kromgo)&nbsp;&nbsp;
[![Nodes](https://img.shields.io/endpoint?url=https%3A%2F%2Fkromgo.rajsingh.info%2Fbadges%2Fcluster_node_count%3Fformat%3Dshields&style=flat-square&label=Nodes)](https://github.com/home-operations/kromgo)&nbsp;&nbsp;
[![Pods](https://img.shields.io/endpoint?url=https%3A%2F%2Fkromgo.rajsingh.info%2Fbadges%2Fcluster_pod_count%3Fformat%3Dshields&style=flat-square&label=Pods)](https://github.com/home-operations/kromgo)&nbsp;&nbsp;
[![CPU](https://img.shields.io/endpoint?url=https%3A%2F%2Fkromgo.rajsingh.info%2Fbadges%2Fcluster_cpu_usage%3Fformat%3Dshields&style=flat-square&label=CPU)](https://github.com/home-operations/kromgo)&nbsp;&nbsp;
[![Memory](https://img.shields.io/endpoint?url=https%3A%2F%2Fkromgo.rajsingh.info%2Fbadges%2Fcluster_memory_usage%3Fformat%3Dshields&style=flat-square&label=Memory)](https://github.com/home-operations/kromgo)&nbsp;&nbsp;
[![Alerts](https://img.shields.io/endpoint?url=https%3A%2F%2Fkromgo.rajsingh.info%2Fbadges%2Fcluster_alert_count%3Fformat%3Dshields&style=flat-square&label=Alerts)](https://github.com/home-operations/kromgo)

</div>

---

Multi-cluster Kubernetes infrastructure managed with FluxCD GitOps. The three
Talos Linux clusters are connected by a Tailscale mesh and share the same
repository, platform conventions, and observability stack.

## Clusters

| Cluster | Site and nodes | Primary role | Storage | <code>${CLUSTER_DOMAIN}</code> |
|---------|----------------|--------------|---------|-------------------------|
| <code>talos-ottawa</code> | Ontario, Canada; 3 control planes (<code>rei</code>, <code>asuka</code>, <code>kaji</code>) plus worker <code>shiro</code> | Primary media, databases, and general workloads | Rook-Ceph, SMB, and local Garage pools | <code>killinit.cc</code> |
| <code>talos-robbinsdale</code> | Minnesota, US; 3 control planes (<code>tank</code>, <code>stone</code>, <code>titan</code>) and no workers | Home automation, media, and general workloads | Rook-Ceph, SMB, and local Garage pools | <code>lukehouge.com</code> |
| <code>talos-stpetersburg</code> | Florida, US; control plane <code>orin-0</code> plus GPU workers <code>spark-0</code> and <code>spark-1</code> | AI/ML inference and GPU workloads | Local-path and local Garage pools | <code>rajsingh.info</code> |

The shared domain is <code>keiretsu.top</code>. Cluster-specific Talos
versions, Kubernetes versions, node schematics, VIPs, and CIDRs are maintained
in each cluster's <code>bootstrap/talos/talconfig.yaml</code> and Flux settings.

## Directory Structure

The app tree is [<code>kubernetes/</code>](kubernetes/README.md): configuration
lives once in <code>base/</code>, and clusters opt in with thin pointer files.
The location of a pointer is the deployment decision: an app is deployed only
to the clusters whose location tree contains that pointer.

~~~text
├── kubernetes/
│   ├── apps/base/<namespace>/<app>/ # real manifests, exactly once
│   ├── apps/ottawa/<namespace>/     # Ottawa Flux Kustomization pointers
│   ├── apps/robbinsdale/<namespace>/
│   ├── apps/stpetersburg/<namespace>/
│   └── components/                  # reusable Kustomize components
├── clusters/
│   ├── common/
│   │   ├── bootstrap/flux/           # one-time Flux install bundle
│   │   └── flux/                     # shared sources and settings
│   ├── talos-ottawa/
│   │   ├── bootstrap/talos/          # talhelper config and patches
│   │   ├── flux/                     # cluster Flux entrypoint and variables
│   │   ├── jobs/                     # site-specific diagnostic jobs
│   │   └── scripts/                  # site operations
│   ├── talos-robbinsdale/            # Talos, Flux, jobs, and site config
│   └── talos-stpetersburg/           # Talos, Flux, jobs, and site config
├── tailscale/                        # HuJSON policy and automation scripts
├── tools/                            # render gate and read-only helpers
├── docs/                             # references, ADRs, and runbooks
└── Makefile                          # rendered diff helper
~~~

Legacy or supplemental content remains under <code>clusters/*/apps</code> while
the application migration is completed. Check references with
<code>tools/refs.sh &lt;name&gt;</code> before changing or removing one of those
files. New apps belong under <code>kubernetes/apps/base/</code> with location
pointers.

## GitOps and Flux Architecture

Each cluster has a GitRepository and three top-level Flux Kustomizations in
<code>clusters/talos-&lt;location&gt;/flux/config/cluster.yaml</code>:

- <code>cluster</code> reconciles the cluster's Flux configuration and variables.
- <code>common-cluster</code> reconciles shared sources and variables from
  <code>clusters/common/flux/</code>.
- <code>kubernetes-apps</code> reconciles
  <code>kubernetes/apps/&lt;location&gt;/</code>, which lists the location's
  namespaces and app pointers.

The app Kustomization enables SOPS decryption and supplies this substitution
stack to child pointers:

~~~text
common-settings
common-secrets
cluster-settings
cluster-secrets
cluster-user-settings       # optional
cluster-user-secrets        # optional
garage-keiretsu-bucket      # optional
~~~

Child Kustomizations can add dependencies, waits, timeouts, and their own
post-build substitution. Flux follows the <code>main</code> branch of this
repository; changes become durable only after they are committed and
reconciled.

### SOPS Secret Encryption

Secret material is encrypted with SOPS/PGP using key
<code>FAC8E7C3A2BC7DEE58A01C5928E1AB8AF0CF07A5</code>. Creation rules live
beside each tree in <code>.sops.yaml</code> files. Never commit plaintext
secrets, kubeconfigs, <code>talsecret.yaml</code>, private keys, or decrypted
<code>*.dec</code> files.

~~~yaml
# representative creation rule
creation_rules:
  - path_regex: .*.yaml$
    encrypted_regex: ^(data|stringData)$
    pgp: FAC8E7C3A2BC7DEE58A01C5928E1AB8AF0CF07A5
~~~

### Tailscale Integration

The official Tailscale Kubernetes operator is deployed through
<code>kubernetes/apps/base/tailscale/operator</code>. Its companion resources
and workloads are split between <code>tailscale/resources</code> and
<code>tailscale-system/tailscale-system-app</code>. The deployment includes:

- subnet-router Connectors for LAN, service, pod, and load-balancer CIDRs;
- per-location PeerRelay and ProxyGroups for common ingress/egress;
- ProxyClasses such as <code>common</code>,
  <code>common-accept-routes</code>, and <code>common-userspace</code>;
- DNSConfig for pod-side tailnet name resolution;
- cross-cluster API and service egress;
- SSH recording and the custom CSI provider.

See [the Tailscale integration reference](docs/reference/tailscale-integration.md)
for the policy, operator resources, egress contract, and CI workflows.

## Platform and Workload Inventory

The exact deployment inventory is derived from location pointers. Use
<code>tools/app.sh --list</code> for the current app-to-base-to-cluster mapping
rather than maintaining a second exhaustive list in this README.

### Shared platform

- Cilium CNI, Hubble, CoreDNS customization, Envoy Gateway, and Gateway API
  routes.
- Flux Operator/Instance, cert-manager, External Secrets, Node Feature
  Discovery, KRO, VPA, snapshot-controller, and local-path storage.
- Tailscale operator/resources/system, Cloudflare ExternalDNS, K8GB, and
  Open Cluster Management agents.
- Agent Sandbox, gVisor, Spegel, GitHub Actions Runner Controller, and
  cluster-specific Kata runtimes where selected by a location pointer.

### Observability

- kube-prometheus-stack, Grafana dashboards/plugins, Mimir, Victoria Logs,
  Fluent Bit, Gatus, Kromgo, blackbox-exporter, smartctl-exporter, and UniFi
  Unpoller.

### Data and storage

- Rook-Ceph is deployed in Ottawa and Robbinsdale.
- Garage provides federated S3-compatible storage across all three clusters,
  with per-node local pools and gateway services.
- CloudNativePG, Dragonfly, CSI SMB, Velero, and Zot provide database,
  backup, file-storage, and registry capabilities.

### Location workloads

- **Ottawa:** media, Immich, Forgejo, Woodpecker, Hermes, agents, Buzz,
  Firecrawl, CLIProxy, TeslaMate, SearXNG, Bhaiya integration, Monz,
  Headlamp, OpenCost, the GPU DRA driver, Kata, and LAN services.
- **Robbinsdale:** media, Immich, Speedtest, and Strimzi/Kafka workloads,
  alongside the shared home and storage platform.
- **St. Petersburg:** DeepSeek/VLLM inference on the DGX Spark workers,
  NVIDIA GPU Operator, LeaderWorkerSet, RDMA device plugin, Home Assistant,
  and the local Garage storage tier.

## Prerequisites

Install the tools through <code>mise</code> from the cluster directory:

~~~bash
brew install mise gpg
cd clusters/talos-<location>
mise install
~~~

The per-cluster <code>.mise.toml</code> installs Task, talhelper, talosctl,
SOPS, kubectl, and Helmfile. Install the Flux CLI separately if you need the
live Flux commands below. Import and trust the repository's SOPS key:

~~~bash
gpg --import /path/to/sops.asc
echo "FAC8E7C3A2BC7DEE58A01C5928E1AB8AF0CF07A5:6:" | gpg --import-ownertrust
gpg --list-secret-keys | grep FAC8E7C3A2BC7DEE58A01C5928E1AB8AF0CF07A5
~~~

## Bootstrap a Talos Cluster

The detailed Talos reference is
[<code>docs/reference/talos.md</code>](docs/reference/talos.md). Ottawa and
St. Petersburg also have hardware-specific guides in their Talos bootstrap
directories.

### Fresh-install sequence

~~~bash
cd clusters/talos-<location>
mise install

# Fresh installations only: create new cluster CA/etcd/node secrets.
mise run init

# Generate clusterconfig/ from talconfig.yaml and the encrypted secret.
mise run genconfig

# Nodes must be in Talos maintenance mode for the initial apply.
mise run apply-insecure
mise run bootstrap
mise run kubeconfig
mise run health
~~~

For an already-installed cluster, use <code>mise run apply</code> for a normal
configuration change or <code>mise run apply-staged</code> when the change
should wait for the next reboot. Generated <code>clusterconfig/</code> files
and kubeconfigs are ignored and must not be committed.

St. Petersburg has two installation images: run <code>mise run iso-spark</code>
for the DGX Spark workers and <code>mise run iso-orin</code> for the Jetson
control plane. Its maintenance-mode task is
<code>mise run apply-orin-insecure &lt;orin-ip&gt;</code> when the Orin address
cannot be resolved yet.

The one-time Flux controller/CRD bundle is kept at
<code>clusters/common/bootstrap/flux/</code>; it is explicitly not reconciled
by Flux. After the initial Flux bootstrap, the committed file
<code>clusters/talos-&lt;location&gt;/flux/config/cluster.yaml</code> creates the
GitRepository and the three reconciliation layers described above. Cilium,
gateways, and workloads are then installed by Flux. Do not
<code>helm install</code>, <code>kubectl apply</code>, or
<code>kubectl create</code> durable cluster resources manually.

## Adding a New Application

Read the complete worked example in
[<code>docs/reference/app-template.md</code>](docs/reference/app-template.md),
and the tree/migration notes in
[<code>kubernetes/README.md</code>](kubernetes/README.md). The short version is:

### 1. Create the base

~~~bash
mkdir -p kubernetes/apps/base/<namespace>/<app>
~~~

Add a <code>kustomization.yaml</code> and the app's raw manifests or
HelmRelease. Add a HelmRepository under
<code>clusters/common/flux/repositories/helm/</code> first when a new chart
source is needed, and list it in that directory's Kustomization.

### 2. Add one pointer per target cluster

<code>kubernetes/apps/&lt;location&gt;/&lt;namespace&gt;/&lt;app&gt;.yaml</code> is
the deployment opt-in. The pointer's Flux Kustomization should retain the app
identity in <code>metadata.name</code>, point at the base, and inherit the
substitution stack:

~~~yaml
apiVersion: kustomize.toolkit.fluxcd.io/v1
kind: Kustomization
metadata:
  name: &app <app>
  namespace: flux-system
spec:
  targetNamespace: <namespace>
  commonMetadata:
    labels:
      app.kubernetes.io/name: *app
  path: ./kubernetes/apps/base/<namespace>/<app>
  prune: true
  sourceRef:
    kind: GitRepository
    name: kubernetes-manifests
  interval: 30m
  retryInterval: 1m
  timeout: 5m
  postBuild:
    substituteFrom:
      - { kind: ConfigMap, name: common-settings }
      - { kind: Secret, name: common-secrets }
      - { kind: ConfigMap, name: cluster-settings, optional: true }
      - { kind: Secret, name: cluster-secrets, optional: true }
~~~

List the pointer in
<code>kubernetes/apps/&lt;location&gt;/&lt;namespace&gt;/kustomization.yaml</code>.
If the app owns a new namespace, list its <code>namespace.yaml</code> there and
retain <code>kustomize.toolkit.fluxcd.io/prune: disabled</code> on the
namespace. Finally list the namespace directory in the location root
Kustomization.

### 3. Ingress

Put an HTTPRoute beside the app when it needs HTTP access. The
<code>ts</code>, <code>private</code>, and <code>public</code> Gateways live in
namespace <code>home</code>. Check listener hostnames before choosing a parent:

~~~bash
tools/kc.sh ot -n home get gateway ts -o jsonpath='{.spec.listeners[*].hostname}'
~~~

Use <code>&lt;app&gt;.${CLUSTER_DOMAIN}</code> for a location domain;
ExternalDNS watches those HTTPRoutes and creates the record.
<code>&lt;app&gt;.${COMMON_DOMAIN}</code> names are not created automatically:
add a DNS-only CNAME to
<code>kubernetes/apps/base/k8gb/k8gb-common/config/cnames.yaml</code>, targeting
<code>ottawa.${COMMON_DOMAIN}</code> for an Ottawa-only route or the appropriate
<code>&lt;name&gt;.cdn.${COMMON_DOMAIN}</code> GSLB name for a multi-cluster route.

### 4. Tailnet targets and auth

For every tailnet hostname consumed by a pod, create an ExternalName Service
with <code>tailscale.com/tailnet-fqdn</code> and
<code>tailscale.com/proxy-group: common-egress</code>. Do not publish Tailscale
CGNAT addresses in public DNS; see the
[Tailscale reference](docs/reference/tailscale-integration.md) for the
readiness and pod-DNS checks.

Tinyauth provides Google authentication for protected web routes. Authorization
is route-local through an Envoy Gateway SecurityPolicy using the identity
headers injected by Tinyauth. The primary instance serves
<code>keiretsu.top</code>; Ottawa's <code>tinyauth-killinit</code> instance
serves <code>killinit.cc</code>. Internal tailnet-only routes can rely on
tailnet policy instead.

### 5. Validate

Run the bounded render gate before committing:

~~~bash
tools/check.sh <location>  # talos-ottawa, talos-robbinsdale, or talos-stpetersburg
~~~

Use <code>tools/check.sh</code> for all three clusters when a shared or
multi-location change is involved. Do not substitute raw
<code>make test</code> or <code>kustomize build</code>; the helper uses the
CI-pinned Flate release and controls noisy output.

## Common Operations

### Flux commands

~~~bash
flux get ks -A
flux get hr -A

flux reconcile kustomization kubernetes-apps -n flux-system
flux reconcile source git kubernetes-manifests -n flux-system

flux suspend ks <name> -n flux-system
flux resume ks <name> -n flux-system
~~~

Do not use <code>--watch</code> or sleep-based polling in CI or operational
checks. Inspect once, then perform other work before checking again.

### Read-only cluster access

Use <code>tools/kc.sh</code>, which selects the repository's canonical
<code>.kube/config</code> and the correct Tailscale API context:

| Alias | Cluster | API operator hostname |
|-------|---------|-----------------------|
| <code>ot</code> | Ottawa | <code>ottawa-k8s-operator.keiretsu.ts.net</code> |
| <code>rb</code> | Robbinsdale | <code>robbinsdale-k8s-operator.keiretsu.ts.net</code> |
| <code>sp</code> | St. Petersburg | <code>stpetersburg-k8s-operator.keiretsu.ts.net</code> |

~~~bash
tools/kc.sh ot get nodes
tools/kc.sh rb get ns
tools/kc.sh sp -n ai get pods
tools/ktriage.sh ot media <pod-name>
~~~

The tracked root <code>.kube/config</code> contains operator URLs and a
placeholder token, not credentials. Never add certificates, keys, or real
tokens. Services are reachable over the Tailscale network; port-forwarding is
normally unnecessary.

### Talos commands

~~~bash
cd clusters/talos-ottawa
mise run health
mise run dashboard
mise run nodes
mise run pods
~~~

The same <code>mise run</code> task names are available in the Robbinsdale and
St. Petersburg directories, with St. Petersburg adding <code>gpu-status</code>
and its two image tasks. Talos control-plane patches are rolling operations:
patch one node, verify etcd, API readiness, and every node's Ready state before
touching the next; handle the VIP holder last.

### Secret management

~~~bash
cd clusters/common
sops flux/vars/common-secrets.sops.yaml

cd ../talos-ottawa
sops flux/vars/cluster-secrets.sops.yaml
~~~

Run SOPS from the directory whose <code>.sops.yaml</code> contains the creation
rules. Use <code>sops -d</code> only for controlled inspection, never to create
a tracked plaintext copy. Values containing <code>$</code> must be stored in a
Secret or ConfigMap and injected through a Flux variable; bare dollar signs are
consumed by Flux envsubst. In raw manifests, write
<code>$${VAR}</code> when the rendered manifest needs a literal
<code>${VAR}</code>.

### Validation and diagnostics

Every PR touching <code>clusters/**</code> or <code>kubernetes/**</code> runs
the Flate matrix and per-cluster rendered-diff workflow. The local gate uses
the same CI-pinned Flate release:

~~~bash
tools/check.sh
make diff
tools/check-versions.sh
tools/orphans.sh
tools/tests/run.sh
~~~

<code>tools/check.sh</code> prints one success line. Exit 124 means the render
process group was killed by the timeout and should be retried; a render that
exits with its own diagnostics is a real failure. The version checker enforces
coupled Talos/tuppr and Rook/Ceph/toolbox versions, while
<code>tools/orphans.sh</code> detects Kustomization-to-disk drift.

## Networking Reference

### Cluster CIDRs

| Cluster | LAN / API VIP | Pod CIDR | Service CIDR | LoadBalancer CIDR |
|---------|---------------|----------|--------------|-------------------|
| <code>talos-ottawa</code> | <code>192.168.169.0/24</code> / <code>192.168.169.25</code> | <code>10.3.0.0/16</code> | <code>10.2.0.0/16</code> | <code>10.169.0.0/16</code> |
| <code>talos-robbinsdale</code> | <code>192.168.50.0/24</code> / <code>192.168.50.25</code> | <code>10.1.0.0/16</code> | <code>10.0.0.0/16</code> | <code>10.50.0.0/16</code> |
| <code>talos-stpetersburg</code> | <code>192.168.73.0/24</code> / <code>192.168.73.25</code> | <code>10.5.0.0/16</code> | <code>10.4.0.0/16</code> | <code>10.73.0.0/16</code> |

### Tailscale DNS and Gateway API

Operator and service names use the <code>keiretsu.ts.net</code> tailnet. Pods
do not automatically receive every MagicDNS device record; workloads that
consume a tailnet name need the common-egress ExternalName Service pattern
described above. A successful laptop <code>dig</code> is not sufficient—verify
the generated egress Service condition and resolution from a pod.

The Gateway API exposure tiers are:

- <code>ts</code>: tailnet ingress;
- <code>private</code>: site-private ingress;
- <code>public</code>: Internet-facing ingress.

<code>ts</code> accepts each cluster's
<code>${CLUSTER_DOMAIN}</code> and <code>*.ts.keiretsu.top</code>.
<code>private</code> and <code>public</code> also accept
<code>*.keiretsu.top</code>. Check the actual listener list before adding a
route.

## Links

- [FluxCD Documentation](https://fluxcd.io/docs/)
- [Talos Linux](https://www.talos.dev/latest/)
- [Talhelper](https://github.com/budimanjojo/talhelper)
- [SOPS](https://github.com/getsops/sops)
- [Tailscale Kubernetes Operator](https://tailscale.com/kb/1236/kubernetes-operator)
- [Flate](https://github.com/home-operations/flate) — offline Flux manifest renderer
- [Repository contribution guide](CONTRIBUTING.md)
- [Talos reference](docs/reference/talos.md)
- [New-app template](docs/reference/app-template.md)
- [Tailscale integration reference](docs/reference/tailscale-integration.md)

## Support

For cluster-specific bootstrap details, start with the Talos reference and the
README files in <code>clusters/talos-*/bootstrap/talos/</code> where present.
For application layout and migration questions, use
[<code>kubernetes/README.md</code>](kubernetes/README.md).
