---

Multi-cluster Kubernetes infrastructure managed with FluxCD GitOps. This repository manages three geographically distributed Talos Linux clusters connected via Tailscale mesh networking.

Architecture Overview

┌─────────────────────────────────────────────────────────────────────────────┐
│                              Tailscale Mesh                                 │
│                             (keiretsu.ts.net)                               │
└─────────────────────────────────────────────────────────────────────────────┘
         │                         │                         │
         ▼                         ▼                         ▼
┌─────────────────┐       ┌─────────────────┐       ┌──────────────────┐
│  talos-ottawa   │       │talos-robbinsdale│       │talos-stpetersburg│
│    (Ontario)    │       │   (Minnesota)   │       │    (Florida)     │
│                 │       │                 │       │                  │
│ • 3 nodes       │       │ • Multi-node    │       │ • Single node    │
│ • Rook-Ceph     │       │ • Rook-Ceph     │       │ • AI/ML workloads│
│ • Home apps     │       │ • Home apps     │       │                  │
│                 │       │                 │       │                  │
└─────────────────┘       └─────────────────┘       └──────────────────┘
         │                         │                         │
         └─────────────────────────┴─────────────────────────┘
                                   │
                  ┌────────────────┴────────────────┐
                  │   Tailscale Services            │
                  │ (acting as East-West Gateways)  │
                  └─────────────────────────────────┘

Clusters

Cluster	Location	Platform	Purpose	Domain
talos-ottawa	Ontario, CA	Talos Linux	Primary site, 3-node MS-A2	killinit.cc
talos-robbinsdale	Minnesota, US	Talos Linux	Primary site	lukehouge.com
talos-stpetersburg	Florida, US	Talos Linux	GPU/AI workloads (NVIDIA)	rajsingh.info

Directory Structure

The app tree is kubernetes/: config lives once in base/, clusters opt in with thin pointer files. Most apps have migrated from the old clusters/*/apps trees (completed June 2026); a few remnants remain (see below). clusters/ now primarily holds cluster bootstrap, Talos config, and Flux entrypoints/vars.

├── kubernetes/                    # the app tree (see kubernetes/README.md)
│   ├── apps/base/<ns>/<app>/      # ALL app config, exactly once
│   │                              #   <app>-<location>/ for cluster-specific variants
│   ├── apps/ottawa/<ns>/          # pointer Flux Kustomizations per cluster
│   ├── apps/robbinsdale/<ns>/     #   app deploys wherever its pointer exists
│   ├── apps/stpetersburg/<ns>/
│   └── components/                # shared kustomize components (oidc-protect, ...)
│
├── kubernetes/                    # the app tree (see kubernetes/README.md)
│   ├── apps/base/<ns>/<app>/      # ALL app config, exactly once
│   │                              #   <app>-<location>/ for cluster-specific variants
│   ├── apps/ottawa/<ns>/          # pointer Flux Kustomizations per cluster
│   ├── apps/robbinsdale/<ns>/     #   app deploys wherever its pointer exists
│   ├── apps/stpetersburg/<ns>/
│   └── components/                # shared kustomize components (oidc-protect, ...)
│
├── clusters/
│   ├── common/
│   │   ├── bootstrap/             # FluxCD bootstrap
│   │   ├── flux/
│   │   │   ├── repositories/     # Helm/OCI/Git repository definitions
│   │   │   └── vars/              # common-settings / common-secrets (sops)
│   │   ├── apps/                  # migration remnants (home/local-gateway, searxng disabled)
│   │   └── scripts/               # utility scripts
│   ├── talos-<location>/          # per cluster: ottawa | robbinsdale | stpetersburg
│   │   ├── bootstrap/talos/       # Talos configuration (talhelper)
│   │   ├── flux/                  # entrypoints (config/cluster.yaml) + vars (sops)
│   │   └── apps/                  # migration remnants (location-specific)
│
├── tailscale/                     # tailnet policy (hujson), scripts, CI tailnets
└── Makefile                       # make test / make diff (flate)

Key Infrastructure Components

FluxCD GitOps

All clusters use FluxCD v2 for GitOps continuous delivery:

• Source: Git repository kubernetes-manifests (this repo)
• Kustomizations: Hierarchical configuration with cluster-specific overrides
• Variable Substitution: Settings injected from ConfigMaps/Secrets
• SOPS Encryption: GPG-encrypted secrets in git

# Configuration hierarchy
clusters/common/flux/vars/common-settings.yaml    # Global settings
clusters/<cluster>/flux/vars/cluster-settings.yaml # Cluster-specific
clusters/<cluster>/flux/vars/cluster-secrets.sops.yaml # Encrypted secrets

SOPS Secret Encryption

All secrets are encrypted with PGP using SOPS:

# clusters/common/.sops.yaml
creation_rules:
  - path_regex: .*.yaml
    encrypted_regex: ^(data|stringData)$
    pgp: FAC8E7C3A2BC7DEE58A01C5928E1AB8AF0CF07A5

Tailscale Integration

Full Tailscale integration via the official k8s-operator:

• Operator: Deployed per-cluster with unique hostnames
• ProxyClass: Custom proxy configurations (userspace, accept-routes)
• Egress Proxies: Cross-cluster access via ExternalName services
• Ingress Proxies: Cross-cluster ingress access via L4 LoadBalancer
• Tailscale Services: HA Ingress Proxies via Service VIP + Static Service-level identity
• Connectors: Subnet routers for site LAN access
• DNS Config: In-cluster DNS resolution for MagicDNS FQDNs

# ProxyClass examples
- common              # Basic proxy with metrics
- common-accept-routes # Accept advertised routes
- common-userspace    # Unprivileged userspace mode

Monitoring Stack

Prometheus-based monitoring with Grafana visualization:

• kube-prometheus-stack: Full monitoring stack (Prometheus, Alertmanager)
• Grafana Operator: Declarative dashboard management
• Grafana Dashboards: Pre-configured dashboards for all components
• ServiceMonitors: Auto-discovery of metrics endpoints

CNI: Cilium

All Talos clusters use Cilium as the CNI:

• eBPF-based Direct routing networking with BGP peering
• Hubble UI for network observability
• Network Policies enforcement
• Service mesh capabilities (optional)

Storage

Cluster	Primary Storage	Secondary
talos-ottawa	Rook-Ceph	SMB (NAS)
talos-robbinsdale	Rook-Ceph	SMB (NAS)
talos-stpetersburg	local-path	-

Gateway API / Envoy Gateway

Cluster ingress via Gateway API:

• Envoy Gateway: Gateway controller
• HTTPRoute: L7 routing with path/header matching
• TCPRoute/UDPRoute: L4 routing
• CDN Integration: Multi-cluster backends with failover

Common Applications (kubernetes/apps/base/, pointers in all three location trees)

Core Infrastructure

• cert-manager - TLS certificate management
• envoy-gateway-system - Gateway API controller
• flux-system - Flux notifications & alerts
• snapshot-controller - Volume snapshots
• spegel - Container registry P2P mirror
• local-path-storage - Local volume provisioner
• external-secrets - External secrets management
• velero - Cluster backup & restore

Networking

• tailscale - Tailscale operator + egress proxies
• cloudflare - External DNS + Tunnel
• core-dns - DNS customization (kube-system)
• cilium - CNI (per-cluster config)

Monitoring & Observability

• monitoring - kube-prometheus-stack
• victoria-logs - Log aggregation
• hubble-ui - Cilium network observability
• mimir - Distributed metrics (per-cluster)
• kromgo - Cluster metrics badges
• blackbox-exporter - Endpoint probing
• unpoller - UniFi metrics
• gatus - Automated status page

Databases & Storage

• cnpg-system - CloudNative PostgreSQL operator
• dragonfly-operator-system - Dragonfly (Redis-compatible)
• garage - S3-compatible object storage
• garage-operator-system - Garage bucket operator

Applications

• home - Homer dashboard, homepage, tailscale gateways
• fluent-bit - Log shipping

CI/CD

• actions-runner-controller - GitHub Actions runners

Cluster Infrastructure

• node-feature-discovery - Hardware feature detection
• vpa-system - Vertical Pod Autoscaler
• kro-system - Kubernetes Resource Orchestrator
• csi-secrets-store - Secrets Store CSI driver
• gvisor - gVisor container runtime sandbox

Cluster-Specific Applications (pointer exists only in that location tree)

talos-ottawa

• rook-ceph - Distributed storage cluster
• immich - Photo management
• media - Media management stack
• forgejo - Git server
• woodpecker - CI/CD
• tempo - Distributed tracing
• zot - OCI registry cache
• teslamate - Tesla data logging
• searxng - Privacy search engine
• k8s-gpu-dra-driver - NVIDIA GPU DRA driver
• kata-containers - Kata Containers runtime
• lan - Internal LAN routing
• headlamp - Kubernetes dashboard (ottawa only)
• opencost - Kubernetes cost monitoring (ottawa only)
• auth/tinyauth - Google SSO auth

talos-robbinsdale

• rook-ceph - Distributed storage cluster
• immich - Photo management
• media - Media management stack
• speedtest - Network speed testing

talos-stpetersburg

• gpu-operator - NVIDIA GPU support
• ai - vLLM inference workloads
• home-assistant - Home automation
• lws-system - LeaderWorkerSet (AI batch scheduling)
• rdma-shared-dp - RDMA shared device plugin
• tailbench - Tailscale benchmarking

Prerequisites

Required Tools

# Package managers
brew install mise  # or use asdf

# Core tools (install via mise)
mise install kubectl flux sops gpg talhelper talosctl task

# Optional
brew install helm kustomize cilium-cli

GPG Key Setup

# Import the SOPS PGP key
gpg --import /path/to/sops.asc

# Trust the key
echo "FAC8E7C3A2BC7DEE58A01C5928E1AB8AF0CF07A5:6:" | gpg --import-ownertrust

# Verify
gpg --list-secret-keys | grep FAC8E7C3A2BC7DEE58A01C5928E1AB8AF0CF07A5

Bootstrap a New Talos Cluster

See clusters/talos-ottawa/bootstrap/talos/README.md for comprehensive instructions.

Quick Start

cd clusters/<cluster-name>

# 1. Generate Talos configs (requires talhelper + sops)
talhelper gensecret > bootstrap/talos/talsecret.sops.yaml
sops --encrypt bootstrap/talos/talsecret.sops.yaml
talhelper genconfig

# 2. Apply configs to nodes
talosctl apply-config -n <node-ip> -f bootstrap/talos/manifests/<node>.yaml
talosctl bootstrap -n <node-ip>

# 3. Install Cilium CNI
helm install cilium cilium/cilium -n kube-system -f ../../kubernetes/apps/base/kube-system/cilium-<cluster>/app/values.yaml

# 4. Install Flux
kubectl apply --server-side --kustomize ../../clusters/common/bootstrap/flux/
kubectl apply -f flux/config/cluster.yaml

# 5. Wait for reconciliation
flux get ks -A --watch

Adding a New Application

One directory of config + one ~20-line pointer file per target cluster. Full recipe and the move guide for old-tree apps: kubernetes/README.md.

1. App config (once)

mkdir -p kubernetes/apps/base/<namespace>/<app>
# helmrelease.yaml / deployment.yaml, httproute.yaml, kustomization.yaml ...
# copy a neighbor in base/ as your starting point

2. Pointer per target cluster

kubernetes/apps/<location>/<namespace>/<app>.yaml — deploys wherever this file exists. The parent Kustomization injects SOPS decryption + the substituteFrom stack, so pointers stay thin and need no postBuild block:

apiVersion: kustomize.toolkit.fluxcd.io/v1
kind: Kustomization
metadata:
  name: &app <app>
  namespace: flux-system
spec:
  targetNamespace: <namespace>
  commonMetadata:
    labels:
      app.kubernetes.io/name: *app
  path: ./kubernetes/apps/base/<namespace>/<app>
  prune: true
  sourceRef:
    kind: GitRepository
    name: kubernetes-manifests
  interval: 30m
  retryInterval: 1m
  timeout: 5m

List it (and a namespace.yaml if the namespace is new) in that directory's kustomization.yaml, and the namespace dir in the location root kustomization.yaml.

3. Ingress

One HTTPRoute next to the app. parentRefs = exposure tiers, hostname = any of the four domains at any location (all gateways terminate all domains):

apiVersion: gateway.networking.k8s.io/v1
kind: HTTPRoute
metadata:
  name: <app>
spec:
  parentRefs:                      # pick any subset: ts | private | public
    - { name: ts, namespace: home }
  hostnames: [<app>.keiretsu.top]
  rules:
    - backendRefs: [{ name: <app>, port: 80 }]

Certs are on the listeners; DNS follows from which gateways you attach to (public → cloudflare, private → unifi, ts → pihole). NOTE: keiretsu.top names need public records (tailnet split-DNS forwards keiretsu.top to 1.1.1.1) — add a CNAME in the k8gb cnames config if the route is not on the public gateway.

4. Auth (optional)

Google-account SSO via tinyauth forward-auth: add the app FQDN + allowed emails to the tinyauth config (kubernetes/apps/base/auth/tinyauth/configmap.yaml) and drop a SecurityPolicy on the route (copy kubernetes/apps/base/authtest/securitypolicy.yaml). Per-FQDN allow-lists in plain repo config; see issue #1547. (The pocket-id OIDC component oidc-protect remains for apps needing real OIDC tokens.)

5. Validate

make test renders all three clusters; the PR gets per-cluster rendered diffs.

Common Operations

Flux Commands

# Check sync status
flux get ks -A
flux get hr -A

# Force reconciliation
flux reconcile ks cluster -n flux-system
flux reconcile source git kubernetes-manifests -n flux-system

# Suspend/resume
flux suspend ks <name> -n flux-system
flux resume ks <name> -n flux-system

Talos Commands (per-cluster)

cd clusters/talos-ottawa/bootstrap/talos
talhelper genconfig  # Regenerate node configs
talosctl health      # Check cluster health
talosctl dashboard   # Open Talos dashboard
talosctl kubeconfig  # Fetch kubeconfig

Secret Management

# Encrypt a secret
sops -e secret.yaml > secret.sops.yaml

# Decrypt for viewing
sops -d secret.sops.yaml

# Edit in place
sops secret.sops.yaml

Validation

Every PR touching clusters/** or kubernetes/** is rendered offline with flate — CI runs the same bounded render gate used locally and comments the rendered manifest diff on the PR. The gate uses the CI-pinned Flate version automatically.

# Render-test all three clusters (what CI runs)
tools/check.sh

# One cluster
tools/check.sh talos-ottawa

# Rendered manifest diff vs origin/main (CI comments this on your PR)
make diff

Gotchas CI will catch for you: helm values schema violations, broken kustomizations, unresolvable chart versions. Two repo rules it enforces by construction:

• Remote git-directory kustomize bases don't render offline — vendor the YAML instead (see legacy examples in clusters/*/apps/ for the pattern: provenance header with upstream ref/SHA and the re-render command).
• HelmRepository URLs need a trailing slash when the index uses relative tgz paths (https://pkgs.tailscale.com/helmcharts/).

SOPS secret values render as placeholders; charts gated on CRD capability checks (e.g. cilium ServiceMonitors) set trustCRDsExist: true in values.

Networking Reference

Cluster CIDRs

Cluster	Pod CIDR	Service CIDR	LB CIDR
talos-robbinsdale	10.1.0.0/16	10.0.0.0/16	10.50.0.0/16
talos-ottawa	10.3.0.0/16	10.2.0.0/16	10.169.0.0/16
talos-stpetersburg	10.5.0.0/16	10.4.0.0/16	10.73.0.0/16

Tailscale DNS

Services are accessible at <hostname>.keiretsu.ts.net:

• ottawa-k8s-operator.keiretsu.ts.net
• robbinsdale-k8s-operator.keiretsu.ts.net
• stpetersburg-k8s-operator.keiretsu.ts.net

Links

• FluxCD Documentation
• Talos Linux
• Talhelper
• SOPS
• Tailscale Kubernetes Operator
• Flate - Offline Flux manifest renderer

Support

For detailed cluster-specific instructions, see the README files in each cluster's bootstrap/talos/ directory.