🏗️ Garage Infrastructure

Multi-region S3-compatible object store · Geo-replicated across 3 datacenters

Garage v2.3.0 · dxflrs/garage
loading metrics… · refreshed on every page load

📊 Live Cluster Metrics

...
Total Bucket Data
...
Total Objects
...
Kubernetes Nodes
...
Replication
...
Garage Nodes
...
Max Queue Depth
...
Block Errors
...
Total Disk Used
across all regions
...
Total Disk Capacity
across all regions
mimir
Data Source
prometheus · server-side

📈 Cluster Charts

🟢 Storage Usage by Zone

🔵 Node Distribution

🟣 Capacity Distribution

🟠 Disk Utilization %

📈 Live Traffic & Health (Last 1 Hour)

🚦 S3 Request Rate

loading...

📤 Data Throughput

loading...

🔄 Block Sync Catch-Up

loading...

⚠️ Block Errors & Queue

loading...

🌐 Multi-Region Topology with Tailscale Proxy Groups

Storage Node Gateway Node common-ingress (inbound) common-egress (outbound) Cross-Region RPC Tailscale VPN
Garage Multi-Region Topology — 3 datacenters with Tailscale VPN and GSLB round-robin Diagram showing three datacenter regions (Ottawa with 4 Talos nodes, Robbinsdale with 3 Talos nodes, St. Petersburg with 3 arm64+GPU nodes) connected via a Tailscale VPN backbone and common-ingress/common-egress Tailscale proxy groups. Each region contains storage pods and gateway pods. Cross-region RPC flows are shown as animated amber lines between regions. A GSLB round-robin DNS (s3.cdn.keiretsu.top) routes public S3 traffic via an Envoy Gateway. Traffic weights: Ottawa 33%, Robbinsdale 33%, St. Petersburg 34%. Replication factor is 2 with consistent mode. 🌐 Envoy Gateway public — s3.cdn.keiretsu.top (GSLB → garage-gateway:3900) 🌐 Tailscale VPN Backbone — keiretsu.ts.net common-ingress ProxyGroup — 3 replicas · Tailscale VPN Inbound → Service ingress-0 ingress-1 ingress-2 9 TS services ingress-0 ingress-1 ingress-2 8 TS services ingress-0 ingress-1 ingress-2 8 TS services common-egress ProxyGroup — 3 replicas · Service → Tailscale (outbound RPC) egress-0 egress-1 egress-2 13 egress TS services egress-0 egress-1 egress-2 13 egress TS services egress-0 egress-1 egress-2 13 egress TS services Ottawa → Robbinsdale RPC 🇨🇦 Ottawa 4 Nodes (Talos) · Services & Media Storage Tier (common-ingress) asuka kaji rei shiro smb Gateway Tier gateway-0 gateway-1 RPC Templates (common-egress) ottawa-gw-{ordinal} robbinsdale-gw-{ordinal} stpetersburg-gw-{ordinal} via s3.cdn.keiretsu.top (33%) 🏠 Robbinsdale 3 Nodes (Talos) · Production Home Lab Storage Tier (common-ingress) stone tank titan smb Gateway Tier gateway-0 gateway-1 RPC Templates (common-egress) ottawa-gw-{ordinal} robbinsdale-gw-{ordinal} stpetersburg-gw-{ordinal} via s3.cdn.keiretsu.top (33%) 🚀 St. Petersburg 3 Nodes (arm64+GPU) · AI/ML Storage (manual layout) spark-0 spark-1 orin-0 (cp) Gateway gateway-0 gateway-1 RPC Templates (common-egress) ottawa-gw-{ordinal} robbinsdale-gw-{ordinal} stpetersburg-gw-{ordinal} via s3.cdn.keiretsu.top (34%) GSLB Round-Robin · s3.cdn.keiretsu.top → ottawa:33 robbinsdale:33 stpetersburg:34 🡺 Public S3: Client → s3.cdn.keiretsu.top (GSLB) → Envoy Gateway public → Gateway pod 🡺 Replication: Gateway → common-egress → Tailscale VPN → other region's common-ingress → Storage pod Each cluster runs all 3 regions' gateway RPC endpoints as local egress services (13 via common-egress) · Tailscale VPN inbound via common-ingress

🏛️ Per-Cluster Breakdown

🇨🇦 Ottawa — Services & Media

K8s Nodes4 ...
Storage Pods5 (4 NVMe localpath + 1 SMB)
Gateway Pods2
common-ingress... replicas · 9 TS svcs
common-egress... replicas · 13 TS svcs (all regions)
... used...
asukakajirei shirosmb gw-0gw-1 ingress×3egress×3

🏠 Robbinsdale — Home Lab

K8s Nodes3 ...
Storage Pods4 (3 localpath + 1 SMB)
Gateway Pods2
common-ingress... replicas · 8 TS svcs
common-egress... replicas · 13 TS svcs (all regions)
... used...
stonetanktitan smb gw-0gw-1 ingress×3egress×3

🚀 St. Petersburg — AI/ML GPU Cluster

K8s Nodes3 ...
Storage3 localpath (spark-0, spark-1, orin-0)
Gateway Pods2
common-ingress... replicas · 8 TS svcs
common-egress... replicas · 13 TS svcs (all regions)
FederationRejoined 2026-06-04
... used...
Notearm64 GPU — timeouts expected
spark-0spark-1orin-0 gw-0gw-1 ingress×3egress×3

⚙️ Architecture

🔑 Replication & Consistency

  • • Replication factor: 2 (read quorum=2)
  • • Consistency mode: consistent — prevents restic AEAD corruption
  • • Layout version v209 — auto-applied via GarageCluster CR

📥 common-ingress (Tailscale VPN Inbound)

  • • 3 replicas per cluster — Tailscale VPN traffic → service
  • • For Tailscale VPN clients accessing Garage S3/admin/web
  • • Garage storage + gateway pods accept VPN inbound via this group
  • • Not used for public S3 CDN traffic (that goes through Envoy Gateway)
  • • 8–9 TS services per cluster with annotation: proxy-group: common-ingress
  • • Migrated from standalone TS pods to proxy group (Jun 9)

📤 common-egress (Outbound RPC)

  • • 3 replicas per cluster — outbound Tailscale connections
  • • Each cluster runs all 3 regions' gateway RPC services (13 total)
  • • Template: {zone}-gw-{ordinal}.keiretsu.ts.net:3901
  • • Cross-region RPC flow: Garage → common-egress → VPN → common-ingress → peer

🌐 GSLB + Envoy Gateway — s3.cdn.keiretsu.top

  • • K8GB round-robin DNS: s3.cdn.keiretsu.top → 3 A records
  • • Weights: ottawa 33%, robbinsdale 33%, stpetersburg 34%
  • • GSLB → Envoy Gateway public → garage-gateway:3900
  • • Tailscale VPN clients hit common-ingress proxy for S3 access
  • • All 3 clusters healthy — GSLB CRs: garage-s3-cdn

Data sourced from Mimir (Prometheus) · Garage operator v0.6.11+ · Garage v2.3.0 · Tailscale operator v1.98

📜 PromQL queries (27 queries across 3 Mimir tenants) 
# Aggregates (talos-ottawa)
sum(garage_bucket_bytes)
sum(garage_bucket_objects)
garage_layout_current_version
max(garage_worker_queue_length)
sum(garage_node_block_errors)
garage_replication_factor

# Per-cluster queries (each tenant)
count(kube_node_info)
count(kube_node_status_condition{condition="Ready",status="true"})
count(garage_local_disk_avail{volume="data"})
sum(garage_local_disk_total{volume="data"})
sum(garage_local_disk_avail{volume="data"})
count(kube_pod_status_phase{phase="Running",namespace="tailscale",pod=~"common-ingress.*"})
count(kube_pod_status_phase{phase="Running",namespace="tailscale",pod=~"common-egress.*"})