# 🏗️ Garage Infrastructure

Multi-region S3-compatible object store · Geo-replicated across 3 datacenters

Garage v2.3.0 · dxflrs/garage
loading metrics… · refreshed on every page load

## 📊 Live Cluster Metrics

...
Total Bucket Data
...
Total Objects
...
Kubernetes Nodes
...
Replication
...
Garage Nodes
...
Max Queue Depth
...
Block Errors
...
Total Disk Used
across all regions
...
Total Disk Capacity
across all regions
mimir
Data Source
prometheus · server-side

## 📈 Cluster Charts

### 🟢 Storage Usage by Zone

### 🔵 Node Distribution

### 🟣 Capacity Distribution

### 🟠 Disk Utilization %

## 📈 Live Traffic & Health (Last 1 Hour)

### 🚦 S3 Request Rate

loading...

### 📤 Data Throughput

loading...

### 🔄 Block Sync Catch-Up

loading...

### ⚠️ Block Errors & Queue

loading...

## 🌐 Multi-Region Topology with Tailscale Proxy Groups

Storage Node Gateway Node common-ingress (inbound) common-egress (outbound) Cross-Region RPC Tailscale VPN

## 🏛️ Per-Cluster Breakdown

### 🇨🇦 Ottawa — Services & Media

K8s Nodes 4 ...
Storage Pods 5 (4 NVMe localpath + 1 SMB)
Gateway Pods 2
common-ingress... replicas · 9 TS svcs
common-egress... replicas · 13 TS svcs (all regions)
... used...
asuka kaji rei shiro smb gw-0 gw-1 ingress×3 egress×3

### 🏠 Robbinsdale — Home Lab

K8s Nodes 3 ...
Storage Pods 4 (3 localpath + 1 SMB)
Gateway Pods 2
common-ingress... replicas · 8 TS svcs
common-egress... replicas · 13 TS svcs (all regions)
... used...
stone tank titan smb gw-0 gw-1 ingress×3 egress×3

### 🚀 St. Petersburg — AI/ML GPU Cluster

K8s Nodes 3 ...
Storage 3 localpath (spark-0, spark-1, orin-0)
Gateway Pods 2
common-ingress... replicas · 8 TS svcs
common-egress... replicas · 13 TS svcs (all regions)
Federation Rejoined 2026-06-04
... used...
Note arm64 GPU — timeouts expected
spark-0 spark-1 orin-0 gw-0 gw-1 ingress×3 egress×3

## ⚙️ Architecture

### 🔑 Replication & Consistency

- • Replication factor: 2 (read quorum=2)
- • Consistency mode: consistent — prevents restic AEAD corruption
- • Layout version v209 — auto-applied via GarageCluster CR

### 📥 common-ingress (Tailscale VPN Inbound)

- • 3 replicas per cluster — Tailscale VPN traffic → service
- • For Tailscale VPN clients accessing Garage S3/admin/web
- • Garage storage + gateway pods accept VPN inbound via this group
- • Not used for public S3 CDN traffic (that goes through Envoy Gateway)
- • 8–9 TS services per cluster with annotation: `proxy-group: common-ingress`
- • Migrated from standalone TS pods to proxy group (Jun 9)

### 📤 common-egress (Outbound RPC)

- • 3 replicas per cluster — outbound Tailscale connections
- • Each cluster runs all 3 regions' gateway RPC services (13 total)
- • Template: `{zone}-gw-{ordinal}.keiretsu.ts.net:3901`
- • Cross-region RPC flow: Garage → common-egress → VPN → common-ingress → peer

### 🌐 GSLB + Envoy Gateway — s3.cdn.keiretsu.top

- • K8GB round-robin DNS: s3.cdn.keiretsu.top → 3 A records
- • Weights: ottawa 33%, robbinsdale 33%, stpetersburg 34%
- • GSLB → Envoy Gateway `public` → garage-gateway:3900
- • Tailscale VPN clients hit common-ingress proxy for S3 access
- • All 3 clusters healthy — GSLB CRs: `garage-s3-cdn`