01 Relevant CNCF projects
A hive is deployed and operated as a native cloud-native workload. The projects below are the load-bearing ones.
Each hive is a single-replica Deployment with a PVC, Service, and Ingress. The hub creates new spoke hives as native Kubernetes objects.
Issues and renews the TLS certificates that terminate every dashboard and the hosted hub, via a Let's Encrypt ClusterIssuer — needed for the long-lived SSE streams.
Runs the single hive OCI image — the Go orchestrator, agent CLIs, and dashboard all travel together as one artifact.
Each hive exposes /metrics — governor mode, queue depth, per-agent token spend, fleet health — for scraping, alongside a live SSE stream to the dashboard.
Hive ships a Kustomize base (namespace, Deployment, Service, PVC, ConfigMap, Secret) so an operator applies one overlay per environment.
02 Organization & team
Hive is an open-source, self-hostable platform — part of the KubeStellar org (github.com/kubestellar/hive) — that keeps software repositories maintained by running a fleet of AI coding agents — triaging issues, writing fixes, opening PRs, and merging on green CI — under a human-controlled autonomy dial and permissions enforced in depth. It runs as a cloud-native workload: each hive is a Kubernetes deployment, and a hosted hub coordinates a fleet of self-hosted spokes.
Hive is run in production by more than one open-source community:
- The KubeStellar Console team (
kubestellar/console) runs a hive against the console's repositories — the flagship end user, where Hive is dogfooded daily. - Project Bluefin (
projectbluefin) runs its own hive (projectbluefin/knuckleand related repos) at a measured ACMM level.
That is the point of the hub-and-spoke design: independent teams each self-host a spoke against their own repositories, while a shared hub (hive.kubestellar.io) provides a registry, a cross-hive leaderboard, and — where reachable — provisioning.
03 Architecture overview & goals
The goal: let a fleet of AI coding agents maintain a repository autonomously — triage issues, write fixes, open PRs, merge on green CI — while keeping a human firmly in control of how much autonomy is granted, and enforcing those limits with technical controls rather than trust.
A hive runs as a single container on Kubernetes with three long-lived processes: a Go orchestrator (governor loop, agent manager, dashboard API, an in-process MITM GitHub proxy, hub heartbeat), a Node.js proxy (the public front door), and ttyd (a web terminal onto the agents' tmux sessions).
Two ideas define the architecture:
- Deterministic before judgment. A pipeline of Go and shell steps enumerates GitHub work, classifies it, gates which PRs are mergeable, and enforces permissions before any model sees a task. Agents only make the judgment calls.
- Autonomy is a dial, enforced in depth. An AI-native Capability Maturity Model (ACMM) with six levels is the single control a human turns; each level maps to a per-agent permission mode enforced at three independent layers.
flowchart LR
github["GitHub
issues · PRs"] --> gov["Governor
queue depth → mode → kick"]
gov --> pipe["Deterministic pipeline
classify · merge-gate · enforce"]
pipe --> agents["AI agents (tmux)
Claude · Copilot · Gemini · Goose"]
agents --> guard["Guardrails
tool deny · scoped token · MITM proxy"]
guard -->|"gated writes"| github
agents -.-> beads["Beads ledger
git-backed work items"]
gov -.->|"heartbeat / 2 min"| hub["Hive Hub
registry · leaderboard · provisioning"]
The autonomy dial (ACMM)
Raising the level is always a human decision. Each rung is a concrete, enforced permission change.
hold for human batch-review. The system proposes; it does not merge.hold label.Enforcement in depth
The ACMM level maps to a per-agent mode (ADVISORY → ISSUES_ONLY → ISSUES_AND_PRS → ISSUES_PRS_MERGE) enforced at three independent layers. A bug in one is caught by the next.
flowchart LR
agent["AI agent"] --> l1["1 · CLI tool deny
(gated by mode)"]
l1 --> l2["2 · Scoped token
(least privilege)"]
l2 --> l3["3 · MITM proxy
(method,path) → min mode
+ repo allowlist"]
l3 --> gh["api.github.com"]
traj["Trajectory review"] -.->|"drift → pause"| agent
The hub holds a registry of all spokes, a cross-hive leaderboard, and provisions reachable spokes with kubectl. Firewalled spokes it cannot reach directly are still fully managed over the 2-minute heartbeat — the control plane that lets independent teams (KubeStellar Console, Project Bluefin) each run a spoke against their own repositories under one shared hub.
The autonomy framework (ACMM) is described in depth in the paper An AI-native Capability Maturity Model.
04 Guiding principles
A handful of principles shape every design decision — and they are why the system can be trusted to run at higher autonomy levels.
A repo climbs the ACMM ladder as its safety net grows — the target is 90%+ test coverage before autonomous merging is on the table. Automation is earned, not flipped on.
Every unit of work is a typed, dependency-aware bead on disk — agents coordinate without collisions and no work is lost across restarts or upgrades.
/clear)Each agent kick starts from a clean slate, so stale context from a prior task can't leak into the next — a cheap, decisive defense against drift.
Anything rules can decide (filtering, classification, merge-gating, enforcement) runs in code before a model sees it. The LLM is for judgment — never fill-in-the-blank "Mad Libs" a program should own.
CI is a pipeline that validates at multiple stages — not a single check that opens the merge door. That's what makes "merge on green" a real signal, not a rubber stamp.
An issue can carry structured context — diagnostics, browser-console errors, OS/browser info, the running SHA — right in its description, which becomes the prompt the agent works from. Better issues, better fixes.
05 Why these projects
- Kubernetes — a hive must be self-hostable anywhere, survive restarts, and be provisioned programmatically. Modeling each hive as a Deployment + PVC + Service + Ingress lets the hub create, upgrade, and delete a spoke with plain API calls.
- cert-manager — removes TLS toil for dashboards that stream over long-lived SSE and must be secured.
- containerd — runs the single self-contained hive image so a hive is one artifact.
- Prometheus / OpenTelemetry — make the fleet observable: a runaway agent or a stalled eval loop is visible before it does harm.
- Kustomize / Helm — keep per-environment deployment declarative.
06 What worked — and what didn't
✓ Worked well
- The three-layer guardrail model. Because the MITM proxy enforces permissions at the network boundary, we can grant real write access with confidence — this made higher ACMM levels safe to run.
- A durable, git-backed work ledger ("beads"). Typed, dependency-aware work items on disk mean agents coordinate without collisions and survive restarts.
- Queue-depth governance. Driving cadence from the actionable-issue count keeps a quiet repo nearly free and gives a flooded one the full fleet — no manual schedules.
- Hub-and-spoke over heartbeat. Managing unreachable (firewalled) spokes purely through the heartbeat response has been robust.
✗ Didn't work (at first)
- Cost attribution across model backends. Reconciling token spend across Anthropic, Copilot, and OpenAI-compatible gateways — especially "auto" model selection — took several iterations to get right.
- Non-reentrant locks on the startup path. A mutex re-taken from the launch path could deadlock startup; the race detector missed it. We moved hot startup state to lock-free atomics.
- Config precedence. The authoritative config lives on the PVC while a ConfigMap seeds only first boot — clear conventions were needed before this stopped confusing operators.
07 The "glue" we had to build
Most of the interesting engineering is in the glue that makes agents safe and coordinated:
- A deterministic shell/Go pipeline that enumerates, classifies, clusters, and merge-gates GitHub work before agents run.
- A
ghwrapper that injects scoped tokens and blocks writes exceeding an agent's tier — the shell-level twin of the proxy. - An in-process MITM proxy (with
iptablesegress redirection and a self-signed CA in the container trust store) that gatesapi.github.comby agent mode and repo allowlist; agent identity is resolved from the connection's owning UID via/proc/net/tcp. - An inference translator that reroutes Anthropic-shaped calls to OpenAI-shaped gateways so one agent CLI works across backends.
- A trajectory reviewer — a periodic second-model check that reads an agent's transcript against its assigned intent and pauses on drift (a defense against prompt-injection-style goal hijacking).
- Agents run inside
tmuxunder per-agent OS users for UID isolation; a "kick" is literally a work order typed into the agent's CLI prompt.
08 Evolution & lessons learned
The central lesson was enforce, don't trust. Early designs relied on prompting and CLI tool-deny lists; that is necessary but not sufficient, because a capable agent finds paths the prompt didn't anticipate. Moving enforcement to the network boundary and to credential scope changed the trust model: we no longer have to believe the agent will behave — we constrain what it can do.
A second lesson: make autonomy a graded, human-owned dial. Rather than "agents on / off", the ACMM ladder lets an operator start advisory-only and climb one rung at a time as trust is earned — every rung a concrete, enforced permission change.
A third: durable state beats clever runtime state. The git-backed beads ledger and PVC-authoritative config both came from painful episodes where in-memory coordination lost work across restarts.
09 What's next
- Planning intelligence (incubating) — automatic decomposition of a high-level goal into a tree of child work items, with a human "approve the plan" gate before execution and stall-triggered re-planning.
- Richer fleet observability — first-class OpenTelemetry traces across the governor → pipeline → agent → PR lifecycle.
- Broadening the guardrail model — an optional content/prompt-injection classifier and a proxy-level request-rate limiter.