## 01 Relevant CNCF projects

A hive is deployed and operated as a native cloud-native workload. The projects below are the load-bearing ones.

☸️ Kubernetes
1.24+

Each hive is a single-replica Deployment with a PVC, Service, and Ingress. The hub creates new spoke hives as native Kubernetes objects.

🔐 cert-manager
v1.x

Issues and renews the TLS certificates that terminate every dashboard and the hosted hub, via a Let's Encrypt `ClusterIssuer` — needed for the long-lived SSE streams.

📦 containerd
cluster runtime

Runs the single hive OCI image — the Go orchestrator, agent CLIs, and dashboard all travel together as one artifact.

📊 Prometheus / OpenTelemetry
exposition

Each hive exposes `/metrics` — governor mode, queue depth, per-agent token spend, fleet health — for scraping, alongside a live SSE stream to the dashboard.

⎈ Helm / Kustomize
v3.x

Hive ships a Kustomize base (namespace, Deployment, Service, PVC, ConfigMap, Secret) so an operator applies one overlay per environment.

## 02 Organization & team

Hive is an open-source, self-hostable platform — part of the KubeStellar org (`github.com/kubestellar/hive`) — that keeps software repositories maintained by running a fleet of AI coding agents — triaging issues, writing fixes, opening PRs, and merging on green CI — under a human-controlled autonomy dial and permissions enforced in depth. It runs as a cloud-native workload: each hive is a Kubernetes deployment, and a hosted hub coordinates a fleet of self-hosted spokes.

Hive is run in production by more than one open-source community:

- The KubeStellar Console team (`kubestellar/console`) runs a hive against the console's repositories — the flagship end user, where Hive is dogfooded daily.
- Project Bluefin (`projectbluefin`) runs its own hive (`projectbluefin/knuckle` and related repos) at a measured ACMM level.

That is the point of the hub-and-spoke design: independent teams each self-host a spoke against their own repositories, while a shared hub (hive.kubestellar.io) provides a registry, a cross-hive leaderboard, and — where reachable — provisioning.

## 03 Architecture overview & goals

The goal: let a fleet of AI coding agents maintain a repository autonomously — triage issues, write fixes, open PRs, merge on green CI — while keeping a human firmly in control of how much autonomy is granted, and enforcing those limits with technical controls rather than trust.

A hive runs as a single container on Kubernetes with three long-lived processes: a Go orchestrator (governor loop, agent manager, dashboard API, an in-process MITM GitHub proxy, hub heartbeat), a Node.js proxy (the public front door), and ttyd (a web terminal onto the agents' `tmux` sessions).

Two ideas define the architecture:

1. Deterministic before judgment. A pipeline of Go and shell steps enumerates GitHub work, classifies it, gates which PRs are mergeable, and enforces permissions before any model sees a task. Agents only make the judgment calls.
2. Autonomy is a dial, enforced in depth. An AI-native Capability Maturity Model (ACMM) with six levels is the single control a human turns; each level maps to a per-agent permission mode enforced at three independent layers.

    flowchart LR
        github["GitHub
    issues · PRs"] --> gov["Governor
    queue depth → mode → kick"]
        gov --> pipe["Deterministic pipeline
    classify · merge-gate · enforce"]
        pipe --> agents["AI agents (tmux)
    Claude · Copilot · Gemini · Goose"]
        agents --> guard["Guardrails
    tool deny · scoped token · MITM proxy"]
        guard -->|"gated writes"| github
        agents -.-> beads["Beads ledger
    git-backed work items"]
        gov -.->|"heartbeat / 2 min"| hub["Hive Hub
    registry · leaderboard · provisioning"]

The maintenance loop: deterministic pre-processing feeds agents whose every write is gated.

### The autonomy dial (ACMM)

Raising the level is always a human decision. Each rung is a concrete, enforced permission change.

L1 Inception (Assisted) — project inception & advisory only; every action needs human approval.
L2 Advisory (Instructed) — observe and report findings as beads; no GitHub writes.
L3 Quality-Gated (Measured) — the quality agent opens hold-gated PRs about testing gaps, coverage, and CI health. This measurement foundation is what earns automation above.
L4 Security-Aware (Adaptive) — all agents file issues (bugs, docs, workflows, vulns); still no PRs.
L5 Semi-Autonomous (Semi-Automated) — all agents open PRs, each auto-labeled `hold` for human batch-review. The system proposes; it does not merge.
L6 Fully Autonomous — agents open PRs and auto-merge on green CI; no `hold` label.

### Enforcement in depth

The ACMM level maps to a per-agent mode (`ADVISORY` → `ISSUES_ONLY` → `ISSUES_AND_PRS` → `ISSUES_PRS_MERGE`) enforced at three independent layers. A bug in one is caught by the next.

    flowchart LR
        agent["AI agent"] --> l1["1 · CLI tool deny
    (gated by mode)"]
        l1 --> l2["2 · Scoped token
    (least privilege)"]
        l2 --> l3["3 · MITM proxy
    (method,path) → min mode
    + repo allowlist"]
        l3 --> gh["api.github.com"]
        traj["Trajectory review"] -.->|"drift → pause"| agent

Every GitHub write crosses all three layers; a trajectory reviewer watches for goal drift.

> The hub holds a registry of all spokes, a cross-hive leaderboard, and provisions reachable spokes with `kubectl`. Firewalled spokes it cannot reach directly are still fully managed over the 2-minute heartbeat — the control plane that lets independent teams (KubeStellar Console, Project Bluefin) each run a spoke against their own repositories under one shared hub.

The autonomy framework (ACMM) is described in depth in the paper An AI-native Capability Maturity Model.

## 04 Guiding principles

A handful of principles shape every design decision — and they are why the system can be trusted to run at higher autonomy levels.

🎯 Earn automation with coverage

A repo climbs the ACMM ladder as its safety net grows — the target is 90%+ test coverage before autonomous merging is on the table. Automation is earned, not flipped on.

🧵 Durable work ledger (beads)

Every unit of work is a typed, dependency-aware bead on disk — agents coordinate without collisions and no work is lost across restarts or upgrades.

🧹 Reset context (`/clear`)

Each agent kick starts from a clean slate, so stale context from a prior task can't leak into the next — a cheap, decisive defense against drift.

⚙️ Deterministic vs. non-deterministic

Anything rules can decide (filtering, classification, merge-gating, enforcement) runs in code before a model sees it. The LLM is for judgment — never fill-in-the-blank "Mad Libs" a program should own.

🔁 Multi-stage CI

CI is a pipeline that validates at multiple stages — not a single check that opens the merge door. That's what makes "merge on green" a real signal, not a rubber stamp.

📝 Issues are prompts

An issue can carry structured context — diagnostics, browser-console errors, OS/browser info, the running SHA — right in its description, which becomes the prompt the agent works from. Better issues, better fixes.

## 05 Why these projects

- Kubernetes — a hive must be self-hostable anywhere, survive restarts, and be provisioned programmatically. Modeling each hive as a Deployment + PVC + Service + Ingress lets the hub create, upgrade, and delete a spoke with plain API calls.
- cert-manager — removes TLS toil for dashboards that stream over long-lived SSE and must be secured.
- containerd — runs the single self-contained hive image so a hive is one artifact.
- Prometheus / OpenTelemetry — make the fleet observable: a runaway agent or a stalled eval loop is visible before it does harm.
- Kustomize / Helm — keep per-environment deployment declarative.

## 06 What worked — and what didn't

### ✓ Worked well

- The three-layer guardrail model. Because the MITM proxy enforces permissions at the network boundary, we can grant real write access with confidence — this made higher ACMM levels safe to run.
- A durable, git-backed work ledger ("beads"). Typed, dependency-aware work items on disk mean agents coordinate without collisions and survive restarts.
- Queue-depth governance. Driving cadence from the actionable-issue count keeps a quiet repo nearly free and gives a flooded one the full fleet — no manual schedules.
- Hub-and-spoke over heartbeat. Managing unreachable (firewalled) spokes purely through the heartbeat response has been robust.

### ✗ Didn't work (at first)

- Cost attribution across model backends. Reconciling token spend across Anthropic, Copilot, and OpenAI-compatible gateways — especially "auto" model selection — took several iterations to get right.
- Non-reentrant locks on the startup path. A mutex re-taken from the launch path could deadlock startup; the race detector missed it. We moved hot startup state to lock-free atomics.
- Config precedence. The authoritative config lives on the PVC while a ConfigMap seeds only first boot — clear conventions were needed before this stopped confusing operators.

## 07 The "glue" we had to build

Most of the interesting engineering is in the glue that makes agents safe and coordinated:

- A deterministic shell/Go pipeline that enumerates, classifies, clusters, and merge-gates GitHub work before agents run.
- A `gh` wrapper that injects scoped tokens and blocks writes exceeding an agent's tier — the shell-level twin of the proxy.
- An in-process MITM proxy (with `iptables` egress redirection and a self-signed CA in the container trust store) that gates `api.github.com` by agent mode and repo allowlist; agent identity is resolved from the connection's owning UID via `/proc/net/tcp`.
- An inference translator that reroutes Anthropic-shaped calls to OpenAI-shaped gateways so one agent CLI works across backends.
- A trajectory reviewer — a periodic second-model check that reads an agent's transcript against its assigned intent and pauses on drift (a defense against prompt-injection-style goal hijacking).
- Agents run inside `tmux` under per-agent OS users for UID isolation; a "kick" is literally a work order typed into the agent's CLI prompt.

## 08 Evolution & lessons learned

The central lesson was enforce, don't trust. Early designs relied on prompting and CLI tool-deny lists; that is necessary but not sufficient, because a capable agent finds paths the prompt didn't anticipate. Moving enforcement to the network boundary and to credential scope changed the trust model: we no longer have to believe the agent will behave — we constrain what it can do.

A second lesson: make autonomy a graded, human-owned dial. Rather than "agents on / off", the ACMM ladder lets an operator start advisory-only and climb one rung at a time as trust is earned — every rung a concrete, enforced permission change.

A third: durable state beats clever runtime state. The git-backed beads ledger and PVC-authoritative config both came from painful episodes where in-memory coordination lost work across restarts.

## 09 What's next

- Planning intelligence (incubating) — automatic decomposition of a high-level goal into a tree of child work items, with a human "approve the plan" gate before execution and stall-triggered re-planning.
- Richer fleet observability — first-class OpenTelemetry traces across the governor → pipeline → agent → PR lifecycle.
- Broadening the guardrail model — an optional content/prompt-injection classifier and a proxy-level request-rate limiter.