Giving Rhodes write access to my infrastructure

Rhodes is an agent that operates my lab infrastructure. It polls Proxmox and vCenter, notices when a resource changes state, builds a remediation plan, and — for a narrow, explicitly enumerated set of actions — executes that plan without asking me. Everything else stops and waits for a human.

It's a TypeScript/Node service, MIT licensed, 185 commits at the head I'm writing from. The README expands the name as "Reasoning, Hybrid Orchestration, Deployment & Execution System." It's been operating against real hardware since May, minus a 52-day outage I'll get to. The positioning I locked after a week of research is "an AI brain for infrastructure" — not an AI SRE, not a DevOps employee, not a migration tool.

The planning was never the hard part. Give an LLM a tool schema and it will produce a plan for anything, confidently. The hard part was deciding what a plan is permitted to do on its own, and enforcing that decision somewhere the model can't talk its way past.

Five tiers, fail closed

Every action routes through one call — GovernanceEngine.evaluate() — and the order inside it is fixed: circuit breaker, tier classification, forbidden checks, guardrail checks, approval decision. Five tiers: read, safe_write, risky_write, destructive, never. `never` is blocked and not approvable. There is no operator override for it.

Classification isn't static per tool. classifyAction() starts from the tool's declared tier and elevates on params:

• count > 1 → risky_write
• RAM > 16384 MB → risky_write
• disk > 500 GB → risky_write
• more than 3 targets → risky_write
• force: true or skip_checks: true → destructive

A tool the classifier has never seen gets `risky_write`, not `read` — the declared tier is read with a fallback: `let tier = tool?.tier ?? "risky_write"`. On top of that, seven name patterns are blocked unconditionally, regardless of who approves:

delete_all
format_storage
modify_host_config
disable_firewall
destroy_cluster
wipe*
rm_rf*

What runs without me

The line: restarting a stopped VM is autonomous. Destructive or cross-system actions — snapshot deletion, config changes, migrations — are gated. My own note from the day I drew it reads "thats good if a vm goes down it dosent need approval to bring it back up."

That line lives in the rule table, not in prose. `vm_auto_restart` ("Auto-restart stopped VMs") is tier safe_write with a 120-second cooldown, and it only ever targets VMs that are already stopped. The rule that restarts a VM which is running but failing its health probe is a separate rule at a higher tier, and the comment says why:

tier intentionally raised: the VM is RUNNING-but-unhealthy, so a power-cycle is materially riskier than the start_vm rule which only ever targets stopped VMs. Forces governance every time.

There's also a rule that deliberately does nothing:

No automatic remediation — adapter connection failures usually mean creds or network, neither of which autopilot can fix safely.

That one matters more than it looks. The temptation with an autonomous agent is to attach a remediation to every condition it can detect. An adapter that can't reach its provider is precisely the case where the agent's model of the world is least trustworthy, so the rule is tier read and it pages me instead.

The safety snapshot is the guardrail

The Proxmox storage-pause playbook — the one that recovers a VM QEMU has suspended on an I/O error — has a shape I now reuse. buildRemediationPlan() prepends a snapshot step before any delete step:

qm snapshot <vmid> rhodes-safety-<ISO-timestamp>

and appends a cleanup step for the previous rhodes-safety-* snapshot. That cleanup runs only after a successful resume and verify. A failed resume leaves the older safety net intact.

A retention floor then excludes the newest non-`current` snapshot from the deletable set, and snapshots with no created_at timestamp are treated as oldest, so the floor can't accidentally preserve an undated entry while pruning a dated one. If only one snapshot is in scope, the plan comes out empty and degrades to "no deletable candidates — escalate to operator." Free more space, lose the safety net is not a trade the agent gets to make unilaterally.

The hard rules are enforced in code, not in the prompt. validateRemediationCandidate() rejects a plan step before it can execute:

"Hard rule: never delete active VM disks (vm-*-disk-*)."
"Hard rule: qm destroy is Tier 5 (NEVER), permanently blocked."
"Hard rule: lvremove on a non-snapshot LV is blocked."
"Hard rule: rhodes-safety-* snapshots can only be deleted via the explicit safety-snap cleanup step."

The comment above it states the threat model plainly: "This guards against an LLM-generated plan that confuses an lvs row for a snapshot." And Rhodes' own safety snapshots are protected from Rhodes — the only path that can delete one is the cleanup step that passes the exact prior snapshot name.

Shadow mode, then the first real save

The production posture before write access was shadow mode:

Shadow / dry-run mode. When true, Tier-1 (read) actions still execute,
but Tier-2+ (safe_write, risky_write, destructive) actions are PLANNED
and LOGGED but NEVER executed.

Reads hit the real cluster. The autopilot polled, classified, built plans, alerted. It just didn't mutate anything. That let me read what it wanted to do before it could do it.

The first real autonomous run was on 2026-05-14, shortly after 02:00 UTC. A nested ESXi host VM had been suspended by QEMU on a storage I/O error. Rhodes hadn't been running when it happened — it found the state at boot and logged "was already in paused_io_error state at RHODES boot — synthesizing discovered_state_change." By then the condition had been open 5.45 hours.

It produced an 11-step plan and graded every step:

• steps 1–6, read: get_vm_status, get_vm_config, list_snapshots, get_node_stats, list_storage, search_logs
• step 7, safe_write: create_snapshot named rhodes-safety-2026-05-14
• step 8, risky_write: an explicit approval checkpoint
• step 9, destructive: delete_snapshot
• step 10, risky_write: resume
• step 11, read: verify

The plan-level gate was raised at tier destructive and the agent sat there. wait_ms: 178650 — two minutes and fifty-eight seconds waiting on me to click Approve. Then it ran.

The safety snapshot landed. The post-run listing shows rhodes-safety-2026-05-14, description "Safety snapshot taken before resuming paused VM due to io-error", with `current` pointing at it as parent. The incident closed on the string "VM esxi-01 state recovered: paused_io_error → running."

Step 10 failed. The planner had chosen `ssh_exec qm resume` over the typed resume tool, and the agent host had no SSH key to the hypervisor node, so it came back exit 255. I finished the resume by hand through the API. Step 9 had already been a no-op — no stale snapshots to prune.

The reason it shelled out is in the plan's own reasoning field: "Step 10 uses ssh_exec to run qm resume directly on the Proxmox node, which is more reliable than attempting to use a non-existent Proxmox API tool." The typed tool existed. Nothing in the planner prompt or the tool descriptions steered the model toward it, so it reached for the command a human would type and then justified the choice with a fact it invented. The fix was hard rule #11 in the planner prompt listing the six typed Proxmox lifecycle tools and forbidding ssh_exec qm when one exists, plus naming the qm equivalent in every lifecycle tool description. Two endpoints that existed on the executor but not the adapter — suspend_vm and reset_vm — had to be added so the typed path was actually complete.

Two things about that run I still can't fully pin down. My own note says dry-run mode was already off; a correctness audit written the same day records the live API still returning dry_run: true and recommends holding the flip until an approval-gate bug was fixed — and I can't reconstruct from the artifacts which order those actually happened in. The dates don't line up cleanly either: a later commit calls this the eleven-step esxi-01 save from May 13th, while the SSE log timestamps read May 14th at 02:08 UTC. Almost certainly a local-versus-UTC gap, but I didn't verify it.

The gate that wasn't a gate

A correctness audit that same week found the thing that actually scared me. ApprovalGate kept `decisions` and `pendingResolvers` keyed by plan_id alone. requestApproval() looked up plan_id, found the earlier plan-level record, and resolved the per-step promise as approved without ever raising a new gate.

So: approve a plan once, and a step inside it classified destructive — a tier listed in explicit_tiers precisely so it forces per-step human review — executes silently, recorded as approved_by "api_operator". The tests missed it because they covered plan-level approval and a single step-level approval, never the combination.

The fix is a composite key:

`${planId}::step:${stepId}`   vs   `${planId}::plan`

with `_step_id` injected alongside `_plan_id` on each step's params, governance forwarding both, and step_id carried through the dashboard POST body and the SSE broadcasts. The commit body names the stakes: the 11-step save "included a deliberate Approval checkpoint step before the destructive delete_snapshot. Under v0.4.5 that step would have been silently auto-approved by the plan-level decision."

A gate you believe in that doesn't fire is worse than no gate at all. I'd have kept approving plans, faster each time.

Knowing who did it

During a live smoke test I stopped a VM from the CLI on purpose. Rhodes opened an incident and its LLM wrote a root cause: "95.28% memory utilization → OOM killer."

Nothing in the pipeline knew who or what caused a state change, so every stop read as a crash and the model filled the causal gap with telemetry-shaped fiction. That's the failure mode that makes an agent unusable in ops — the postmortem reads exactly like a diagnosis, and it's fabricated.

Attribution shipped as v0.6.5. Per-substrate event-source adapters (Proxmox task log, vCenter event API) write into a SQLite store, and a correlator matches the observed transition against recent events on the same resource before the narrative is written. The transition table is explicit, not learned: running→stopped expects vm_stop or vm_delete, running→paused expects vm_suspend, running→unreachable expects vm_migrate or host_disconnect.

Confidence is a three-rung ladder. `high` = exact resource id plus an event type matching the transition inside a 30-second window. `medium` = same resource inside the 300-second lookback but the event type doesn't match. `low` = provider-level activity with no specific resource id — defined, and deliberately never used, because it's too easy to mis-attribute.

The goal, verbatim from the code: show "stopped by <operator> via Proxmox UI at 14:32" instead of "crashed."

It's still log-only at HEAD. shouldSuppress() exists and nothing calls it — the correlator annotates the incident after it opens rather than deciding whether it opens at all. The module header still describes the suppress-first design, which is a fair reading of intent and a wrong reading of the wiring.

Four bugs between a Slack button and an executed plan

Slack is the approval surface for anything that isn't me at the dashboard, and it's filtered rather than a firehose. Only approval_needed, execution_failed, health_check_failed, the three ticket kinds and upgrade_approval reach the team channel; per-step execution_complete spam is silently dropped. plan_generated is intentionally omitted as noise on a public channel. Anything carrying a thread timestamp bypasses the allowlist, because that's a reply scoped to a specific operator conversation.

Getting one Approve click to actually execute an upgrade took four fixes, in this order.

One: the click did nothing. The confirm modal rendered `*cluster*` and backticked version strings with literal asterisks and backticks, I clicked Approve, a blue spinner ran about three seconds, then cleared with no error. The request never reached the interactivity URL. The confirm dialog's text field was type: "mrkdwn"; Block Kit requires plain_text for confirm dialogs. The wrong type passes validation when the message is posted and fails only when a human clicks — Slack drops the callback with no error to the user and no failed-delivery indication in the app dashboard. Fixed, with a comment carrying the incident context so a later "make it prettier" pass can't flip it back, plus a regression test asserting every confirm sub-field is plain_text.

Two: the click landed, HTTP 200, correct run id, and nothing executed. The orchestrator SQLite DB had two runs at phase="pending", current_host_index=-1, started_at=null. approveUpgradePlan called createRun then drive(), and createRun seeds at pending — the FSM requires an explicit `approve` event to leave it, so drive() inferred action "none" forever. Every approval was persisting an approval record and an orphan run. The runner's own test fixture already documented the requirement, with the comment "Move past pending — drive() picks up at approved." The bootstrap had the same requirement and no fixture to remind it.

Three: the whole flow was absent from production. The wiring had been added inside `case "dashboard"` of the mode switch; the service runs `full`. The tell was a boot log line that never printed. The fix extracted the block into a helper called from both modes, and the helper now reports which of three states it's in:

[rhodes] Upgrade orchestrator: ON

The commit is blunt that no test could catch it — the bug is reachability, and the only confirmation that matters is that boot log surfacing after a deploy.

Four: the same class recurred two months later. `full` mode never called attachAttributionCorrelator; only dashboard mode did. So the attribution pollers ran and collected actor data that nothing consumed. Same shape, different subsystem. I don't have a structural fix beyond asserting on the boot log.

The parts that hold it together

Two subsystems did most of the work of making the rest testable. The graph shipped as one commit — 33 files, 7,419 insertions, 138 new tests — a SQLite-backed model of Resources, Relationships and Conditions with per-type closed state enums that each carry a mandatory `unknown` member. Cross-provider identity uses explicit typed `manifests_as` edges instead of trying to merge records: each provider gets its own Resource and an edge connects the perspectives, with a separate resolver inferring those edges so they can be audited apart from directly observed ones. The whole thing shipped behind an env gate, defaulting off, so the substrate couldn't destabilize the running deploy.

It also shipped broken in a way that only showed up against live vCenter: 1 of 8 VMs discovered and zero vSphere relationships. getVmPlacement and getHostPlacement were stubs returning empty ids, so every runs_on / member_of edge failed its foreign-key check — and because the error escaped the VM loop, one bad edge aborted the entire inventory pass. Implementing placement by inverting the filtered list endpoints, and making relationship upserts skip-and-record instead of abort, took vSphere relationships from 0 to 79 against a live 4-host vCenter.

The orchestrator ships its FSM separately from its runner on purpose. `transition(run, event) → { nextRun, nextAction }` is pure — no I/O, no primitive calls, no DB writes. The runner does all the I/O and feeds results back through transition(). The stated reason: "This keeps the FSM testable in isolation: thousands of transitions per test, deterministic, no mocks." Phases run pending → approved → preflight → executing → (completed | rolling_back → failed) | aborted, with per-host substates inside executing, and four invariants the FSM enforces — terminal phases reject everything idempotently, `abort` wins in any non-terminal phase, currentHostIndex is -1 outside executing, timestamps are set on entry and exit.

Where it runs, and what's still broken

Rhodes does not run on the infrastructure it manages. I learned that the expensive way: the box hosting it dropped off the network in June and never came back, and Rhodes was down 52 days. An orchestrator hosted on the cluster dies exactly when its self-heal should fire. It now lives on a small dedicated machine outside the cluster, dashboard bound to the private network only, API calls 401 without a token, and only the Slack webhook path publicly reachable — which now verifies request signatures: HMAC-SHA256 over `v0:{ts}:{body}`, five-minute replay window, timing-safe compare, failing closed when no signing secret is configured. Before that, those routes were exempt from operator auth because an upstream relay shim was assumed to be verifying Slack signatures before forwarding. The shim no longer existed. An unverified POST there is remote control of the fleet.

Open items, honestly:

• The secret scanner that gates every commit hangs on large staged diffs. The 33-file, 7,419-insertion graph commit reproducibly pinned it at 100% CPU on a ~260KB staged candidate; two orphaned scanner processes from that run were still burning CPU four days later, 5,935 minutes each. I worked around it with --no-verify after manually reviewing the staged files. At HEAD the script still has no timeout and no watchdog — I checked the code rather than re-running it, so I can only say the fix isn't in, not that the hang is gone.
• package.json still says 0.4.7 while the feature work is at v0.7.3.5. Git tags stop at v0.5.0.
• 165 test files. The last count I recorded in a commit message was 2,665 passing and 16 skipped, plus one known pre-existing dashboard-static failure; several commits have landed since, including 86 new tests for the network self-heal path, so the real total is a little higher now.
• The upgrade orchestrator is the piece I can least prove end to end. The approval to runner to progress-thread path works and the per-host Proxmox primitives are implemented, but I have no artifact of a finished multi-host cluster upgrade — built, not yet proven.
• Serving the 20 MCP tools over streamable HTTP needed a fresh McpServer and transport per request — reusing one instance leaves handshake state stale, so `initialize` succeeds and every call after it silently returns empty. The stdio path was never affected.

There's one switch I haven't flipped. shouldSuppress() is written and tested and sits one call site away from live. The moment I wire it, Rhodes stops opening incidents for changes I made myself — which means it starts deciding on its own that a state change was intentional. That's a larger grant of authority than restarting a stopped VM, and the confidence ladder hasn't seen enough real data yet for me to hand it over.