Agents are a new class of workload. Private cloud is where you contain them.

A while back I built an infrastructure agent, wired it up to real credentials against a lab cluster, and then spent an evening trying to talk it into doing something destructive.

It worked. I asked it to delete production and it agreed.

The governance layer I had written was not broken. It did exactly what I designed it to do: it classified every proposed action by risk and it knew that destroying a VM was a dangerous verb. What it did not know was that this particular VM was the one thing in the environment that could not come back. I had authorized verbs. The agent was operating on nouns.

I wrote that incident up at the time (the full postmortem is here). The bug itself was a one-afternoon fix. The uncomfortable part took longer to sit with, and it is the reason this series exists.

Every security control I had reached for was built for software that computes. Agents do not compute. They act.

Two conversations, one gap

The AI security discussion right now runs on two tracks, and neither one covers the thing that will actually page you at 2 AM.

The first track is model security. Prompt injection, jailbreaks, training data poisoning, extraction attacks. Important work, genuinely. It is also almost entirely about what goes into and comes out of a model.

The second track is data security. Where do the embeddings live, who can query the vector store, does the RAG pipeline leak one tenant's documents into another tenant's answer. Also important. Also, again, about information.

Here is what is missing. An agent with a kubeconfig, a service account token, and a vCenter session is not an information system. It is an operator. It holds credentials, it makes plans, it calls APIs, and it changes the state of your infrastructure while nobody is watching. Prompt injection is not the interesting failure mode there. The interesting failure mode is a perfectly reasonable model, doing exactly what it was asked, against a target it did not understand.

Nobody is jailbreaking your agent. Your agent is going to be helpful, and that is the problem.

What an agent actually is, in infrastructure terms

Strip away the framing and an agent is a workload with three properties that nothing else in your estate has at the same time:

It holds standing credentials. Not a request-scoped token issued for one transaction. A durable identity that lets it act whenever it decides to act, including at 3 AM when the self-healing loop fires and no human is in the room.

Its action space is emergent. A microservice has an API surface you can enumerate. You can read the OpenAPI spec and know every call it will ever make. An agent's next call is a function of a model, a prompt, a tool registry, and whatever context it just pulled in. You can enumerate the tools. You cannot enumerate the plans.

It is nondeterministic under identical conditions. Run the same job twice against the same cluster state and you can get two different plans. Both might be correct. One might not be. Your entire testing methodology assumes this is not true.

Now look at how your existing controls are built.

Least privilege assumes you can write down what a workload needs in advance. RBAC authorizes an identity to perform a verb on a resource type, which means delete pod looks identical whether it is a rolling restart or an outage. Audit logs record what happened, in past tense, with no record of why the actor thought it was a good idea.

Three foundational controls, all of them quietly assuming the actor is deterministic software or a human being who can be asked to explain themselves later. An agent is neither.

Why this lands differently in private cloud

I want to be careful here, because the easy version of this argument is a cheap shot at hyperscalers, and that is not the argument.

Public cloud is very good at this problem right up until the boundary you need to enforce sits below the line where your responsibility ends. When your agent runs in a managed Kubernetes service, calling a managed model endpoint, over a network you configure but do not own, the containment surface you actually control is the IAM policy and the security group. That is a real surface. It is also thin, and it lives entirely at the application layer.

Agentic workloads need boundaries at layers most teams do not normally get to touch: the scheduler, the network data path, the storage policy, the snapshot chain, the identity broker. On VMware Cloud Foundation running vSphere Kubernetes Service, you own all of them. Not because VCF has some special AI security feature. Because the primitives that contain a nondeterministic actor turn out to be the primitives VCF has been building for fifteen years, and they are already sitting under your Kubernetes clusters.

That is the actual thesis of this series. The best agent containment platform available today is a well-configured private cloud, and most platform teams already own one without realizing what it is good for.

The three properties that matter

Prevention is the wrong goal. You cannot enumerate every bad plan an agent might generate, and if you try, you will build a control so restrictive that the agent stops being useful and your developers route around it. I have watched teams do this. The agent gets a read-only role, produces nothing but suggestions, and quietly gets abandoned inside a quarter.

The goal is not an agent that cannot do damage. The goal is an agent whose damage is bounded, reversible, and attributable.

Bounded authority. Not "what verbs may it call" but "what is the maximum extent of the change it can make before something stops it." Authority is a blast radius question, not a permissions question. Two agents with identical RBAC roles can have wildly different blast radii depending on which namespace they sit in, which storage policy backs their volumes, and which network segments they can reach.

Reversibility. Every action an agent takes should have a defined path back. Some of that is snapshots. Some of it is GitOps, where the desired state lives in a repository and the agent's change is a commit you can revert. Some of it is refusing to let the agent take actions that have no undo, which is a design decision you make at the tool registry, long before the model gets involved.

Attribution. When something breaks, you need to reconstruct not just the action but the reasoning. Which model version. Which prompt. What context was in the window. What the agent believed the cluster state was at decision time, and whether it was right. A Kubernetes audit log tells you a service account deleted a StatefulSet. It does not tell you the agent had stale state because a watch had silently dropped forty seconds earlier.

Bounded, reversible, attributable. Everything in the rest of this series maps back to one of those three.

Where VKS puts the boundaries

VKS is the successor to Tanzu Kubernetes Grid Service, and the important architectural detail for our purposes is where it sits. VKS clusters are provisioned by the vSphere Supervisor, a Kubernetes control plane embedded in vCenter, and the worker nodes are VMs that Cluster API bootstraps onto ESXi hosts inside a vSphere Namespace.

That last part is the one people skim past. The vSphere Namespace is not a Kubernetes namespace with a nicer UI. It is a resource-bounded partition of a cluster, enforced by the hypervisor, with its own quotas, its own storage policy assignments, and its own RBAC scope. When you drop an agent into one, the boundary is not advisory.

Here is the mapping I use when I am designing an agent deployment:

• Agent consumes more than its share of the cluster: vSphere Namespace resource quotas, enforced below Kubernetes
• Agent reaches a system it has no business touching: vDefend distributed firewall policy pushed into Antrea by label, plus distributed IDS/IPS at the pod level
• Agent's credential gets reused or copied: VCF Identity Broker with OIDC, short-lived projected service account tokens, no long-lived kubeconfigs on disk
• Agent destroys persistent state: vSAN storage policies, PVC retain semantics, snapshot and immutable snapshot policy on the backing datastore
• Agent quietly drifts the environment: Argo CD Operator running as a Supervisor Service, scoped per vSphere Namespace, with Git as the source of truth
• Agent burns the GPU budget on a runaway job: vGPU profile assignment and MIG partitioning, plus namespace-level quotas on GPU resources
• Nobody can explain what happened afterward: vCenter events, Kubernetes audit log, and the agent's own decision log, correlated on a shared trace ID

Not one row of that table is an AI feature. Every row is infrastructure you already have, applied to an actor it was not originally designed for.

The Kubernetes half of this is the part most teams get wrong, so a concrete example. This is the shape of the RBAC role I give an agent that is allowed to remediate workloads but not to remove them:

apiVersion: rbac.authorization.k8s.io/v1
kind: Role
metadata:
  name: agent-remediator
  namespace: platform-agents
rules:
  # Read broadly. Observation is cheap and the agent needs context to plan well.
  - apiGroups: ["", "apps", "batch"]
    resources: ["pods", "deployments", "statefulsets", "jobs", "events"]
    verbs: ["get", "list", "watch"]

  # Restart by patching, never by deleting. Same outcome, recoverable path.
  - apiGroups: ["apps"]
    resources: ["deployments"]
    verbs: ["patch"]

  # Named resources only. The agent cannot invent a target it was never scoped to.
  - apiGroups: ["apps"]
    resources: ["statefulsets"]
    resourceNames: ["ingest-worker", "index-builder"]
    verbs: ["patch"]

Two decisions in there are worth calling out.

The agent restarts pods by patching a deployment annotation rather than by deleting pods, because a patch is a state change you can read back and reason about, and a delete is a hole. The behavior is identical. The recoverability is not.

And resourceNames is doing quiet, heavy lifting. It is the difference between "may patch StatefulSets" and "may patch these two StatefulSets," which is the difference between an agent that can restart a stuck ingest worker and an agent that can restart your database because the model decided that was a reasonable interpretation of "fix the pipeline."

Egress is the other half, and it is the one that gets skipped. An agent calls model endpoints, tool servers, and increasingly MCP servers it discovered at runtime. Default-deny on egress with an explicit allowlist is not paranoid here, it is table stakes:

apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata:
  name: agent-egress-default-deny
  namespace: platform-agents
spec:
  podSelector:
    matchLabels: { role: agent }
  policyTypes: ["Egress"]
  egress:
    - to:
        - namespaceSelector:
            matchLabels: { name: model-runtime }
      ports:
        - { protocol: TCP, port: 8000 }

That NetworkPolicy is only the in-cluster half, and on VKS the other half is the part most Kubernetes people have never had access to.

VKS ships Antrea as its default CNI, and Antrea connects back to NSX through an adapter pod running inside the cluster. You author the distributed firewall policy once, in NSX Manager, against Kubernetes labels rather than IP addresses. The adapter translates it into AntreaClusterNetworkPolicy resources, and the Antrea agent programs it into OVS flow rules at the pod's virtual interface, which is the first hop the packet ever takes. Policy follows the pod. Reschedule it, scale it, let it come back on another node with a different address, and the rule still applies, because the rule was never about the address.

Antrea Egress is the piece that matters most for agents. It gives an authorized pod a dedicated, routable, predictable source address on the way out, so pod identity survives the trip across the cluster boundary and the infrastructure firewall can enforce on it too.

That leaves you with two independent enforcement points expressing one intent. The in-cluster policy decides which pods are permitted to egress at all. The infrastructure firewall independently validates the egress identity as traffic leaves. An attacker who lands inside the agent's pod has to defeat both, and the second one is not running anywhere they can reach.

Broadcom's own team has published a hands-on version of exactly this, micro-segmenting a NIM inference pipeline down to which pods may reach the vector database, with default-deny egress and Antrea Egress identity checked again at the infrastructure layer. It is a good piece of work and you should steal from it.

It is also, notably, entirely about packets. Every component in that architecture is a service that exchanges data with another service. None of them wake up at 3 AM and decide to delete a StatefulSet. The segmentation story for AI pipelines is well covered at this point. The authority story for AI actors is not, and that is the gap this series is walking into.

The part that makes private cloud non-optional

There is a version of this where the agent runs on VKS and calls a model API somewhere on the internet, and honestly that is where most teams will start. It is fine. It is also a strange architecture once you look at it directly, because you have carefully bounded the agent's authority over your infrastructure and then handed its entire reasoning process to a system you cannot audit, version-pin, or subpoena.

VCF 9 ships Private AI Services as a standard entitlement, and the pieces that matter for this conversation are the Model Store, the Model Runtime, and the Vector Database, all of them running as a multi-tenant layer on GPU-enabled workload domains. The agent's brain and the agent's hands end up inside the same governance boundary. The model version is a thing you pinned. The inference logs are yours. The context that went into a decision never left the building.

For regulated environments this stops being an architectural preference and becomes the only workable answer. For everyone else, it is the difference between a postmortem you can actually complete and one that ends with "the model probably did this because."

I will spend a full part of this series on that, because running the model next to the agent changes what attribution can look like, and most of the interesting work is downstream of that choice.

What is coming

This is Part 1 of a series. The plan:

Part 2: Identity. Killing the kubeconfig-in-an-environment-variable pattern. Workload identity for agents through the VCF Identity Broker, short-lived tokens, and what it takes to revoke an agent in under a minute at 3 AM.

Part 3: The network. Egress control for agents that call models and tools you did not write. NSX distributed firewall patterns, IDPS on east-west agent traffic, and the specific problem of MCP servers discovered at runtime.

Part 4: Reversibility. Snapshots, GitOps through the Argo CD Supervisor Service, and what "undo" means when the actor is nondeterministic. Includes the tool registry design question of which actions should simply not exist.

Part 5: Attribution. Building an audit trail that survives a real postmortem. Correlating vCenter events, Kubernetes audit logs, and agent decision traces into one story.

Part 6: Running the brain inside the boundary. Private AI Services, the Model Store, vGPU profile strategy for inference workloads, and why co-locating the model with the agent is a security decision before it is a performance one.

Why I actually care about this

I did not arrive here from a threat model. I arrived here because I want to hand my lab to an agent and go to sleep.

That is the honest version. I have a vSphere estate I care about and an agent called Rhodes running against it, planning changes, executing them, verifying its own work, and escalating when it hits something it should not decide alone. Every time I widen what Rhodes is allowed to do, I hit the same wall. I trust the model's judgment more than I trust my own ability to write down, in advance, every single thing it must never touch. Permissions are a language for describing the known. Agents work in the unknown. You do not close that gap by writing a longer Role, and I have tried.

So here is what I want, roughly in the order I want it.

An agent that can restart a stuck workload at 3 AM without asking me, and cannot destroy a datastore at any hour, where that difference is enforced by the platform underneath it rather than by a paragraph in a prompt. An agent whose network reach is a property of the namespace it was born into, not a config file it could rewrite. An agent whose every action has a defined path back, so that being wrong is survivable rather than terminal. And when it does something I did not expect, an audit trail that tells me what it believed at the time, not just which API it called.

None of that is exotic. Nearly all of it is sitting in VCF already, built for a completely different reason, waiting for someone to point it at a new kind of actor.

The one piece I would tell anyone to build first is the least interesting: a small evaluation harness. A suite of genuinely dangerous requests, run against your own agent, scored on whether it refuses, escalates, or cheerfully complies. Mine is embarrassingly simple and it has caught more real problems than any amount of architecture diagramming. You cannot bound something you have never tried to break.

More in Part 2.

Views are my own and do not represent my employer.