Self-Hosted AI Agents: Running the Model Inside Your Cluster

Self-hosted AI agents look like this. The model runs as a pod in the same cluster as the workloads it reports on. Weights sit on a PersistentVolumeClaim, pulled once at startup and never fetched again. Beside it, a small HTTP service holds a ServiceAccount that can get and list pods, events and logs, and nothing else — so when an engineer asks why a deployment keeps restarting, the service queries the Kubernetes API first and hands the live state to the model inside the prompt. Nothing about the cluster crosses the network boundary. The architecture below follows a published open-source walkthrough of exactly this setup by engineer Maryam Tavakkoli — an Ollama pod serving Mistral 7B behind a FastAPI pod, with a read-only ClusterRole in between.

When your data cannot leave your own network

Most AI tooling sold to platform and operations teams is a hosted service. You point it at your telemetry and it ingests your pod events and log lines, then hands back advice. For plenty of companies that trade is fine and the product is good.

For a hospital group, or a payments company whose auditors will ask where production logs went, the trade is not on offer. The project dies in the vendor security review, for a reason unrelated to product quality: production data has to move somewhere the contract cannot fully follow, and a named person has to sign for that.

Self-hosting takes the question off the table. There is no new sub-processor for your register, and no argument to have about what a vendor retains for model improvement. The security review becomes a review of a workload your team already knows how to reason about, running under a ServiceAccount you issued, in a namespace you control. The reviewer’s questions become questions about RBAC and network policy.

That is a smaller claim than it sounds. Where the data genuinely cannot leave, it is also the entire commercial case. If your data can leave, a hosted frontier model is stronger and takes less of your team to run, and you should use it. Settle the hosted-versus-custom trade-off on paper before anyone provisions a GPU node.

What a small local model is good enough for

Two questions that look almost identical put very different demands on a language model.

Ask a model what ImagePullBackOff means and you are asking it to recall something from training. Open-weight models in the 7B class are mediocre at recall next to the frontier, and prompt engineering does not close that gap. Recall is a capability you buy.

Ask which of your pods is in that state right now and why, with the pod list and its recent events and log tail already in the prompt, and the task narrows: read this text, find the pattern, say it back in order. A small model is often enough for that. In the walkthrough the answer comes back as “Pod API-7b8d has restarted 14 times in the last hour with ImagePullBackOff against registry.local. Run kubectl describe pod API-7b8d to confirm.” Not one fact in that sentence is something the model had to know. The restart count came from the API. So did the registry name. The model supplied the ordering and the English.

Which sets the design rule for anything self-hosted: the model is never the source of facts. Every fact in the output should already be in the prompt, put there by code you wrote and can point at. A hosted frontier model will sometimes carry a thin prompt on general knowledge alone. A local 7B model handed the same thin prompt invents, so the retrieval step has to be tight enough that inventing is never necessary.

That constraint decides which internal jobs qualify. Summarizing a case file, or drafting a first-pass answer out of documents you supply — grounded work over data you hand it. Open-ended reasoning and long multi-step planning still belong to the large models.

Why retrieval matters more than model size

The retrieval step is where the engineering actually sits, and it is the part a vendor demo never shows.

For a cluster agent it means deciding, in code, exactly what gets fetched before the model sees anything. In the published walkthrough that is every pod in the target namespace with phase and restart counts, the reason any of them is waiting, the last ten events, and twenty lines of logs from any pod not in a Running state. Those bounds are the product. Fetch less and the model fills the gap with plausible fiction. Fetch everything and the context window goes on healthy pods nobody asked about, and the answer degrades in a way that reads as the model being stupid.

There is no vector database in that loop. The retrieval target is a live API queried at request time, which beats an embedding index for anything whose truth changes minute to minute — cluster state, or an order status. An index built at midnight is wrong by breakfast and gives no sign of it. Index the documents that sit still, query the systems that move. If the choice in front of you is grounding versus training a model on your own data, retrieval versus fine-tuning covers where that line falls.

A retrieval layer that returns clean, structured context is model-agnostic. Freeze the context schema before you pick a model tag, so swapping the tag later is a one-line change and a re-run of your evaluation set.

Give the first agent read access only

A ServiceAccount with get and list and nothing else is what makes a first version deployable without waiting on an AI governance committee. The API server enforces the boundary, and no output the model produces can cross it. If the agent hallucinates a delete, the call fails at authorization. The walkthrough’s author puts the reasoning plainly: hallucinations multiplied by write access is a poor combination.

Read-only also makes iteration cheap. The worst case is a wrong answer, which an engineer catches by reading it. Prompts and model tags become things you try on a Tuesday afternoon rather than things that need a change-advisory board. Write access gets earned later, one verb at a time, each with its own rule and its own review.

Read access carries its own exposure, and the RBAC boundary does nothing about it. An agent that can read across log streams and document stores becomes an extraction path once someone can get text into its context, and for a support-facing agent that is anyone who can file a ticket. Scope reads per source and redact before anything hits a log. Record which documents each run retrieved — the same discipline that keeps an agent’s write permissions defensible later.

Where the prompt and the model version should live

In Git, in the same repository as the manifests.

The system prompt is configuration. When someone rewrites the line that tells the agent how to behave, that change belongs in a pull request, reviewable and revertable, the way a Helm value is. You can git blame a regression in agent output the same way you would git blame a broken chart value. Kept in application code or an environment variable, the prompt becomes drift nobody can explain to an auditor.

The model tag is configuration too, and it should be pinned to a digest. Between model versions the judgment shifts before anything visibly breaks, which is the hardest class of regression to spot — the agent still answers and still sounds right, while weighing the evidence differently. Pin the version. Keep a set of real cases with known-good answers and re-run them before a new tag reaches production.

Do that and git log becomes the behavioral audit trail — the artifact a regulated review asks for, and the one nobody has when the prompt lives in a .env file. Wiring this into an existing GitOps pipeline is DevOps and cloud work more than AI work. Keep the prompt file next to the image tag and the Role, so one pull request moves all of them or none.

What a self-hosted AI agent costs

The compute footprint is modest: one pod with enough memory to hold the weights, plus a lightweight API pod beside it and a volume for the model. No per-token bill and no egress line item. That is the part finance notices first.

Operational cost replaces it. Someone owns model updates and the evaluation set. Someone decides when CPU inference stops being acceptable and a GPU node has to be budgeted. None of that shows up in a vendor comparison, and all of it lands on a team that already carries an on-call rotation. The question worth asking is which named person absorbs those two jobs, because if the answer is nobody, the deployment rots quietly and still passes its security review.

Build cost is bounded, because a read-only agent with a scoped retrieval layer is bounded work — the systems it touches get listed before anything is written. The cheapest honest way to buy it is the first slice on its own. In our agent cost guide that is Tier One — Pilot, ₹1L – ₹1.5L: one job, proven against real past cases in a sandbox, with a written verdict on whether it holds up. A full self-hosted deployment for data that legally cannot leave sits at the top of that same ladder, and the guide is explicit that it does not start there without a passed pilot. Your figure comes out of a paid discovery against a written scope.

When a hosted model is the better call

If nothing legal or contractual stops your data leaving, let it leave. You get a stronger model and a shorter route to production, with no GPU capacity planning to own.

Local hosting is also the wrong answer when the task genuinely needs frontier reasoning. Hand a small model a vague question with thin context and it returns something confident and wrong; the fix for that is a bigger model, not a better prompt.

And if the task is a fixed sequence with fixed rules, a model of any size is an expensive way to run an if statement. Deterministic workflow code is cheaper and easier to defend, so settle the agents versus automation question first — it decides whether a model belongs in the design at all.

Where to start with a self-hosted agent

Pick one question your team asks a system repeatedly and answers by hand: which of these is failing and why, or what this customer has already told support. Write the retrieval step first, before any model is involved — the code that fetches exactly the context a competent colleague would want, and nothing beyond it. Print the output, read it, and check whether you could answer the question yourself from that text alone. If you can, a small local model probably can too, and what remains is a Deployment and a scoped ServiceAccount, plus the evaluation set that tells you when a new tag has changed the agent’s mind. If you cannot, no model rescues it, and the work is still in the retrieval layer. That first slice is the pilot stage of an AI agent engagement, and it is the cheapest place to learn whether the idea holds at all.