Your In-Cluster LLM Is Still Trying to Phone Home

A familiar failure mode goes like this. The LLM is running on your Kubernetes cluster, the API has no public ingress, and everyone agrees the data is private. Then somebody checks the firewall logs.

The model server has been calling a model registry. The tracing library is exporting spans to a hosted service. An agent tool can reach arbitrary internet addresses. Prompts are sitting in application logs beside bearer tokens and retrieved document fragments.

The inference happened in-cluster. The data boundary did not.

That distinction is the whole job. Running a private LLM is not mainly about putting a GPU pod behind a ClusterIP service. It is about proving every path by which data, metadata, credentials, and model artifacts can cross the boundary.

Start by deciding what private actually means

Before choosing a model or inference server, write down the boundary. Be painfully literal.

Does private mean prompts cannot leave the cluster? Can encrypted telemetry leave? Are model weights allowed to download at pod startup? Can an agent call a public search API? May administrators inspect prompts during an incident? Which namespaces, storage systems, and backup locations are inside the boundary?

If nobody answers those questions, the architecture will answer them accidentally.

For a strict private AI deployment, I usually start with a stronger position: runtime workloads get no general internet egress. Exceptions must name a destination and a reason. That forces hidden dependencies into view early, while they are still cheap to remove.

Kubernetes does not deny egress by default. A baseline policy should.

apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata:
 name: deny-all-egress
 namespace: private-ai
spec:
 podSelector: {}
 policyTypes:
 - Egress

This only works if the cluster networking implementation enforces NetworkPolicy. Verify that rather than admiring the YAML. Cilium, Calico, and cloud CNI implementations differ in features and operational details.

Run an egress test from the actual namespace and service account:

kubectl -n private-ai run egress-test \
 --rm -it --restart=Never \
 --image=curlimages/curl \
 --serviceaccount=llm-runtime \
 -- https://example.com

For a closed runtime namespace, success is a failed connection. Add narrow policies for internal DNS, object storage, the vector database, and an OpenTelemetry collector only when the workload needs them.

Model weights are part of the supply chain

Many otherwise private deployments download weights from Hugging Face or another registry whenever a pod starts. That creates an availability dependency, weakens reproducibility, and gives the runtime somewhere external to talk.

Fetch the model during a controlled build or promotion process. Scan the files, record the license, pin the revision, calculate digests, and place the approved artifact in an internal registry or object store. Do not use a moving branch name such as main as a production version.

For Hugging Face based runtimes, offline mode helps expose accidental downloads:

env:
 - name: HF_HUB_OFFLINE
 value: "1"
 - name: TRANSFORMERS_OFFLINE
 value: "1"
 - name: HF_HOME
 value: /models/cache

Those variables are guardrails, not proof. Enforced egress policy is the proof.

The same discipline applies to container images. Pin images by digest, generate an SBOM, scan them, and promote approved images through an internal registry. A model server built from an unpinned image is not reproducible MLOps. It is a surprise scheduled for a later date.

Serving the model is the easy bit

vLLM is a sensible default for many decoder-only models because it provides an OpenAI-compatible API and handles continuous batching. Other workloads may fit NVIDIA Triton, Text Generation Inference, llama.cpp, or a custom PyTorch server better. Pick from measured workload behavior, model support, and operational fit, not whichever project had the loudest release week.

A minimal GPU request looks like this:

resources:
 requests:
 cpu: "4"
 memory: 24Gi
 nvidia.com/gpu: "1"
 limits:
 nvidia.com/gpu: "1"

That is not capacity planning. Account for weight memory, KV cache, context length, concurrency, quantization, and headroom during rollout. A second replica is useful only if another suitable GPU can actually schedule it.

Keep inference nodes in a dedicated pool. Use taints and tolerations, restrict who can launch GPU workloads, and monitor GPU memory, queue depth, time to first token, token throughput, request failures, and pod restarts. Avoid logging prompt bodies just because they make debugging convenient at 2 a.m. Convenience has a habit of becoming retention policy.

RAG creates more exits than the model does

A RAG system adds an embedding model, document ingestion, chunk storage, a vector database, retrieval, reranking, and prompt assembly. Each stage can leak data without the LLM doing anything wrong.

Keep embeddings local if the source material is sensitive. Treat embeddings as derived sensitive data, not harmless numeric soup. Restrict vector database access by workload identity, encrypt its volumes and backups, and separate collections where authorization boundaries differ.

Authorization must happen before retrieved text reaches the prompt. Filtering results after retrieval is too late. The model has already seen them.

Ingestion deserves the same scrutiny. A document parser may execute external helpers, follow URLs, or consume hostile files. Run parsers with tight CPU and memory limits, a read-only root filesystem, no unnecessary service account token, and no egress. Scan uploaded files before processing them.

Agents need a shorter leash

Agents turn model output into actions. That changes the risk substantially.

Do not give an agent a general HTTP tool and call the system private. Put tools behind typed internal APIs with explicit authentication, authorization, input validation, timeouts, and audit records. Use allowlisted destinations for any outbound connector. High-impact actions should require approval outside the model loop.

The agent should receive the least authority needed for the current task. A service account that can read every namespace because the prototype needed one secret is how prototypes become incident reports.

Private agents also need protection from prompt injection. Retrieved documents and tool responses are untrusted input. They must not be able to rewrite tool policy, expose credentials, or expand the agent's permissions. Tool enforcement belongs in code and infrastructure, never solely in a system prompt.

Observability can quietly undo the design

Prompt logging is useful until somebody pastes a production token, medical record, source file, or contract into the chat box. Then the log platform becomes another sensitive data store.

Record operational metadata by default: model version, request ID, latency, token counts, result status, queue time, and policy decisions. Make content capture an explicit, access-controlled diagnostic mode with short retention and a visible audit trail. Apply the same rule to traces and exception reporting. SDK defaults deserve inspection.

Good AI engineering joins application controls to the same hardened cloud foundations used for other production systems. Identity, network policy, admission control, secret handling, encrypted storage, image provenance, patching, backups, and incident response still apply. An LLM does not suspend any of them. It merely creates new places to get them wrong.

If you only do one thing this week

Apply default-deny egress to a non-production copy of the full AI stack, including RAG ingestion, agents, telemetry, and model startup. Then watch what breaks. Every failed connection is a dependency you can name, remove, or permit deliberately. That exercise will tell you more about whether your private AI is actually private than another architecture diagram ever will.

If this is on your plate, TecLeads does exactly this as part of our AI & Applied ML work. If you'd like a second pair of eyes on your setup, book a 30-minute call or explore what we do.