Your First Model Needs a Release Process, Not a Notebook

The model worked in the notebook. Then it reached production and started returning nonsense for requests containing empty fields.

Nobody had tested that path because the training data contained no empty fields. The API stayed healthy, Kubernetes reported green pods, and the dashboards showed normal CPU usage. From an infrastructure perspective, everything was fine. From the user's perspective, the product was broken.

This is where teams discover what MLOps is actually for.

It is not a fashionable layer of tooling around machine learning. It is the engineering needed to answer four uncomfortable questions: what exactly did we deploy, why is it behaving differently, can we prove the replacement is better, and how quickly can we put the previous version back?

For a team shipping its first model, the goal is not to build an elaborate platform. The goal is to create a short, repeatable path from experiment to production that leaves evidence behind.

Treat the model as one part of the release

A model artifact on its own is nearly meaningless. Its behaviour also depends on preprocessing code, feature definitions, tokenizer versions, inference parameters, system prompts, retrieval settings, and the data used to train or evaluate it.

That becomes especially obvious with LLM applications. Change a prompt template, embedding model, chunk size, or RAG retrieval filter and you have changed the product, even if the underlying model name remains identical. Agents add more moving parts: tool schemas, permissions, retry rules, memory, and termination conditions.

Version the complete inference package. At minimum, every release should identify:

• Application commit SHA
• Model or provider model version
• Training dataset or evaluation dataset version
• Prompt and configuration version
• Container image digest
• Feature or preprocessing code version

Put those identifiers somewhere queryable. Image labels are useful, but runtime metadata is better because operators can inspect the running service without guessing what the deployment manifest resolved to.

BUILD_INFO = {
 "git_sha": os.environ["GIT_SHA"],
 "model_version": os.environ["MODEL_VERSION"],
 "eval_set": os.environ["EVAL_SET_VERSION"],
 "image_digest": os.environ["IMAGE_DIGEST"],
}

@app.get("/version")
def version():
 return BUILD_INFO

Do not use latest for models, datasets, prompts, or containers. latest is an instruction to forget what happened.

Your pipeline needs gates that understand model behaviour

Normal CI tests still apply. Test the API contract, authentication, timeouts, dependency failures, and malformed input. Then add tests for the things ordinary application code does not expose.

For a classifier, that might mean checking per-class precision and recall against an approved evaluation set. A single aggregate score can hide a bad release. For a forecasting model, compare error across useful time windows, not only the full dataset. For a RAG service, test whether retrieval returns the expected documents before grading the generated answer. Otherwise, you will spend hours blaming the LLM for a broken index or filter.

Keep the first gate simple and deterministic:

model_gate:
 dataset: eval/customer-intent-v3.jsonl
 candidate: models/intent-2026-08-07.onnx
 thresholds:
 macro_f1_min: 0.84
 max_regression: 0.02
 p95_latency_ms: 180

The exact thresholds belong to the product and its failure costs. Set them before seeing the candidate's result. Moving a threshold after a failed run is not evaluation. It is negotiation with yourself.

Generative AI evaluation is less tidy, but that is not an excuse to skip it. Start with a small set of real, sanitised tasks and record expected properties. Does the answer cite retrieved material? Does it refuse requests outside scope? Can an agent call only the tools permitted for that role? Does private AI data stay out of logs and external traces?

Automated scoring helps with volume. Human review is still needed for ambiguous failures. Store both the input and evaluation result so a regression can be reproduced later.

Production telemetry must describe answers, not just containers

Prometheus can tell you that inference latency increased. It cannot tell you that a fraud model stopped flagging a particular transaction type or that an agent began looping between two tools.

Infrastructure metrics remain necessary: request rate, errors, saturation, queue depth, GPU memory, token usage, and dependency latency. Add model-level signals alongside them.

Depending on the system, useful signals include prediction distribution, confidence distribution, missing feature rates, retrieval hit rate, documents returned per query, tool call count, refusal rate, and structured-output validation failures.

Be careful with labels. Putting user IDs, prompts, document names, or model responses into Prometheus labels creates cardinality trouble and a security problem. Detailed traces belong in controlled storage with retention rules, redaction, and access logging.

For externally hosted LLMs, log the provider request ID when available. During an incident, that identifier is often more useful than another screenshot of an error message.

Deploy so rollback is boring

A model release should be independently reversible. If rolling back requires rebuilding an image, fetching an unversioned artifact, or asking the person who trained it which file they used, the release process is unfinished.

Package smaller models inside the immutable container image when practical. For large artifacts, fetch by exact version and verify a checksum before serving traffic.

sha256sum -c /models/model.sha256
exec uvicorn app:api --host 0.0.0.0 --port 8080

Use a readiness probe that performs a cheap model operation, not merely an HTTP response from the web framework. A process can be alive while its model is absent, corrupt, or incompatible with the runtime.

Roll out gradually. A Kubernetes canary deployment, service-mesh traffic split, or application-level shadow request can all work. The mechanism matters less than comparing the candidate with the current release on the same traffic shape.

Shadowing deserves caution. Duplicating requests into a candidate environment can duplicate personal or commercially sensitive data too. For private AI systems, confirm that the shadow path has the same network controls, logging policy, and data handling rules as production.

Define rollback triggers before deployment. Error rate and latency are obvious. Model-specific triggers might include a sudden change in prediction distribution, a fall in retrieval success, or an increase in agent tool failures. Someone also needs the authority to roll back without arranging a committee meeting while customers wait.

Keep the platform smaller than the problem

Teams sometimes respond to their first model by installing an entire MLOps catalogue: experiment tracking, feature stores, orchestration, model registries, specialist serving frameworks, and several dashboards nobody owns.

Start with the gaps you actually have. Git, object storage with versioning, a container registry, CI, Kubernetes, and Prometheus may be enough. MLflow is useful when experiment and model lineage has become difficult to manage. DVC can help when datasets need versioning alongside code. KServe or Seldon can earn their place when serving patterns and rollout controls are repeated across teams.

Every platform component creates upgrades, permissions, backups, and incident paths. AI engineering is still production engineering. The tool must remove more operational work than it creates.

If you only do one thing this week

Create a release manifest for the model currently closest to production. Record the code commit, artifact checksum, evaluation dataset, configuration, container digest, owner, and rollback target. Make the deployment emit that manifest through a version endpoint or structured startup log.

That one change will expose the missing pieces quickly. If you cannot fill in a field, you have found an unmanaged dependency. Fix those dependencies before adding another orchestration layer.

Your first model does not need a grand platform. It needs provenance, evaluation, observable behaviour, and a rollback path that works when everyone is tired.

If this is on your plate, TecLeads does exactly this as part of our AI & Applied ML work. If you'd like a second pair of eyes on your setup, book a 30-minute call or explore what we do.