The production-readiness checklist for AI agents

Shipping an agent to production is not the same as shipping a service. The checklist that keeps a web service healthy misses the failure modes that hurt agents most. Here is the one that does not, grouped by the three questions you have to be able to answer.

Matt King

July 2, 2026 9 min read

There is a checklist most teams use to decide a service is ready for production. Health check, error tracking, latency budget, runbook, on-call rotation. It is a good checklist, and for an agent it is not enough, because it was built for things that fail loudly and an agent's worst failures are silent. The agent returns valid-looking output that is wrong, stops running without erroring, or runs up a bill without any single request failing. None of those trip a normal checklist.

This is the checklist that does. It is organised around the three questions you have to be able to answer about an agent in production at any moment: is it running, what is it costing, and is it any good. Each question maps to a failure mode that does not throw, which is exactly why each needs its own signal. The three are the pillars the rest of this blog keeps coming back to, drawn together in three blind spots when running AI agents in production.

The foundation: one event per run

Before the three questions, the thing they all rest on. Every signal below derives from a single primitive: one event per agent run, carrying enough to identify, cost, and place it in time.

A run id, client-generated, so writes are idempotent and you can correlate the run across systems.
An agent_id, stable, so spend and quality can be grouped by agent.
started_at and finished_at in client time, plus a server received_at.
A status of ok, fail, or partial.
A cost, or the model and token counts to compute it against a rate card.

That is the floor. From those fields alone you get cost attribution, schedule freshness, and a baseline to alert on. Adding inputs and outputs unlocks quality scoring and drill-down. Get the per-run event right and everything else is a query on top of it; skip it and no amount of downstream tooling can reconstruct what you did not capture.

Question 1: is it running?

The failure: a scheduled agent goes silent. No exception, because it produced nothing to throw. Standard monitoring is blind to it because you cannot poll for an event that did not happen.

Every scheduled agent has a declared expected cadence (a cron expression or interval).
A heartbeat fires on every run, reporting ok or fail, so both "stopped running" and "ran and failed" are visible.
A grace window is set per job, tight for low-variance jobs, looser for jobs that legitimately vary.
A missed window pages a real destination, routed by severity (PagerDuty for customer-facing, Slack for internal).
The missed-run alert carries the inputs the run would have taken, so on-call can replay or amend rather than just acknowledge.

This is schedule freshness, and it is the cheapest pillar to instrument and the most expensive to skip. The worked version is catch a silent AI agent before your customer does.

Question 2: what is it costing?

The failure: cost runs away while every individual request succeeds. A retry loop, a ballooning context window, a model left too large for the job. The invoice tells you a month late and at the account level, which is too late and too coarse.

Cost is attributed per agent, computed at run time against a rate card, not read back from the provider bill.
A spend baseline is set per agent, and a deviation from it alerts, rather than only a fixed monthly budget.
Cost-per-successful-run is tracked, so waste hidden inside a healthy average surfaces.
Where the agent knows them, customer_id and feature are tagged, so cost-to-serve and per-feature cost are answerable.
Multi-agent calls propagate a parent run id so the call tree's cost rolls up correctly.
Cached-token counts are captured if prompt caching is on, so a broken cache shows up as a spend anomaly.

This is the Spend pillar. The incident it prevents is the anatomy of an £8,900 token bill.

Question 3: is it any good?

The failure: quality drifts. Everything runs, the bill is normal, the output is just worse. It is the failure that hides longest because nothing about it looks like a failure, and it surfaces in your support inbox rather than your dashboard.

Deterministic checks run on every run: schema match, required fields, value ranges. Free, and they catch structural breakage immediately.
An LLM judge runs on a sample for nuance, against an anchored rubric you wrote and validated.
The rubric is versioned and runs are scored against the version live when they ran, never silently re-scored.
A calibration set exists and is replayed when the rubric or judge model changes.
The judge model is pinned to an explicit version, not a floating alias.
Drift detection runs on the score distribution, with deploy markers, so a quality drop is attributable to the change that caused it.

This is the Verify pillar, covered in depth in how to write an LLM rubric that survives a model upgrade and LLM-as-judge in production.

The cross-cutting pieces

A few items that do not belong to a single question but gate readiness all the same.

Scoped ping tokens for anything that reports in from a cron or external tool; the full API key reserved for your own application code behind secrets management.
Run ids generated client-side before the first network call, so a write can be retried idempotently and correlated even if the response is lost.
The observability path cannot block or crash the agent: bounded queue, hard timeout, drop rather than stall. Telemetry that can take down the thing it measures is worse than none.
At least 30 days of run retention, so an incident can be investigated after the fact and a baseline has history to compare against.

How to read the checklist

You do not have to land every box before launch. The honest minimum for a customer-facing agent is: the per-run event, schedule freshness on anything scheduled, per-agent cost with a baseline, and deterministic checks on output. That covers the silent failure, the runaway bill, and structural quality breakage, which are the three that hurt soonest. The judge layer, the calibration set, and per-feature attribution can follow within the first weeks.

What you should not do is launch with none of it and plan to add observability after the first incident, because the first incident is precisely the thing the checklist exists to prevent, and by then you are reconstructing from provider logs instead of reading a dashboard. The three questions are not a maturity model to grow into; they are the definition of being able to operate the agent at all.

Every item here is something you can build yourself or get from AgentPing, where one SDK call emits the per-run event and the three pillars (Pulse, Spend, Verify) derive from it. Either way, the checklist is the same, because the failure modes are the same. Get started and put a live signal on all three questions before your agent meets its first real incident.

What makes an AI agent production-ready?

An agent is production-ready when you can answer three questions at any moment: is it running, what is it costing, and is its output any good. Each maps to a failure mode that does not raise an exception and so will not be caught by ordinary application monitoring: a scheduled agent going silent, cost running away, and quality drifting. Readiness is having a live signal for all three before launch, not the ability to investigate them after an incident.

How is an agent readiness checklist different from a normal service checklist?

A normal service checklist assumes failures are loud: crashes, error rates, latency spikes, all of which throw or show up in standard metrics. An agent's worst failures are silent. It returns valid-looking output that is wrong, it stops running without erroring, its cost climbs without any single request failing. So the agent checklist adds three things a service checklist does not have: schedule freshness, per-agent cost attribution, and continuous output quality scoring.

What is the minimum instrumentation to ship an agent safely?

One event per agent run carrying a run id, an agent id, start and finish times, a status, and either a cost or the model and token counts to compute it. Those few fields give you cost attribution, schedule freshness, and a baseline to alert on. Adding inputs, outputs, and a rubric unlocks quality scoring. You do not need all of it on day one, but you do need the per-run event, because every other signal derives from it.

Do I need quality scoring before launch or can it wait?

You need at least the cheap layer before launch: deterministic checks on every run for shape and required content, which cost nothing and catch structural failures immediately. The LLM-judge layer for nuance can follow shortly after, on a sample. What you should not do is launch a customer-facing agent with no output signal at all, because quality drift is the failure mode that hides longest and surfaces in your support inbox rather than your dashboard.

What credentials and identifiers should be in place before launch?

Client-generated run ids so writes are idempotent and you can correlate across systems, scoped per-agent tokens for anything that reports in from a cron or external tool, and the full API key reserved for your own application code behind secrets management. Getting the identifier and credential model right before launch avoids a painful retrofit later, because run ids in particular are hard to backfill once traffic is flowing.