The Four Numbers That Tell You An AI Agent Is Actually Working

There is a specific moment in most AI deployments where everyone stops looking. The pilot worked, the rollout happened, and then the thing just… runs. Nobody checks it, because nobody decided what “working” would look like.

Months later someone notices the output quality has drifted, or the bill has tripled, or a workflow has been quietly failing on a subset of inputs for six weeks. Nobody noticed because there was nothing to notice it with.

The fix is not a monitoring platform. It is four numbers, tracked daily, on one screen.

1. Attempted

How many items did the agent try to process yesterday?

This is your volume baseline, and it is the first number because without it the others are meaningless. “98% success rate” on 4 items is noise. The same rate on 4,000 items is a system.

It also catches the failure mode nobody expects: the agent silently stopped receiving work. A broken upstream job, an expired credential, a cron that stopped firing — these do not raise errors. They just make the count go to zero. If you are not watching attempted volume, a dead integration looks exactly like a quiet day.

2. Succeeded

How many completed correctly, by whatever definition you agreed on?

The hard part is “correctly”. For a structured task — extract these six fields, classify this ticket — you can validate automatically. For anything fuzzy you need either a schema, a confidence threshold, or periodic human sampling.

What you must not do is use “the agent returned something” as the definition of success. A language model will always return something. Confident nonsense counts as success under that definition, which makes the metric worse than useless.

3. Escalated

How many items went to a human?

This is the number teams most often skip, and it is the most actionable. An escalation log is a prioritised backlog of your next improvements: each entry is a real item the system could not handle, with a real frequency attached.

Watch for the two failure shapes:

• Escalation rate too high — the automation is not earning its cost. Either narrow the scope or improve the prompt/validation.
• Escalation rate suddenly too low — often worse. It usually means the agent has started guessing instead of admitting uncertainty. Plausible-but-wrong output does not escalate; it just flows downstream.

A healthy system has a stable, explained escalation rate. Not zero.

4. Cost per task

What does one processed item cost in model and infrastructure spend?

This is the number that decides whether the whole thing scales. A workflow that is economically brilliant at 100 items/day can be ruinous at 10,000 if the cost curve is not flat.

Track it as a ratio, not a total, so growth does not mask degradation. Then set two limits:

• A per-task ceiling that triggers an alert when breached — usually a sign of retry loops or a prompt that has grown.
• A hard daily cap that stops spend outright. Without this, one runaway loop can consume a month’s budget in an afternoon.

Put them on one screen

Four numbers, daily, somewhere you will actually see them:

attempted   succeeded   escalated   cost/task
   1,842        1,791         51         $0.031

That single line answers “is it working?” without opening anything. And the trends are where the value is — attempted flat while escalated climbs means the input distribution has shifted. Cost per task creeping up means retries are growing. These are invisible in a dashboard you never open.

Why four and not forty

Instrumentation is subject to the same trap as the rest of software: the dashboard that tries to show everything gets abandoned. Four numbers get looked at. Forty get bookmarked and forgotten.

You can add more later, once the four are habit. The order matters: volume, correctness, human load, unit cost. That is also the order in which they will surprise you.

The uncomfortable question

If you cannot answer all four right now for an AI system you have in production, that is the finding. Not a monitoring gap — an observability gap, and it means you are running an automation you do not actually understand.

That is not a reason to panic. It is a reason to spend an afternoon adding a counter, a log line, and a small daily report. The numbers are cheap. The alternative — finding out from a customer — is not.