Where To Put The Human: Designing The Review Step In An AI Workflow

Every workflow that puts a language model in charge of a task eventually forces the same design decision: where does a human sit?

Teams tend to answer it by default rather than by design. Either everything routes through an approver — and the “automation” becomes a queue of work a person still has to touch — or nothing does, because the demo worked, and the first confidently wrong output reaches a customer before anyone has a plan for it.

The question is not whether a human should be involved. It is where, how often, and with what information.

Three positions, and they are not interchangeable

Before the action (blocking approval). The agent prepares the work; a person releases it. Nothing leaves unseen. This is the safe position and the expensive one: throughput is capped by reviewer availability, and the automation’s real contribution is drafting rather than deciding.

After the action (audit). The output ships, and a person reviews a sample afterwards. Throughput is uncapped and errors still get caught — but only the ones the sample finds. Wrong output that reaches a customer before the audit is a cost you have consciously accepted.

On exception (escalation queue). The agent handles ordinary cases alone and hands a person the items it cannot handle. Best of both worlds, but only if “cannot handle” is defined honestly. A system that guesses instead of admitting uncertainty never escalates, never queues anything, and never tells you it is struggling.

Match the position to the cost of being wrong

Two questions settle most cases.

If the output is wrong, what happens next? An incorrect internal classification is found and corrected by the people who use it. An incorrect email to a customer, invoice or regulator cannot be unsent. Reversible and internal → review after. Irreversible and external → review before.

How would we find out? If errors surface through a downstream check — a reconciliation, a validation rule, a second system that rejects bad input — you can afford sampling. If the only detector is the customer complaining, you cannot.

Where the answer to both is “badly and late”, the honest conclusion is that the task is not ready to be automated at all. That is a finding, not a failure: pick a workflow with a cheaper error profile first — the same logic that governs choosing a first candidate at all.

Four review patterns that actually work

Full review. Every item, every time. This is the right setting for the first couple of weeks of any new workflow, whatever the eventual design will be — not because it is sustainable, but because it is the only way to learn what reviewers actually reject.

Threshold review. The agent passes items that meet a condition — a validation rule, a value below a set amount, a confidence score — and holds the rest. Workable, but treat the threshold as a dial you tune against real review outcomes, not as a line the model draws. Calibrate it against what humans reject, not against what the model claims to be sure of.

Sampled review. A fixed share of all traffic, plus full coverage of defined categories: high-value items, new counterparties, inputs outside the usual shape. Sampling gives you a continuous read on quality without gating throughput.

Exception queue. The agent declines and routes the item to a person. Cheap to run and easy to get wrong, because an escalation without context is simply another task to reconstruct from scratch.

Build the escalation payload deliberately

When an item lands with a human, that person needs four things on one screen: the input, what the agent proposed to do, why it stopped, and a suggested next action.

An escalation that forces the reviewer to re-read the source document, re-run the reasoning and then decide has not saved time. It has moved the work, added a context switch, and created a queue people learn to avoid.

The queue is also the cheapest improvement backlog you will ever have. Every entry is a real case the system could not handle, with a real frequency attached — which makes it the natural companion to the four numbers that tell you whether an agent is working. The count tells you how much human load the workflow creates; the contents tell you what to fix next.

Watch for approval theatre

The characteristic failure of human review is not that it is unsafe. It is that it stops being review.

A person approving the same shape of item hundreds of times in a row will stop reading them. Rubber-stamping is the worst of both worlds: you keep the latency and headcount of full review while losing the quality benefit. If a reviewer has rejected nothing in weeks, one of two things is true — the system has become genuinely reliable, or nobody is looking. Either way the setting should change: relax the review if it is reliable, fix the queue design if nobody is looking.

The opposite mistake is review that never ends. If every workflow you ship keeps a permanent 100% sign-off step, you have automated the drafting and kept the deciding. That can be a legitimate design — but be honest about it, because it is not a labour-saving automation and should not be measured as one.

Start wide, then relax with evidence

The reliable sequence:

1. Ship at full review for a fixed period. At any decent volume, two weeks is enough.
2. Read what reviewers actually changed — not what they were asked to check, what they corrected.
3. Automate the categories they never touch; keep human review on the ones they do.
4. Revisit whenever volume, input shape or the underlying model changes.

Every relaxation step is backed by a sample you took yourself, which is the only evidence that survives the question “why does this no longer need sign-off?”.

Human review is not the opposite of automation. It is the part of the design that decides whether an agent earns trust quickly or loses it slowly — and it is the placement decision we work through with teams as part of our services. The difference between a workflow that ships and one that stalls in approval is rarely the model. It is where the human was put.