Human-in-the-Loop AI Agents: When to Hand Off

"Should a person review it?" is the wrong question

Teams usually start with one of two positions. Either a person reviews everything the AI produces, or the AI runs on its own and someone looks at a dashboard now and then. Neither one holds up.

If a person reviews everything, you haven't automated the work, you've added a step to it. Worse, reviewers who approve hundreds of correct drafts in a row stop reading closely, so the one bad draft gets approved too. If nobody reviews anything, errors are silent, and in workflows like refunds, invoices or customer messages a silent error is one somebody else finds for you.

The useful question is narrower: which specific cases need a person, and how will the system know? That has an answer you can write down and test.

Four rules for when an AI agent hands off

Almost every handoff we've designed comes down to one of four triggers:

• The AI isn't sure. A confidence score below a threshold sends the case to a person. Be careful here: a confidence number the model reports about itself is a weak signal on its own. Set the threshold by checking it against cases where you already know the right answer, and never let it be the only rule.
• The AI contradicts the records. If the draft disagrees with facts your system can check directly, a person decides. In Proofly, if the model recommends contesting a chargeback but the merchant's database shows a refund was already issued, the case goes to manual review no matter how confident the model sounds.
• Being wrong is expensive or can't be undone. Sending money, deleting data, messaging a customer, changing a contract or anything with legal or medical weight gets a person by default. The test isn't how hard the case is, it's what a mistake costs and whether you can reverse it.
• The case is outside what the AI was built for. A new product category, a missing field, an unexpected language, or the model choosing "other" all mean the input doesn't look like anything you tested. Those go to a person until they're understood.

A case only proceeds on its own when none of the four fire. That keeps the automatic path to the cases you've actually tested, and sends everything else to someone who can judge it.

This is the difference people mean by human-in-the-loop and human-on-the-loop. For the flagged cases, a person approves before anything happens: they're in the loop. For everything else, the agent acts inside its guardrails and a person supervises from the side, through logs and spot checks: they're on the loop. Most agentic workflows need both, and the four rules decide which mode each case gets.

Approval gates belong in code, not in the prompt

It's tempting to write the handoff rules into the prompt: "If the customer has an open return, recommend manual review." We did exactly that in Proofly, and models still confidently recommended contesting orders with an unresolved return. A prompt instruction is a strong suggestion. It is not a guarantee.

So the rules run after the model answers, as ordinary code that checks the draft against data the system read for itself. Here's the shape of it, simplified from patterns we use:

type Draft = { action: string; confidence: number; category: string };
type Facts = { refundIssued: boolean; knownCategories: Set<string> };

const IRREVERSIBLE = new Set(["issue_refund", "delete_record", "send_to_customer"]);

// Returns the reasons a person must look. Empty means the case can proceed.
function handoffReasons(draft: Draft, facts: Facts, threshold: number): string[] {
  const reasons: string[] = [];
  if (draft.confidence < threshold) reasons.push("Low confidence");
  if (draft.action === "contest" && facts.refundIssued) reasons.push("A refund was already issued");
  if (IRREVERSIBLE.has(draft.action)) reasons.push("This action can't be undone");
  if (!facts.knownCategories.has(draft.category)) reasons.push("Unfamiliar case type");
  return reasons;
}

Two details matter. First, the function returns reasons, not just yes or no, because the reviewer needs to know why the case landed on their desk. Second, the facts come from your own systems, not from the model's description of them. A model saying "no refund found" is not the same as your database saying it.

Design the handoff, not just the trigger

Sending a case to a person is only useful if that person can act on it quickly. A good review screen shows:

• The AI's draft, ready to approve as-is, so the reviewer edits rather than starts over.
• The reason it was flagged, in plain language, taken straight from the rule that fired.
• The evidence the draft was based on, with links back to the source records.
• Clear choices: approve, edit and approve, or reject with a short reason.

This is also roughly what regulators are asking for. Article 14 of the EU AI Act says high-risk AI systems must be designed so a person can understand what the system is doing, interpret its output, override it and stop it. Even if your workflow isn't high-risk under the Act, it's a good bar for any review screen.

Handoffs also need an owner and a deadline. A queue nobody watches is just a slower way of dropping cases. If the work has a time limit, like a chargeback response window, the queue should show what's due first.

Keep an eye on both paths

Cases that proceed automatically still need checking. Pull a random sample of them on a regular schedule and have a person review those too. That's how you catch the mistakes your rules didn't anticipate.

On the human side, watch for rubber-stamping, also called automation bias: trusting the AI because it's usually right. If reviewers approve nearly everything and spend only a few seconds on each case, either the rules are flagging too much or people have stopped reading. Both are worth fixing, and both show up in your logs if you record how long each review took and what changed.

Every human decision is training data for your rules

Each time a person approves, edits or rejects a draft, you learn something. Save the case, the draft and the final decision. Over time that becomes your evals: a test set you can run every prompt or model change against before it ships.

It also tells you where to move the line. If people approve a certain kind of case unchanged, week after week, that kind of case may be ready to run on its own. If they keep correcting another, it needs a tighter rule or a better prompt. Start with more handoffs than you think you need and loosen them as the evidence builds up. Going the other way means finding the gaps through mistakes.

A short checklist

Before an AI workflow goes live, we check that we can answer each of these:

• Which actions are irreversible, and do they always go to a person?
• Can a person override or stop the agent, and do they know how?
• Which facts can the system check for itself, and does the code compare the AI's draft against them?
• What confidence threshold are we using, and which known cases was it checked against?
• What does an unfamiliar case look like, and where does it go?
• Who owns the review queue, and what's the deadline?
• How are automatic cases sampled for spot checks?
• Where are human decisions saved, and when do we review them?

See how these rules work in a real build in our Proofly case study and the Proofly engineering post. If you're planning an AI agent or workflow and want help deciding where the handoffs and guardrails go, take a look at our AI automation work or get in touch.