What is human-in-the-loop review?

Human-in-the-loop review means building a person into an AI process at the points where judgement or accountability matters. IBM defines it as "a system or process in which a human actively participates in the operation, supervision or decision-making of an automated system". In AI, it means people check, correct or approve outputs before they count.

Why it is still needed

AI systems are useful long before they are reliable enough to act alone. TheAgentCompany, a benchmark first published in December 2024 that simulates a small software company, found that "the most competitive agent can complete 30% of tasks autonomously". Agents have improved since, but some outputs still need a person before they move money or reach a customer.

Where to put the person

Review adds most where errors are costly and hard to catch later: a price change, a payment, a volume that settles a contractor's pay. It adds least where errors are cheap and caught downstream anyway.

Checking everything has a known weakness. IBM notes that people "can get tired, distracted or confused". The research on automation bias describes "the propensity for humans to favor suggestions from automated decision-making systems". A reviewer faced with a long queue of mostly correct results may start to approve without looking.

Routing and its limit

The usual answer is to route only what needs judgement: low-confidence fields, anomalies and large amounts. That depends on the model knowing when it is wrong, which it often doesn't. So add a random sample of what was accepted automatically. The sample measures the error rate among results nobody reviewed. For fields that move money, add deterministic checks, such as invoice lines against the total.

What the reviewer needs

The source beside the result, such as the scanned ticket next to the extracted weight. A clear choice to accept, correct or reject. A record of each decision, so anyone can see who approved what.

In a timber business

Good candidates are extracted figures that settle payments, changes to master data and prices, and AI-drafted messages to customers or contractors. Routine lookups rarely need review.

How Quarri works explains the platform as a layer over existing systems, not a migration.

Quarri is an AI-native data platform for the timber supply chain. It connects buying, production, sales and inventory for forest management, sawmill, wood products and pulp, paper and packaging operators.