Book a 30-min call
← cd ../blogs
$ cat posts/how-we-built-hallucination-detection-pipeline.mdx

How We Built a Production Hallucination Detection Pipeline

April 14, 2026 · updated August 8, 2026 · ImmovableTech Team

  • Production AI
  • Evaluation
  • Retrieval

The problem: content at scale needs fact-checking

When an enterprise content platform processes tens of thousands of articles, social posts, and marketing assets every day, small inaccuracies compound fast. A single hallucinated statistic in a published piece erodes reader trust. Multiply that across 30,000+ pieces daily and you have a brand-risk problem that manual review cannot solve.

Our client needed an automated verification layer that could sit between content generation and publication, flag factual inconsistencies, and do it without slowing down the editorial pipeline.

Why we chose a multi-agent architecture

The obvious first approach — run each piece through a single LLM prompt and ask “is this factually correct?” — fails in practice for three reasons:

  1. Monolithic prompts hallucinate about hallucinations. A single model evaluating its own outputs (or outputs from a similar model) tends to rubber-stamp plausible-sounding claims.
  2. Different verification tasks need different strategies. Checking a statistical claim requires source retrieval. Checking logical consistency requires structural reasoning. Checking attribution requires entity resolution. One prompt cannot do all three well.
  3. Latency budgets vary by content type. A 500-word social post needs sub-second verification. A 5,000-word report can tolerate 10 seconds. A monolithic pipeline cannot adapt.

We chose LangGraph because it gave us explicit control over the agent execution graph — which stages run in parallel, which are conditional, and where human review gets injected.

Where agents are now

We built this pipeline before MCP became the default standard for tool-calling agents. MCP was donated to the Linux Foundation’s Agentic AI Foundation in December 2025 with over 97 million monthly SDK downloads behind it, and the core SDKs have grown several-fold since. If we were rebuilding today, we’d structure the retrieval and scoring agents as MCP tool servers rather than tightly coupled LangGraph nodes, so that any MCP-compatible client could invoke our verification tools without model-specific function-calling code.

We would not reach for A2A here, and that’s a change of view from how we first wrote this up. The claim-extraction-to-retrieval handoff looks like an agent-to-agent handoff on a whiteboard, but both stages run in one deployment and share a typed state object. Putting a network protocol between them would buy us serialisation overhead, a second failure domain and a worse trace, in exchange for an interoperability we have no use for. A2A is the right tool when the two agents are operated by different teams. Ours aren’t.

The core architecture — decomposed agents with explicit routing — holds up well. The wiring between them is what we’d modernise, and less of it than we originally thought.

The pipeline: three stages, five agents

Stage 1: claim extraction

The first agent parses incoming content into discrete, verifiable claims. A 2,000-word article might yield 15-40 claims ranging from “Company X reported $2.3B in Q3 revenue” to “This approach reduces latency by 40%.”

We used structured output parsing with a strict JSON schema so every downstream agent receives claims in a consistent format — claim text, claim type (statistical, causal, attributive), confidence that the claim is actually a factual assertion (vs. opinion), and the source sentence.

Stage 2: source retrieval and cross-reference

For each extracted claim, a retrieval agent searches a curated knowledge base and, where permitted, the open web. We built three retrieval strategies:

  • Vector similarity search against an internal knowledge base (Pinecone) for domain-specific facts
  • Structured database lookups for financial figures, dates, and named entities
  • Web search fallback for claims that reference recent events or external data

The retrieval agent returns ranked evidence passages with relevance scores. Claims with no retrievable evidence get flagged as “unverifiable” rather than “false” — an important distinction that reduced false-positive rates by 34%.

Stage 3: factuality scoring

The scoring agent takes each claim paired with its retrieved evidence and produces a factuality verdict: confirmed, contradicted, partially supported, or unverifiable. It also generates a human-readable explanation citing the specific evidence that supports or contradicts the claim.

We fine-tuned this stage on 8,000 manually labelled claim-evidence pairs from the client’s domain. The fine-tuned model outperformed the zero-shot frontier model we had been using by 11 percentage points on our evaluation set.

Key engineering decisions

Batching over streaming. Content arrives in bursts. We batch claims into groups of 50 for retrieval and scoring, which reduced per-claim latency from 3.2 seconds to 0.8 seconds by amortising embedding and API call overhead.

Async pipeline with Redis queues. Each stage runs independently. If the retrieval service is slow, claim extraction continues filling the queue. This decoupling let us scale each stage independently and eliminated cascading timeouts.

Confidence thresholds, not binary gates. Rather than blocking all content below a threshold, we route to three paths: auto-publish (high confidence), editor review (medium confidence), and auto-hold (low confidence). This preserved editorial velocity — 78% of content passes through without human intervention.

Continuous calibration. Editor corrections feed back into the training set. Every month we re-evaluate the scoring model against the latest corrections and retrain if accuracy drops below 90%.

Under the hood: LangGraph agent definition

Here’s a simplified version of how the verification graph is wired together:

from langgraph.graph import StateGraph, END
from typing import TypedDict, List

class VerificationState(TypedDict):
    content: str
    claims: List[dict]
    evidence: List[dict]
    verdicts: List[dict]

graph = StateGraph(VerificationState)
graph.add_node("extract_claims", extract_claims_agent)
graph.add_node("retrieve_evidence", retrieve_evidence_agent)
graph.add_node("score_factuality", score_factuality_agent)
graph.add_node("route_decision", route_decision_agent)

graph.set_entry_point("extract_claims")
graph.add_edge("extract_claims", "retrieve_evidence")
graph.add_edge("retrieve_evidence", "score_factuality")
graph.add_edge("score_factuality", "route_decision")
graph.add_conditional_edges("route_decision", decide_next, {
    "auto_publish": END,
    "editor_review": END,
    "auto_hold": END,
})

pipeline = graph.compile()

The key thing to notice: the graph is deterministic. We know exactly which node runs after which. Unlike prompt-chained agent systems, there’s no ambiguity about execution order — and when something fails, we know exactly where.

Results

After six weeks from kickoff to production:

  • 30,000+ content pieces verified daily without editorial bottleneck
  • 92% factuality accuracy on our held-out evaluation set
  • 78% auto-publish rate — most content passes through without manual review
  • 34% reduction in false positives compared to the single-prompt baseline we benchmarked against

What the 92% does and does not mean

That headline number needs a caveat we should have published with it the first time. The 92% is raw agreement between our pipeline’s verdict and the human label on a held-out set, across a four-way verdict space that is heavily skewed towards “confirmed”. Raw agreement on a skewed label distribution flatters any system, because a large share of the agreement is available by chance — always guessing the majority class scores well above zero.

A chance-corrected statistic such as Cohen’s κ is the honest measure, because it divides out the agreement you would get from both raters simply favouring the same majority label. The largest systematic study of LLM judges to date — 21 judges from nine providers, roughly 541,000 individual judgments — found that every single judge’s exact-match score overstated its Cohen’s κ, by between 33.8 and 41.3 percentage points on MT-Bench, and that judge rankings shifted by as many as 14 positions depending on which benchmark you used. Their headline example is blunt: a judge reporting 85% agreement on MT-Bench has a κ of about 0.48. We have no reason to think our pipeline is immune to the same arithmetic.

So read 92% as “agrees with our editors most of the time on our content mix”, not as “is right 92% of the time”. The number that actually justified the system to the client was different anyway: the false-positive reduction and the 78% auto-publish rate, both of which are measured against a baseline rather than against chance, and both of which map directly onto editor workload. If you are standing up something similar, report a chance-corrected figure alongside agreement from the start — retrofitting the honesty later is a worse conversation.

Lessons learned

Start with evaluation, not architecture. We spent the first week building the labelled evaluation dataset before writing a single line of pipeline code. Every architecture decision was tested against this dataset, which prevented us from shipping a system that “felt right” but scored poorly.

Multi-agent is not always better. We initially had seven agents. We merged two and eliminated one when analysis showed they added latency without improving accuracy. The final five-agent graph was the result of pruning, not additive design.

The “unverifiable” category is your best friend. Forcing binary true/false verdicts on every claim creates noise. Allowing “unverifiable” as a legitimate outcome made the system more trustworthy to editors, which drove adoption.

What we’d do differently

Report chance-corrected agreement from day one. Covered above, and it’s the one we’d fix first. We shipped a number that was true and misleading, which is worse than shipping a smaller number that is hard to argue with.

Use different model families for extraction and verification. We ran both stages on the same model family, which means correlated failure: if the extractor and the verifier share a blind spot, the verification step is theatre. Two independent families give you a genuine second opinion. We didn’t discover this from first principles — we found a class of claim where the pipeline was confidently, consistently wrong, and the common cause was that the same model had produced and then blessed the reasoning.

Add traces from day one. We retrofitted observability in week four and immediately found the scoring agent making three times more API calls than necessary on long-form content, because it was re-embedding evidence passages it had already seen. That’s the kind of waste you only catch with end-to-end tracing, and we spent three weeks debugging blind for no reason.

Expose the pipeline as an MCP server. The verification pipeline is only reachable through our custom frontend and internal API. Exposing it as an MCP tool server would let any compatible client call it — an editor’s assistant, an external content platform, a CMS integration. The pipeline is the product; the interface shouldn’t be the bottleneck.

References

This project is part of our AI & Machine Learning Engineering practice. Read the full case study or talk to us about a similar challenge.