Book a 30-min call
← cd ../blogs
$ cat posts/reading-order-is-the-document-ai-metric.mdx

Reading order is the document AI metric that matters

June 23, 2026 · ImmovableTech Team

  • Document AI
  • Retrieval
  • Computer Vision
  • Evaluation

Every character correct, every answer wrong

A document parser can transcribe a two-column page with perfect character accuracy and still destroy the retrieval system sitting downstream of it. That is not a hypothetical. It is the most common way we have seen document RAG pipelines fail quietly, and the reason it stays quiet is that the metric everyone reports — character or word error rate — cannot see it.

The mechanism is simple. A model that flattens a page in raster-scan order reads left to right across the full page width, so on a two-column layout it takes the first line of column A, then the first line of column B, then the second line of column A, and so on. Every glyph is right. The CER is excellent. What comes out is two unrelated argument threads zipped together into a single block of text.

What the interleaving does to a chunker

Page (two columns)              Raster-scan output
┌───────────┬───────────┐
│ A1        │ B1        │       A1 B1 A2 B2 A3 B3
│ A2        │ B2        │       └─ one "paragraph", two arguments
│ A3        │ B3        │
└───────────┴───────────┘

The damage compounds at every stage after the parser. The chunker splits on token count, because that is what chunkers do, and it has no signal that the boundary between A1 and B1 is a topic boundary rather than a sentence boundary. So a chunk lands containing half of one argument and half of another.

Then you embed it. An embedding model given text that genuinely mixes two topics does not fail loudly — it returns a vector that sits somewhere between the two regions of the space, belonging to neither. That chunk is now a magnet for queries about either topic and a good answer to neither. At query time the retriever surfaces it with a respectable similarity score, the generator reads a passage that is fluent and grammatical and internally incoherent, and it produces a confident answer stitched from two sources that were never adjacent in the document.

This is the same class of problem we wrote about in our hallucination detection pipeline, with one nasty difference. There, the model invented something not in the source. Here the model is faithfully reporting text that your own pipeline invented, so a groundedness check against the parsed corpus passes cleanly. The corruption happened upstream of the thing doing the verifying.

What the benchmarks actually score

Here is where teams get misled, and it is worth being precise rather than sweeping, because the claim “no benchmark measures this” is false.

OmniDocBench does measure reading order. It annotates the reading sequence of every detected block — more than 16,000 reading-order annotations across 981 pages in the original release — and scores predictions with a normalised edit distance over the block sequence. That is exactly the right metric, and it exists.

What it does not do is put that metric in the headline number. The Overall score on the v1.5 leaderboard is defined as ((1 − TextEdit) × 100 + TableTEDS + FormulaCDM) / 3. Three terms: text edit distance, table structure, formula recognition. Reading order is reported in its own column and contributes nothing to the score everyone quotes. A model can rank near the top of that leaderboard while scrambling multi-column pages, and nothing in the number you compare will tell you.

The benchmark’s own 2024 evaluation shows how wide that gap gets. Broken out by column layout, the general-purpose VLMs of that generation degrade catastrophically as columns multiply: InternVL2 posts a reading-order edit distance of 0.082 on single-column pages, 0.312 on two-column and 0.682 on three-column. Qwen2-VL runs 0.098 to 0.248 to 0.517 across the same three. Their text recognition on those pages is fine. Their reading order collapses by roughly a factor of five to eight, and a single mean score across a corpus that is mostly single-column will hide all of it.

So the honest framing is not that reading order is unmeasured. It is that it is measured and then excluded from the ranking, which for practical purposes is worse — the number exists, so nobody thinks to look for it.

Finding it in your own corpus without a labelled benchmark

You almost certainly do not have reading-order ground truth for your documents, and you do not need it. Three diagnostics get you most of the way, in increasing order of cost.

The cheapest is an internal-coherence sweep over chunks you have already embedded. Split each chunk into sentences, embed them, and take the mean cosine similarity between consecutive pairs. Genuine prose is locally coherent; interleaved columns are not, because every sentence boundary is also a topic switch.

import numpy as np
from sentence_transformers import SentenceTransformer

model = SentenceTransformer('BAAI/bge-small-en-v1.5')

def coherence(sentences: list[str]) -> float:
    """Mean cosine similarity between consecutive sentences in one chunk."""
    if len(sentences) < 2:
        return 1.0
    embeddings = model.encode_document(sentences, normalize_embeddings=True)
    return float(np.mean(model.similarity_pairwise(embeddings[:-1], embeddings[1:])))

# The bottom of this list is where reading order broke.
suspects = sorted(chunks, key=lambda c: coherence(c.sentences))[:50]

This is a ranking, not a verdict — tables, reference lists and figure captions score badly for legitimate reasons. The point is to put the fifty worst chunks in front of a human in an afternoon rather than sampling randomly and finding nothing.

The second diagnostic is to stratify retrieval quality by layout. Tag every source page with its column count, which a layout detector gives you cheaply and which for many corpora you can infer from the source template. Then compute hit rate separately for questions whose gold passage came from a single-column page versus a multi-column one. If the multi-column stratum is materially worse, you have your answer, and you have it in the language your stakeholders already use. This is the diagnostic that changed minds for us, because a reading-order edit distance means nothing to a product owner and a retrieval hit rate split by page type means everything.

The third is to spot-read. Actually open the parsed markdown for your worst pages next to the source PDF. It is unglamorous and it is the only step that tells you which of several failure modes you have, because interleaved columns, dropped sidebars and captions welded into body text all look similar in aggregate metrics and need completely different fixes.

There is a fourth signal worth stealing from the DeepSeek-OCR 2 paper, which faces the same problem we do: in production there is no ground truth, so they track repetition rate as their primary observable quality metric, reporting 4.17% on online user-log images and 2.88% on batch PDF processing for the newer model. Degenerate repetition is a decent proxy for a model that has lost the thread of a page, and it costs nothing to compute on every document you parse.

Three ways out, in increasing order of effort

The cheapest fix is to stop using a flat-text parser on layout-heavy documents. A pipeline tool that runs layout detection first, extracts each region independently and then emits regions in a detected reading order will beat a raster-scan VLM on multi-column pages by construction, because it never had the opportunity to interleave anything. This is the approach behind the regulated claim-extraction engine we built on LayoutLMv3 and ColPali, and for structured, templated documents it remains the right default.

The middle path is to keep the region decomposition but make the ordering explicit and auditable — emit each region with its bounding box and an ordinal, chunk within regions rather than across them, and never let a chunk span a region boundary without a deliberate rule saying it may. The value here is less accuracy than debuggability: when retrieval goes wrong you can point at a region and an ordinal instead of guessing.

The most interesting option is a model that treats ordering as part of the architecture rather than a post-processing step. DeepSeek open-sourced DeepSeek-OCR 2 on 28 January 2026 (arXiv 2601.20552), and it was merged into Hugging Face transformers on 1 June 2026, so it is straightforward to try. Its contribution is DeepEncoder V2, which replaces the CLIP component of the previous encoder with a Qwen2-0.5B LLM-style encoder. Visual tokens get bidirectional attention; a set of learnable “causal flow” queries gets causal attention and can attend to every visual token plus the preceding queries. Only those reordered queries are passed to the decoder. In effect the encoder does the reading-order reasoning before the language model ever sees the page, instead of the language model trying to unscramble a raster-flattened sequence.

The design constraint we found most telling is the token budget. The model caps visual tokens at 1,120 per page — a deliberate ceiling, chosen to match Gemini-3 Pro’s maximum — against the 6,000 to 7,000 that comparable end-to-end models spend. The claim is not that ordering costs more compute. It is that ordering the tokens well is worth more than having five times as many of them.

On the results, be careful which number you repeat. The paper reports 91.09% Overall on OmniDocBench v1.5 with reading-order edit distance falling from 0.085 to 0.057 against its own predecessor. The OmniDocBench maintainers’ own harness run, published on the v1.5 leaderboard in March 2026, puts the same model at 89.17 Overall with reading order at 0.060. The two-point gap sits almost entirely in formula and table scoring rather than reading order, and the paper is explicit that its own two rows are self-run while every other row comes from the benchmark repository. The reading-order improvement replicates. Treat the headline as vendor-reported.

And it is not solved. On the paper’s own per-document-type breakdown, newspapers — the densest multi-column case there is — still sit at 0.176 reading-order edit distance, an order of magnitude worse than academic papers at 0.013. If your corpus looks like a newspaper, no current model rescues you from evaluating this yourself.

What we’d do differently

We would have looked at the parser output before touching the retriever. On a document pipeline where retrieval quality was visibly poor on a subset of sources, we spent real effort tuning the things that are easy to tune — chunk size, the reranker, the number of passages passed to the generator — on the assumption that extraction was solved because our field-level extraction F1 was high. It was high. Field-level F1 is computed per region, so it is structurally incapable of detecting a cross-region ordering error. We had chosen a metric that could not fail in the way our system was failing, and then trusted it.

The specific thing that made it worse before it got better: our first intervention was to increase chunk overlap, on the reasonable-sounding theory that more context per chunk would help the generator recover. On interleaved text, overlap propagates the contamination — the same scrambled span now appears in three chunks instead of one, so the retriever has three mediocre candidates competing with the one good chunk instead of one. Hit rate went down. We reverted it, and the reversion is what finally prompted someone to open the parsed markdown and read it.

The second thing we would change is cheaper. Log the column count and page type alongside every chunk at ingestion time. It costs nothing, it is impossible to reconstruct later once chunks are in the index, and it is the single field that turns “retrieval is bad sometimes” into a stratified answer in an afternoon.

References

We build document extraction and retrieval pipelines as part of our AI & Machine Learning Engineering practice. Talk to us if your RAG system answers confidently from documents nobody has read since they were parsed.