The Benchmark Creator Is Almost Always on the Podium

By Hashir Omer

We aggregated thirteen public OCR and document-parsing leaderboards into a single table. Then, instead of asking the usual question “Which parser is Best?”, we asked a much simpler one:

When the organization behind a benchmark enters its own model, where does that model finish?

In our collection, ten benchmarks feature an identifiable creator or sponsor-affiliated entry. All ten place their affiliated model in the top three.

Seven places it at number one. This is an extraordinarily consistent home-field advantage.

The Scoreboard of Scoreboards

Here is how the home teams fare, based on each benchmark’s headline results:

(Results snapshot taken on September 1st 2026)

The cynical takeaway is that document benchmarks are rigged. However, several well-understood mechanisms explain this convergence:

• Problem Selection: Teams build benchmarks precisely because existing evaluations overlook failure modes they care about. Their models are naturally built to resolve those same edge cases. For example, the team behind LlamaParse criticized OmniDocbench and olmOCR, two of the most popular OCR parsing benchmarks before publishing their own benchmark.

• Configuration Asymmetry: A creator knows the optimal configuration for its own tool and has little incentive to find the best configuration for competing tools. Running a rival's engine with default settings while running your own with hand-tuned parameters isn't fraud; it's just what happens when nobody is paid to make the competition look good.

• Hill-Climbing as an Engineering Loop: Once a benchmark exists, it becomes the team's internal CI/CD regression suite. Every release is implicitly optimized against it, release after release, while competitors' models are evaluated cold.

• Publication Filter: Organizations rarely publish an evaluation unless it validates the core value proposition of their research or product. The benchmarks where the home team lost quietly never ship. What we see is heavily shaped by survivorship bias.

In short, a benchmark embodies its creator’s definition of the problem and the creator’s system is systematically engineered to solve that exact definition. This dynamic occurs naturally, even when everyone acts in good faith.

Away Games

Cross-benchmark comparisons highlight the volatility of these scores and show how easily cross-table narratives can mislead if versions are conflated.

Take HunyuanOCR. The baseline model scores 92.42 on MORE (1st place), 89.95 on OmniDocBench (18th place), and 68.3 on MDPBench. It dominates its home benchmark but slips significantly when evaluated on alternative corpora.

olmOCR 2 shows an equally stark divergence. Ai2 reports an 82.4 on olmOCR-Bench (3rd overall). On PDF Parse Bench's table-extraction track, however, olmOCR-2-7B records a 3.51, placing 21st of 22 engines.

The discrepancy does not mean one benchmark is wrong; it means the tasks and penalty functions evaluate entirely different capabilities. An accuracy score is not an intrinsic, portable property of a model.

This suggests the following approach for reading leaderboards:

Use vendor leaderboards, and then just subtract the vendor. A leaderboard built by a model's creator is not worthless as significant engineering effort goes into constructing these benchmarks. They curate documents, define scoring functions, run dozens of competing engines, and that investment produces genuinely useful data. The catch is that the one row you should ignore is the maintainer's own! So strike out the home team, and what remains is a well-funded, well-maintained ranking of everyone else. Ai2's olmOCR-Bench, for instance, may be an unreliable referee for olmOCR itself, but it's still a serious instrument for comparing the other 20+ engines on its board.

By following this approach we are evaluating a tool by its away record, not its home record. When assessing a specific model, looking up its score everywhere except the leaderboard its creators publish, should give more balanced results because a model that holds up on a rival's benchmark is passing a hostile audit. However, this may introduce the inverse bias: competitors have incentives to construct tasks where their own model shines and others stumble. This is why, if possible, independent leaderboards, despite their limitations, remain the most trustworthy signal when they exist.

If we eventually publish a document-parsing benchmark, we fully expect our own engine to score near the top. We will have designed the test around the failures we find intolerable, trained our models on those distributions, and spent months fixing those exact mistakes. That will prove our product aligns with our benchmark, however it will not prove we "solved" document parsing!

A Buyer’s Guide to Document-AI Leaderboards

Benchmarks remain valuable diagnostic tools if interpreted within their proper scope:

• Check the affiliations first. Read the author list and corporate sponsorships before reading the scores. If the maintainer has an entry, mentally strike that row. For each commercial tool, check how it fares in their competitors benchmarks or ideally benchmarks maintained by independent people.
• Triangulate every commercial tool. Look up how it fares on competitors' benchmarks and, ideally, on independently maintained ones. A tool that only wins at home is telling you something.
• Evaluate on your own data. This beats every public benchmark, because your corpus is the problem definition that actually matters. It's time-consuming, but even a small sample of your real documents will surface failure modes no leaderboard will.

Evaluate on Your Own Data

At the end of the day, evaluating from your own documents will prove which solution delivers the best results for you. We recommend finding what document parsing success looks like for your document corpus and then building a small ground-truth set by hand for that definition.

In this way when you benchmark against your own data you'll have the most meaningful benchmarking results as it will suit your context.

So, by all means benchmark, but do so with care!