Playing an Away Game with the New PyMuPDF Benchmark Page
September 22, 2026
In our recent article, The Benchmark Creator Is Almost Always on the Podium, we examined 13 document-AI leaderboards. In all 10 cases where we identified an entry affiliated with the benchmark's creator or sponsor, that entry finished in the top three. That raised a question for our own benchmark page. How do we publish results while holding ourselves to the same standard?
The new PyMuPDF benchmark page uses ParseBench, the benchmark suite from LlamaIndex, the company behind LlamaParse. Using a competitor's benchmark gives us an external test, but our choices of parsers, versions, and configurations still matter. We make those choices visible alongside the results.
What the page shows
A parser that extracts a document accurately may still be too slow or too large for your deployment. The page brings quality scores together with processing speed, cold-start time, installation size, API fees, and support for local execution.
In the recorded run covering 2,078 documents, PyMuPDF has the highest overall quality score among the local parsers we measured. The comparison with Docling shows how quality, speed, and installation size differ.
| Metric | PyMuPDF 1.28.2 | Docling 2.124.0 |
|---|---|---|
| Overall quality score | 53.6 | 48.0 |
| Table extraction score | 72.0 | 58.5 |
| Visual grounding score | 61.9 | 62.8 |
| Median processing time per page | 0.47 seconds | 11 seconds |
| Installed packages and dependencies | 278 MB | 1.6 GB |
Figures are rounded and apply to the configurations recorded in that run. Docling was measured through its local docling-serve interface.
The page also includes published cloud-parser results with higher overall quality scores, shown separately as external references with their listed API costs. These results help readers weigh extraction quality against throughput, budget, and whether processing must stay within their infrastructure.
How the numbers are made
We measure the local parsers through an automated pipeline on a fixed AWS instance type, with engines tested in the same controlled session. Each run records the software versions, configuration, benchmark revision, and environment. The evaluation methodology remains unchanged, and the overall score follows ParseBench's definition.
The measurement scripts, configurations, and results down to individual documents are available in our public ParseBench fork. Any necessary compatibility fixes are tracked explicitly. Readers can inspect how the results were produced and repeat the measurements.
Versions matter. In our tests, Docling's table score differed by almost nine points between versions 2.122.0 and 2.124.0, as documented in the measurement notes. A quality score needs its version and configuration attached. Timing results need the hardware details too.
Why we start with non-VLM parsers
Our direct comparisons currently focus on local parsers that do not use vision-language models (VLMs). VLM pipelines address the same extraction tasks with different compute requirements and operating costs. A useful comparison needs to show quality, speed, and cost together.
As our GPU-based models become available, we plan to extend the page to cover VLM pipelines under the same reporting principles.
Put the results to work
The advice from our leaderboard review applies to this page too. Check who ran the benchmark, read the version and configuration details, and test a representative sample of your own documents.
The page gives you results to inspect and a process you can repeat. Your own documents remain the most useful test.
Discuss This Article with the Community
Have a question, a different approach, or something you built after reading this? Share it on the forum or join the Discord, we'd love to hear from you.
