Performance characteristics and optimization

[cau-git]

This thread is to outline and discuss the details for two goals:

1. Assess the conversion speed and resource footprint of docling in different scenarios, and compare to other open-source tools with the same purpose.
2. Further optimize docling's conversion speed and resource efficiency

We are happy to receive your feedback on the items outlined below! The results of the evaluation outlined here will be reflected in an updated Technical Report for docling 2.x.

1. Conversion speed and resource footprint

There are several dimensions to establish when assessing the runtime characteristics of docling and alternatives:

• Dataset: We need a diverse dataset of PDF samples with a representative distribution of features
• Hardware / OS: Which hardware to run on (e.g. which CPU, with or without GPU), on which thread budget? Which OS?
• Conversion options: With/without OCR, with/without table structure recognition, profile how much time is spent in which model/pipeline stage?
• Software: Apart from docling, which alternatives to consider? How can we harmonize different tool's configuration options to enable fair comparison?

Dataset

We are considering a benchmark dataset sourced from DocLayNet and a portion of random samples from ccpdf mixed in, to make it more diverse.
The target size is ~4000 pages (i.e. <100 docs), such that benchmark runs would typically complete in one to a few hours even with a low-resource hardware profile.

Hardware / OS

To keep the number of experiments we need to drive at a reasonable level, we plan to benchmark on two system configurations:

1. AWS EC2 VM (g6.2xlarge flavour) with Ubuntu 22.04 image, providing 8 core vCPU, 32 GB RAM, and Nvidia L4 GPU (24 GB VRAM)
2. Apple Macbook Pro M3 Max, on macOS, with option to enable MPS accelerator

Conversion options

We plan to define several profiles for conversion options, to show the impact of different models in the pipeline on conversion speed, e.g. with or without OCR, with or without table structure, with different PDF backends. For docling, we intend to profile the pipeline to analyze the contribution of different models/stages to the overall runtime.

Software

Docling is not out here alone. We want to evaluate and compare with popular open-source alternatives. Currently, we considder comparing to the following projects:

• Marker
• MinerU (magic-pdf)
• unstructured.io

2. Speed optimization

From the above evaluation, we want to learn which parts of the docling pipeline need to be targeted first to achieve even better speed.

One known issue to address in this context is accelerator support. Currently, docling can partially use GPU acceleration, and will do so by default when a CUDA device is detected by torch. Among other things, we plan to:

• Provide explicit configuration controls for accelerator support
• Enable the layout model to run on CUDA/MPS (currently CPU)
• Fix bugs currently seen when using docling on a GPU-enabled device
• Improve batching in models for GPU inference
• Allow multiple page batches to process in parallel with multi-threading

[MHDataExpert]

how can we run layout model on GPU?