🆕 Haystack 3.4 is here! Python 3.15 support and ToolResultOffloadCompactor to keep agent context lean without losing tool results
Maintained by deepset

Integration: MiniCPM-OCR

Convert PDFs and images to Markdown with MiniCPM-OCR, an open-weight OCR model running on your own inference server

Authors
deepset

Table of Contents

Overview

MiniCPM-OCR is a 1.3B open-weight OCR model by OpenBMB, specialized in document parsing. It recognizes text, tables, and formulas in Chinese and English documents and returns them as Markdown. You run the model on your own inference server, such as vLLM, llama.cpp, or Ollama, so documents never leave your infrastructure.

This integration lets you use MiniCPM-OCR to convert PDFs and images into Haystack documents.

Components

  • MiniCPMOCRDocumentConverter. This component detects the layout of each page locally and sends the detected blocks to a MiniCPM-OCR inference server. It returns one document per source, with pages separated by form feed characters (\f) so you can split them by page.

Installation

pip install minicpm-ocr-haystack

The integration supports Python 3.11 to 3.13.

You also need a running MiniCPM-OCR inference server. With a GPU, you can use vLLM:

vllm serve openbmb/MiniCPM-OCR-1.0 --served-model-name minicpm-ocr --port 8000

Without a GPU, for example on a laptop or Apple Silicon, llama.cpp can serve the quantized weights:

llama-server -hf openbmb/MiniCPM-OCR-1.0-gguf:Q4_K_M --host 127.0.0.1 --port 8080

For llama.cpp, initialize the converter with server_url="http://127.0.0.1:8080" and backend="llamacpp".

You can also serve the quantized weights with Ollama. Download the model and its vision projector, then register them with a Modelfile:

mkdir minicpm-ocr && cd minicpm-ocr
hf download openbmb/MiniCPM-OCR-1.0-gguf MiniCPM-OCR-1_0-Q8_0.gguf mmproj-model-f16-4x.gguf --local-dir .

cat > Modelfile <<'EOF'
FROM ./MiniCPM-OCR-1_0-Q8_0.gguf
FROM ./mmproj-model-f16-4x.gguf
PARAMETER num_ctx 8192
PARAMETER temperature 0
EOF

ollama create minicpm-ocr -f Modelfile

Then point the converter at the Ollama server:

from haystack_integrations.components.converters.minicpm_ocr import MiniCPMOCRDocumentConverter

converter = MiniCPMOCRDocumentConverter(
    server_url="http://localhost:11434",
    backend="ollama",
    model="minicpm-ocr",
)

Usage

How to use the MiniCPMOCRDocumentConverter

Basic usage with a local file:

from pathlib import Path

from haystack_integrations.components.converters.minicpm_ocr import MiniCPMOCRDocumentConverter

converter = MiniCPMOCRDocumentConverter(server_url="http://localhost:8000")
converter.warm_up()

result = converter.run(sources=[Path("my_document.pdf")])
documents = result["documents"]

The first call to warm_up() downloads the layout model into ~/.cache/minicpm-ocr/layout.

Here’s an example of an indexing pipeline that converts PDFs and images, splits them by page, and writes them to a Document Store:

from haystack import Pipeline
from haystack.components.preprocessors import DocumentSplitter
from haystack.components.writers import DocumentWriter
from haystack.document_stores.in_memory import InMemoryDocumentStore
from haystack_integrations.components.converters.minicpm_ocr import MiniCPMOCRDocumentConverter

document_store = InMemoryDocumentStore()

pipeline = Pipeline()
pipeline.add_component("converter", MiniCPMOCRDocumentConverter(server_url="http://localhost:8000"))
pipeline.add_component("splitter", DocumentSplitter(split_by="page", split_length=1))
pipeline.add_component("writer", DocumentWriter(document_store=document_store))

pipeline.connect("converter", "splitter")
pipeline.connect("splitter", "writer")

file_paths = ["invoice.pdf", "receipt.jpg", "contract.pdf"]
pipeline.run({"converter": {"sources": file_paths}})

License

minicpm-ocr-haystack is distributed under the terms of the Apache-2.0 license.