Integration: MiniCPM-OCR
Convert PDFs and images to Markdown with MiniCPM-OCR, an open-weight OCR model running on your own inference server
Table of Contents
Overview
MiniCPM-OCR is a 1.3B open-weight OCR model by OpenBMB, specialized in document parsing. It recognizes text, tables, and formulas in Chinese and English documents and returns them as Markdown. You run the model on your own inference server, such as vLLM, llama.cpp, or Ollama, so documents never leave your infrastructure.
This integration lets you use MiniCPM-OCR to convert PDFs and images into Haystack documents.
Components
-
MiniCPMOCRDocumentConverter. This component detects the layout of each page locally and sends the detected blocks to a MiniCPM-OCR inference server. It returns one document per source, with pages separated by form feed characters (\f) so you can split them by page.
Installation
pip install minicpm-ocr-haystack
The integration supports Python 3.11 to 3.13.
You also need a running MiniCPM-OCR inference server. With a GPU, you can use vLLM:
vllm serve openbmb/MiniCPM-OCR-1.0 --served-model-name minicpm-ocr --port 8000
Without a GPU, for example on a laptop or Apple Silicon, llama.cpp can serve the quantized weights:
llama-server -hf openbmb/MiniCPM-OCR-1.0-gguf:Q4_K_M --host 127.0.0.1 --port 8080
For llama.cpp, initialize the converter with server_url="http://127.0.0.1:8080" and backend="llamacpp".
You can also serve the quantized weights with Ollama. Download the model and its vision projector, then register them with a Modelfile:
mkdir minicpm-ocr && cd minicpm-ocr
hf download openbmb/MiniCPM-OCR-1.0-gguf MiniCPM-OCR-1_0-Q8_0.gguf mmproj-model-f16-4x.gguf --local-dir .
cat > Modelfile <<'EOF'
FROM ./MiniCPM-OCR-1_0-Q8_0.gguf
FROM ./mmproj-model-f16-4x.gguf
PARAMETER num_ctx 8192
PARAMETER temperature 0
EOF
ollama create minicpm-ocr -f Modelfile
Then point the converter at the Ollama server:
from haystack_integrations.components.converters.minicpm_ocr import MiniCPMOCRDocumentConverter
converter = MiniCPMOCRDocumentConverter(
server_url="http://localhost:11434",
backend="ollama",
model="minicpm-ocr",
)
Usage
How to use the MiniCPMOCRDocumentConverter
Basic usage with a local file:
from pathlib import Path
from haystack_integrations.components.converters.minicpm_ocr import MiniCPMOCRDocumentConverter
converter = MiniCPMOCRDocumentConverter(server_url="http://localhost:8000")
converter.warm_up()
result = converter.run(sources=[Path("my_document.pdf")])
documents = result["documents"]
The first call to warm_up() downloads the layout model into ~/.cache/minicpm-ocr/layout.
Here’s an example of an indexing pipeline that converts PDFs and images, splits them by page, and writes them to a Document Store:
from haystack import Pipeline
from haystack.components.preprocessors import DocumentSplitter
from haystack.components.writers import DocumentWriter
from haystack.document_stores.in_memory import InMemoryDocumentStore
from haystack_integrations.components.converters.minicpm_ocr import MiniCPMOCRDocumentConverter
document_store = InMemoryDocumentStore()
pipeline = Pipeline()
pipeline.add_component("converter", MiniCPMOCRDocumentConverter(server_url="http://localhost:8000"))
pipeline.add_component("splitter", DocumentSplitter(split_by="page", split_length=1))
pipeline.add_component("writer", DocumentWriter(document_store=document_store))
pipeline.connect("converter", "splitter")
pipeline.connect("splitter", "writer")
file_paths = ["invoice.pdf", "receipt.jpg", "contract.pdf"]
pipeline.run({"converter": {"sources": file_paths}})
License
minicpm-ocr-haystack is distributed under the terms of the
Apache-2.0 license.
