docling-project / ChemicalOCR

ChemicalOCR is a compact vision-language model fine-tuned specifically for optical character recognition (OCR) in chemical structure images. It extracts text and bounding boxes from molecular drawings, enabling the recognition of atom labels, abbreviations, and descriptive text within chemical structures.

Overview

ChemicalOCR is fine-tuned from SmolDocling , a lightweight 256M-parameter model originally developed for end-to-end document conversion. By specializing for chemical images, ChemicalOCR substantially outperforms general-purpose OCR systems on chemical structure recognition tasks.

What it does

• Extracts character-level text and bounding boxes from chemical structure images
• Recognizes atom labels , chemical abbreviations , and descriptive text near or below molecular structures

Architecture

• Base model : SmolDocling-256M (Idefics3ForConditionalGeneration)
• Parameters : 256M
• Input : Chemical structure image
• Output : Text detections with bounding boxes

Performance

ChemicalOCR substantially outperforms existing OCR models on chemical structure benchmarks:

Detailed Metrics

Usage

from transformers import AutoProcessor, AutoModelForVision2Seq
from PIL import Image

model = AutoModelForVision2Seq.from_pretrained("docling-project/ChemicalOCR")
processor = AutoProcessor.from_pretrained("docling-project/ChemicalOCR")

image = Image.open("chemical_structure.png")
inputs = processor(images=image, return_tensors="pt")
outputs = model.generate(**inputs)
result = processor.decode(outputs[0], skip_special_tokens=True)

Part of MarkushGrapher-2

ChemicalOCR is a key component of MarkushGrapher-2 , an end-to-end model for multimodal recognition of chemical structures from patent documents. In the MarkushGrapher-2 pipeline, ChemicalOCR provides the text and layout information that is fused with visual features for Markush structure recognition.

Citation

@inproceedings{strohmeyer2026markushgrapher2,
  title     = {MarkushGrapher-2: End-to-end Multimodal Recognition of Chemical Structures},
  author    = {Strohmeyer, Tim and Morin, Lucas and Meijer, Gerhard Ingmar and Weber, Valery and Nassar, Ahmed and Staar, Peter W. J.},
  booktitle = {Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)},
  year      = {2026}
}

License

This model is released under the Apache 2.0 License .