ATH-MaaS / OvisOCR3

OvisOCR3

Online Demo

Introduction

OvisOCR3 is a unified 0.8B multimodal model for OCR and document understanding . It integrates document parsing, general OCR, key information extraction, document figure captioning, and cross-page table reconstruction into a single compact model. These capabilities are selected through task-specific prompts, bringing recognition, extraction, and understanding together without requiring separate model deployments for each task.

Built on Qwen3.5-0.8B, OvisOCR3 is trained in four stages: continued pretraining, mid-training, on-policy distillation, and reinforcement learning. This training process builds a strong foundation in text recognition and document structure understanding, then refines these capabilities through domain-specific supervision and rewards.

OvisOCR3 achieves state-of-the-art performance among models of comparable size across all five domains , demonstrating that a single compact model can combine strong document parsing with a broad range of OCR and document-understanding capabilities.

Capabilities

The task prompts specify the output contracts. The examples below use these files directly.

Performance

Performance comparisons across five task domains are reported below. -- denotes an unreported value, not zero. Bold scores indicate the best result in each column.

Document Parsing

General OCR

Key Information Extraction

Document Figure Captioning

Cross-Page Table Reconstruction

Inference

The example uses a local model directory containing the weights, configuration, tokenizer, processor files, and prompts/ . Run it from that directory, or set MODEL_PATH to its location. A GPU environment supported by vLLM is required.

pip install "vllm==0.22.1" pillow

import warnings
from pathlib import Path

from PIL import Image
from vllm import LLM, SamplingParams


MODEL_PATH = Path(".")

model = LLM(
    model=str(MODEL_PATH.resolve()),
    tensor_parallel_size=1,
    gpu_memory_utilization=0.8,
    gdn_prefill_backend="triton",
    limit_mm_per_prompt={"image": 2},
)
tokenizer = model.get_tokenizer()
sampling_params = SamplingParams(max_tokens=16384, temperature=0.0)


def load_prompt(name: str) -> str:
    return (MODEL_PATH / "prompts" / f"{name}.txt").read_text(encoding="utf-8")


def infer(
    images: list[Image.Image], task_prompt: str, *,
    return_metadata=False,
):
    if not 1 <= len(images) <= 2:
        raise ValueError("This example accepts one or two images per request.")

    content = [{"type": "image"} for _ in images]
    content.append({"type": "text", "text": task_prompt})
    prompt = tokenizer.apply_chat_template(
        [{"role": "user", "content": content}],
        tokenize=False,
        add_generation_prompt=True,
        enable_thinking=False,
    )
    request = {
        "prompt": prompt,
        "multi_modal_data": {"image": [image.convert("RGB") for image in images]},
        "mm_processor_kwargs": {
            "images_kwargs": {
                "min_pixels": 448 * 448,
                "max_pixels": 2880 * 2880,
            }
        },
    }
    output = model.generate([request], sampling_params)[0].outputs[0]
    if output.finish_reason == "length":
        warnings.warn("Generation reached a token limit; the output may be incomplete.")
    return (output.text, output.finish_reason) if return_metadata else output.text


page = Image.open("test1.jpg").convert("RGB")
text = infer([page], load_prompt("plain_ocr"))
Path("output.txt").write_text(text, encoding="utf-8")
print(text)

infer returns the raw generated text by default; return_metadata=True also returns the finish reason. It does not remove image tags, trim repetitions, repair JSON, or otherwise rewrite the response. All task examples share the inference settings above. These are usage settings, not benchmark reproduction settings.

An image list passed to infer belongs to one request . For independent pages, call infer([page], prompt) for each page. The two-image form is used for cross-page reconstruction below.

Task-Specific Usage

The following snippets reuse page , infer , and load_prompt from the inference example. Replace placeholder values with the fields, regions, or questions needed for your document. Use literal placeholder replacement rather than Python str.format , since prompts also contain JSON and LaTeX braces.

Document Parsing

Compared with OvisOCR2's Markdown parsing, OvisOCR3 adds structured document parsing and document layout detection. Structured parsing returns region-level records with bounding boxes, categories, and content; Markdown parsing returns a continuous document without explicit bounding boxes or category labels for individual text regions.

For document_markdown , follow the OvisOCR2 postprocessing order: strip leading and trailing whitespace, optionally filter image-tag blocks (enabled by default), and then apply repeated-tail cleanup to every response regardless of the finish reason. Set filter_imgtags=False to retain visual-region references for rendering. Keep raw_markdown for inspection; structured and layout JSON outputs are left unchanged. The helper checks its own length and repetition thresholds. It does not recover missing content or make a truncated response complete.

def _clean_truncated_repeats(
    text: str,
    min_text_len: int = 8000,
    max_period: int = 200,
    min_period: int = 1,
    min_repeat_chars: int = 100,
    min_repeat_times: int = 5,
) -> str:
    n = len(text)
    if n < min_text_len:
        return text

    max_period = min(max_period, n - 1)
    for unit_len in range(min_period, max_period + 1):
        if text[n - 1] != text[n - 1 - unit_len]:
            continue

        match_len = 1
        idx = n - 2
        while idx >= unit_len and text[idx] == text[idx - unit_len]:
            match_len += 1
            idx -= 1

        total_len = match_len + unit_len
        repeat_times = total_len // unit_len
        tail_len = total_len % unit_len

        if repeat_times >= min_repeat_times and total_len >= min_repeat_chars:
            return text[: n - total_len + unit_len] + text[n - tail_len :]

    return text


# Format 1: Markdown, with optional image-tag filtering and repeated-tail cleanup.
raw_markdown = infer([page], load_prompt("document_markdown"))
filter_imgtags = True
text = raw_markdown.strip()
if filter_imgtags:
    text = "\n\n".join(
        block
        for block in text.split("\n\n")
        if not block.strip().startswith('<img src="images/bbox_')
    )
markdown = _clean_truncated_repeats(text)

# Format 2: Structured JSON with bbox, category, and content.
structured_json = infer([page], load_prompt("document_structured"))

# Format 3: Layout JSON with bbox and category only, without transcription.
layout_json = infer([page], load_prompt("document_layout"))

Structured and layout outputs follow natural reading order. The allowed category lists and category-specific content formats are defined in their respective prompts.

Coordinates and Visual Regions

Bounding boxes use [left, top, right, bottom] order with integer coordinates in [0, 1000) , relative to the image supplied to the model. The origin is the top-left corner. Convert to pixels with x * image_width / 1000 and y * image_height / 1000 . For cropped inputs, coordinates are relative to that crop, not the original page.

Markdown and structured parsing can reference visual regions using <img src="/ATH-MaaS/OvisOCR3/resolve/main/images/bbox_left_top_right_bottom.jpg" /> . These paths describe crops to create from the input image; the model does not write image files. Preserve the image orientation used for inference when extracting crops.

import re


BBOX_IMAGE_PATTERN = re.compile(r'images/bbox_(\d+)_(\d+)_(\d+)_(\d+)\.jpg')


def save_markdown_with_crops(markdown: str, page_image: Image.Image, output_dir: str):
    output_dir = Path(output_dir)
    images_dir = output_dir / "images"
    images_dir.mkdir(parents=True, exist_ok=True)
    width, height = page_image.size

    for left, top, right, bottom in set(BBOX_IMAGE_PATTERN.findall(markdown)):
        x1 = max(0, min(width, round(int(left) * width / 1000)))
        y1 = max(0, min(height, round(int(top) * height / 1000)))
        x2 = max(0, min(width, round(int(right) * width / 1000)))
        y2 = max(0, min(height, round(int(bottom) * height / 1000)))
        if x2 <= x1 or y2 <= y1:
            warnings.warn(f"Skipping invalid crop: {(left, top, right, bottom)}")
            continue
        crop_path = images_dir / f"bbox_{left}_{top}_{right}_{bottom}.jpg"
        page_image.crop((x1, y1, x2, y2)).convert("RGB").save(crop_path)

    (output_dir / "output.md").write_text(markdown, encoding="utf-8")


save_markdown_with_crops(markdown, page, "output")

General OCR

All ten tasks read the linked .txt prompt files. Image classification takes a target description and candidate labels, and position-aware VQA uses its own answer-and-box instruction. Fill only the requested task inputs in each template.

import re


def general_ocr_prompt(name: str) -> str:
    prompt = load_prompt(name)
    return prompt if name == "position_aware_vqa" else prompt.strip()


def general_vqa_prompt(question: str) -> str:
    prompt = general_ocr_prompt("visual_question_answering").replace("{question}", question)
    return re.sub(r"\n{3,}", "\n\n", prompt).strip()

import json

text = infer([page], general_ocr_prompt("plain_ocr"))
words_json = infer([page], general_ocr_prompt("word_spotting"))

references = json.dumps(["Invoice number", "Total"], ensure_ascii=False, separators=(",", ":"))
grounding_prompt = general_ocr_prompt("reference_grounding").replace("{references}", references)
grounding_json = infer([page], grounding_prompt)

region = json.dumps([100, 100, 900, 300], separators=(",", ":"))
region_prompt = general_ocr_prompt("region_ocr").replace("{target_region}", region)
region_text = infer([page], region_prompt)

formula_latex = infer([page], general_ocr_prompt("formula_recognition"))
chart_json = infer([page], general_ocr_prompt("chart_parsing"))

translation_prompt = general_ocr_prompt("text_translation").replace("{target_region}", region)
translation_prompt = translation_prompt.replace("{target_language}", "English")
translated_text = infer([page], translation_prompt)

target_description = "the main subject"
candidate_labels = json.dumps(["cat", "dog", "bird", "fish"], ensure_ascii=False)
classification_prompt = general_ocr_prompt("image_classification").replace(
    "{target description}", target_description
).replace("{candidate_labels}", candidate_labels)
selected_label = infer([page], classification_prompt)

question = "What is the invoice number?"
vqa_prompt = general_vqa_prompt(question)
answer_with_reasoning = infer([page], vqa_prompt)

position_prompt = general_ocr_prompt("position_aware_vqa").replace("{question}", question)
answer_json = infer([page], position_prompt)

References and regions use compact JSON. Supply references expected to be visible in the image. For classification, {target description} specifies what to classify, and {candidate_labels} is a JSON array of candidate labels. VQA accepts the complete question, including answer choices when applicable; its visible reasoning ends with a boxed answer. Position-aware VQA accepts the question text and uses its own answer-and-box prompt. Formula recognition and chart parsing need no template substitutions.

Spotting and grounding boxes, as well as input regions, use [x_min,y_min,x_max,y_max] with integer coordinates in [0,1000) , normalized independently by image width and height. The origin is the top-left corner, and coordinates refer to the supplied image, including when it is a crop. Use [0,0,999,999] for whole-image region OCR or translation.

Chart parsing represents scalar data values as strings and uses the literal string "None" for missing title, source, or axis metadata. Position-aware VQA uses "bbox": "(x1,y1,x2,y2)" , unlike the numeric arrays used by spotting and grounding.

Key Information Extraction

Schema-based extraction uses an example of the desired JSON output, not a formal JSON Schema declaration. Empty strings serve as value placeholders, while nested objects and arrays define the output structure.

import json

open_json = infer([page], load_prompt("kie_open"))

keys = json.dumps(["invoice_number", "date", "total"], ensure_ascii=False)
fields_prompt = load_prompt("kie_fields").replace("{keys_json}", keys)
fields_json = infer([page], fields_prompt)

schema = {
    "invoice_number": "",
    "date": "",
    "seller": {
        "name": "",
        "address": "",
    },
    "items": [
        {
            "name": "",
            "quantity": "",
            "unit_price": "",
            "amount": "",
        }
    ],
    "total": "",
}
schema_prompt = load_prompt("kie_schema").replace(
    "{schema_json}", json.dumps(schema, ensure_ascii=False)
)
schema_json = infer([page], schema_prompt)

Values should retain the visible source text. Open and specified-field extraction use arrays for repeated values and empty strings for empty values; specified-field extraction also uses empty strings for missing fields. Schema-based extraction follows the supplied template and uses empty strings for missing values. Adherence to the template is prompted, not enforced by constrained decoding in this example; validate generated JSON before downstream use.

Document Figure Captioning

Use the full page image with figure_captioning.txt :

caption_json = infer([page], load_prompt("figure_captioning"))

The model returns a structured page JSON array with bbox , category , and content . For visual categories such as image and chart , content contains a concise description inside <caption>...</caption> , with a maximum of 100 words requested by the prompt. These captions replace the image references used for visual regions in structured document parsing. Flowcharts additionally include a Mermaid representation inside <mermaid>...</mermaid> . Other page elements retain their task-specific text, formula, or table content.

This is localized figure description within a document page, not just transcription of a printed figure caption.

Cross-Page Table Reconstruction

OvisOCR3 reconstructs cross-page tables in two stages using the same checkpoint:

1. Stage 1: page parsing. Parse each page independently with document_markdown.txt or document_structured.txt . For structured JSON output, concatenate each element’s content in reading order to obtain Markdown, preserving HTML tables, formulas, and image references.
2. Stage 2: table reconstruction. Supply two adjacent page images and their parsed Markdown with crosspage_table.txt . The model identifies whether a table continues across the page boundary and reconstructs continued fragments as a single HTML table; otherwise, it returns a negative prediction.

import re

pages = [
    Image.open("page1.jpg").convert("RGB"),
    Image.open("page2.jpg").convert("RGB"),
]
page_results = [
    infer([image], load_prompt("document_markdown"), return_metadata=True)
    for image in pages
]
page_markdowns = []
for index, (raw, reason) in enumerate(page_results, start=1):
    if reason not in ("stop", "length"):
        raise ValueError(f"Page {index}: finish_reason={reason}")
    markdown = _clean_truncated_repeats(raw.strip())
    markdown = re.sub(r'^```(?:markdown|html)?\s*\n|\n```\s*$', '', markdown.strip(), flags=re.I)
    if not markdown:
        raise ValueError(f"Page {index}: empty Markdown")
    page_markdowns.append(re.sub(
        r'(?<![\w/])(?:page_\d+/)?(images/bbox_\d+_\d+_\d+_\d+\.jpg)',
        rf'page_{index}/\1', markdown,
    ))

if all(re.search(r'<table\b[^>]*>.*?</table\s*>', md, re.I | re.S) for md in page_markdowns):
    page_context = "\n\n".join(
        f'<INPUT_PAGE index="{index}">\n{markdown}\n</INPUT_PAGE>'
        for index, markdown in enumerate(page_markdowns, start=1)
    )
    prompt = load_prompt("crosspage_table").removesuffix("\n") + "\n\nInput Markdown:\n\n" + page_context
    tables_json, reason = infer(pages, prompt, return_metadata=True)
    if reason != "stop":
        raise ValueError(f"Cross-page output is incomplete: finish_reason={reason}")
else:
    tables_json = '[{"source_pages": [], "html": "No cross-page table exists between these pages."}]'
print(tables_json)

The result is a JSON array. Each record has source_pages , using input-local, 1-based page numbers , and html , containing one complete HTML table. Images within tables retain page-qualified references such as page_1/images/bbox_100_200_300_400.jpg .

When no cross-page table exists, the specified response is:

[{"source_pages": [], "html": "No cross-page table exists between these pages."}]

For tables spanning more than two pages, Stage 2 is applied recursively , processing adjacent pairs from the end of each consecutive sequence of table-bearing pages toward its beginning. After each merge, the reconstructed table is written into the earlier page and its updated Markdown is re-rendered before the next step, keeping the image and text consistent. For example, the model first merges pages 2 and 3, then pairs the updated page 2 with page 1. Each step uses only two page images while preserving the original source-page associations. A ready-to-run document-level implementation is available in document_crosspage.py , supporting either document_markdown.txt or document_structured.txt for Stage 1 and converting structured JSON to Markdown before reconstruction.

Stage 2 first reconstructs the table across pages 2 and 3, then pairs the re-rendered page 2 with page 1. Each step uses two page images and their current Markdown.

The script uses the same vLLM version and inference settings as the example above. Install the additional dependencies and run the command below from the model repository root, supplying page images in reading order.

python -m pip install beautifulsoup4 markdown-it-py playwright
python -m playwright install --with-deps chromium

python document_crosspage.py \
    --model . \
    --stage1-prompt document_structured.txt \
    --output crosspage_output \
    page1.jpg page2.jpg page3.jpg

Use a new output directory for each run; it contains document.md , per-page Markdown, raw responses, and result.json with table provenance and any processing errors.

Output Handling

Validation and Rendering

• Preserve raw outputs before parsing, rendering, or applying cleanup. JSON, HTML, LaTeX, and Mermaid may need validation, and generation can stop before an output is complete.
• Render Markdown with support for HTML tables and LaTeX. Figure captions and Mermaid content require their own handling inside structured JSON.
• Treat generated HTML and Mermaid as untrusted content. Sanitize or isolate them before rendering in a browser.
• For evaluation, freeze prompts, preprocessing, generation parameters, and postprocessing. These usage examples do not define a benchmark protocol.

Limitations

OvisOCR3 can produce incorrect, incomplete, repeated, or malformed outputs. Small or blurred text, severe perspective distortion, dense layouts, complex formulas, and long tables may affect recognition and structure recovery. Captions and answers can include unsupported details; verify them against the source image.

Cross-page reconstruction also depends on the quality of the first-stage page parsing and correct page order. Missing table fragments, inconsistent page context, or truncated generations can lead to incorrect merges. Review results before using them in critical applications.

Technical Report and Citation

If you find OvisOCR3 useful, please consider citing our technical report:

@misc{ovisocr3,
  title  = {{OvisOCR3 Technical Report}},
  author = {{ATH-MaaS}},
  year   = {2026}
}

The technical report link will be added when available.

License

This project is licensed under the Apache License, Version 2.0 (SPDX-License-Identifier: Apache-2.0).