datalab-to / surya-parsebench-rl

Surya ParseBench RL

This is a research checkpoint, not a product model. We built it to test one question: how much of a ParseBench score comes from reading the page, and how much comes from matching the benchmark's output conventions?

We took Surya OCR 2 , a 650M-parameter OCR model, and trained it with 70 steps of RL. We did not train on any ParseBench documents. It went from 23rd to 3rd on the ParseBench leaderboard (mean of the four non-chart dimensions). Almost all of the gain came from formatting. Content faithfulness, the score closest to "did the model read the page correctly", moved 0.4 points.

For OCR, use Surya OCR 2 or Chandra OCR 2 , not this checkpoint.

Results

ParseBench, our run of the public harness ( parse-bench==1.0.4 ), with the provider in code/ . Charts are left out of every number.

On the leaderboard at commit b72c714 , 83.15 ranks 3rd and 82.00 ranks 5th.

What moved:

• Semantic formatting: +10.9 from RL.
• Content faithfulness: +0.4.
• Tables: +0.9 from RL, and +1.1 more from the inference fixes.
• Visual grounding: +2.2 from RL, and +2.0 more from the inference fixes.

Notes:

• These are our runs, not the leaderboard's pipeline.
• The "before RL" row uses the same converter as the other rows, so the rows compare the model only.
• The parse-bench layout loader cannot score 64 of the 500 layout documents for any provider. Visual grounding averages over the other 436.

Training

70 steps of RL from Surya OCR 2. We did not train on any ParseBench documents.

How to run it

The score needs the model, the exact prompt, the retry rule, and the converter in code/ .

1. Serve with vLLM:

   vllm serve datalab-to/surya-parsebench-rl --served-model-name surya --max-model-len 32768 \
     --limit-mm-per-prompt '{"image":1}'
   

2. Render each PDF page at 192 DPI. Send the image with this exact prompt: OCR this image to HTML. Each block is a div with data-label and data-bbox (x0 y0 x1 y1, normalized 0-1000).
3. Decode greedy with max_tokens 28000. If the output does not end with </div> , the page looped. Retry with repetition_penalty 1.1, then 1.25, and keep the first output that closes.
4. Convert the layout HTML with code/surya_convert.py . The ParseBench provider in code/pbext/__init__.py does steps 2 to 4. code/bench.sh shows how we ran the four groups and built the scorecard.

config.json sets text_config.eos_token_id to 2. This checkpoint has no MTP head, so run it without speculative decoding.

License

The model weights use a modified AI Pubs Open Rail-M license, the same as Surya OCR 2: free for research, personal use, and startups under $5M in funding or revenue. The code in code/ is Apache 2.0.