FluidInference / unlimited-ocr-mlx-coreml

Unlimited-OCR (MLX + Core ML)

Baidu's Unlimited-OCR (DeepSeek-OCR architecture: SAM-B + CLIP-L page encoder, 12-layer DeepseekV2 MoE decoder, 3B parameters / ~0.5B active) for Apple silicon, including its multi-page mode : up to 16 pages go in as one prompt and the whole document comes out of a single generation as markdown, <PAGE> -separated, with <|det|>label [x1, y1, x2, y2]<|/det|> layout boxes (0–999 page coordinates). Offline.

Two runtimes share the Core ML page encoder:

• MLX decoder ( mlx/ , recommended): 8-bit, upstream's R-SWA cache, ~200–260 tokens/s on an M5 Pro. Baidu's 14-page paper in one pass: 13,420 tokens in ~70 s.
• MLX turbo decoder ( mlx-turbo/ ): routed experts 4-bit (group 32), rest 8-bit. ~15% faster (14 pages in ~50 s, ~270 tokens/s); text within ~1 point of the 8-bit decoder, with small differences in formulas and tables.
• Core ML decoder ( UnlimitedOcrDecoder.mlpackage ): stateful 8-bit, ~50 tokens/s.

Runtime: UnlimitedOcrFastManager (MLX) and UnlimitedOcrManager (Core ML) in FluidUse (Swift, macOS 15+), with a SwiftUI demo ( UnlimitedOcrDemo ): drop a PDF, every page is read in one pass while the page thumbnails highlight the block being written.

Packages

Upstream's R-SWA attention keeps the whole prefix (prompt + page tokens) plus a ring of the last 128 generated tokens, so the cache has a fixed size per prefix bucket: k_cache / v_cache are Core ML state, [12 × (P + 128), 1280] . P = 384 / 640 / 1152 / 2304 holds 1 / 2 / 4 / 8 pages (each page is 273 tokens). Decoder inputs: x [1, C, 1280] embeddings, cos / sin [C, 128] , additive mask [1, 1, C, P + 128 + C] (cached slots, then the C new tokens), write_idx [C] cache slots, out_index [1] ; outputs the top-16 logits of each vocab half. The host (Swift runtime is the reference) assembles the prefix, places generated token t in slot P + t % 128 , and applies upstream's 35-gram no-repeat ban.

Quality

Reference: upstream's own code ( trust_remote_code ), fp32 on CPU, greedy, same pixels.

The page encoder must run on the GPU: on the Neural Engine its output is wrong (row cosine down to 0.66).

Speed (M5 Pro, 24 GB, GPU, Swift)

The decoder computes all 64 experts per token (routing weights zero the unselected ones), which keeps the graph static but reads every expert's weights on each step.

Usage (Swift)

import FluidUse
import PDFKit
import UnlimitedOcrMLX

let manager = try await UnlimitedOcrFastManager.load(from: bundleDirectory)  // MLX; UnlimitedOcrManager = Core ML
let pdf = PDFDocument(url: pdfURL)!
let pages = try (0..<min(pdf.pageCount, manager.maxPages)).map {
    try UnlimitedOcrPagePreprocessor.page(from: pdf.page(at: $0)!)
}
var text = ""
for try await event in manager.read(pages: pages) {
    if case .text(let delta) = event { text += delta }
}
print(UnlimitedOcrDocument(parsing: text).markdown)

License and credits

MIT, same as baidu/Unlimited-OCR ( LICENSE is Baidu's). All model weights are Baidu's; this repository only changes their format and precision. Conversion by Fluid Inference .