cnmoro / tinyzchunk

tinyzchunk weights

Tiny MLP weights for the tinyzchunk chunker — a GPU-free, numpy-only chunker distilled from the zChunk algorithm. English and Brazilian Portuguese, robust to noisy/OCR'd text.

The package source lives on GitHub: https://github.com/cnmoro/tinyzchunk

Install

pip install tinyzchunk

The pip package is weight-free: on first use the weights are fetched from this repo automatically (and cached). No GPU, no tokenizer.

Usage

from tinyzchunk import Chunker

chunker = Chunker()                    # auto-fetches weights from this repo
chunks = chunker.chunk(long_document)  # -> list[str]

# or fetch explicitly
chunker = Chunker.from_pretrained("cnmoro/tinyzchunk")

Files

• weights.npz — char-level sentence/paragraph boundary model (fallback)
• line_weights.npz — line-level "does a new unit start here?" model (primary)

Distillation

Trained from Qwen2.5-7B zChunk teachers on prose + structured corpora (Q&A roteiros, schedules, bios, FAQs, legal, contact lists) plus simulated-PDF-noise training. Event schedules boundary F1 0.92–0.98; Q&A roteiros 0.74–1.0; noisy text 0.89–0.97. ~14 ms/document on CPU.