AlexWortega / tinyvla

TinyVLA: a dual-rate VLA that beats OpenVLA-7B on LIBERO-Spatial at 0.64B params

Headline result: 86% success on LIBERO-Spatial (n=100) in dual-rate mode (semantic latent refreshed at 1 Hz), 79% full-rate — vs OpenVLA-7B's 84.7% — with 140M parameters on the online control path. Built and trained end-to-end in ~5 days on a single RTX A6000.

1. Architecture

                SLOW / SMART path (0.5–2 Hz, cached)
instruction ─────────────────────┐
camera → Qwen3.5-0.8B vision tower → Qwen3.5 language stack (first 12 of 24 layers)
                  │                       │
                  │            8 readout tokens → linear → semantic latent (8×512)
                  │                       │ cached between refreshes
──────────────────┼───────────────────────┼──────────────────────────
                  │       FAST / MOTOR path (10–20 Hz, ~140M online)
                  └── 64 spatial tokens per camera ─┐
proprio (padded 16d) ───────────────────────────────┤
embodiment-id embedding ────────────────────────────┤
                                                    ▼
                    flow-matching action expert (38M, 12 blocks,
                    interleaved cross-/self-attention, AdaLN-free)
                                                    ▼
                              50-step action chunk (padded 8d + validity mask)

• Single shared encoder : the Qwen3.5-0.8B native ViT (100.6M) feeds both paths.
• Slow path : image + instruction + 8 trailing pad-token "readout" slots through the truncated language stack (L/2 layer-skip a la SmolVLA); their hidden states → latent 8×512.
• Fast path : raw vision-tower spatial tokens (bypassing the LM) + proprio + embodiment token, cross-attended by the flow expert. 10 Euler steps at inference.
• Flow matching : linear-OT path, t ~ Beta(1.5, 1), velocity target, masked MSE.
• Totals: 643M stored / 39.5M–575M trainable depending on stage / ~140M online per tick.
• Implemented as an out-of-tree LeRobot policy plugin ( --policy.type=tinyvla --policy.discover_packages_path=tinyvla , lerobot 0.6.1).

2. Training recipe

Ablation FTs: ft1 (all frozen) → 0% closed-loop; ft2 (vision unfrozen) → 60%; ft3 (= ft2 from distilled ckpt) → 61%; ft4 (LM+vision unfrozen) → 79/86% . Unfreezing the LM was the single largest gain; distillation was neutral on LIBERO (likely overwritten by vision unfreezing during FT — bake it into pretraining next time).

Cross-embodiment handling: canonical schema (cam0=scene/cam1=wrist via regex mapping, per-dataset mean/std normalization, action pad-to-8 + validity mask, state pad-to-16, learned embodiment-id token). Weighted mixture sampler with per-source retry on corrupt frames (IndexError/KeyError/decode RuntimeError).

3. Results

LIBERO — all four suites (dual-rate, latent @ 1 Hz)

Beats OpenVLA-7B on every suite at 1/11 the parameters and ~1/50 the online compute. XS-VLA (90.3 avg) remains ahead — their latent flow matching is future work for us.

LIBERO-Spatial (100 episodes, 10 per task, init states 0-9)

Dual-rate ≥ full-rate consistently across all evals (also 65% vs 58% at ft3, n=50) — a stable cached latent appears to act as temporal regularization.

Context (published numbers, their own eval protocols)

MolmoAct2 context (from their cards/repo, no numeric tables published outside the LIBERO-LeRobot card): 5B params on Molmo2-ER backbone + flow expert; claims to beat π0.5; Molmo2-ER reportedly surpasses GPT-5 / Gemini-Robotics-ER-1.5 on 13 embodied-reasoning benchmarks; #1 VLA on the MolmoSpace leaderboard. Inference needs ~26GB fp32 (16GB bf16), cuda-graphs for ~2× speedup — H100-class serving vs our Raspberry-Pi-class 140M fast path. Notably they ship MolmoAct2-SO100_101 (absolute joint control) — a 5B competitor on our target embodiment; a head-to-head offline eval on SO101 held-out data is an obvious next test.

Honesty notes: our n=100 (±8pp) vs their 500; single suite; eval covers init states 0–9 per task (demos of same tasks/init states are in training data — this is the standard LIBERO protocol used by all listed models). Multi-suite evals (object/goal/long) and an identically-protocoled SmolVLA baseline run are in progress.

Dual-rate validation (the architectural claim)

• Offline: held-out chunk MSE with latent from a frame 2s earlier == fresh (0.772 vs 0.772).
• Closed-loop: 86% @ 1 Hz refresh vs 79% full-rate.
• Deployment implication (Raspberry Pi 5): the 0.5B slow path can run at 0.5–1 Hz (llama.cpp GGUF Q4 territory), only the 140M fast path needs real-time rates.

4. Pitfalls we hit (read before reproducing)

1. robosuite/LIBERO renders are 180°-rotated vs recorded datasets. A vertical flip alone leaves a horizontal MIRROR — the policy reaches to the wrong side and scores 0%. Use img[::-1, ::-1] . (lerobot's LiberoEnv.render() flips for visualization but _format_raw_obs feeds policies raw rotated frames — check your eval path.)
2. eef orientation antipodal ambiguity : env quat → rotvec gives −π-ish x-component, datasets store +π-ish. Canonicalize (flip antipodal representation when rotvec_x < 0) or your normalized state input is off by ~17σ.
3. HuggingFaceVLA/libero meta says fps=10 but frames are native 20 Hz (~161 frames/episode ≈ 8s demos). Do NOT action-repeat at eval.
4. GT-replay harness is the decisive convention test : replay dataset actions in the env (first episode per task ↔ init_state 0). 4/5 success = conventions right; policy-independent.
5. HuggingFaceVLA community_dataset v1/v3 are aggregates of v2.1 sub-datasets → convert with lerobot's convert_dataset_v21_to_v30.py ; 4/44 sources have meta≠parquet row counts (broken at source) — validate len(reader.hf_dataset) == meta.total_frames .
6. Qwen3.5 custom forwards need mm_token_type_ids for M-RoPE; patchify must match the official processor bit-exact (verify!). Linear-attention needs fla-core + causal-conv1d or you get a 5× slower torch fallback.
7. Flow-matching loss can drop while conditioning is ignored — A/B with shuffled observations to verify the model actually uses them.

5. Repository layout

src/tinyvla/            # LeRobot plugin: configuration/modeling/processor + modules
scripts/train.py        # accelerate loop: weighted mixture, staleness aug, distill loss, resume
scripts/eval_libero.py  # closed-loop LIBERO eval (all convention fixes, dual-rate, video)
scripts/replay_libero.py, diag_obs_gap.py   # convention diagnostics
scripts/prepare_community_v3.py, label_spatial.py, eval_offline.py
configs/                # stage2/stage3/libero_ft* yaml
checkpoints/            # stage2_final, stage3_final, libero_ft4_final (main)
videos/                 # sample eval episodes

6. Quick start

from tinyvla.modeling_tinyvla import TinyVLAPolicy
policy = TinyVLAPolicy.from_pretrained("AlexWortega/tinyvla")  # ft4 checkpoint at root
# closed-loop LIBERO: python scripts/eval_libero.py --checkpoint <ckpt> \
#   --suite libero_spatial --episodes 100 --n-action-steps 10 --refresh-s 1.0

Trained 2026-08-14 → 2026-08-19 on 1× RTX A6000 (48GB), lerobot 0.6.1, torch 2.10, transformers 5.5.4.