eren23 / lewm-models

LeWM Model Collection

Quantized and architecture-variant world models derived from LeWM (Lucas Maes et al., Mila/NYU/Samsung SAIL/Brown).

All models are inference-ready checkpoints with full training provenance, quantization experiments, and hardware benchmark data.

---

TL;DR

See full comparison table →

---

What is LeWM?

LeWM (Latent Encoder World Model) is a JEPA-based vision world model: a ViT encoder compresses 224×224 images into a compact latent vector, and a DiT-style predictor forecasts the next latent given the current state and robot action. It was trained on the PushT environment.

These variants explore:

• Architecture changes : Latent bottleneck dimensions, layer counts, hybrid ALAL attention
• Quantization : INT8, Q4, ternary, WANDA pruning
• Hardware specializations : ESP32-P4 LQ40 binary, FPGA hardwired shift-add, browser WASM

---

Model Index

Slim Architecture Variants (epoch 1)

Hybrid ALAL Encoder Variants (epoch 1)

Baseline + Quantizations (epoch 100+ expert)

Note : All slim variants are epoch 1. Quality will improve with longer training (100+ epochs like the baseline expert). See Training Status .

---

Hardware Benchmarks

Apple Silicon (M5, Zig SIMD NEON)

ESP32-P4 (32MB PSRAM @ 200MHz, PIE SIMD)

Browser WASM (no SIMD, pure Rust)

---

Format Guide

safetensors (standard)

All variants exported as HuggingFace-compatible safetensors + config.json:

slim_96d_4e_4p/
├── config.json          # Architecture config
├── lejepa_weights.safetensors  # f32 weights
└── README.md            # This model's card

Use with Synapse , transformers , or any safetensors-compatible loader.

LQ40 (edge/embedded)

Custom binary format for microcontrollers. No JSON parsing at load time, no filesystem needed.

┌─────────────────────────────────────┐
│ Magic: "LQ40"      (4 bytes)        │
│ config_len: uint32   (4 bytes)       │
│ JSON config          (config_len)    │
│ Weight data          (binary)         │
└─────────────────────────────────────┘

Supports: f32, INT8 (encoder), Q4 (predictor), full Q4, WANDA-pruned Q4.

Use with:

• Synapse ESP32-P4 target
• Synapse browser WASM
• Synapse native runtime

---

Quantization Deep Dive

See docs/quantization.md for full details.

What Was Tried

Why Full Q4 Loses Quality

The encoder's INT8 path uses per-channel symmetric quantization with f32 scales. When we skip INT8 and quantize the encoder directly to Q4:

• Encoder INT8: cos=0.9998 vs f32
• Encoder Q4: cos=0.93 vs f32 (7% quality drop)

The issue is the ViT encoder has high dynamic range in intermediate activations. INT8 preserves more signal per channel. Q4's 32-element block granularity doesn't match the encoder's channel statistics.

Why Ternary Underperforms

Ternary weights ({-1, 0, +1}) theoretically compress 8x more than Q4. In practice:

• Q4 cos: 0.998 vs f32
• Ternary cos: ~0.85 vs f32

The predictor's adaLN modulation is sensitive to weight magnitude, not just sign. Ternary destroys the scale information that adaLN relies on.

---

Hardwired / ASIC Exploration

See docs/hardwired_lewm.md .

Q4 weights (integers -8 to 7) decompose into shift-and-add trees. Multiplication by a constant becomes a wire + adder network — no multiplier circuit, no memory fetch.

Results (RTL synthesized with Yosys):

• 0 BRAM, 0 multipliers for weight storage
• 98.3% of all multiplier circuits eliminated
• cos=1.000 vs f32 (mathematically identical)
• Fits in $129 Arty A7 at 12% LUT utilization

---

Conversion Pipeline

1. Download from W&B

pip install wandb
wandb login
python scripts/download_from_wandb.py \
  --project eren23/crucible-lewm \
  --name lewm_slim_96d_4e_4p_epoch_1 \
  --output-dir slim_96d_4e_4p/

2. Convert .ckpt → safetensors + config.json

python scripts/convert_lewm_ckpt.py \
  --input slim_96d_4e_4p/lewm_slim_96d_4e_4p_epoch_1.ckpt \
  --output slim_96d_4e_4p/

The converter uses a stub unpickler — no jepa or module package required.

3. Export to LQ40 (edge formats)

cd synapse
cargo run --release -p synapse-inference --example export_lewm_q4 -- \
  --checkpoint ../lewm-models/slim_96d_4e_4p/lejepa_weights.safetensors \
  --config ../lewm-models/slim_96d_4e_4p/config.json \
  --mode full \
  --output ../lewm-models/slim_96d_4e_4p/lewm-slim-96d-full.bin

Modes: full (INT8+Q4), q4-pred (Q4 predictor only), wanda20-q4 , wanda40-q4 .

---

Citation

If you use these models or the quantization results, cite the original LeWM paper:

@article{maes2025lewm,
  title={LeWorldModel: Stable End-to-End Joint-Embedding Predictive Architecture from Pixels},
  author={Maes, Lucas and Le Lidec, Quentin and Scieur, Damien and LeCun, Yann and Balestriero, Randall},
  journal={arXiv},
  year={2025}
}

For the quantization and architecture experiments, cite this collection:

@misc{lewm_models_2026,
  title={LeWM Model Collection: Quantized and Architecture Variants},
  author={Attocoder Team},
  year={2026},
  publisher={GitHub},
  url={https://github.com/attocode/lewm-models}
}

---

License

All models are derived from LeWM, which is licensed under CC BY-NC 4.0 .

You are free to:

• Share : copy and redistribute the material
• Adapt : remix, transform, and build upon the material

Under the following terms:

• Attribution : You must give appropriate credit to the original LeWM authors
• NonCommercial : You may not use the material for commercial purposes

See CC BY-NC 4.0 for details.