dejanseo / fanout-diffusion

Fanout Diffusion Retriever (540k)

A sub-millisecond, hardware-accelerated continuous consistency retrieval engine that expands broad search queries into coherent 10-query fan-out sets in dense embedding space ($D = 768$) in a single forward pass.

👉 Try the Live Web Playground on Hugging Face Spaces

Built with 1-bit Tensor Core PTX MMA kernels , 1-step consistency distillation , and INT4 outer quantization , delivering:

• 0.329 ms single-query latency (RTX 4090, 1000x faster than legacy multi-step EDM)
• 417,035 QPS native throughput (4,170,350 fanout vectors/sec in standalone C++)
• 1.64 MB model checkpoint (INT4 outer + 1-bit weights)
• 97.5% prompt alignment parity with ground truth targets across all 55,819 queries in the 540k dataset
• Zero-dependency deployment via native C++ executable, ONNX Runtime, and Triton Inference Server

Performance & Architecture Comparison

Semantic Quality Evaluation (540k Dataset)

Evaluated across all 55,819 test queries (558,190 fanout vectors) in data/diffusion_dataset_540k.pt :

1-Bit Architectural Frontier Sweep (Ablation Study)

An empirical frontier sweep (F1–F7) was executed across backbone depth, direct 768-dim embedding projections, wide MLPs, Gram matrix diversity loss, and multi-step consistency leaps:

Repository Files

• checkpoints/champion_b1_consistency_1step_qat.pt (1.64 MB): The champion INT4 QAT 1-step consistency checkpoint.
• checkpoints/champion_b1_consistency_1step.onnx (30.64 MB): Standalone ONNX export compatible with ONNX Runtime & Triton.
• models/fanout_1bit_weights.bin (7.35 MB): Raw packed binary weights for native C++ mma.sync execution.
• bin/fanout_engine.exe (0.48 MB): Standalone native Windows C++ binary with CUDA PTX MMA kernel.
• fanout_deployment_bundle.zip (26.26 MB): Self-contained production deployment bundle.
• data/taxonomy_embeddings.pt (17.14 MB): Pre-computed Google Product Taxonomy embeddings ($D = 768$).
• data/diffusion_dataset_540k.pt (1.8 GB): 55,819 query sets (558,190 query-fanout pairs) embedded via google/embeddinggemma-300m .
• diffusion_540k_best.pt (165.2 MB): Legacy 16-step EDM FP32 checkpoint for baseline comparison.
• load_model.py : Universal loader supporting ONNX, PyTorch 1-step, and legacy EDM.

Quickstart

1. ONNX Runtime (Zero Build Dependencies)

pip install huggingface_hub onnxruntime-gpu sentence-transformers

import torch
from sentence_transformers import SentenceTransformer
from load_model import load_fanout_onnx, sample_consistency_onnx

# 1. Load ONNX model directly from Hugging Face Hub
session = load_fanout_onnx(repo_id="dejanseo/fanout-diffusion")

# 2. Encode broad query using Google embeddinggemma-300m
embedder = SentenceTransformer("google/embeddinggemma-300m", device="cuda")
query_vector = embedder.encode(["running shoes and athletic sneakers"], convert_to_tensor=True, normalize_embeddings=True)

# 3. Generate 10 continuous fan-out vectors in a single pass (sub-millisecond)
fanout_vectors = sample_consistency_onnx(session, query_vector)
# fanout_vectors shape: [1, 10, 768]

2. Standalone C++ Engine ( fanout_engine.exe )

# Run 10,000 queries through native PTX Tensor Core kernel
fanout_engine.exe --model models/fanout_1bit_weights.bin --batch 256 --warmup 200 --benchmark 1000

Output:

Batch 256 Latency: 0.614 ms | Throughput: 417,035 queries/sec (4,170,350 vectors/sec)
Single-Query Latency: 0.329 ms

3. Triton Inference Server

The repository includes a production-ready Triton configuration:

docker run --gpus all --rm -p 8000:8000 -p 8001:8001 -p 8002:8002 \
  -v $(pwd)/triton_model_repository:/models \
  nvcr.io/nvidia/tritonserver:24.08-py3 \
  tritonserver --model-repository=/models