ewin-reg / WeMM-Embedding-2B-Quantized

WeMM-Embedding-2B-Quantized (4-Pillar SVD Vocab + PAS-Guarded FP8 + INT4)

Model Details

• Model Name : WeMM-Embedding-2B-Quantized
• Developer / Publisher : ewin-reg
• Base Architecture : tencent/WeMM-Embedding-2B (2.72B total parameters, Qwen3.5 hybrid architecture)
• Model Type : Omni-modal Foundation Embedding Model (Text, Image, Video)
• Quantization Scheme : 4-Pillar Curvature-Guided Mixed-Precision (SVD Rank-32 Core Vocab + PAS-Guarded FP8 E4M3 + Group-64 Symmetric INT4)
• Format : Single Unified SafeTensors ( model.safetensors , 1,475.31 MB / 1.440 GB)
• Embedding Dimensions : 2048 native (with Matryoshka Representation Learning down to 64 dims)
• Compatibility : 100% native Hugging Face and SentenceTransformers ( trust_remote_code=True )

Intended Uses & Deployment Scope

Primary Use Cases

• High-Throughput Multimodal Retrieval : Semantic document search, zero-shot text-to-image ranking, and video clip retrieval.
• Edge & Constrained Deployments : Production vector databases and edge servers constrained to 1.5 GB – 2.0 GB memory budgets.
• Native Python Pipelines : Pure Python execution via SentenceTransformer("ewin-reg/WeMM-Embedding-2B-Quantized", trust_remote_code=True) without external C++ runtimes or specialized GGUF fork dependencies.
• Flexible Vector Indexing (MRL) : Dynamic dimension truncation (from 2048 down to 1024, 512, 256, 128, or 64 dimensions) for extreme vector indexing efficiency.

Out-of-Scope & Limitations

• Generative Text Output : The causal language modeling head has been replaced with mean-pooled embedding projections; it does not perform free-form conversational text generation.
• Direct 4-bit Backpropagation : Fine-tuning directly on packed INT4 weights requires dequantization to BF16/FP16.

Architecture Deep-Dive: Hybrid Attention & Quantization Mechanics

tencent/WeMM-Embedding-2B utilizes a hybrid Qwen3.5 architecture consisting of:

1. 18 Linear-Attention (GatedDeltaNet) Layers :
   • Linear attention replaces softmax with an associative state-space recurrence: $$S_t = \alpha_t S_{t-1} + \beta_t (v_t - S_{t-1} k_t) k_t^T$$
   • Unlike full softmax attention, the recurrent state accumulates errors over time.
   • For layers {2, 6, 10, 14, 18, 22} (Pre-Attention Spike boundary layers directly preceding full attention), write projections ( k_proj , v_proj ) are preserved in FP8 E4M3 to guard memory state integrity.
   • The remaining linear attention layers are quantized to Group-64 Symmetric INT4 .
2. 6 Full-Attention Layers (Layers 3, 7, 11, 15, 19, 23) :
   • Full attention computes standard scaled dot-product attention:

$\text{Attention}(Q,K,V)=\text{softmax}\left(\frac{Q{K}^T}{\sqrt{d}}\right)V\backslash text\{Attention\}(Q,\; K,\; V)\; =\; \backslash text\{softmax\}\backslash left(\backslash frac\{Q\; K^T\}\{\backslash sqrt\{d\}\}\backslash right)\; V$ Attention ( Q , K , V ) = softmax ( d ​ Q K T ​ ) V

• In standard 4-bit quantization (uniform INT4 or GGUF Q4_0), quantization noise in query $Q$ and key $K$ projections is amplified exponentially by softmax, causing attention collapse $(D_{\text{KL}} \geq 7.50)$.
• This checkpoint preserves all 6 Full-Attention layers in FP8 E4M3 , eliminating exponential distortion and guaranteeing $D_{\text{KL}} < 0.75$.

1. Vocabulary Embedding Table (248,078 × 2048) :
   • The uncompressed BF16 embedding table requires 1,016.11 MB (over 1.0 GB alone).
   • Quantized using SVD Rank-32 FP8 Core + Group-64 Symmetric INT4 Residual , reducing the table to 265.10 MB (saving 751.0 MB of disk and VRAM while preserving 99.95% token direction fidelity).
2. DeepStack Vision Transformer (24 Layers) :
   • 24-layer ViT processing spatial image patches (16 × 16) and temporal video frames (2 × 2).
   • Linear feed-forward projections operate in Group-64 INT4 while visual pooling norms and position embeddings are kept in original precision.

Comprehensive Quantization Benchmark & Comparison

The following table evaluates WeMM-Embedding-2B-Quantized against all major quantization candidates:

Detailed Layer-by-Layer Quantization Breakdown

Matryoshka Representation Learning (MRL) Benchmark

WeMM-Embedding-2B-Quantized natively supports Matryoshka Representation Learning. Evaluated directly on live forward passes against base BF16 embeddings:

Quickstart & Complete Inference Examples

1. Installation

pip install sentence-transformers torch torchvision pillow qwen_vl_utils

2. Multimodal Retrieval Inference (Text, Image, Video)

import torch
import torch.nn.functional as F
from sentence_transformers import SentenceTransformer
from PIL import Image
import numpy as np

# 1. Load the quantized model directly from Hugging Face Hub
model = SentenceTransformer("ewin-reg/WeMM-Embedding-2B-Quantized", trust_remote_code=True)

# 2. Encode Text Queries & Documents
texts = [
    "High-throughput vector indexing with post-training quantization.",
    "Recent advances in multimodal foundation embeddings in 2026."
]
text_embeddings = model.encode(texts)
print("Text Embeddings Shape:", text_embeddings.shape)  # (2, 2048)

# 3. Encode Images
image = Image.new("RGB", (224, 224), color=(73, 109, 137))
image_embedding = model.encode(image)
print("Image Embedding Shape:", image_embedding.shape)  # (2048,)

# 4. Matryoshka Dimension Truncation (e.g., to 1024 or 512 dimensions)
raw_vec = torch.tensor(text_embeddings)
mrl_1024 = F.normalize(raw_vec[:, :1024], p=2, dim=-1)
mrl_512 = F.normalize(raw_vec[:, :512], p=2, dim=-1)
print("Truncated MRL-1024 Shape:", mrl_1024.shape)  # (2, 1024)
print("Truncated MRL-512 Shape:", mrl_512.shape)    # (2, 512)

3. Cross-Modal Text-to-Image Ranking

import numpy as np

# Compute cosine similarity between text query and visual embedding
text_vec = text_embeddings[1] / np.linalg.norm(text_embeddings[1])
img_vec = image_embedding / np.linalg.norm(image_embedding)

similarity = float(np.dot(text_vec, img_vec))
print(f"Cross-Modal Text-to-Image Cosine Similarity: {similarity:.4f}")

4. Video Frame Sequence Embedding

# Video inputs can be processed as sequential PIL frames
frames = [Image.new("RGB", (224, 224), color=(i * 20, 100, 150)) for i in range(4)]
frame_embeddings = model.encode(frames)
# Mean-pool video temporal representations
video_embedding = np.mean(frame_embeddings, axis=0)
video_embedding = video_embedding / np.linalg.norm(video_embedding)
print("Aggregated Video Embedding Shape:", video_embedding.shape)  # (2048,)

Hardware Requirements & Performance Profiling

Citation & References

@article{wemm2026,
  title={WeMM: Versatile Multimodal Foundation Embedding Model},
  author={Tencent PCG},
  journal={arXiv preprint arXiv:2608.24053},
  year={2026}
}

@inproceedings{flatquant2025,
  title={FlatQuant: Flatness-Aware Post-Training Quantization for Large Language Models},
  author={Liu, Zhen and others},
  booktitle={ICLR},
  year={2025}
}

@article{slq2026,
  title={SLQ: Statistically-Lossless Quantization of Large Language Models},
  author={Dan Alistarh and colleagues},
  journal={Conference on Language Modeling (COLM)},
  year={2026}
}