aufklarer / EmbeddingGemma-300M-MLX-8bit

EmbeddingGemma 300M — MLX 8-bit

MLX conversion of google/embeddinggemma-300m for on-device transcript search on Apple Silicon. It produces the checkpoint's own 768-dimensional sentence embeddings: prompt, bidirectional Gemma 3 encoder, mean pooling, dense head, L2 normalization.

Exported from the upstream checkpoint at revision 57c266a740f537b4dc058e1b0cda161fd15afa75 , not from a third-party MLX conversion.

Usage (Swift / MLX)

Used by speech-runtime 's TextEmbedding module:

import TextEmbedding

let embedder = try await EmbeddingGemma.load(directory: bundleDirectory)
let passages = embedder.embed(documents: ["The deadline moved to March."], batchSize: 16)
let query = embedder.embed(query: "when is the deadline")

A query and the documents that answer it are embedded under different prompts, which embed(query:) and embed(documents:) apply.

Model details

• Architecture : Gemma 3 bidirectional encoder — 24 layers, hidden size 768, 3 attention heads over 1 key/value head, head dimension 256
• Head : mean pooling, dense 768 → 3072 → 768, L2 normalization
• Embeddings : 768 dimensions, float32
• Precision : 8-bit, group 64
• Size : 327 MB
• Context : 512 tokens including <bos> and <eos>
• Prompts : title: none | text: for a document, task: search result | query: for a query

Fidelity

Cosine against the float32 source through sentence-transformers, on identical token ids, over 240 meeting-transcript passages and 800 utterances: 0.9989 minimum, 0.9996 mean .

What a runtime has to match

Three details decide whether a port reproduces this model, and each was measured against the source:

• Bidirectional window 257. Full layers see every token; sliding layers see tokens fewer than 257 positions away, which is what transformers derives from the stored sliding_window: 512 for a bidirectional Gemma 3. It is written into config.json as bidirectional_window . Using 512 fell to 0.9985 cosine on a 294-token passage, and a causal mask scored 0.54–0.65.
• Padding is masked in attention , not only in pooling.
• Norm weights include Gemma's +1 ( norm_weights_include_offset in config.json ), so every norm is a plain RMSNorm.

Conversion

From soniqo/speech-models , models/embeddinggemma/export :

poetry run python convert_mlx.py --output EmbeddingGemma-300M-MLX-8bit \
    --bits 8

The exporter's tests hold the MLX implementation to the source checkpoint through sentence-transformers in float32.

License

The base model is Gemma, and this is a derivative of it: use is governed by the Gemma Terms of Use and the Gemma Prohibited Use Policy , which apply to these weights as they apply to the original. The weights are converted and quantized; nothing else about the model is changed.