Aquiles-ai / MoonViT-3D

The model weights and their respective implementation were taken from moonshotai/Kimi-K2.6 , so all credit and thanks go to the Moonshot AI team.

Example usage:

from transformers import AutoImageProcessor, AutoModel
import torch
from PIL import Image

img = Image.open("a_red_car.png")
processor = AutoImageProcessor.from_pretrained("Aquiles-ai/MoonViT-3D", trust_remote_code=True)
model = AutoModel.from_pretrained("Aquiles-ai/MoonViT-3D", trust_remote_code=True,
    dtype=torch.bfloat16, attn_implementation="eager" # or flash_attention_2
  ).to("cuda")

inputs = processor.preprocess([{"type": "image", "image": img}], return_tensors="pt").to("cuda")
out = model(pixel_values=inputs["pixel_values"], grid_thws=inputs["grid_thws"])
print([o.shape for o in out])

Batch processing:

from transformers import AutoImageProcessor, AutoModel
import torch
from PIL import Image

img = Image.open("a_red_car.png")
img2 = Image.open("a_gray_cat.png")

medias = [
    {"type": "image", "image": img},
    {"type": "image", "image": img2},
]

processor = AutoImageProcessor.from_pretrained("Aquiles-ai/MoonViT-3D", trust_remote_code=True)
model = AutoModel.from_pretrained("Aquiles-ai/MoonViT-3D", trust_remote_code=True,
    dtype=torch.bfloat16, attn_implementation="eager" # or flash_attention_2
  ).to("cuda")

inputs = processor.preprocess(medias, return_tensors="pt").to("cuda")
out = model(pixel_values=inputs["pixel_values"], grid_thws=inputs["grid_thws"])
print([o.shape for o in out])

Actual use cases for this model:

• Image embedding extraction for similarity search (image retrieval), dataset deduplication, or image clustering.
• Frozen backbone for downstream fine-tuning: attach a classification, detection, or regression head on top of the features and train only that part.
• Building block for your own VLM: connect it to an LLM (via a projection layer such as a Q-Former or MLP) if you want to replicate something similar to Kimi-VL but with your own text decoder.
• Multimodal RAG: generate image embeddings to index alongside text in a vector database and perform cross-modal text-to-image retrieval.
• Frame-by-frame video processing: since it supports grid_thws with a temporal dimension, in theory you can pass sequences of frames and get spatio-temporal features, although the modelcard doesn't include an example of this — you'd need to build it yourself.