rednote-hilab / dots.vlm1.inst

dots.vlm1

🤗 Hugging Face |    📄 Blog
🖥️ Demo |   💬 WeChat (微信) |   📕 rednote

Visit our Hugging Face (click links above) or check out our live demo to try dots.vlm1! Enjoy!

Introduction

We are excited to introduce dots.vlm1 , the first vision-language model in the dots model family. Built upon a 1.2 billion-parameter vision encoder and the DeepSeek V3 large language model (LLM), dots.vlm1 demonstrates strong multimodal understanding and reasoning capabilities.

Through large-scale pretraining and carefully tuned post-training, dots.vlm1 achieves near state-of-the-art performance in both visual perception and reasoning , setting a new performance ceiling for open-source vision-language models—while still maintaining competitive capabilities in pure-text tasks.

Model Summary

This repo contains the instruction-tuned dots.vlm1 model which has the following features:

• Type: A multimodal vision-language model with 1.2B vision encoder and DeepSeek V3 LLM
• Training Stages: Vision encoder pretraining, VLM pretraining, and supervised fine-tuning (SFT)
• Architecture: NaViT vision encoder + MLP adapter + DeepSeek V3 MoE language model
• Vision Encoder: 1.2B parameters, 42 transformer layers with RMSNorm, SwiGLU, and 2D RoPE
• Supported Languages: English, Chinese
• Context Length: 65,536 tokens
• License: MIT

Model Highlights :

• NaViT Vision Encoder : Trained entirely from scratch rather than fine-tuning an existing vision backbone. It natively supports dynamic resolution and incorporates pure visual supervision in addition to traditional text supervision, thereby enhancing the upper bound of perceptual capacity. Beyond image captioning datasets, a large amount of structured image data was introduced during pretraining to improve the model's perceptual capabilities—particularly for tasks such as OCR.
• Multimodal Training Data : In addition to conventional approaches, dots.vlm1 leverages a wide range of synthetic data strategies to cover diverse image types (e.g., tables, charts, documents, graphics) and descriptions (e.g., alt text, dense captions, grounding annotations). Furthermore, a strong multimodal model was used to rewrite web page data with interleaved text and images, significantly improving the quality of the training corpus.

Example Usage

Environment Setup

You have two options to set up the environment:

Option 1: Using Base Image + Manual Installation

# Use the base SGLang image
docker run -it --gpus all lmsysorg/sglang:v0.4.9.post1-cu126

# Clone and install our custom SGLang branch
# IMPORTANT: Only our specific SGLang version supports dots.vlm1 models
# We have submitted a PR to the main SGLang repository (currently under review):
# https://github.com/sgl-project/sglang/pull/8778
git clone --branch dots.vlm1.v1 https://github.com/rednote-hilab/sglang sglang
pip install -e sglang/python

Option 2: Using Pre-built Image (Recommended)

# Use our pre-built image with dots.vlm1 support
docker run -it --gpus all rednotehilab/dots.vlm1_sglang:v0.4.9.post1-cu126

Multi-Node Deployment

Our model supports distributed deployment across multiple machines. Here's how to set up a 2-node cluster:

Prerequisites:

• Model: rednote-hilab/dots.vlm1.inst
• Node 1 IP: 10.0.0.1 (master node)
• Node 2 IP: 10.0.0.2 (worker node)

Node 1 (Master - rank 0):

export HF_MODEL_PATH="rednote-hilab/dots.vlm1.inst"

python3 -m sglang.launch_server \
    --model-path $HF_MODEL_PATH \
    --tp 16 \
    --dist-init-addr 10.0.0.1:23456 \
    --nnodes 2 \
    --node-rank 0 \
    --trust-remote-code \
    --host 0.0.0.0 \
    --port 15553 \
    --context-length 65536 \
    --max-running-requests 64 \
    --disable-radix-cache \
    --mem-fraction-static 0.8 \
    --chunked-prefill-size -1 \
    --chat-template dots-vlm \
    --cuda-graph-max-bs 64 \
    --quantization fp8

Node 2 (Worker - rank 1):

export HF_MODEL_PATH="rednote-hilab/dots.vlm1.inst"

python3 -m sglang.launch_server \
    --model-path $HF_MODEL_PATH \
    --tp 16 \
    --dist-init-addr 10.0.0.1:23456 \
    --nnodes 2 \
    --node-rank 1 \
    --trust-remote-code \
    --host 0.0.0.0 \
    --port 15553 \
    --context-length 65536 \
    --max-running-requests 64 \
    --disable-radix-cache \
    --mem-fraction-static 0.8 \
    --chunked-prefill-size -1 \
    --chat-template dots-vlm \
    --cuda-graph-max-bs 64 \
    --quantization fp8

Configuration Parameters

Key parameters explanation:

• --tp 16 : Tensor parallelism across 16 GPUs per node
• --nnodes 2 : Total number of nodes in the cluster
• --node-rank : Node identifier (0 for master, 1+ for workers)
• --context-length 65536 : Maximum context length
• --quantization fp8 : Use FP8 quantization for efficiency
• --chat-template dots-vlm : Use custom chat template for dots.vlm model

API Usage

Once the server is launched, you can access the model through OpenAI-compatible API:

curl -X POST http://localhost:15553/v1/chat/completions \
    -H "Content-Type: application/json" \
    -d '{
        "model": "model",
        "messages": [
            {
                "role": "user",
                "content": [
                    {
                        "type": "text",
                        "text": "Please briefly describe this image"
                    },
                    {
                        "type": "image_url",
                        "image_url": {
                            "url": "https://upload.wikimedia.org/wikipedia/commons/thumb/d/dd/Gfp-wisconsin-madison-the-nature-boardwalk.jpg/2560px-Gfp-wisconsin-madison-the-nature-boardwalk.jpg"
                        }
                    }
                ]
            }
        ],
        "temperature": 0.1,
        "top_p": 0.9,
        "max_tokens": 55000
    }'

Performance

On major visual benchmarks, dots.vlm1 has achieved overall performance comparable to leading models such as Gemini 2.5 Pro and Seed-VL1.5 thinking . In particular, it demonstrates strong visual-text understanding and reasoning capabilities on datasets like MMMU , MathVision , and OCR Reasoning .

For typical text-based reasoning tasks (e.g., AIME , GPQA , LiveCodeBench ), dots.vlm1 performs roughly on par with DeepSeek-R1-0528 , showing competitive general capability in mathematics and coding.

Overall, dots.vlm1 approaches state-of-the-art levels in multimodal visual understanding and achieves mainstream performance in text reasoning.

Detailed evaluation results are available in our blog .