Aquiles-ai / Qwen2.5-VL-3B-Instruct-Img2Code

Qwen2.5-VL-3B-Instruct-Img2Code 🎨→💻

Qwen2.5-VL-3B-Instruct-Img2Code is a fine-tuned version of Qwen/Qwen2.5-VL-3B-Instruct specialized in generating clean, functional HTML/CSS code from webpage screenshots .

🎯 Model Description

This model transforms webpage screenshots into semantically correct HTML/CSS code, supporting:

• Tailwind CSS integration
• Responsive design layouts
• Semantic HTML5 structure
• Complex UI components (navigation bars, forms, cards, grids)
• Modern CSS features (flexbox, grid, animations)

🔧 Training Details

• Base model: Qwen/Qwen2.5-VL-3B-Instruct
• Method: LoRA (r=8, alpha=16)
• Target modules: q_proj, k_proj, v_proj, o_proj, gate_proj, up_proj, down_proj
• Dataset: HuggingFaceM4/WebSight
  • 8,100 training examples
  • 900 evaluation examples
• Configuration:
  • Total steps: 438
  • Learning rate: 2e-4 (cosine with warmup)
  • Gradient accumulation: 16
  • Image resolution: 256×28×28 to 512×28×28 pixels
• Hardware: NVIDIA L4 24GB VRAM
• Training time: ~5.5 hours

📊 Performance Metrics

💻 Usage

Installation

pip install transformers torch pillow qwen-vl-utils torchvision
# Optional but recommended:
pip install flash-attn --no-build-isolation

Basic Inference

from transformers import Qwen2_5_VLForConditionalGeneration, AutoProcessor
from qwen_vl_utils import process_vision_info
from PIL import Image
import torch

# Load model
model = Qwen2_5_VLForConditionalGeneration.from_pretrained(
    "Aquiles-ai/Qwen2.5-VL-3B-Instruct-Img2Code",
    dtype=torch.bfloat16,
    attn_implementation="flash_attention_2",  # Requires flash-attn
    device_map="auto"
)

# Without flash-attn:
# model = Qwen2_5_VLForConditionalGeneration.from_pretrained(
#     "Aquiles-ai/Qwen2.5-VL-3B-Instruct-Img2Code",
#     dtype="auto",
#     device_map="auto"
# )

# Load processor
processor = AutoProcessor.from_pretrained(
    "Aquiles-ai/Qwen2.5-VL-3B-Instruct-Img2Code",
    min_pixels=256*28*28,
    max_pixels=1280*28*28
)

# Load image
image = Image.open("screenshot.jpg")

# Prepare messages
messages = [
    {
        "role": "user",
        "content": [
            {
                "type": "image",
                "image": image,
            },
            {"type": "text", "text": "Generate the HTML/CSS code for this webpage screenshot."},
        ],
    }
]


text = processor.apply_chat_template(
    messages, tokenize=False, add_generation_prompt=True
)
image_inputs, video_inputs = process_vision_info(messages)
inputs = processor(
    text=[text],
    images=image_inputs,
    videos=video_inputs,
    padding=True,
    return_tensors="pt",
)
inputs = inputs.to("cuda")

# Inference: Generation of the output
generated_ids = model.generate(**inputs, max_new_tokens=2048)
generated_ids_trimmed = [
    out_ids[len(in_ids) :] for in_ids, out_ids in zip(inputs.input_ids, generated_ids)
]
output_text = processor.batch_decode(
    generated_ids_trimmed, skip_special_tokens=True, clean_up_tokenization_spaces=False
)
print(output_text)

Streaming Inference

from transformers import TextIteratorStreamer
from threading import Thread

# ... (previous loading code)

streamer = TextIteratorStreamer(
    processor,
    skip_prompt=True,
    skip_special_tokens=True
)

generation_kwargs = dict(
    inputs,
    streamer=streamer,
    max_new_tokens=2048
)

thread = Thread(target=model.generate, kwargs=generation_kwargs)
thread.start()

print("Generating code:")
for new_text in streamer:
    print(new_text, end="", flush=True)

thread.join()

Production Deployment with vLLM

Start server

vllm serve Aquiles-ai/Qwen2.5-VL-3B-Instruct-Img2Code \
  --host 0.0.0.0 \
  --port 8000 \
  --api-key dummyapikey \
  --mm-encoder-tp-mode data \
  --limit-mm-per-prompt '{"image":2,"video":0}' \
  --max-model-len=16384 \
  --gpu-memory-utilization=0.90

Request to the server from the OpenAI client

from openai import OpenAI
import base64

def encode_image(image_path):
    with open(image_path, "rb") as image_file:
        return base64.b64encode(image_file.read()).decode("utf-8")

client = OpenAI(api_key="dummyapikey", base_url="http://127.0.0.1:8000/v1")
image_base64 = encode_image("screenshot.jpg")

stream = client.chat.completions.create(
    model="Aquiles-ai/Qwen2.5-VL-3B-Instruct-Img2Code",
    messages=[{
        "role": "user",
        "content": [
            {"type": "text", "text": "Generate the HTML/CSS code for this webpage screenshot."},
            {"type": "image_url", "image_url": {"url": f"data:image/jpeg;base64,{image_base64}"}}
        ]
    }],
    max_tokens=2048,
    stream=True
)

for chunk in stream:
    if chunk.choices[0].delta.content:
        print(chunk.choices[0].delta.content, end="", flush=True)

vLLM Benefits: 20-30x faster, OpenAI-compatible API, continuous batching.

🚀 Capabilities & Limitations

✅ Supported Features

• Landing pages, navigation bars, card layouts, forms
• Dashboards with data visualization placeholders
• Multi-column responsive layouts
• Tailwind CSS and modern CSS (gradients, shadows, animations)
• Semantic HTML5 structure

⚠️ Limitations

• Static HTML/CSS only (no JavaScript logic)
• Uses placeholder images (doesn't extract actual images)
• OCR limitations may affect text accuracy
• Requires manual review before production use
• Best for: Starting point for development, not production-ready code

🔗 Related Products

Aquiles-RAG - High-Performance Retrieval-Augmented Generation

• Repository: https://github.com/Aquiles-ai/Aquiles-RAG
• PyPI: pip install aquiles-rag
• Features:
  • Vector search (Redis HNSW, Qdrant, PostgreSQL pgvector)
  • FastAPI REST API
  • Embedding-agnostic architecture
  • Sync & async Python clients
  • Interactive setup wizard
  • Optional re-ranking

Perfect for: Building intelligent code search systems, documentation assistants, or UI component libraries with semantic search capabilities.

📚 Dataset

WebSight (HuggingFace M4): High-quality webpage screenshots paired with source code, including landing pages, dashboards, blogs, and e-commerce sites.

📄 Citation

@misc{aquiles-qwen-img2code,
  author = {Aquiles-ai},
  title = {Qwen2.5-VL-3B-Instruct-Img2Code: Automated Webpage Screenshot to Code Generation},
  year = {2025},
  publisher = {HuggingFace},
  url = {https://huggingface.co/Aquiles-ai/Qwen2.5-VL-3B-Instruct-Img2Code}
}

🙏 Acknowledgments

• Qwen Team for the excellent base model Qwen2.5-VL-3B-Instruct
• HuggingFace M4 for the WebSight dataset

📜 License

Same license as the base model Qwen2.5-VL-3B-Instruct

Contact: https://aquiles.vercel.app/
Version: 1.0
Last Updated: October 2025