llm-jp / llm-jp-clip-vit-base-patch16

Model Card for llm-jp-clip-vit-base-patch16

Model Details

Japanese CLIP model trained with OpenCLIP on relaion2B-en-research-safe-japanese-translation , a Japanese translation of the English subset of ReLAION-5B ( https://huggingface.co/datasets/laion/relaion2B-en-research-safe ), translated by gemma-2-9b-it .

The total number of parameters of this model is 248M.

How to Use

Installation

$ pip install open_clip_torch

Zero-shot Image Classification

import open_clip

model, preprocess = open_clip.create_model_from_pretrained('hf-hub:llm-jp/llm-jp-clip-vit-base-patch16')
tokenizer = open_clip.get_tokenizer('hf-hub:llm-jp/llm-jp-clip-vit-base-patch16')

import torch
from PIL import Image
import requests

url = 'http://images.cocodataset.org/val2017/000000039769.jpg'
image = Image.open(requests.get(url, stream=True).raw)
image = preprocess(image).unsqueeze(0)
text = tokenizer(["猫", "犬", "鳥"])

with torch.no_grad(), torch.cuda.amp.autocast():
    image_features = model.encode_image(image)
    text_features = model.encode_text(text)
    image_features /= image_features.norm(dim=-1, keepdim=True)
    text_features /= text_features.norm(dim=-1, keepdim=True)

    text_probs = (100.0 * image_features @ text_features.T).softmax(dim=-1)

print("Label probs:", text_probs)
# Label probs: tensor([[9.9425e-01, 5.2273e-03, 5.2600e-04]])

Reference:

• Using OpenCLIP at Hugging Face , HuggingFace Docs
• OpenCLIP repository

Training Details

Model Architecture

• Text Encoder: RoBERTa base with llm-jp-tokenizer
• Image Encoder: ViT-B/16

Training Data

This model is trained on relaion2B-en-research-safe-japanese-translation . Due to a 70% success rate in image downloads, the dataset size was 1.45 billion samples, and we processed it over 9 epochs (13 billion samples in total).

Evaluation

Evaluation Code: https://github.com/llm-jp/clip-eval

Table: Performance of each model in zero-shot image classification and image-text retrieval tasks. Bold indicates first place, and underline indicates second place.

LICENSE

The Apache License, Version 2.0

Please refer to the Gemma Terms of Use , as the training data was translated using gemma-2-9b-it. We utilizes Gemma solely for translation purposes. According to the definition of "Model Derivatives" in Section 1.1(e), our model does not fall under the category of a "model in order to cause that model to perform similarly to Gemma." Therefore, we have concluded that it is not necessary to inherit the Gemma license.

Citation

Bibtex:

@inproceedings{sugiura2025clip,
author = {杉浦 一瑳 and 栗田 修平 and 小田 悠介 and 河原大輔 and 岡崎 直観},
month = mar,
series = {言語処理学会第31回年次大会 (NLP2025)},
title = {オープンLLMによる翻訳を活用した日本語 CLIP の開発},
year = {2025}
}