nm-testing / Qwen2-VL-72B-Instruct-FP8-dynamic

huggingface.co
Total runs: 927
24-hour runs: 0
7-day runs: 0
30-day runs: 0
Model's Last Updated: February 27 2025
image-text-to-text

Introduction of Qwen2-VL-72B-Instruct-FP8-dynamic

Model Details of Qwen2-VL-72B-Instruct-FP8-dynamic

Creation
from transformers import AutoProcessor, Qwen2VLForConditionalGeneration

from llmcompressor.modifiers.quantization import QuantizationModifier
from llmcompressor.transformers import oneshot, wrap_hf_model_class

MODEL_ID = "Qwen/Qwen2-VL-72B-Instruct"

# Load model.
model_class = wrap_hf_model_class(Qwen2VLForConditionalGeneration)
model = model_class.from_pretrained(MODEL_ID, device_map="auto", torch_dtype="auto")
processor = AutoProcessor.from_pretrained(MODEL_ID)

# Configure the quantization algorithm and scheme.
# In this case, we:
#   * quantize the weights to fp8 with per channel via ptq
#   * quantize the activations to fp8 with dynamic per token
recipe = QuantizationModifier(
    targets="Linear",
    scheme="FP8_DYNAMIC",
    ignore=["re:.*lm_head", "re:visual.*"],
)

# Apply quantization and save to disk in compressed-tensors format.
SAVE_DIR = MODEL_ID.split("/")[1] + "-FP8-dynamic"
oneshot(model=model, recipe=recipe, output_dir=SAVE_DIR)
processor.save_pretrained(SAVE_DIR)

# Confirm generations of the quantized model look sane.
print("========== SAMPLE GENERATION ==============")
input_ids = processor(text="Hello my name is", return_tensors="pt").input_ids.to("cuda")
output = model.generate(input_ids, max_new_tokens=20)
print(processor.decode(output[0]))
print("==========================================")

Runs of nm-testing Qwen2-VL-72B-Instruct-FP8-dynamic on huggingface.co

927
Total runs
0
24-hour runs
0
3-day runs
0
7-day runs
0
30-day runs

More Information About Qwen2-VL-72B-Instruct-FP8-dynamic huggingface.co Model

More Qwen2-VL-72B-Instruct-FP8-dynamic license Visit here:

https://choosealicense.com/licenses/apache-2.0

Qwen2-VL-72B-Instruct-FP8-dynamic huggingface.co

Qwen2-VL-72B-Instruct-FP8-dynamic huggingface.co is an AI model on huggingface.co that provides Qwen2-VL-72B-Instruct-FP8-dynamic's model effect (), which can be used instantly with this nm-testing Qwen2-VL-72B-Instruct-FP8-dynamic model. huggingface.co supports a free trial of the Qwen2-VL-72B-Instruct-FP8-dynamic model, and also provides paid use of the Qwen2-VL-72B-Instruct-FP8-dynamic. Support call Qwen2-VL-72B-Instruct-FP8-dynamic model through api, including Node.js, Python, http.

Qwen2-VL-72B-Instruct-FP8-dynamic huggingface.co Url

https://huggingface.co/nm-testing/Qwen2-VL-72B-Instruct-FP8-dynamic

nm-testing Qwen2-VL-72B-Instruct-FP8-dynamic online free

Qwen2-VL-72B-Instruct-FP8-dynamic huggingface.co is an online trial and call api platform, which integrates Qwen2-VL-72B-Instruct-FP8-dynamic's modeling effects, including api services, and provides a free online trial of Qwen2-VL-72B-Instruct-FP8-dynamic, you can try Qwen2-VL-72B-Instruct-FP8-dynamic online for free by clicking the link below.

nm-testing Qwen2-VL-72B-Instruct-FP8-dynamic online free url in huggingface.co:

https://huggingface.co/nm-testing/Qwen2-VL-72B-Instruct-FP8-dynamic

Qwen2-VL-72B-Instruct-FP8-dynamic install

Qwen2-VL-72B-Instruct-FP8-dynamic is an open source model from GitHub that offers a free installation service, and any user can find Qwen2-VL-72B-Instruct-FP8-dynamic on GitHub to install. At the same time, huggingface.co provides the effect of Qwen2-VL-72B-Instruct-FP8-dynamic install, users can directly use Qwen2-VL-72B-Instruct-FP8-dynamic installed effect in huggingface.co for debugging and trial. It also supports api for free installation.

Qwen2-VL-72B-Instruct-FP8-dynamic install url in huggingface.co:

https://huggingface.co/nm-testing/Qwen2-VL-72B-Instruct-FP8-dynamic

Url of Qwen2-VL-72B-Instruct-FP8-dynamic

Qwen2-VL-72B-Instruct-FP8-dynamic huggingface.co Url

Provider of Qwen2-VL-72B-Instruct-FP8-dynamic huggingface.co

nm-testing
ORGANIZATIONS

Other API from nm-testing