Model Details of Qwen2-VL-72B-Instruct-FP8-dynamic
Creation
from transformers import AutoProcessor, Qwen2VLForConditionalGeneration
from llmcompressor.modifiers.quantization import QuantizationModifier
from llmcompressor.transformers import oneshot, wrap_hf_model_class
MODEL_ID = "Qwen/Qwen2-VL-72B-Instruct"# Load model.
model_class = wrap_hf_model_class(Qwen2VLForConditionalGeneration)
model = model_class.from_pretrained(MODEL_ID, device_map="auto", torch_dtype="auto")
processor = AutoProcessor.from_pretrained(MODEL_ID)
# Configure the quantization algorithm and scheme.# In this case, we:# * quantize the weights to fp8 with per channel via ptq# * quantize the activations to fp8 with dynamic per token
recipe = QuantizationModifier(
targets="Linear",
scheme="FP8_DYNAMIC",
ignore=["re:.*lm_head", "re:visual.*"],
)
# Apply quantization and save to disk in compressed-tensors format.
SAVE_DIR = MODEL_ID.split("/")[1] + "-FP8-dynamic"
oneshot(model=model, recipe=recipe, output_dir=SAVE_DIR)
processor.save_pretrained(SAVE_DIR)
# Confirm generations of the quantized model look sane.print("========== SAMPLE GENERATION ==============")
input_ids = processor(text="Hello my name is", return_tensors="pt").input_ids.to("cuda")
output = model.generate(input_ids, max_new_tokens=20)
print(processor.decode(output[0]))
print("==========================================")
Runs of nm-testing Qwen2-VL-72B-Instruct-FP8-dynamic on huggingface.co
927
Total runs
0
24-hour runs
0
3-day runs
0
7-day runs
0
30-day runs
More Information About Qwen2-VL-72B-Instruct-FP8-dynamic huggingface.co Model
More Qwen2-VL-72B-Instruct-FP8-dynamic license Visit here:
Qwen2-VL-72B-Instruct-FP8-dynamic huggingface.co is an AI model on huggingface.co that provides Qwen2-VL-72B-Instruct-FP8-dynamic's model effect (), which can be used instantly with this nm-testing Qwen2-VL-72B-Instruct-FP8-dynamic model. huggingface.co supports a free trial of the Qwen2-VL-72B-Instruct-FP8-dynamic model, and also provides paid use of the Qwen2-VL-72B-Instruct-FP8-dynamic. Support call Qwen2-VL-72B-Instruct-FP8-dynamic model through api, including Node.js, Python, http.
Qwen2-VL-72B-Instruct-FP8-dynamic huggingface.co is an online trial and call api platform, which integrates Qwen2-VL-72B-Instruct-FP8-dynamic's modeling effects, including api services, and provides a free online trial of Qwen2-VL-72B-Instruct-FP8-dynamic, you can try Qwen2-VL-72B-Instruct-FP8-dynamic online for free by clicking the link below.
nm-testing Qwen2-VL-72B-Instruct-FP8-dynamic online free url in huggingface.co:
Qwen2-VL-72B-Instruct-FP8-dynamic is an open source model from GitHub that offers a free installation service, and any user can find Qwen2-VL-72B-Instruct-FP8-dynamic on GitHub to install. At the same time, huggingface.co provides the effect of Qwen2-VL-72B-Instruct-FP8-dynamic install, users can directly use Qwen2-VL-72B-Instruct-FP8-dynamic installed effect in huggingface.co for debugging and trial. It also supports api for free installation.
Qwen2-VL-72B-Instruct-FP8-dynamic install url in huggingface.co: