helenai / Phi-4-mini-instruct-ov-asym

microsoft/Phi-4-mini-instruct converted to OpenVINO with asymmetric INT4 weight compression, for inference with OpenVINO on CPU and Intel GPU.

Quick start with OpenVINO GenAI:

pip install huggingface-hub[cli]
pip install --pre -U openvino openvino-tokenizers openvino-genai --extra-index-url https://storage.openvinotoolkit.org/simple/wheels/nightly
curl -O https://raw.githubusercontent.com/helena-intel/snippets/refs/heads/main/llm_chat/python/llm_chat_manual.py
huggingface-cli download helenai/Phi-4-mini-instruct-ov-asym --local-dir Phi-4-mini-instruct-ov-asym
python llm_chat_manual.py Phi-4-mini-instruct-ov-asym CPU

In the last line, change CPU to GPU to run on Intel GPU.

Model export command:

pip install optimum-intel[openvino]@git+https://github.com/huggingface/optimum-intel.git
pip install --pre -U openvino openvino-tokenizers --extra-index-url https://storage.openvinotoolkit.org/simple/wheels/nightly
optimum-cli export openvino -m microsoft/Phi-4-mini-instruct --weight-format int4 phi-4-mini-instruct-ov-asym

Model export versions and parameters:

openvino_version         : 2025.3.0-19419-3932807324e
nncf_version             : 2.17.0
optimum_intel_version    : 1.25.0.dev0+8f127ce
optimum_version          : 1.26.1
pytorch_version          : 2.6.0
transformers_version     : 4.51.3

advanced_parameters      : {'statistics_path': None, 'awq_params': {'subset_size': 32, 'percent_to_apply': 0.002, 'alpha_min': 0.0, 'alpha_max': 1.0, 'steps': 100, 'prefer_data_aware_scaling': True}, 'scale_estimation_params': {'subset_size': 64, 'initial_steps': 5, 'scale_steps': 5, 'weight_penalty': -1.0}, 'gptq_params': {'damp_percent': 0.1, 'block_size': 128, 'subset_size': 128}, 'lora_correction_params': {'adapter_rank': 8, 'num_iterations': 3, 'apply_regularization': True, 'subset_size': 128, 'use_int8_adapters': True}, 'lora_adapter_rank': 256, 'backend_params': {}}
all_layers               : False
awq                      : True
backup_mode              : int8_asym
compression_format       : dequantize
gptq                     : False
group_size               : 64
ignored_scope            : []
lora_correction          : False
mode                     : int4_asym
ratio                    : 1.0
scale_estimation         : True
sensitivity_metric       : max_activation_variance