This model is a quantized version of
microsoft/Phi-3-mini-4k-instruct
and is converted to the OpenVINO format. This model was obtained via the
nncf-quantization
space with
optimum-intel
.
First make sure you have
optimum-intel
installed:
pip install optimum[openvino]
To load your model you can do as follows:
from optimum.intel import OVModelForCausalLM
model_id = "helenai/Phi-3-mini-4k-instruct-openvino-4bit"
model = OVModelForCausalLM.from_pretrained(model_id)