py-feat / mp_facemesh_v2

MP_FaceMesh_V2

Model Description

MP_FaceMesh_V2 is a pytorch port of tensorfolow FaceMeshV2 model from Google's mediapipe library. The model takes a cropped 2D face with 25% margin on each side resized to 256 x 256 pixels and outputs a dense 473 landmark coordinates in a 3D (x,y,z) coordinate space.

The original tensorflow model was ported to ONNX and then to pytorch using onnx2torch . Currently, we are serializing the converted model, which requires onnx2torch as a dependency.

See the mediapipe model card for more details.

Model Details

• Model Type : Convolutional Neural Network (MobileNetV2-like)
• Framework : pytorch

Model Sources

• Repository : GitHub Repository
• Model Card : Mediapipe FaceMesh model card
• Paper : Attention Mesh: High-fidelity Face Mesh Prediction in Real-time

Citation

If you use the mp_facemesh_v2 model in your research or application, please cite the following paper:

Grishchenko, I., Ablavatski, A., Kartynnik, Y., Raveendran, K., & Grundmann, M. (2020). Attention mesh: High-fidelity face mesh prediction in real-time. arXiv preprint arXiv:2006.10962.

@misc{grishchenko2020attentionmeshhighfidelityface,
      title={Attention Mesh: High-fidelity Face Mesh Prediction in Real-time}, 
      author={Ivan Grishchenko and Artsiom Ablavatski and Yury Kartynnik and Karthik Raveendran and Matthias Grundmann},
      year={2020},
      eprint={2006.10962},
      archivePrefix={arXiv},
      primaryClass={cs.CV},
      url={https://arxiv.org/abs/2006.10962}, 
}

Example Useage

import torch
from huggingface_hub import hf_hub_download

device = 'cpu'

# Load model and weights
landmark_model_file = hf_hub_download(repo_id='py-feat/mp_facemesh_v2', filename="face_landmarks_detector_Nx3x256x256_onnx.pth")
landmark_detector = torch.load(landmark_model_file, map_location=device, weights_only=False)
landmark_detector.eval()
landmark_detector.to(device)


# Test model
face_image = "path/to/your/test_image.jpg"  # Replace with your extracted face image that is [224, 224]

# Extract Landmarks
landmark_results = landmark_detector(torch.tensor(face_image).to(device))