py-feat / face_multitask_v2

face_multitask_v2

A single multi-task convolutional model for facial behavior analysis, used by py-feat 's Detectorv2 . From one face crop it jointly predicts action units, categorical emotion, valence/arousal, eye gaze, a 478-point face mesh, 6-DoF head pose, and 52 MediaPipe/ARKit blendshapes (the v2.7 model; replaces v2.6).

• Backbone: ConvNeXt-V2 Tiny (FCMAE + IN-22k/IN-1k pretrained)
• Heads: ME-GraphAU AU graph (AFG/FGG/SC) + unified-feature emotion/V-A heads
  • landmark, pose, and blendshape regression heads + the v2.6 eye-aware gaze head : RoI-pooled eye features (localized by the predicted mesh), conditioned on predicted head pose (6D), with an L2CS-style binned prediction over the full ±180° range (v2.7: 2° bins)
• Params: ~42M · Input: 224×224 RGB (from a 256×256 face crop)
• File: face_multitask_v27.safetensors (safetensors; ModelV2Config JSON in the file metadata)
• Weights: soup (equal average) of five consecutive fine-tuning checkpoints

Outputs

Benchmarks (held-out, file-verified — v2.7 deployed checkpoint)

All gaze splits are identity-disjoint from training (held-out subjects), and EYEDIAP is never trained on by any py-feat model.

Notes: v2.7 = the v2.6 architecture retrained with a rescaled gaze loss, a rebalanced within-gaze data mix, per-source augmentation, 2° gaze bins, and a head-pose label fix. Vs v2.6 it improves out-of-distribution gaze (EYEDIAP −0.7°, Columbia −1.3°), RAF-DB macro-F1 end-to-end (+2.7), AFEW-VA arousal, blendshape fidelity (best of any release), and occlusion robustness, at the cost of ~0.6° on frontal MPIIGaze and 0.01 on the 12-AU set.

Known limitation (v2.6 and v2.7): continuous valence/arousal on video-frame corpora (Aff-Wild2, AFEW-VA) is substantially below v2.5 (e.g. Aff-Wild2 CCC-V 0.85 → 0.33). AffectNet (still-image) V/A is unaffected. If frame-wise continuous V/A on video is your primary measure, prefer the v2.5 weights ( face_multitask_v2.safetensors , still published in this repo). A fix is under investigation. Numbers are from the deployed checkpoint (v2.7 stage-3 soup ep05-09), weight-verified against the published .safetensors .

Usage

from feat import Detectorv2
detector = Detectorv2(device="cuda")
fex = detector.detect("image.jpg")   # returns a py-feat Fex

The model expects a face crop produced by RetinaFace + py-feat's extract_face_from_bbox_torch(frame, bbox, face_size=256, expand_bbox=1.2) , then resized to 224 and ImageNet-normalized. Detectorv2 handles this.

License

Research / non-commercial use only. Trained on datasets (AffectNet, DISFA+, RAF-DB, Aff-Wild2, BP4D, etc.) whose licenses restrict use to academic research. The ConvNeXt-V2 backbone is MIT-licensed. Confirm each constituent dataset's terms before any non-research use.