Arko007 / deepfake-detector-checkpoints

Deepfake Triage Plugin — Detector Checkpoints

Four proxy-architecture deepfake detector checkpoints trained for the model-agnostic uncertainty + explainability triage plugin described in the research proposal "A Model-Agnostic Uncertainty and Explainability Plugin for Generalizable Deepfake Detection" (Bhumika Tewari, TBVL Lab, IISER Bhopal). Each checkpoint is loaded by the plugin's FastAPI backend ( backend/main.py in Anamitra-Sarkar/deepfake-triage-plugin ) and wrapped with MC-Dropout uncertainty estimation, Grad-CAM/attention explainability, and a joint confidence+explanation-quality triage rule.

Update (2026-07-24): retrained on the full corrected dataset. An earlier version of these checkpoints was trained on data affected by a bug in restore_crops_from_hf.py that silently excluded ~750 real videos' worth of already-extracted crops, leaving that run with an artificially severe ~1:35 real:fake ratio instead of FF++'s actual ~1:6. That bug is fixed; the checkpoints and metrics below are from a full retrain on the corrected, complete dataset. The old real-class F1 numbers (xception 0.725, sbi 0.673, vit 0.627, lsda 0.725 at video level) are superseded by the numbers in this card, which are all equal or higher.

Files

Important scope note: these are architecturally-diverse proxy backbones standing in for the four architectures named in the research proposal (Xception, SBI, UIA-ViT, LSDA) — they reproduce each paper's general architecture family (CNN / augmentation-based CNN / Vision Transformer / latent-augmented CNN) but not each paper's exact published training recipe (e.g. SBI's self-blending augmentation pipeline, UIA-ViT's patch-consistency loss, or LSDA's specific latent-space augmentation method). Treat these as a working proof-of-concept for the plugin architecture, not a reproduction of the original papers' benchmark numbers.

Training data

Real FaceForensics++ (c23) videos via the xdxd003/ff-c23 Kaggle dataset mirror — folder layout: DeepFakeDetection , Deepfakes , Face2Face , FaceShifter , FaceSwap , NeuralTextures (fake) and original (real).

This run used the full ~7000-video dataset (all 6 fake methods + the full real set), face-cropped via MTCNN and persisted to the Arko007/deepfake-ff-face-crops HF dataset repo across multiple preprocessing sessions (resumable, processed_videos.txt -tracked, no video reprocessed twice). The held-out validation split used for the metrics below has 11,666 frames across 1,049 videos (150 real / 899 fake) — consistent with FF++'s ~1:6 real:fake ratio, confirming the corrected restore actually pulled in the full real class this time.

Training setup (from training/train_ddp.py / the training notebook)

All 4 models: --epochs 20 --patience 5 (early stopping on validation loss), AdamW optimizer, ReduceLROnPlateau scheduler, BCEWithLogitsLoss(pos_weight=n_real/n_fake) for class-imbalance correction, plus a WeightedRandomSampler (per-class weight 1/n_class ) during training.

• Xception / SBI / LSDA: --batch_size 64 --lr 1e-4
• ViT: --batch_size 32 --lr 5e-5

Train/val split: 85/15, video-level stratified (not frame-level — see split_samples() in train_ddp.py ), so frames from the same video never leak across the split.

Evaluation methodology and results (real, computed — not illustrative)

Computed by training/evaluate_models.py , which reconstructs the exact held-out validation split ( seed=42 , val_fraction=0.15 ) and reports accuracy, per-class precision/recall/F1, macro-F1, AUROC, and confusion matrices, at both frame level and video level (video-level = mean probability across a video's frames, since frames from the same video are near-duplicates and accuracy alone is misleading under FF++'s class imbalance).

Caveat, stated plainly: this held-out split was also used during training for checkpoint selection (best validation loss / early stopping). It is not a separate, from-scratch generalization test set. Treat these numbers as trustworthy validation-time performance, not an independent-test-set claim.

Video-level metrics (the numbers that matter for real-world triage)

Frame-level metrics

Reading these honestly: accuracy alone would be misleading here (FF++ is fake-heavy) — that's why real-class F1 and AUROC are the headline numbers. Xception is the strongest all-around (real F1 0.855, AUROC 0.989). SBI and ViT show the largest real-precision vs. real-recall gap (they over-flag real videos as fake more often) but their AUROC (0.94-0.97) shows the underlying probability ranking is still strongly separated — that gap is a threshold-calibration property of those two architectures on this data, not evidence the model failed to learn. LSDA sits in between. No model's F1 collapsed under the class imbalance; the pos_weight + WeightedRandomSampler combination held up.

Full machine-readable results (including confusion matrices) are in eval_results.json in this repo.

Cross-architecture calibration, explanation-quality, and triage study (2026-07-25)

Full research-questions study (RQ1-RQ3, see the paper/report in research/ ), run via 20-pass MC-Dropout across the full held-out validation split (11,666 frames / 1,049 videos), plus explanation-quality and triage-transferability metrics on a class-balanced ~4,000-sample draw per model. Raw output: research_results.json in this repo.

RQ1 — Expected Calibration Error (lower is better):

All four are reasonably calibrated (ECE <0.12), but not uniformly — SBI's ECE is ~3.5x Xception's.

H1 test (does MC-Dropout actually improve calibration over raw softmax?): NOT SUPPORTED. A raw single-pass (dropout OFF) baseline was computed separately on the identical val split ( raw_baseline_results.json in this repo) specifically to test H1's literal comparative claim:

MC-Dropout's ECE is statistically indistinguishable from the raw baseline for every architecture, and marginally worse for SBI.

Correction: an earlier version of this card claimed Xception/UIA-ViT's null result was "mechanically guaranteed" by zero dropout probability. That's stale — build_model() was patched to pass drop_rate=0.2 to both, and direct inspection confirms one real Dropout(p=0.2) module exists in each ( head.drop / head_drop ). The actual issue: best_xception.pth / best_vit.pth were trained before that patch and are evaluated here after it — an accidental train/test dropout mismatch, itself the invalid-MC-Dropout scenario, not a zero-variance guarantee.

Valid-config retest ( dropoutfix_eval_results.json , matched train/test dropout via the _dropoutfix checkpoints):

Both retrains converged cleanly and are equal-or-better classification quality than the originals (Xception: 0.924 vs 0.913 macro-F1, 0.998 vs 0.989 AUROC; ViT: 0.862 vs 0.793 macro-F1, 0.987 vs 0.942 AUROC — ViT's first attempt diverged at 47.2% accuracy due to a batch_size/lr mismatch against the original's proven config; a second attempt matching it batch_size=32 lr=5e-5 converged cleanly). Both confirm H1's null result under fully valid, matched train/test dropout conditions — all four architectures now have a valid H1 confirmation, unanimous: MC-Dropout provides no measurable calibration benefit under any tested configuration.

Promoted 2026-07-25 : best_xception.pth and best_vit.pth now are these dropout-fix checkpoints (per the user's explicit go-ahead), re-verified working correctly on the live backend afterward (both fake and real test images, in-browser). The pre-promotion originals are preserved, non-destructively, as best_xception_predropoutfix_backup.pth / best_vit_predropoutfix_backup.pth in this same repo.

RQ2 — Spearman correlation, predictive entropy vs. explanation stability:

Higher uncertainty correlates with less stable explanations, significantly, in 3/4 architectures (not SBI) — small effect sizes throughout.

RQ3 — Triage false-negative capture (same untuned entropy=0.6, stability=0.65 threshold pair for all four models):

The triage rule escalates 88.8-96.1% of true false negatives across every architecture without any per-architecture recalibration — the core transferability claim holds cleanly.

Live deployment verification (2026-07-24)

Both the FastAPI backend and the React frontend were deployed to Modal (T4 GPU, CPU fallback if CUDA raises a RuntimeError mid-request) purely to verify the full product end-to-end with these corrected checkpoints — not a permanent hosting solution (the client's proposal asked for the working product, not hosted infrastructure; the Modal deployment was stopped again after verification).

• /health : {"status": "healthy", "cuda_available": true, ...} .
• /detect on a real (non-fake) FF++ validation frame: returned is_fake: false , probability: 0.00069 (correctly confident this is real), weights_source: "trained" (confirms the real checkpoint loaded — not a silently-failed fallback to ImageNet weights), full triage response (entropy/stability/Grad-CAM heatmap) returned correctly.
• Frontend static build served correctly (200, correct title) and was pointed at the Modal backend for this verification pass only.

As of 2026-07-25, both backend and frontend are deployed to Modal ( deepfake-triage-backend / deepfake-triage-frontend ) for user testing; HF Spaces now only supports Gradio so it is no longer used for hosting this FastAPI+React app, and Render/Vercel are not the live path either (see repo frontend/src/App.jsx MODEL_ENDPOINTS , which points at the Modal backend).

Uncertainty, explainability, and triage (implementation, not just checkpoints)

See plugin_core/ in the repo:

• uncertainty.py — MCDropoutPlugin (stochastic forward passes → mean probability, variance, entropy), plus calculate_ece / generate_reliability_data for calibration analysis — now run against the full labeled held-out split (see the RQ1-RQ3 study section above); the deployed UI's Calibration tab shows these same measured numbers, not illustrative ones.
• explainability.py — Grad-CAM (CNN backbones) / saliency-based attention (ViT), with stability-under-perturbation and spatial-entropy quality metrics.
• triage.py — joint rule combining entropy, explanation stability, and borderline-probability checks into VERIFIED_SAFE / VERIFIED_FAKE / ESCALATE_TO_HUMAN.