cyberagent / multimodal-adversarial-training

[WACV'26] Multimodal Adversarial Training — Resources

This repository hosts model checkpoints and data resources for the paper:

Multimodal Adversarial Defense for Vision-Language Models by Leveraging One-To-Many Relationships Futa Waseda, Antonio Tejero-de-Pablos, Isao Echizen IEEE/CVF Winter Conference on Applications of Computer Vision (WACV), 2026

For the source code and training scripts , please refer to the GitHub repository: 👉 https://github.com/CyberAgentAILab/multimodal-adversarial-training

📘 Overview

This work proposes Multimodal Adversarial Training (MAT) for Vision-Language Models (VLMs). MAT is a unified adversarial training pipeline for image-text retrieval models. The extended version, MAT+ , additionally leverages one-to-many relationships in image-text pairs to improve robustness.

Highlights

• Unified MAT pipeline for image-text retrieval models (CLIP, ALBEF, BLIP).
• MAT+ leverages one-to-many relationships in image-text pairs.
• Reproducible results on Flickr30k and COCO benchmarks.

📘 Directory structure

resources/
├── checkpoints/                          # MAT/MAT+ model checkpoints
│     ├── ALBEF_flickr_MAT_HumanCaps.pth
│     ├── BLIP_flickr_MAT_HumanCaps.pth
│     ├── CLIP_B_coco_MAT_HumanCaps.pth
│     ├── CLIP_B_coco_MAT_base.pth
│     └── CLIP_B_flickr_MAT_HumanCaps.pth
└── augmentations/                        # Data augmentations for MAT+
      ├── dataset_json.zip                # Text augmentation annotations
      └── flickr_SD_I2I_0.5.zip          # Image augmentations (SD img2img)

📘 Checkpoints

Adversarially trained model checkpoints for image-text retrieval:

The base models used for training are:

• ALBEF : salesforce/ALBEF
• BLIP : salesforce/BLIP
• CLIP : openai/CLIP (ViT-B/16)

📘 Augmentations

Data augmentations used to reproduce MAT+ results:

📘 Usage

1. Clone or download this repository:

   # Using the Hugging Face CLI
   hf download cyberagent/multimodal-adversarial-training --local-dir ./resources
   
   # Or using git with LFS
   git lfs install
   git clone https://huggingface.co/cyberagent/multimodal-adversarial-training
   

2. Clone the code repository and follow its setup instructions.

3. Update the checkpoint and data paths in configs/ to point to the downloaded resources.

📘 Citation

If you find these resources useful, please cite:

@inproceedings{waseda2026multimodal,
  title={Multimodal Adversarial Defense for Vision-Language Models by Leveraging One-To-Many Relationships},
  author={Waseda, Futa and Tejero-de-Pablos, Antonio and Echizen, Isao},
  booktitle={Proceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision (WACV)},
  year={2026}
}

📘 Acknowledgements

This work builds upon the following repositories:

• Models : ALBEF , BLIP
• Attacks : Co-Attack , SGA

📘 License

This repository is licensed under the GNU General Public License v3.0 .