tahamajs / Inductive_Biases_on_Shortcut_Learning

Inductive Biases on Shortcut Learning

A research codebase investigating how optimization-induced inductive biases (batch size, learning rate, and loss function) influence the tendency of deep networks to rely on shortcuts and the resulting group‑robustness of their predictions.

The goal of this project is to provide a reliable , configurable , and reproducible PyTorch pipeline that accompanies a paper exploring shortcut learning across several benchmark datasets.

🚀 Features

• Datasets supported : Waterbirds (WILDS), CelebA, Colored MNIST (CMNIST, automatically generated), with an optional domino‑style synthetic dataset for controlled experiments.
• Model zoo : ResNet‑18, ResNet‑50, a small convolutional network, and a simple multilayer perceptron (MLP).
• Loss functions : standard cross‑entropy, focal loss, label smoothing, and a gradient‑norm‑regularized objective used in the paper.
• Evaluation metrics : overall accuracy and worst‑group accuracy (WGA) following the group‑distributional robustness literature.
• Reproducibility : deterministic seeds, YAML configuration files, hyperparameter sweep scripts, CSV logging, and complete unit test coverage.
• Extensible : easy to plug in new datasets, architectures, or loss functions via the config system.

📁 Repository Structure

.
├── configs/                # yaml experiment configurations
├── data/                   # dataset root (user provides raw files here)
├── models/                 # model definitions and factory
├── paper/                  # LaTeX source for the accompanying paper
├── scripts/                # main training/evaluation/sweep/analysis scripts
├── tests/                  # unit and integration tests (pytest)
├── .github/workflows/      # continuous‑integration pipeline
└── requirements.txt        # python dependencies

🛠️ Setup & Installation

This project targets Python 3.8+ and assumes a Unix‑like environment (Linux/macOS).

1. Create a virtual environment and activate it

   python -m venv .venv
   source .venv/bin/activate      # or `.\.venv\Scripts\activate` on Windows
   

2. Install dependencies

   pip install --upgrade pip
   pip install -r requirements.txt
   # optional: `pip install torch torchvision` to install a specific CUDA build
   

3. (Optional) install extra packages for paper compilation and plotting

   pip install matplotlib seaborn pandas pyyaml
   

📥 Data Preparation

Each dataset has slightly different requirements; place the raw files under data/ as specified below.

Waterbirds (WILDS)

1. Download the dataset from WILDS or use the Kaggle mirror.
2. Place the extracted folder at data/waterbirds/ . The directory should contain:
   • metadata.csv with column img_filename pointing to image files.
   • the images/ subdirectory with all bird photographs.

CelebA

1. Obtain the official CelebA release from LFWA .
2. Arrange files as follows under data/celebA/ :
   • list_attr_celeba.csv
   • list_eval_partition.csv
   • img_align_celeba/ (all aligned face images)

Colored MNIST (CMNIST)

The CMNIST images are generated automatically when you run the training script; you do not need to download anything manually. The code uses torchvision's MNIST loader and adds random color correlations on the fly.

You can also pre‑download the base MNIST files for offline use:

kaggle datasets download -d hojjatk/mnist-dataset -p data/cmnist
unzip data/cmnist/mnist-dataset.zip -d data/cmnist

Domino (synthetic)

No files are required; the dataset is synthesized in memory by scripts/data_loader.py when --dataset domino is specified.

🏃‍♀️ Running Experiments

All configurations live under configs/ and can be overridden via command‑line flags.

Train a model

python scripts/train.py --config configs/base.yaml

To vary individual parameters without editing the YAML file:

python scripts/train.py \
    --config configs/base.yaml \
    --dataset cmnist \
    --model small_cnn \
    --loss gradnorm \
    --batch_size 128 \
    --lr 0.001 \
    --seed 42

Checkpoints and log files are written to results/<dataset>_<experiment_name>/ by default; see configs/base.yaml for path settings.

Evaluate a trained checkpoint

python scripts/eval.py \
    --checkpoint results/cmnist_baseline/best_model.pt \
    --dataset cmnist \
    --model small_cnn \
    --loss ce

The evaluation script computes both overall accuracy and worst‑group accuracy and prints a concise summary.

Hyperparameter sweeps

The sweep.py script automates over a grid of learning rates, batch sizes, seeds, etc.

python scripts/sweep.py --config configs/base.yaml

After sweeps finish, aggregate the results and produce plots:

python scripts/aggregate_results.py --results_root results --output_csv results/all_results.csv
python scripts/plot_results.py \
    --input_csv results/all_results.csv \
    --output_png results/wga_vs_lr_bs.png

The plotting utilities rely on matplotlib and pandas .

🧪 Testing

Run the full test suite (requires the virtual environment):

pytest -q

Unit tests cover data loaders, model factories, loss implementations, and basic training loops. New code should include corresponding tests to maintain >90% coverage.

📄 Paper

The narrative accompanying these experiments lives in paper/main.tex . To compile:

cd paper
pdflatex main.tex     # or use your favorite LaTeX toolchain

Figures generated by the analysis scripts can be included by pointing LaTeX at results/ .

🤝 Contributing

Contributions are welcome! A few guidelines:

1. Open an issue before adding large features to discuss design.
2. Follow the YAML configurability pattern when introducing new options.
3. Add tests for new functionality and ensure pytest passes.
4. Update this README with any new instructions or requirements.

📜 License

This repository is released under the MIT license. See LICENSE for details.

✨ Acknowledgements

This codebase was developed as part of the project described in the paper "Inductive Biases on Shortcut Learning" by [authors].

If you use this code in your research, please cite the paper accordingly. Feel free to contact the author(s) for questions or collaborations.