jc-builds / CalorieCLIP

🍎 CalorieCLIP: Accurate Food Calorie Estimation

CalorieCLIP is a fine-tuned CLIP model that estimates calories from food images with state-of-the-art accuracy. It outperforms all tested VLMs (including GPT-4o and Claude) while running entirely on-device.

🎯 Key Results

🍽️ Example Predictions

Real predictions from our validation set across multiple datasets:

🚀 Quick Start

Installation

pip install open-clip-torch torch pillow

Python Usage

# Clone or download this repo first, then:
from calorie_clip import CalorieCLIP

# Load model from local directory
model = CalorieCLIP.from_pretrained(".")

# Predict calories
calories = model.predict("food_photo.jpg")
print(f"Estimated: {calories:.0f} calories")

# Batch prediction
images = ["breakfast.jpg", "lunch.jpg", "dinner.jpg"]
results = model.predict_batch(images)

Direct Usage (no wrapper)

import torch
import open_clip
from PIL import Image

# Load CLIP
clip, _, preprocess = open_clip.create_model_and_transforms('ViT-B-32', pretrained='openai')
checkpoint = torch.load('calorie_clip.pt', map_location='cpu', weights_only=False)
clip.load_state_dict(checkpoint['clip_state'], strict=False)

# Load regression head
import torch.nn as nn
class RegressionHead(nn.Module):
    def __init__(self):
        super().__init__()
        self.net = nn.Sequential(
            nn.Linear(512, 512), nn.BatchNorm1d(512), nn.ReLU(), nn.Dropout(0.4),
            nn.Linear(512, 256), nn.BatchNorm1d(256), nn.ReLU(), nn.Dropout(0.3),
            nn.Linear(256, 64), nn.ReLU(), nn.Linear(64, 1)
        )
    def forward(self, x): return self.net(x)

head = RegressionHead()
head.load_state_dict(checkpoint['regressor_state'])
clip.eval(); head.eval()

# Predict
img = preprocess(Image.open('food.jpg')).unsqueeze(0)
with torch.no_grad():
    features = clip.encode_image(img)
    calories = head(features).item()
print(f"{calories:.0f} calories")

Command Line

python calorie_clip.py my_food_image.jpg
# Output: my_food_image.jpg: 342 calories

📊 Training Progress

The model was trained for 30 epochs on the Nutrition5k dataset with:

• Huber Loss for robustness to outliers
• Strong augmentation (rotation, color jitter, flips)
• Fine-tuning last 2 CLIP transformer blocks (9.4% of parameters)
• Differential learning rates (1e-5 for CLIP, 1e-3 for regression head)

🔬 Technical Details

Architecture

┌─────────────────┐     ┌──────────────┐     ┌─────────────┐
│   Food Image    │────▶│  CLIP ViT-B  │────▶│  Regression │────▶ Calories
│   (224×224)     │     │   Encoder    │     │    Head     │
└─────────────────┘     │  (fine-tuned)│     │  (4 layers) │
                        └──────────────┘     └─────────────┘
                              │
                              ▼
                        512-dim features

Model Specs

• Base Model : OpenAI CLIP ViT-B/32
• Fine-tuned Layers : Last 2 transformer blocks + regression head
• Trainable Parameters : 9.4% (8.5M of 90M)
• Input Size : 224×224 RGB
• Output : Single float (calories)

Comparison to VLMs

We tested multiple Vision-Language Models on the same test set:

Key Finding : All tested local VLMs (Qwen, Pixtral) suffered from mode collapse, outputting the same calorie value for all images. CalorieCLIP's regression approach avoids this entirely.

📁 Files

CalorieCLIP/
├── config.json           # Model configuration
├── calorie_clip.pt       # Model weights (PyTorch)
├── calorie_clip.py       # Inference code
├── requirements.txt      # Dependencies
└── assets/
    ├── training_progress.png
    ├── model_comparison.png
    ├── accuracy_breakdown.png
    └── error_distribution.png

📋 Training Data

Trained on a combined dataset of:

• Nutrition5k : 5,006 real cafeteria food images with professional calorie measurements
• Food-101 subset : 8,000+ food images with estimated calories
• Total: 13,004 samples (11,053 train / 1,951 validation)
• Diverse foods : beignets, prime rib, ramen, hamburgers, bruschetta, chicken wings, pork chops, greek salads, sashimi, and more

⚠️ Limitations

• Trained on cafeteria food; may be less accurate for restaurant/home-cooked meals
• Single-dish focused; complex multi-item plates may have higher error
• Portion size estimation is inherently challenging from 2D images
• Not a replacement for professional nutrition advice

🙏 Citation

@software{calorieclip2024,
  author = {Haplo LLC},
  title = {CalorieCLIP: Accurate Food Calorie Estimation from Images},
  year = {2024},
  url = {https://huggingface.co/jc-builds/CalorieCLIP}
}

📄 License

MIT License - free for commercial and personal use.

---

Made with ❤️ by Haplo LLC