kingabzpro / gemma4-emotion-lora

gemma4-emotion-lora

This repository contains a LoRA adapter built on top of google/gemma-4-E4B-it for emotion classification on the dair-ai/emotion dataset. The model is prompted to return exactly one label from: sadness , joy , love , anger , fear , or surprise .

Model Details

• Base model: google/gemma-4-E4B-it
• Adapter type: LoRA
• Framework: transformers + peft + trl
• Dataset: dair-ai/emotion
• Labels: sadness , joy , love , anger , fear , surprise

Training in the final notebook used a shuffled subset of the dataset with 4,000 training examples, 400 validation examples, and 400 test examples.

Training Setup

The adapter was trained for 1 epoch with 4-bit loading and bf16 compute. Key settings from the notebook:

• LoRA rank: 16
• LoRA alpha: 32
• LoRA dropout: 0.05
• Learning rate: 1e-4
• Batch size: 8
• Gradient accumulation steps: 2
• Max sequence length: 256
• Optimizer: paged_adamw_8bit

Results

Evaluation was run on the 400-example test split before and after fine-tuning:

This shows a clear improvement for label-only emotion classification with a lightweight adapter.

Intended Use

This model is intended for short English text emotion classification in the six-label setup used by dair-ai/emotion . It is best suited for experimentation, education, and lightweight downstream NLP workflows.

Limitations

• This is a LoRA adapter, not a standalone full model.
• Results are based on a reduced train/validation/test subset, not the full dataset.
• The model is trained for a narrow six-label task and may not generalize well outside this label space.
• Generative decoding can still produce invalid labels in some cases.

Usage

#!pip install -U transformers accelerate bitsandbytes peft
import torch
from transformers import AutoTokenizer, BitsAndBytesConfig
from peft import AutoPeftModelForCausalLM

model_id = "kingabzpro/gemma4-emotion-lora"

bnb_config = BitsAndBytesConfig(
    load_in_4bit=True,
    bnb_4bit_quant_type="nf4",
    bnb_4bit_compute_dtype=torch.bfloat16,
    bnb_4bit_use_double_quant=True,
)

tokenizer = AutoTokenizer.from_pretrained(model_id, use_fast=True)

if tokenizer.pad_token is None:
    tokenizer.pad_token = tokenizer.eos_token

model = AutoPeftModelForCausalLM.from_pretrained(
    model_id,
    quantization_config=bnb_config,
    device_map="cuda:0",
    dtype=torch.bfloat16,
)

messages = [
    {
        "role": "system",
        "content": (
            "You are an emotion classification assistant. "
            "Read the user's text and answer with exactly one label. "
            "Only choose from: sadness, joy, love, anger, fear, surprise. "
            "Return only the label and nothing else."
        ),
    },
    {
        "role": "user",
        "content": "Classify the emotion of this text:\n\nThis is the best day of my life!",
    },
]

inputs = tokenizer.apply_chat_template(
    messages,
    tokenize=True,
    add_generation_prompt=True,
    return_dict=True,
    return_tensors="pt",
).to(model.device)

input_len = inputs["input_ids"].shape[-1]

with torch.no_grad():
    outputs = model.generate(**inputs, max_new_tokens=4)

print(tokenizer.decode(outputs[0][input_len:], skip_special_tokens=True).strip())
#joy

Check out the Kaggle notebook for testing here: Testing Gemma 4 on Human Emotions

Source

• Final training notebook: fine-tune-gemma-4-on-emotions_final.ipynb
• Hub repository: https://huggingface.co/kingabzpro/gemma4-emotion-lora
• Base model: https://huggingface.co/google/gemma-4-E4B-it