mjpsm / roadblock-classifier-v2

🚧 Roadblock Classification Model (v2)

📌 Overview

The Roadblock Classification Model (v2) is a fine-tuned transformer-based model built on BERT to classify student check-ins into two categories:

• ROADBLOCK → The student cannot move forward
• NOT_ROADBLOCK → The student is still making progress

This model is designed to understand semantic meaning , not just keywords, enabling it to differentiate between difficulty and true blockage .

---

🧠 Motivation

❌ Problem with Version 1

The first version of this model attempted to classify:

• struggles
• confusion
• being stuck

all under one label

This created a major issue:

The model could not distinguish between temporary difficulty and actual inability to proceed

---

🔥 Why Version 2 Was Created

Version 2 was developed to separate definitions clearly :

---

💥 Key Insight

Not all struggles are roadblocks.

Example:

---

⚙️ Model Architecture

• Base Model: bert-base-uncased
• Task: Binary Classification
• Framework: Hugging Face Transformers
• Training Environment: Google Colab (GPU)

---

📊 Dataset Design

The dataset was synthetically generated and refined iteratively to ensure:

✅ Semantic Accuracy

• Focus on meaning, not keywords

✅ Balanced Classes

• ROADBLOCK vs NOT_ROADBLOCK distribution controlled

✅ Language Diversity

• Includes:
  • formal phrasing
  • informal/slang expressions
  • varied sentence structures

---

🚨 Bias Identification and Correction

🔍 Initial Problem

Early versions of the dataset showed strong keyword bias , such as:

• "problem" → always NOT_ROADBLOCK
• "can't" → always ROADBLOCK
• "stuck" → always ROADBLOCK

---

⚠️ Why This Was Dangerous

The model learned:

❌ keyword → label
instead of
✅ meaning → label

This caused incorrect predictions in real-world scenarios.

---

🔧 Bias Mitigation Strategy

To eliminate bias, the dataset was redesigned to include:

1. Keyword Symmetry

Each keyword appears in both labels :

---

2. Contrastive Examples

Pairs of sentences with similar wording but different meanings:

• "I can't fix it and I'm stuck" → ROADBLOCK
• "I can't fix it yet but I'm making progress" → NOT_ROADBLOCK

---

3. Pattern Diversity

Avoided over-reliance on patterns like:

• "but" → NOT_ROADBLOCK

Instead included:

• "and I fixed it"
• "and it's working now"
• "and I solved it"

---

✅ Result

The model now learns:

progress vs no progress
instead of relying on surface-level patterns.

---

🧪 Model Evaluation

The model was tested on:

1. Clean Synthetic Data

• Achieved near-perfect validation scores (expected due to dataset similarity)

2. Edge Cases

• Handled ambiguous phrasing correctly

3. Realistic Language

Test examples:

---

⚠️ Observed Limitation

Minor generalization gap:

• "I was confused but it's working now" → incorrectly predicted ROADBLOCK

---

🔧 Fix Approach

Instead of regenerating the dataset:

Add targeted examples to cover missing language patterns

---

🔁 Active Learning Strategy

This model is designed to serve as a base model for active learning .

---

🔥 Active Learning Workflow

1. Model predicts on real check-ins
2. Identify incorrect predictions
3. Collect high-value error samples
4. Add corrected examples to dataset
5. Retrain model

---

💥 Key Principle

High-confidence errors are more valuable than random samples

---

🎯 Goal

Continuously improve the model using real-world feedback , not just synthetic data.

---

🚀 Future Improvements

• Integrate real Slack check-in data
• Expand dataset with informal and noisy text
• Add confidence-based filtering for active learning
• Combine with a Struggle Detection Model for multi-signal analysis

---

🧠 Final Insight

This model represents a shift from:

❌ pattern-based classification
to
✅ meaning-based understanding

---

💯 Conclusion

The Roadblock Classification Model (v2):

• Correctly distinguishes difficulty vs blockage
• Handles diverse language patterns
• Minimizes keyword bias
• Serves as a strong foundation for active learning systems

---

🔥 This is not just a model — it is a continuously improving system.