llm-semantic-router / mmbert-feedback-detector

mmBERT Feedback Detector

A high-performance multilingual 4-class feedback classification model fine-tuned on mmBERT-base using AMD MI300X GPU .

Model Description

This model classifies user feedback into 4 categories:

Performance

Training Data

• Dataset : llm-semantic-router/feedback-detector-dataset
• Size : 51,694 examples (46,524 train / 5,170 validation)
• Languages : English, Japanese, Turkish
• Labeling : GPT-OSS-120B via vLLM on AMD MI300X
• Sources : MultiWOZ, SGD, INSCIT, MIMICS, Hazumi, Consumer Complaints

Training Configuration

Hardware

Usage

Quick Start

from transformers import pipeline

classifier = pipeline("text-classification", model="llm-semantic-router/mmbert-feedback-detector")

result = classifier("Thank you, that was exactly what I needed!")
print(result)  # [{'label': 'SAT', 'score': 0.99}]

Full Example

from transformers import AutoTokenizer, AutoModelForSequenceClassification
import torch

model = AutoModelForSequenceClassification.from_pretrained("llm-semantic-router/mmbert-feedback-detector")
tokenizer = AutoTokenizer.from_pretrained("llm-semantic-router/mmbert-feedback-detector")

labels = ["SAT", "NEED_CLARIFICATION", "WRONG_ANSWER", "WANT_DIFFERENT"]

def classify(text):
    inputs = tokenizer(text, return_tensors="pt", truncation=True, max_length=512)
    with torch.no_grad():
        outputs = model(**inputs)
    probs = torch.softmax(outputs.logits, dim=-1)
    pred = probs.argmax(-1).item()
    return labels[pred], probs[0][pred].item()

# Test
label, confidence = classify("Thank you, that was helpful!")
print(f"Label: {label}, Confidence: {confidence:.2%}")

Multilingual Examples

# English - Satisfied
classify("Thanks, that's exactly what I needed!")
# => ('SAT', 0.99)

# English - Need clarification
classify("Can you explain that in more detail?")
# => ('NEED_CLARIFICATION', 0.97)

# English - Wrong answer
classify("That's incorrect, the information you gave me was wrong.")
# => ('WRONG_ANSWER', 0.95)

# English - Want different
classify("Can you show me other options instead?")
# => ('WANT_DIFFERENT', 0.94)

# Japanese - Need clarification
classify("もう少し詳しく教えてください")
# => ('NEED_CLARIFICATION', 0.96)

# Turkish - Wrong answer  
classify("Bu yanlış bilgi, düzeltin lütfen")
# => ('WRONG_ANSWER', 0.93)

# German (zero-shot)
classify("Können Sie mir eine andere Option zeigen?")
# => ('WANT_DIFFERENT', 0.89)

# Spanish (zero-shot)
classify("Gracias, eso es exactamente lo que necesitaba!")
# => ('SAT', 0.95)

Use Cases

• Chatbot feedback analysis : Detect user satisfaction in real-time
• Customer service : Route dissatisfied users to human agents
• Dialogue systems : Adapt responses based on user feedback
• Quality monitoring : Track satisfaction metrics across conversations

Limitations

• Best performance on conversational/dialogue text
• May have reduced accuracy on very short inputs (<5 words)
• Cross-lingual transfer works best for Romance and Germanic languages

Related Models

• LoRA Adapter : llm-semantic-router/mmbert-feedback-detector-lora (27 MB)
• Base Model : jhu-clsp/mmBERT-base

Citation

@model{mmbert_feedback_detector,
  title={mmBERT Feedback Detector},
  author={LLM Semantic Router Team},
  year={2025},
  url={https://huggingface.co/llm-semantic-router/mmbert-feedback-detector}
}

License

Apache 2.0