llm-semantic-router / mom-v1

MoM-v1

Model Description

This is a LoRA (Low-Rank Adaptation) fine-tuned model based on Qwen/Qwen3-0.6B for academic question category classification. Unlike traditional classification approaches, this model uses a generative instruction-following paradigm where the model generates category names as text, which is more natural for causal language models.

This model is part of the semantic-router project and classifies academic questions into 14 categories from the MMLU-Pro dataset.

Why "mom"?

The model name "mom" stands for "Mixture-of-Models" , a core concept in the semantic-router project. The semantic-router implements an intelligent routing system that:

1. Classifies incoming queries into categories (using models like this one)
2. Routes each category to specialized models optimized for that domain
3. Dynamically selects the best model based on the query type and performance scores

This category classifier is a critical component of the MoM architecture, enabling the router to understand query intent and direct it to the most appropriate specialized model (e.g., routing law questions to legal-specialized LLMs, math questions to math-optimized models, etc.).

Key MoM Benefits:

• 🎯 Better Performance : Domain-specific models outperform general-purpose models
• 💰 Cost Efficiency : Route simple queries to smaller models, complex ones to larger models
• ⚡ Optimized Latency : Select faster models when appropriate
• 🔧 Flexibility : Easily add/remove specialized models without retraining

Model Details

• Base Model : Qwen/Qwen3-0.6B (752M parameters, 28 layers, 32k context)
• Task : Academic Question Category Classification (Generative)
• Framework : PyTorch + Transformers + PEFT
• Training Method : LoRA (Low-Rank Adaptation)
• LoRA Rank : 16
• LoRA Alpha : 32
• LoRA Dropout : 0.05
• Target Modules : q_proj, k_proj, v_proj, o_proj, gate_proj, up_proj, down_proj
• Training Epochs : 8
• Batch Size : 4 (with gradient accumulation)
• Learning Rate : 3e-4
• Dataset Size : ~4,200 samples (300 per category × 14 categories)
• Expected Accuracy : 70-85%

Categories

The model classifies questions into 14 academic categories:

1. biology - Biological sciences, life sciences
2. business - Business, management, corporate strategy
3. chemistry - Chemical sciences, molecular chemistry
4. computer science - Computing, algorithms, programming
5. economics - Economic theory, markets, finance
6. engineering - Engineering principles, design
7. health - Medicine, healthcare, public health
8. history - Historical events, periods, analysis
9. law - Legal systems, jurisprudence, regulations
10. math - Mathematics, statistics, calculus
11. other - Miscellaneous topics not fitting other categories
12. philosophy - Philosophical thought, ethics, logic
13. physics - Physical sciences, mechanics, quantum physics
14. psychology - Psychological sciences, behavior, cognition

How It Works

This model uses an instruction-following format where it generates the category name as text:

Input Format:

You are an expert academic classifier. Classify the following question into exactly ONE category. Respond with ONLY the category name.

Categories: biology, business, chemistry, computer science, economics, engineering, health, history, law, math, other, philosophy, physics, psychology

Examples:
Q: What is the optimal capital structure for a corporation?
A: business

Q: How do neurons transmit signals?
A: biology

Q: What are the principles of contract law?
A: law

Now classify this question:
Q: {your question here}
A:

Output: The model generates the category name (e.g., "law", "biology", "computer science")

Performance

Expected Performance

• Validation Accuracy : 70-85% (typical for generative classification)
• Inference Speed : ~50-100ms per question (on GPU)
• Model Size : ~752M base parameters + ~500K trainable LoRA parameters (99%+ parameter reduction)
• Memory Footprint : ~1.5GB (FP16) - Suitable for edge devices and consumer GPUs

Advantages of This Configuration

✅ Compact & Efficient : 0.6B base model is 3x smaller than 1.7B variant
✅ Faster Inference : Lower latency for real-time applications
✅ Lower Memory : Can run on consumer GPUs (RTX 3060, 4060, etc.)
✅ Balanced Dataset : 300 samples/category provides good generalization
✅ Standard LoRA Rank : Rank 16 is the recommended default for most tasks
✅ Production-Ready : Optimized for deployment in resource-constrained environments

Advantages of Generative Approach

✅ Natural for Decoder Models : Qwen3 is a causal language model, so generating text is its native capability
✅ Instruction-Following : Leverages pre-training on instruction-following tasks
✅ Better Generalization : Can handle out-of-distribution questions more gracefully
✅ Explainable : Can be prompted to explain reasoning (with modified prompts)
✅ Higher Accuracy : Expected 70-85% vs 40-60% for classification head approaches

Files

• adapter_model.safetensors : LoRA adapter weights
• adapter_config.json : LoRA-specific configuration
• config.json : Model configuration
• tokenizer_config.json : Tokenizer configuration
• special_tokens_map.json : Special tokens mapping
• label_mapping.json : Category label mappings and instruction template

Comparison with Other Variants

When to use mom-v1:

• ✅ Resource-constrained environments (consumer GPUs, edge devices)
• ✅ Real-time inference requirements (low latency)
• ✅ Cost-sensitive deployments (lower compute costs)
• ✅ Batch processing with limited GPU memory

When to use mom-r1:

• ✅ Maximum accuracy requirements
• ✅ Server-side deployment with ample resources
• ✅ Complex or ambiguous questions
• ✅ Research and experimentation

Limitations

• Single Category Output : Model is trained to output only one category per question
• English Only : Trained on English academic questions
• Domain-Specific : Optimized for MMLU-Pro style academic questions
• Smaller Model : May have slightly lower accuracy than larger variants on edge cases
• Temperature Sensitivity : Performance may vary with different generation parameters

Ethical Considerations

• This model is designed for academic question classification and should not be used for high-stakes decision-making without human oversight
• The model may reflect biases present in the MMLU-Pro dataset
• Users should validate model outputs, especially for edge cases or ambiguous questions
• The smaller model size may lead to more errors on complex or nuanced questions

Citation

If you use this model, please cite:

@misc{qwen3-generative-classifier-lora-v1,
  title={Qwen3-0.6B Generative Category Classifier with LoRA},
  author={Semantic Router Team},
  year={2025},
  url={https://github.com/vllm-project/semantic-router},
  note={Fine-tuned on MMLU-Pro dataset with LoRA rank 16, optimized for efficiency}
}

References

• Base Model : Qwen/Qwen3-0.6B
• Dataset : TIGER-Lab/MMLU-Pro
• LoRA Paper : LoRA: Low-Rank Adaptation of Large Language Models
• PEFT Library : Hugging Face PEFT
• Semantic Router : vllm-project/semantic-router

License

Apache 2.0

Acknowledgments

• Thanks to the Qwen team for the excellent base model
• Thanks to TIGER-Lab for the MMLU-Pro dataset
• Thanks to Hugging Face for the PEFT library and model hosting