Kiy-K / Fyodor-StarCoder2-7B-Instruct-Agentic

Fyodor-StarCoder2-7B-MoE

A Mixture-of-Experts (MoE) enhanced version of StarCoder2-7B with 8 experts and top-2 routing.

Model Description

This model extends StarCoder2-7B with a custom Mixture-of-Experts architecture, adding 8 expert networks with top-2 routing. The MoE layers are strategically placed throughout the model to enhance performance on code generation and function calling tasks.

Key Features

• 🎯 8 Expert Networks : Specialized sub-networks for different coding patterns
• 🔀 Top-2 Routing : Dynamic expert selection with load balancing
• 💡 Enhanced Code Understanding : Trained on diverse coding and instruction datasets
• 🚀 Efficient Architecture : Only MoE layers and LM head trained, base model frozen
• 🎓 Function Calling : Strong capabilities in structured function calling and API usage

Architecture Details

MoE Configuration

• Number of Experts : 8
• Active Experts per Token : 2
• MoE Interval : Every 16 layers
• Expert Network : FC1 → GELU → Dropout → FC2 with residual connections
• Load Balancing : Advanced gating with noise injection for training stability

Training Strategy

• Base Model : StarCoder2-7B (frozen)
• Trainable Parameters : ~2-3B (MoE layers + LM head)
• Quantization : 4-bit NF4 quantization for base model
• Precision : BFloat16 mixed precision training
• Final Loss : N/A

Training Data

The model was trained on a diverse mixture of datasets:

1. Function Calling : Salesforce/xlam-function-calling-60k (10k samples)
2. Python Code : flytech/python-codes-25k (5k samples)
3. Code Instructions : sahil2801/CodeAlpaca-20k (5k samples)
4. Python Tutorials : iamtarun/python_code_instructions_18k_alpaca (5k samples)
5. General Chat : HuggingFaceH4/ultrachat_200k (5k samples)
6. Reasoning : Open-Orca/OpenOrca (5k samples)
7. Uncensored : cognitivecomputations/dolphin (3k samples)
8. Multi-task : teknium/OpenHermes-2.5 (5k samples)

Total Training Samples : ~43,000

Usage

Installation

pip install torch transformers accelerate bitsandbytes

Loading the Model

from transformers import AutoModelForCausalLM, AutoTokenizer
import torch

# Load tokenizer
tokenizer = AutoTokenizer.from_pretrained("Kiy-K/Fyodor-StarCoder2-7B-Instruct-Agentic")

# Load model
model = AutoModelForCausalLM.from_pretrained(
    "your-username/fyodor-starcoder2-7b-moe",
    torch_dtype=torch.bfloat16,
    device_map="auto",
    trust_remote_code=True,
    ignore_mismatched_sizes=True
)

Code Generation Example

prompt = """### Instruction:
Write a Python function that calculates the factorial of a number recursively.

### Response:
"""

inputs = tokenizer(prompt, return_tensors="pt").to(model.device)

with torch.no_grad():
    outputs = model.generate(
        **inputs,
        max_new_tokens=256,
        temperature=0.7,
        top_p=0.95,
        do_sample=True,
        pad_token_id=tokenizer.eos_token_id
    )

print(tokenizer.decode(outputs[0], skip_special_tokens=True))

Function Calling Example

prompt = """### Instruction:
Generate a function call to search for restaurants near a location.

Available functions:
- search_restaurants(location: str, cuisine: str, price_range: str)

### Response:
"""

inputs = tokenizer(prompt, return_tensors="pt").to(model.device)
outputs = model.generate(**inputs, max_new_tokens=128)
print(tokenizer.decode(outputs[0], skip_special_tokens=True))

Training Details

Hyperparameters

• Batch Size : 16
• Gradient Accumulation Steps : 16
• Effective Batch Size : 256
• Learning Rate : 5e-5
• Epochs : 3
• Warmup Steps : 100
• Max Sequence Length : 384
• Optimizer : AdamW (weight_decay=0.01)
• Scheduler : Linear with warmup
• Mixed Precision : BFloat16

Hardware

• GPU : NVIDIA A100 (recommended) or equivalent
• Memory : ~40GB VRAM with 4-bit quantization
• Training Time : ~8-12 hours (depends on hardware)

Evaluation

The model shows strong performance on:

• ✅ Python code generation
• ✅ Function calling and API usage
• ✅ Code explanation and documentation
• ✅ Multi-turn coding conversations
• ✅ Instruction following

Limitations

• Base model knowledge cutoff applies (training data up to 2023)
• May occasionally generate incorrect or suboptimal code
• Best performance on Python; other languages may vary
• Requires proper prompt formatting for optimal results

Ethical Considerations

• This model can generate code that may have security implications
• Users should review and test all generated code before deployment
• The model inherits biases from training data
• Not suitable for safety-critical applications without human oversight

Citation

If you use this model, please cite:

@misc{fyodor-starcoder2-moe,
  author = {Khoi},
  title = {Fyodor-StarCoder2-7B-MoE: A Mixture-of-Frozen-Experts Code Model},
  year = {2025},
  publisher = {HuggingFace},
  howpublished = {\url{https://huggingface.co/Kiy-K/Fyodor-StarCoder2-7B-Instruct-Agentic}}
}

Also cite the base model:

@misc{lozhkov2024starcoder2,
  title={StarCoder 2 and The Stack v2: The Next Generation},
  author={Lozhkov, Anton and others},
  year={2024},
  eprint={2402.19173},
  archivePrefix={arXiv}
}

License

This model inherits the BigCode OpenRAIL-M license from StarCoder2-7B.

Acknowledgements

• Built on top of StarCoder2-7B by BigCode
• Training datasets from various contributors on HuggingFace
• Inspired by Mixtral and other MoE architectures

Model Card Contact

For questions or issues, please open an issue on the model repository.