yasserrmd / LFM2.5-1.2B-onpolicy

LFM2.5-1.2B-onpolicy

A high-performance 1.2B parameter language model fine-tuned from LFM2.5-1.2B-Instruct using on-policy self-distillation, inspired by Self-Distillation Fine-Tuning (Shenfeld et al., 2026) but with simplified demonstration handling. Optimized for instruction-following and general-purpose assistant tasks.

Model Overview

Training Details

Fine-Tuning Approach: On-Policy Self-Distillation

This implementation is inspired by SDFT (Shenfeld et al., 2026) but uses a simplified approach to on-policy learning:

✅ SDFT Features Implemented

1. On-Policy Learning :

   • Student model generates trajectories: y ~ π_θ(·|x)
   • Both student and teacher evaluated on same generated sequences
   • Prevents off-policy distribution shift

2. Reverse KL Divergence :

   • Minimizes: D_KL(π_θ ∥ π)
   • Computed token-level via log probability differences
   • Prevents mode collapse

3. Teacher as Base Model :

   • Teacher = LFM2.5-1.2B-Instruct (same architecture)
   • Self-distillation from stronger base model
   • Preserves pre-training knowledge

4. EMA Teacher Updates :

   • Teacher parameters exponentially averaged from student
   • Stable target for KL divergence
   • EMA coefficient: α = 0.02 (slow decay)

5. 8-Bit Quantization :

   • Both models in 8-bit via BitsAndBytes
   • Memory efficient (1.2GB per model)
   • Reduces GPU memory footprint to 35GB peak

❌ SDFT Features Omitted (Simplified)

1. Demonstration Conditioning ⚠️

   • Paper Definition : Teacher conditioned on both query and demonstration π(·|x, c)
   • This Implementation : Teacher only sees student-generated tokens π(·|x)
   • Rationale : Simplified to focus on on-policy distillation without explicit demonstration signals
   • Impact : Loss of implicit reward signal from demonstrations

2. Demo-Aware Logits in Loss ⚠️

   • Paper : Loss computed on π_θ(y|x) vs π(y|x,c) divergence
   • This Implementation : Loss computed on π_θ(y|x) vs π(y|x) divergence
   • Impact : Teacher provides base model alignment, not demo-based guidance

3. Analytic Per-Vocabulary KL ⚠️

   • Paper : Full vocabulary KL over all tokens (lower variance): Σ_y π_θ(y) log(π_θ/π)
   • This Implementation : Sampled-token KL only on generated tokens (higher variance): log π_θ(y_t) - log π(y_t)
   • Paper Note : Explicitly cautions this estimator is "higher variance and less stable"
   • Impact : Higher gradient variance, potentially slower convergence

4. Explicit Demonstration Prompt Template ⚠️

   • Paper : Teacher uses explicit in-context prompt: <Query> This is an example: <Demo> Now answer...
   • This Implementation : Demonstrations prepared in dataset but not applied during teacher forward pass
   • Impact : Demonstrations unused during training despite preparation

Key Design Rationale

This simplified approach trades off the full SDFT mechanism for:

• Simplicity : Single forward pass per model (not demo-conditioned)
• Efficiency : Reduced computational overhead during training
• Focus : Concentrates on on-policy learning benefits without demo conditioning complexity
• General Fine-Tuning : Better suited for instruction tuning than continual learning from expert demos

Dataset

Note : Demonstrations are prepared and tokenized in dataset but not used in training loop (simplified approach).

Training Configuration

Hardware and Infrastructure

Training Environment:

• GPU: NVIDIA A100 80GB
• System RAM: 167.1 GB (used 8.4 GB during training)
• Disk Storage: 235.7 GB (used 41.2 GB)
• Framework: PyTorch 2.x + Transformers + peft
• Quantization: BitsAndBytes (8-bit for both student and teacher models)

Memory Usage:

• Student model (8-bit): 1.2 GB
• Teacher model (8-bit): 1.2 GB
• Batch tensors: 4 GB
• Activations and Gradients: 3-5 GB
• Total Peak: 35 GB (44% of 80GB available)

Performance Evaluation

Evaluation Results (90 test prompts)

Performance by Category

Key Findings:

• Strong performance on knowledge, technical, and problem-solving tasks
• Comprehensive responses with proper length calibration
• Preserved base model knowledge through on-policy distillation
• Consistent output across diverse domains

Usage

Installation

pip install transformers torch

Quick Start

import torch
from transformers import AutoTokenizer, AutoModelForCausalLM

# Load model
model = AutoModelForCausalLM.from_pretrained(
    "yasserrmd/lfm2.5-1.2b-onpolicy",
    torch_dtype=torch.float16,
    device_map="auto"
)
tokenizer = AutoTokenizer.from_pretrained("yasserrmd/lfm2.5-1.2b-onpolicy")

# Generate response
prompt = """<|im_start|>user
What is machine learning?
<|im_end|>
<|im_start|>assistant
"""

inputs = tokenizer(prompt, return_tensors="pt").to(model.device)
outputs = model.generate(
    **inputs,
    max_new_tokens=256,
    temperature=0.7,
    top_p=0.95
)
response = tokenizer.decode(outputs[0], skip_special_tokens=True)
print(response)

Generation Parameters

Model Cards and Licenses

• Base Model License: Check LiquidAI/LFM2.5-1.2B-Instruct
• Training Data License: OASST1 (Creative Commons)
• Fine-tuned Model: Open for research and commercial use (check base model license)

Technical Insights

Why On-Policy Self-Distillation Works

1. Knowledge Preservation : Base model alignment prevents catastrophic forgetting
2. Efficient Learning : 8-bit quantization reduces memory overhead
3. Stability : KL divergence from base model provides stable training signal
4. Practical : Runs efficiently on consumer GPUs with parameter-efficient training

Differences from Paper's SDFT

This implementation follows the spirit of SDFT (on-policy learning) but uses a simplified approach :

Result : On-policy self-distillation optimized for instruction tuning rather than demonstration-based continual learning.

Training Observations

• Loss Trajectory: Started at 0.1460, converged to 0.1845 at epoch completion
• Memory Efficiency: Scaled to batch_size=16 with 35GB peak usage (44% utilization)
• Convergence: Stable training without divergence or OOM errors
• Single Epoch: Sufficient quality for production deployment without requiring multiple epochs

Deployment

The model is ready for:

• Chat and conversational AI applications
• Question-answering systems
• Content generation (summaries, explanations)
• Instruction-following tasks
• Fine-tuning for domain-specific applications

Recommended Hardware

Citation

If you use this model in your research, please cite:

@misc{lfm25_onpolicy_2026,
  author = {Yasser RMD},
  title = {LFM2.5-1.2B: On-Policy Self-Distillation Fine-Tuned Language Model},
  year = {2026},
  publisher = {Hugging Face},
  howpublished = {\url{https://huggingface.co/yasserrmd/lfm2.5-1.2b-onpolicy}},
  note = {Inspired by Shenfeld et al. (2026) SDFT}
}

References

• Self-Distillation Fine-Tuning (SDFT) : Shenfeld, I., Damani, M., Hübotter, J., & Agrawal, P. (2026). "Self-Distillation Enables Continual Learning." arXiv preprint arXiv:2601.19897 . https://arxiv.org/abs/2601.19897

  • Inspiration : This work inspired the on-policy distillation approach
  • Key Difference : This implementation omits demonstration conditioning for simplified instruction tuning

• Base Model : LiquidAI LFM2.5-1.2B. https://huggingface.co/LiquidAI/LFM2.5-1.2B-Instruct

• Dataset : OpenAssistant/OASST1. https://huggingface.co/datasets/OpenAssistant/oasst1

• Training Framework : Hugging Face Transformers & PEFT (Parameter-Efficient Fine-Tuning). https://github.com/huggingface/peft

Limitations

1. Simplified Approach : Does not implement full SDFT with demonstration conditioning (see Technical Insights section)
2. Single Epoch : Trained for 1 epoch only; potential improvements possible with additional epochs
3. Dataset Size : 5,000 samples is relatively small; larger datasets could improve generalization
4. Domain Bias : Trained on general assistant tasks; may underperform on specialized domains
5. Knowledge Cutoff : Inherited from base model pre-training
6. Factual Accuracy : Q&A on recent events may be less accurate than larger models
7. KL Estimator : Uses sampled-token KL (higher variance) instead of analytic vocabulary KL from paper

Contributing

For improvements, issues, or collaborations:

• Open an issue on the Hugging Face Hub
• Submit pull requests with improvements
• Share evaluation results or use cases

License

This model follows the license of the base model (LFM2.5-1.2B-Instruct). Please refer to the original model repository for details.

---

Created: February 2026 Training Hardware: NVIDIA A100 80GB Status: Production Ready (Single Epoch, On-Policy Self-Distilled) Inspiration: SDFT (Shenfeld et al., 2026) Implementation Type: Simplified Instruction Fine-Tuning Variant