reaperdoesntknow / LFM2.5-1.2B-Distilled-SFT

LFM2.5-1.2B-Distilled-SFT

A 1.2B hybrid model (SSM + attention) built in two stages: knowledge distillation from a 24B MoE hybrid teacher on STEM chain-of-thought data, then supervised fine-tuning on logical inference. The first proof-weighted distillation + SFT pipeline on a non-transformer architecture.

Liquid Foundation Models run at 239 tok/s on AMD CPU and fit under 1GB of RAM. This model adds structured STEM reasoning and formal logical inference to that efficiency substrate.

"Structure beats scale, collaboration beats hierarchy, observation beats theory." — Convergent Intelligence LLC: Research Division

Training Pipeline

Stage 1: Knowledge Distillation (STEM Reasoning Backbone)

LFM2.5-1.2B distilled from LFM2-24B-A2B — a 24B MoE hybrid (SSM + attention) with only 2B active parameters per token. Teacher and student share the LFM hybrid architecture, so the KL divergence transfers reasoning patterns between architecturally compatible models.

Data: 2,802 STEM CoT samples from 5 domains:

All from 0xZee . Focused subset — core mathematical reasoning domains that share the most structural overlap with logical inference.

Loss function:

1. Proof-Weighted Cross-Entropy (55%) — 2.5x → 1.5x on derivation tokens
2. Knowledge Distillation KL Divergence (45%) — T=2.0, scaled by T²

Training format:

Solve the following problem carefully and show a rigorous derivation.

Problem:
{question}

Proof:
{CoT}

Final Answer:
{response}

Stage 1 hyperparameters:

---

Stage 2: Logical Inference SFT

Fine-tuned on KK04/LogicInference_OA — a reproduction of the LogicInference dataset (Santiago Ontañón, Google Research) formatted for instruction-following. IID split, LOGICINFERENCEe format (inference first, answer at end). 5,491 unique inference problems extended to ~54,607 instruction-response pairs.

Why logical inference on a hybrid architecture? SSM components excel at sequential state propagation — exactly what formal logical inference requires. Each premise updates a logical state, and the conclusion follows from the final state. The hybrid architecture's inductive bias naturally aligns with propositional logic chains. SFT activates this alignment explicitly.

Training format:

### Instruction:
{instruction}

### Response:
{response}

Stage 2 hyperparameters:

Model Details

Usage

from transformers import AutoTokenizer, AutoModelForCausalLM
import torch

model_id = "reaperdoesntknow/LFM2.5-1.2B-Distilled-SFT"

tokenizer = AutoTokenizer.from_pretrained(model_id)
model = AutoModelForCausalLM.from_pretrained(
    model_id,
    torch_dtype=torch.bfloat16 if torch.cuda.is_available() else torch.float32,
    device_map="auto",
)

# Logical inference (Stage 2)
prompt = """### Instruction:
Consider the following premises: If p then q. If q then r. p is true. What can we infer?

### Response:
"""

# STEM derivation (Stage 1 still works)
prompt_stem = """Solve the following problem carefully and show a rigorous derivation.

Problem:
Solve the system of linear equations: 2x + y = 5, x - y = 1.

Proof:
"""

inputs = tokenizer(prompt, return_tensors="pt").to(model.device)
with torch.no_grad():
    outputs = model.generate(**inputs, max_new_tokens=512, do_sample=False)
print(tokenizer.decode(outputs[0], skip_special_tokens=True))

GGUF

Quantized versions at reaperdoesntknow/LFM2.5-1.2B-Distilled-SFT-GGUF .

Prompt Formats

STEM derivation (Stage 1):

Solve the following problem carefully and show a rigorous derivation.

Problem:
[Your problem]

Proof:

Logical inference (Stage 2):

### Instruction:
[Your question or logical inference problem]

### Response:

Intended Uses

Good for: On-device logical inference and STEM reasoning, mobile/edge/IoT deployment, formal reasoning tasks, educational tutoring, embedded inference pipelines, anywhere you need structured reasoning under 1GB.

Not for: Formal proof verification, safety-critical systems, complex multi-step proofs beyond model capacity, or long-context tasks beyond 1024 tokens.

Limitations

1.2B hybrid model. The SSM components give excellent inference speed but the model has hard capacity limits. Trained on 2,802 STEM samples (smaller than the 6,122 used for Qwen3 variants). Logical inference strongest on propositional logic patterns in the training data. Complex nested quantifiers may exceed capacity. Always verify.

Related Models

Citation

@misc{colca2026lfmsft,
  title={Hybrid SSM/Attention Distillation + Logical Inference: LFM2-24B to LFM2.5-1.2B},
  year={2026},
  publisher={HuggingFace},
  url={https://huggingface.co/reaperdoesntknow/LFM2.5-1.2B-Distilled-SFT},
  note={Convergent Intelligence LLC: Research Division}
}

References

Santiago Ontañón. "LogicInference: A Large-Scale Dataset for Logical Inference." ICLR 2023 Workshop on Mathematical and Empirical Understanding of Foundation Models. Paper | Code

---

Convergent Intelligence LLC: Research Division "Where classical analysis fails to see, we begin."

---

Convergent Intelligence Portfolio

Part of the Liquid Foundation Model Series by Convergent Intelligence LLC: Research Division

Top Models from Our Lab

Total Portfolio: 41 models | 2,781 total downloads

Last updated: 2026-03-28 12:47 UTC