Asilarkness / test150m

CubicV11 150M Reasoning LM

Custom 156.9M-parameter causal language model trained with the CubicV11 sparse long-context layer and the Cubic Muon optimizer. This revision was exported at optimizer step 6,500 from kabsis/AM-DeepSeek-R1-0528-Distilled/code.jsonl using assistant-only loss over the teacher's <think> and <answer> output.

Architecture

• 14 layers, width 896, 14 query heads and 7 KV heads
• 4,096-token causal local attention window
• causal pooled global summaries every 256 tokens
• compressed cross-layer depth memory, rank 224
• RoPE base 500,000, SwiGLU and tied embeddings
• trained context length: 32,768 tokens
• ByteLevel BPE vocabulary: 16,384 tokens

This is custom PyTorch code, not a drop-in Transformers model. Use the included cubic_v11_chat_gradio.py application:

python cubic_v11_chat_gradio.py

It downloads this repository, loads model.safetensors , uses incremental local, global-summary and depth caches, supports a 32K total context window, reserves up to 4,096 tokens for the answer and launches a public Gradio chat link.

training_checkpoint.pt contains model and optimizer state for continuing training. cubic_v11_deepseek_distill_32k.py is the matching trainer.

Limitations and use

This is an experimental research checkpoint. It can produce incorrect, repetitive or unsafe code and reasoning. Generated code must be reviewed and tested before use. The source distillation dataset and its upstream model have their own terms; users are responsible for checking those terms. No claim is made that this checkpoint is production-ready.