CobrIX / CobrIX-1.5-preview-Coder-Full-72B-A18B

PREVIEW 1.5 — preview version for testing. Final release will follow after think-SFT, native MTP and GGUF quants.

CobrIX-1.5-preview-Coder-Full-72B-A18B

Mixture-of-Experts decoder (~72B total / ~18B active per token) built from the dense Qwen 3.5 base empero-ai/Qwythos-9B-v2 and 5 dense Qwen 3.5 fine-tuned experts (DPO-aligned + SFT-reinforced for code and cybersecurity, in Portuguese and English), without mergekit.

Architecture

• Qwen35MoEForCausalLM (decoder-only), model_type=qwen35_moe .
• Every transformer layer replaces the dense mlp with a sparse MoE block:

input_layernorm -> linear_attn -> post_attention_layernorm -> gate/experts[0..12]/shared_expert -> residual

• num_local_experts=13 , num_experts_per_tok=2 (top-2 routing, softmax over the 13 experts).
• Always-active shared expert (copy of the base MLP) gated by sigmoid(x @ w) , num_shared_experts=1 .
• Router logits computed in float32 .
• Router initialisation: random ; shared-expert gates are zeros (new parameters; every copied weight is untouched).
• Hybrid layers: 24× linear_attention + 8× full_attention (every 4th layer), 32 layers total.

Weights

• Base model contributes: embed_tokens , linear_attn.* , self_attn.* , layernorms, norm , rotary_emb , lm_head .
• Each expert contributes only gate_proj , up_proj , down_proj of every layer.
• DPO + SFT deltas fused into every expert (see training notes below).
• No weight is modified, averaged, or interpolated beyond the fused deltas.

Expert layout

The 13 slots reuse the 5 trained experts, weighted toward code and cybersecurity (which cover 10 of the 13 slots, so ~96% of tokens route through at least one of them):

slot:    0  1  2  3  4  5  6  7  8  9  10 11 12
expert:  3  4  1  5  2  3  4  5  3  2  4  5  3

• Expert 1 — model identity (Alpaca)
• Expert 2 — context and reasoning traces (distillation)
• Expert 3 — programming, cybersecurity, long horizon + reasoning (4 coding datasets)
• Experts 4 and 5 — cybersecurity

Training notes

• Experts: DPO (preference alignment, 750 steps, coding/cyber mix) + SFT reinforcement pass, fused per expert.
• Router: freshly initialised (not trained) — routing quality improves with router-only fine-tuning, planned for a later release.

Usage

from transformers import AutoTokenizer, AutoModelForCausalLM

tokenizer = AutoTokenizer.from_pretrained("CobrIX/CobrIX-1.5-preview-Coder-Full-72B-A18B")
model = AutoModelForCausalLM.from_pretrained(
    "CobrIX/CobrIX-1.5-preview-Coder-Full-72B-A18B", torch_dtype="auto", trust_remote_code=True,
)

inputs = tokenizer("Olá! Quem é você?", return_tensors="pt")
out = model.generate(**inputs, max_new_tokens=64)
print(tokenizer.decode(out[0], skip_special_tokens=True))

The folder contains the self-contained configuration_qwen35_moe.py / modeling_qwen35_moe.py modules used through auto_map , so no monkey patch is required.

Hardware note: the full bf16 checkpoint needs ~150 GB RAM/VRAM to load. For most machines, wait for the Q4_K_M/Q5_K_M GGUF releases.

Limitations (preview)

• No native MTP head (planned for 1.5-final alongside inference optimisations).
• No GGUF quants yet — for local inference, wait for the Q4_K_M/Q5_K_M releases or convert with llama.cpp ( convert_hf_to_gguf.py + architectures[0]=Qwen3_5MoeForCausalLM , --no-mtp ).

Feedback — help make it better

This is a preview, and your testing shapes the final 1.5. If you find a flaw — a wrong answer, insecure code suggestion, broken reasoning, bad Portuguese/English, formatting issues, or anything the model could do better — please tell us.

📧 Send to: [email protected]

To make your report actionable, please include:

1. What you asked — the exact prompt (or a minimal version that reproduces the issue).
2. What it answered — the model's output (or the relevant part).
3. What you expected — what a good answer would look like.
4. How to improve it — your suggestion: better data, different behavior, new capability, etc.
5. Context — language used (PT/EN), code language if applicable, and your setup (transformers, vLLM, GGUF quant, etc.).

Every report is read by the team and directly influences the think-SFT data, the router training and the final release. Thank you for testing! 🙏

Building it

python convert.py --base <base> --experts <e0> <e1> ... <e12> \
    --output CobrIX-1.5-preview-Coder-Full-72B-A18B \
    --num-local-experts 13 --num-experts-per-tok 2
python verify.py --model CobrIX-1.5-preview-Coder-Full-72B-A18B
python test.py --model CobrIX-1.5-preview-Coder-Full-72B-A18B

Interest List

If you want to know when the AI ecosystem and CobrIX Code will be available to the public, visit:

https://cobrix.vercel.app/coder

On the site, you will find a field to enter your email and join the early access waitlist .

Registered users will receive an email when the CobrIX AI ecosystem and CobrIX Code become available .

We currently have the funds to host CobrIX-1.0-Coder-Flash-33B-A13B on an RTX 6000 Ada Generation GPU (48GB VRAM), supporting between 50 and 100 concurrent users. Please help us by donating (links below); help us create AI that is usable for everyone—open-source AI forever—so we can evolve together.

🚀 Join the waitlist and get notified when CobrIX Code is released.

Donations for Infrastructure

The development of custom AI architectures such as CobrIX-1.5-Preview-Coder-Full-MoE requires ongoing computing resources, research, experimentation, and maintenance.

If this project brings value to your work or company, you can support the development and infrastructure of the CobrIX ecosystem through the wallets below.

All donations help fund:

• AI infrastructure
• GPU computing - Nvidia RTX PRO 6000 Blackwell 96 gb vram
• Open-source model development
• New model training
• Research and experimentation
• Community support

Bitcoin (BTC)

bc1q8mu8fjak4y84qj4dlk8pu4d3zhknm92zra4r4m

Ethereum (ETH / ERC-20)

0x8D9187dEa0a77390ef668361cd5b236DE54af2BB

Solana (SOL)

GQR2jZnWuWP1c3dbuz4mC7ZnyveacBKy63q8qf9nj8bp

Thank you to all developers, researchers, enthusiasts, and partners supporting the evolution of open and accessible artificial intelligence.

CobrIX — Open AI infrastructure and custom model research.