CobrIX / CobrIX-1.0-Full-72B-A18B

CobrIX-1.0-Full-MoE

This is a Mixture-of-Experts (MoE) decoder built directly from the dense Qwen 3.5 base ( empero-ai/Qwythos-9B-v2 ) and 5 dense Qwen 3.5 fine-tuned experts, without relying on CobrIXKit. The model features 72B total parameters , 18B active parameters (A13B) per token, and a native 1,048,576-token context window .

Architecture

• Model Class: Qwen35MoEForCausalLM (decoder-only), model_type=qwen35_moe .
• Context Window: Native support for up to 1,048,576 tokens ( max_position_embeddings=1048576 ).
• Layer Design: Every transformer layer replaces the dense mlp with a sparse MoE block:

  input_layernorm -> linear_attn -> post_attention_layernorm -> gate/experts[0..4]/shared_expert -> residual
  

• Routing: num_local_experts=13 , num_experts_per_tok=2 (top-2 routing, softmax over the 13 experts).
• Shared Expert: An always-active shared expert (copy of the base MLP) gated by sigmoid(x @ w) , where num_shared_experts=1 .
• Precision: Router logits are computed in float32 .
• Initialization: Router initialization is random . Shared-expert gates are initialized as zeros.
• Layer Types:

  layer_types = [
      'linear_attention', 'linear_attention', 'linear_attention', 'full_attention',
      'linear_attention', 'linear_attention', 'linear_attention', 'full_attention',
      'linear_attention', 'linear_attention', 'linear_attention', 'full_attention',
      'linear_attention', 'linear_attention', 'linear_attention', 'full_attention',
      'linear_attention', 'linear_attention', 'linear_attention', 'full_attention',
      'linear_attention', 'linear_attention', 'linear_attention', 'full_attention',
      'linear_attention', 'linear_attention', 'linear_attention', 'full_attention',
      'linear_attention', 'linear_attention', 'linear_attention', 'full_attention'
  ]
  

Weights

• Base Model Contributes: embed_tokens , linear_attn.* , layernorms, norm , rotary_emb , and lm_head .
• Experts Contribute: Only gate_proj , up_proven , and down_proj for every layer.
• Integrity: No weight is modified, averaged, or interpolated during assembly.

Usage

The model directory contains self-contained configuration_qwen35_moe.py and modeling_qwen35_moe.py modules. These are loaded automatically via Hugging Face's auto_map , meaning no monkey patching is required .

from transformers import AutoTokenizer, AutoModelForCausalLM

model_id = "CobrIX/CobrIX-1.0-Full-72B-A18B"
tokenizer = AutoTokenizer.from_pretrained(model_id, trust_remote_code=True)
model = AutoModelForCausalLM.from_pretrained(
    model_id, 
    torch_dtype="auto", 
    trust_remote_code=True
)

inputs = tokenizer("Hello, How are you?", return_tensors="pt")
out = model.generate(**inputs, max_new_tokens=64)
print(tokenizer.decode(out[0], skip_special_tokens=True))

Building & Verification

To replicate the build or verify the model structure, use the provided scripts:

# Build the MoE model from base and experts
python convert.py --base empero-ai/Qwythos-9B-v2 --experts <e0> <e1> <e2> <e3> <e4> --output Qwen35-CobrIX-MoE

# Verify architecture integrity
python verify.py --model Qwen35-CobrIX-MoE

# Run generation tests
python test.py --model Qwen35-CobrIX-MoE