CobrIX / CobrIX-1.0-Flash-33B-A13B

CobrIX-1.0-Flash-MoE

This is a Mixture-of-Experts (MoE) decoder built directly from the dense Qwen 3.5 base ( empero-ai/Qwythos-9B-v2 ) and 5 dense Qwen 3.5 fine-tuned experts, without relying on CobrIXKit.

The model features 33B total parameters , 13B active parameters (A13B) per token, and a native 1,048,576-token context window .

Capabilities & Usage Style

The Flash version is designed to act as a highly efficient AI Copilot , delivering optimal performance when utilized with human supervision and real-time interaction. It excels in:

• Advanced Reasoning (Think): Possesses a high capacity for complex logical decomposition and the resolution of extremely complex problems; trained on datasets from the world's best artificial intelligences—such as Claude Fable 5, Kimi K3, and Qwen 3.8 Max, among various other cutting-edge models.
• Expert Programming: Exceptional skills in writing, debugging, and refactoring code across various languages, with particular strength in backend and frontend web development—specifically React.js and TypeScript.
• Cybersecurity: A solid foundation for analyzing security contexts, code vulnerabilities, and scripting, but with a greater focus on programming than on cybersecurity.

Architecture

• Model Class: Qwen35MoEForCausalLM (decoder-only), model_type=qwen35_moe .
• Context Window: Native support for up to 1,048,576 tokens ( max_position_embeddings=1048576 ).
• Layer Design: Every transformer layer replaces the dense mlp with a sparse MoE block:

input_layernorm -> linear_attn -> post_attention_layernorm -> gate/experts[0..4]/shared_expert -> residual

• Routing: num_local_experts=5 , num_experts_per_tok=2 (top-2 routing, softmax over the 5 experts).
• Shared Expert: An always-active shared expert (copy of the base MLP) gated by sigmoid(x @ w) , where num_shared_experts=1 .
• Precision: Router logits are computed in float32 .
• Initialization: Router initialization is random . Shared-expert gates are initialized as zeros.
• Layer Types:

layer_types = [
    'linear_attention', 'linear_attention', 'linear_attention', 'full_attention',
    'linear_attention', 'linear_attention', 'linear_attention', 'full_attention',
    'linear_attention', 'linear_attention', 'linear_attention', 'full_attention',
    'linear_attention', 'linear_attention', 'linear_attention', 'full_attention',
    'linear_attention', 'linear_attention', 'linear_attention', 'full_attention',
    'linear_attention', 'linear_attention', 'linear_attention', 'full_attention',
    'linear_attention', 'linear_attention', 'linear_attention', 'full_attention',
    'linear_attention', 'linear_attention', 'linear_attention', 'full_attention'
]

Weights

• Base Model Contributes: embed_tokens , linear_attn.* , layernorms, norm , rotary_emb , and lm_head .
• Experts Contribute: Only gate_proj , up_proven , and down_proj for every layer.
• Integrity: No weight is modified, averaged, or interpolated during assembly.

Usage

The model directory contains self-contained configuration_qwen35_moe.py and modeling_qwen35_moe.py modules. These are loaded automatically via Hugging Face's auto_map , meaning no monkey patching is required .

from transformers import AutoTokenizer, AutoModelForCausalLM

model_id = "CobrIX/CobrIX-1.0-Flash-33B-A13B"

tokenizer = AutoTokenizer.from_pretrained(model_id, trust_remote_code=True)
model = AutoModelForCausalLM.from_pretrained(
    model_id,
    torch_dtype="auto",
    trust_remote_code=True
)

inputs = tokenizer("Hello, How are you?", return_tensors="pt")
out = model.generate(**inputs, max_new_tokens=64)
print(tokenizer.decode(out[0], skip_special_tokens=True))

Building & Verification

To replicate the build or verify the model structure, use the provided scripts:

# Build the MoE model from base and experts
python convert.py --base empero-ai/Qwythos-9B-v2 --experts <e0> <e1> <e2> <e3> <e4> --output Qwen35-CobrIX-MoE

# Verify architecture integrity
python verify.py --model Qwen35-CobrIX-MoE

# Run generation tests
python test.py --model Qwen35-CobrIX-MoE