inference-optimization / gemma-4-1B-0.8B-tiny

Gemma4-26B-1.1B-tiny

A tiny version of google/gemma-4-26B-A4B for testing and development.

Model Details

• Base Model: google/gemma-4-26B-A4B
• Architecture: gemma4 (multimodal vision-language with Mixture of Experts)
• Total Parameters: 1.04B
• Activated Parameters: 0.89B (MoE with top-k=8 out of 16 experts)

Configuration Comparison

Checkpoint Structure

The model is saved as a single safetensors file ( model.safetensors ) containing all weights. The architecture maintains the same structure as the original Gemma4 model with:

• Vision embedding projection
• Language model with text layers (MoE + standard MLP)
• Mixed attention types (sliding_attention for local context, full_attention for global context)
• Router for MoE expert selection

Validation

The model has been validated to:

• Load successfully with AutoModelForCausalLM.from_pretrained()
• Achieve low perplexity on training data (0.95)
• Generate coherent text completions
• Inference perplexity: 1.21

Fine-tuning Results

The model was fine-tuned on a toy dataset of internet copypastas:

• Initial perplexity: ~12.5
• Final perplexity: 0.95 (target: 3.0)
• Training steps: 200
• The model successfully memorized the training data and generates appropriate completions

Example Generation

Prompt: "According to all known laws"
Output: "According to all known laws of aviation, there is no way a bee should be able to fly. Its wings are too small to get its fat little body off the ground. The bee, of course, flies anyway because bees don't care what humans think is impossible."

Usage

from transformers import AutoModelForCausalLM, AutoTokenizer

model = AutoModelForCausalLM.from_pretrained(
    "inference-optimization/Gemma4-26B-1.1B-tiny",
    trust_remote_code=True,
    device_map="auto"
)
tokenizer = AutoTokenizer.from_pretrained(
    "inference-optimization/Gemma4-26B-1.1B-tiny",
    trust_remote_code=True
)

prompt = "According to all known laws"
inputs = tokenizer(prompt, return_tensors="pt").to(model.device)
outputs = model.generate(**inputs, max_length=100)
print(tokenizer.decode(outputs[0], skip_special_tokens=True))

Additional Notes

• This is a tiny model for testing purposes only - it is not trained for production use
• The model maintains the architectural characteristics of Gemma4 including:
  • Mixture of Experts (MoE) with sparse activation
  • Mixed attention patterns (sliding + full attention)
  • Vision-language capabilities (though vision components are not fine-tuned)
• Useful for:
  • Testing quantization and compression techniques
  • Validating transformers integration
  • Development and debugging without large model overhead
• The model uses the same tokenizer and processor as the base model
• Vision capabilities are present but not validated in fine-tuning (text-only dataset used)

License

Same as the base model: Gemma License

Created With

This model was created using the llm-compressor create-tiny-model skill.