bharatgenai / Param-1-5B

Param-1-5B

Param-1-5B is a bilingual ( English–Hindi ) large language model developed under the Param-1 family. With 5 billion parameters , this model extends the capabilities of Param-1-2.9B by incorporating enhanced mathematical reasoning and code understanding/generation .

The model is pretrained from scratch and designed to serve as a strong foundation for downstream tasks such as mathematical problem solving, and code-related understanding / generation.

Key Highlights

• 5B parameter dense Transformer model
• Bilingual : English and Hindi
• Enhanced domains : Math and Code (compared to Param-1-2.9B)
• Updated dataset mixture and percentage ratios
• Designed as a pretrained (PT) base model

Model Inference

from transformers import AutoTokenizer, AutoModelForCausalLM
import torch

model_name = "bharatgenai/Param-1-5B"

tokenizer = AutoTokenizer.from_pretrained(model_name, trust_remote_code=False)
model = AutoModelForCausalLM.from_pretrained(
    model_name,
    trust_remote_code=True,
    torch_dtype=torch.bfloat16 if torch.cuda.is_available() else torch.float32,
    device_map="auto"
)

prompt = "Your prompt here"
inputs = tokenizer(prompt, return_tensors="pt").to(model.device)

with torch.no_grad():
    output = model.generate(
        **inputs,
        max_new_tokens=300,
        do_sample=True,
        top_k=50,
        top_p=0.95,
        temperature=0.6,
        eos_token_id=tokenizer.eos_token_id,
        use_cache=False
    )

print(tokenizer.decode(output[0], skip_special_tokens=True))

Benchmarks

Model Architecture

• Architecture: Transformer (Decoder-only)
• Number of parameters: ~5B
• Hidden size: 4096
• Intermediate size: 14336
• Number of layers: 16
• Number of attention heads: 32
• Number of key-value heads: 8
• Maximum position embedding: 4096
• Positional embeddings: Rotary (RoPE)
• Attention mechanism: Grouped-query attention
• Activation function: silu
• Precision: bf16-mixed

Training Data

The model is pretrained on a large-scale bilingual corpus with a strong focus on English and Hindi , along with dedicated Math and Code data .

Dataset Composition (Total ≈ 6.61T tokens )

• English Natural Language: 3.6T
• Hindi Natural Language: 2.77T
• Math & Code: 238.4B
  • Math: 40%
  • Code: 60%

Compared to Param-1-2.9B , Param-1-5B includes:

• A larger overall training corpus
• Explicit inclusion of Math and Code datasets
• Improved performance in mathematical and code understanding/generation
• Continued emphasis on English–Hindi bilingual capability

Training Details

• Training framework: NVIDIA NeMo
• Training infrastructure: Yotta's Shakti Cloud
• Hardware: NVIDIA H100 - 1024 GPUs
• Total training tokens: 6.61 Trillion

Limitations

• This is a pretrained base model and may require fine-tuning for instruction-following or chat use cases.
• The model may reflect biases present in large-scale web and code data.
• Performance on low-resource Hindi domains may vary depending on task complexity.

License

This model is released under the BharatGen non-commercial license .

Please refer to the LICENSE file for detailed terms and conditions.