bharatgenai / AyurParam

---

AyurParam

BharatGen introduces AyurParam, a domain-specialized large language model fine-tuned from Param-1-2.9B-Instruct on a high-quality Ayurveda dataset. It is designed to handle Ayurvedic queries, classical text interpretation, clinical guidance, and wellness knowledge. Ayurveda offers vast traditional medical wisdom, yet most language models lack domain-specific understanding. AyurParam bridges this gap by combining Param-1’s bilingual strengths with a curated Ayurvedic knowledge base, enabling contextually rich and culturally grounded responses.

🏗 Model Architecture

AyurParam inherits the architecture of Param-1-2.9B-Instruct:

• Hidden size: 204
• Intermediate size: 7168
• Attention heads: 16
• Hidden layers: 32
• Key-value heads: 8
• Max position embeddings: 2048
• Activation: SiLU
• Positional Embeddings: Rotary (RoPE, theta=10000)
• Attention Mechanism: Grouped-query attention
• Precision: bf16-mixed
• Base model: Param-1-2.9B-Instruct

📚 Data Preparation

AyurParam’s training corpus was carefully crafted to ensure deep Ayurvedic knowledge, Sanskrit/English bilingual accessibility, and clinical relevance. Steps involved:

1. Source Gathering

• 15k+ passages from classical Ayurvedic texts (digitized and curated).
• 10k+ passages from AYUSH ministry guidelines, research papers, and clinical case discussions.

1. Question Generation

• 5 curated Q&A pairs generated per passage using an open-source LLM + domain expert review.

1. Domain Taxonomy & Personas

• Built an Ayurveda-specific taxonomy (Dosha, Dhatu, Mala, Srotas, Nidana, Chikitsa, etc.).
• Defined multiple personas: student, vaidya (physician), researcher, policymaker, wellness coach.

1. Dataset Construction

• 1.5M Q&A pairs grounded in taxonomy and personas.
• 4M multi-turn conversation samples created.
• Sanskrit terminology preserved with transliteration and explanations.

🏋️ Training Setup

• Base model: Param-1-2.9B-Instruct
• Training framework: Hugging Face + TRL (SFT) + torchrun multi-node setup
• Prompt template: Custom-designed for Ayurvedic inference
• Scheduler: Linear with warmup
• Epochs: 3
• Total training samples: ~8M
• Test samples: ~800k
• Base learning rate: 5e-6
• Minimum learning rate: 0
• Additional tokens: , , ,
• Vocab size: 256k + 4
• Global batch size: 1024
• Micro batch size: 4
• Gradient accumulation steps: 32

🚀 Inference Example

from transformers import AutoTokenizer, AutoModelForCausalLM
import torch

model_name = "bharatgenai/AyurParam"
tokenizer = AutoTokenizer.from_pretrained(model_name, trust_remote_code=False)
model = AutoModelForCausalLM.from_pretrained(
    model_name,
    trust_remote_code=True,
    torch_dtype=torch.bfloat16 if torch.cuda.is_available() else torch.bfloat32,
    device_map="auto"
)

# Example Ayurvedic query
user_input = "What is the Samprapti (pathogenesis) of Amavata according to Ayurveda?"

# Prompt styles
# 1. Generic QA: <user> ... <assistant>
# 2. Context-based QA: <context> ... <user> ... <assistant>
# 3. Multi-turn conversation (supports up to 5 turns): <user> ... <assistant> ... <user> ... <assistant>

prompt = f"<user> {user_input} <assistant>"
inputs = tokenizer(prompt, return_tensors="pt").to(model.device)

with torch.no_grad():
    output = model.generate(
        **inputs,
        max_new_tokens=300,
        do_sample=True,
        top_k=50,
        top_p=0.95,
        temperature=0.6,
        eos_token_id=tokenizer.eos_token_id,
        use_cache=False
    )

print(tokenizer.decode(output[0], skip_special_tokens=True))

📊 Benchmark Results: Ayur Param vs Baselines

• BhashaBench-Ayur benchmark

---

1. Overall Performance

Similar Range Models

Larger Models

---

2. Question Difficulty

Similar Range Models

Larger Models

---

3. Question Type

Similar Range Models

Larger Models

---

From the above results, Ayur Param not only outperforms all similar-sized models but also achieves competitive or better performance than larger models across multiple metrics.

Contact

For any questions or feedback, please contact:

• Sravan Kumar ( [email protected] )
• Kundeshwar Pundalik ( [email protected] )