bharatgenai / FinanceParam_FP32

---

FinanceParam

BharatGen introduces FinanceParam , a domain-specialized large language model fine-tuned from Param-1-2.9B-Instruct on a high-quality finance dataset. FinanceParam is designed to deliver accurate, bilingual (English-Hindi) Indian financial knowledge for personal finance, taxation, banking, investments, and policy guidance.

---

💰 Motivation

Finance touches every aspect of daily life, from household budgeting to national economic policy. Yet, existing language models lack deep domain expertise in Indian finance, regulatory frameworks, and cultural nuances. FinanceParam bridges this gap by combining Param-1’s bilingual capabilities with a meticulously curated financial knowledge base tailored for India.

---

🏗 Model Architecture

FinanceParam inherits the architecture of Param-1-2.9B-Instruct:

• Hidden size : 2048
• Intermediate size : 7168
• Attention heads : 16
• Hidden layers : 32
• Key-value heads : 8
• Max position embeddings : 2048
• Activation : SiLU
• Positional Embeddings : Rotary (RoPE, theta=10000)
• Attention Mechanism : Grouped-query attention
• Precision : bf16-mixed
• Base model : Param-1-2.9B-Instruct

---

📚 Data Preparation

FinanceParam’s training corpus was carefully crafted to ensure deep Indian Finance knowledge, cultural relevance, and bilingual (English-Hindi) accessibility.

Steps involved:

1. Source Gathering

   • 10K+ open-source, India-focused finance news & information passages.

2. Question Generation

   • Generated 5 curated Q&A pairs per passage using an open-source LLM.

3. Domain Taxonomy & Personas

   • Built an exhaustive, India-specific financial taxonomy.
   • Defined CA, policy-maker, business and multiple such personas.

4. Dataset Construction

   • 2M Q&A pairs grounded in taxonomy and personas.
   • Complete dataset translated into Hindi.
   • 6M multi-turn conversation samples created.

5. Source Gathering

   • Collected 25,000+ finance-focused passages from trusted Indian sources: government portals (Income Tax Dept., RBI, SEBI, IRDAI), banking reports, investment advisories, policy documents, and financial news.

6. Knowledge-Enriched Question Generation

   • For each passage, an open-source LLM generated 5 high-quality Q&A pairs, refined to cover personal finance, taxation, banking, insurance, and investment topics.

7. Domain Taxonomy & Personas

   • Built a comprehensive Indian finance taxonomy spanning income, budgeting, taxation, insurance, banking, and investments.
   • Defined diverse user personas: salaried professionals, students, investors, small business owners, retirees, and policy-makers.

8. Dataset Construction

   • Compiled 9M Q&A pairs grounded in taxonomy and personas.
   • Translated the entire dataset into Hindi to ensure accessibility across India’s multilingual audience.
   • Expanded into 8M multi-turn dialogues

---

🏋️ Training Setup

• Base model : Param-1-2.9B-Instruct
• Training framework : Transformer Framework + pytorch multi-node setup
• Prompt template : Custom-designed for financial system inference
• Scheduler : Linear
• Epochs : 1
• Total training samples : 24M
• Learning rate : 2e-4
• Vocab size : 256K
• Batch size : 512

---

🚀 Inference Example

from transformers import AutoTokenizer, AutoModelForCausalLM
import torch

model_name = "bharatgenai/FinanceParam"
tokenizer = AutoTokenizer.from_pretrained(model_name, trust_remote_code=False)
model = AutoModelForCausalLM.from_pretrained(
    model_name,
    trust_remote_code=True,
    torch_dtype=torch.bfloat16 if torch.cuda.is_available() else torch.bfloat32,
    device_map="auto"
)

# Example Finance query
user_input = "How to file income tax return. Tell me in detail"

# Based on your requirements use the type of prompt (refere the above examples)
# Append assistant and user for chat model.
prompt = [{"role": "user", "content": user_input}]

inputs = tokenizer.apply_chat_template(prompt, tokenize=True, return_tensors="pt").to(model.device)

with torch.no_grad():
    output = model.generate(
        inputs,
        max_new_tokens=300,
        eos_token_id=tokenizer.eos_token_id,
        use_cache=False
    )

print(tokenizer.decode(output[0], skip_special_tokens=True))

📊 Benchmarks

Overall BBF Performance

This table shows the average BBF (Benchmark for Finance) performance across all tasks, split by English and Hindi subsets.

---

Domain-Wise Performance

This table highlights how models perform across specific finance-related domains such as banking, taxation, insurance, economics, etc.

---

Difficulty-Level Performance

This table breaks down performance across Easy, Medium, and Hard difficulty levels .

---

Question-Type Performance

This table reports results by question type (e.g., MCQ, comprehension, reasoning).

---

📜 License

This SFT checkpoint is released under the BharatGen non-commercial license .
Please refer to the LICENSE for terms and conditions.