AICrossSim / clm-200m

Model Card for AICrossSim/clm-200m

This is a 200M parameter causal language model trained on the FineWeb dataset, designed for general language modeling tasks.

Model Details

Model Description

This is a 200M parameter transformer-based language model trained using the 🤗 transformers library. It uses RMSNorm for normalization and is trained on the FineWeb dataset.

• Developed by: AICrossSim
• Funded by: ARIA
• Shared by: AICrossSim
• Model type: Causal Language Model
• Language(s) (NLP): English
• License: odc-by
• Finetuned from model: No, trained from scratch

Model Sources

• Repository: https://github.com/AICrossSim/NewComputeBench

Training Details

Training Data

The model was trained on the FineWeb dataset (HuggingFaceFW/fineweb).

Training Procedure

Training Hyperparameters

• Learning rate: 0.0001
• Optimizer: AdamW
• Batch size: 38
• Sequence length: 2048
• Training steps: 31,686
• Warmup steps: 6,337
• Training regime: bfloat16 mixed precision with float32 reduction
• Gradient clipping: 1.0
• Random seed: 42

Speeds, Sizes, Times

• Checkpoint interval: Every 500 steps
• Checkpoint format: float32 export
• Parallelization:
  • Data parallel sharding: 2
  • Tensor parallel degree: 1
  • Pipeline parallel degree: 1