sail / data-mixture-pile-cc-1b

Models Trained with Human Selection

This is a collection of the language models trained using Pile-CC, each with approximately 1B parameters, trained on different seeds. This project aims to validate the generalization capabilities of the RegMix approach ( https://huggingface.co/papers/2407.01492 ) from small-scale (e.g., 1M parameters) to large-scale (e.g., 1B parameters) models.

Key Features

• Model Size : 5 separate models trained with different seeds, each with ~1B parameters
• Training Data : The pile-cc only data mixture on the RegMix-Data dataset

Dataset

The models were trained using the RegMix-Data dataset, which is split into different domains from The Pile dataset.

Training Hyperparameters

How to Load a Model

You can load any model using the corresponding branch with the Hugging Face Transformers library:

from transformers import AutoModel, AutoTokenizer

model = AutoModel.from_pretrained("sail/data-mixture-pile-cc-1b", revision="seed-1")
tokenizer = AutoTokenizer.from_pretrained("sail/data-mixture-pile-cc-1b", revision="seed-1")

Data Mixture

The specific data mixture used for training this 1B model is as follows, which can be also found in our code :

train:
  train_the_pile_pile_cc: 1.0
valid:
  valid_the_pile_pile_cc: 1.0
model_name: tinyllama_1_1b

Model Variants

To access different model variants, simply change the revision parameter in the from_pretrained method to the desired seed (e.g., "seed-2", "seed-3"), and the maxium seed is 5.

Model Performance

We evaluated each model using lm-evaluation-harness . The performance metric for each task is the average of 0-shot to 5-shot accnorm (accuracy normalized, if available) or acc (accuracy) scores.

Usage Notes

• These models are primarily intended for research purposes.
• Performance may vary depending on the specific task and domain.

Citation

If you use these models in your research, please cite the RegMix paper:

@article{liu2024regmix,
  title={RegMix: Data Mixture as Regression for Language Model Pre-training},
  author={Liu, Qian and Zheng, Xiaosen and Muennighoff, Niklas and Zeng, Guangtao and Dou, Longxu and Pang, Tianyu and Jiang, Jing and Lin, Min},
  journal={arXiv preprint arXiv:2407.01492},
  year={2024}
}

For more information about the RegMix methodology and its applications, please refer to the original paper .