Total runs:
8.7K
Run Growth:
-8
Growth Rate:
-0.09%
Sponsored by
AdsCreator.com
Weyaxi / Einstein-v6.1-Llama3-8B
Model's Last Updated: July 24 2024
text-generation
Introduction of Einstein-v6.1-Llama3-8B
Model Details of Einstein-v6.1-Llama3-8B
🔬 Einstein-v6.1-Llama3-8B
This model is a full fine-tuned version of meta-llama/Meta-Llama-3-8B on diverse datasets.
This model is finetuned using
8xRTX3090
+
1xRTXA6000
using
axolotl
.
This model's training was sponsored by sablo.ai .
See axolotl config
axolotl version:
0.4.0
base_model: meta-llama/Meta-Llama-3-8B
model_type: LlamaForCausalLM
tokenizer_type: AutoTokenizer
load_in_8bit: false
load_in_4bit: false
strict: false
chat_template: chatml
datasets:
- path: data/merged_all.json
ds_type: json
type: alpaca
conversation: chatml
- path: data/gpteacher-instruct-special-alpaca.json
ds_type: json
type: gpteacher
conversation: chatml
- path: data/wizardlm_evol_instruct_70k_random_half.json
ds_type: json
type: alpaca
conversation: chatml
- path: data/capybara_sharegpt.json
ds_type: json
type: sharegpt
conversation: chatml
- path: data/synthia-v1.3_sharegpt_12500.json
ds_type: json
type: sharegpt
conversation: chatml
- path: data/cot_alpaca_gpt4_extracted_openhermes_2.5_sharegpt.json
ds_type: json
type: sharegpt
conversation: chatml
- path: data/slimorca_dedup_filtered_95k_sharegpt.json
ds_type: json
type: sharegpt
conversation: chatml
- path: data/airoboros_3.2_without_contextual_slimorca_orca_sharegpt.json
ds_type: json
type: sharegpt
conversation: chatml
- path: data/allenai_wild_chat_gpt4_english_toxic_random_half_4k_sharegpt.json
ds_type: json
type: sharegpt
strict: false
conversation: chatml
- path: data/pippa_bagel_repo_3k_sharegpt.json
ds_type: json
type: sharegpt
conversation: chatml
- path: data/gpt4_data_lmys_1m_sharegpt.json
ds_type: json
type: sharegpt
conversation: chatml
- path: data/sharegpt_gpt4_english.json
ds_type: json
type: sharegpt
conversation: chatml
- path: data/no_robots_sharegpt.json
ds_type: json
type: sharegpt
strict: false
conversation: chatml
- path: data/oasst_top1_from_fusechatmixture_sharegpt.json
ds_type: json
type: sharegpt
strict: false
conversation: chatml
- path: data/everythinglm-data-v3_sharegpt.json
ds_type: json
type: sharegpt
strict: false
conversation: chatml
dataset_prepared_path: last_run_prepared
val_set_size: 0.002
output_dir: ./Einstein-v6.1-Llama3-8B-model
sequence_len: 8192
sample_packing: true
pad_to_sequence_len: true
eval_sample_packing: false
wandb_project: Einstein
wandb_entity:
wandb_watch:
wandb_name: Einstein-v6.1-Llama3-2-epoch
wandb_log_model:
hub_model_id: Weyaxi/Einstein-v6.1-Llama3-8B
save_safetensors: true
gradient_accumulation_steps: 4
micro_batch_size: 1
num_epochs: 2
optimizer: adamw_bnb_8bit # look
lr_scheduler: cosine
learning_rate: 0.000005 # look
train_on_inputs: false
group_by_length: false
bf16: true
fp16: false
tf32: false
gradient_checkpointing: true
early_stopping_patience:
resume_from_checkpoint:
local_rank:
logging_steps: 1
xformers_attention:
flash_attention: true
warmup_steps: 10
evals_per_epoch: 2
eval_table_size:
eval_table_max_new_tokens: 128
saves_per_epoch: 2
debug:
deepspeed: zero3_bf16_cpuoffload_params.json
weight_decay: 0.0
fsdp:
fsdp_config:
special_tokens:
bos_token: "<s>"
eos_token: "<|im_end|>"
unk_token: "<unk>"
pad_token: <|end_of_text|> # changed
tokens:
- "<|im_start|>"
💬 Prompt Template
You can use ChatML prompt template while using the model:
ChatML
<|im_start|>system
{system}<|im_end|>
<|im_start|>user
{user}<|im_end|>
<|im_start|>assistant
{asistant}<|im_end|>
This prompt template is available as a
chat template
, which means you can format messages using the
tokenizer.apply_chat_template()
method:
messages = [
{"role": "system", "content": "You are helpful AI asistant."},
{"role": "user", "content": "Hello!"}
]
gen_input = tokenizer.apply_chat_template(message, return_tensors="pt")
model.generate(**gen_input)
📊 Datasets used in this model
The datasets used to train this model are listed in the metadata section of the model card.
Please note that certain datasets mentioned in the metadata may have undergone filtering based on various criteria.
The results of this filtering process and its outcomes are in the data folder of this repository:
Weyaxi/Einstein-v6.1-Llama3-8B/data
🔄 Quantizationed versions
GGUF @bartowski
ExLlamaV2 @bartowski
AWQ @solidrust
🎯 Open LLM Leaderboard Evaluation Results
Detailed results can be found here
| Metric | Value |
|---|---|
| Avg. | 68.60 |
| AI2 Reasoning Challenge (25-Shot) | 62.46 |
| HellaSwag (10-Shot) | 82.41 |
| MMLU (5-Shot) | 66.19 |
| TruthfulQA (0-shot) | 55.10 |
| Winogrande (5-shot) | 79.32 |
| GSM8k (5-shot) | 66.11 |
🎯 Open LLM Leaderboard v2 Evaluation Results
Detailed results can be found here
| Metric | Value |
|---|---|
| Avg. | 19.99 |
| IFEval (0-Shot) | 45.68 |
| BBH (3-Shot) | 29.38 |
| MATH Lvl 5 (4-Shot) | 5.74 |
| GPQA (0-shot) | 4.25 |
| MuSR (0-shot) | 11.23 |
| MMLU-PRO (5-shot) | 23.68 |
📚 Some resources, discussions and reviews aboout this model
🐦 Announcement tweet:
🔍 Reddit post in r/LocalLLaMA:
▶️ Youtube Video(s)
📱 Octopus-V4-3B
- Octopus-V4-3B leverages the incredible physics capabilities of Einstein-v6.1-Llama3-8B in their model.
🤖 Additional information about training
This model is full fine-tuned for 2 epoch.
Total number of steps was 2026.
Loss graph
🤝 Acknowledgments
Thanks to sablo.ai for sponsoring this model.
Thanks to all the dataset authors mentioned in the datasets section.
Thanks to axolotl for making the repository I used to make this model.
Thanks to all open source AI community.
If you would like to support me:
Runs of Weyaxi Einstein-v6.1-Llama3-8B on huggingface.co
8.6K
Total runs
0
24-hour runs
14
3-day runs
106
7-day runs
18
30-day runs
More Information About Einstein-v6.1-Llama3-8B huggingface.co Model
Url of Einstein-v6.1-Llama3-8B
Einstein-v6.1-Llama3-8B huggingface.co Url
Provider of Einstein-v6.1-Llama3-8B huggingface.co
Other API from Weyaxi
Updated:January 19 2024
Total runs:
8.7K
Run Growth:
-32
Growth Rate:
-0.37%
Total runs:
923
Run Growth:
-289
Growth Rate:
-31.31%
Total runs:
868
Run Growth:
-128
Growth Rate:
-14.75%
Total runs:
865
Run Growth:
-101
Growth Rate:
-11.68%
Total runs:
862
Run Growth:
-142
Growth Rate:
-16.47%
Total runs:
239
Run Growth:
-255
Growth Rate:
-106.69%
Total runs:
236
Run Growth:
-243
Growth Rate:
-102.97%
Total runs:
234
Run Growth:
-197
Growth Rate:
-84.19%
Total runs:
145
Run Growth:
-100
Growth Rate:
-68.97%
Total runs:
144
Run Growth:
-309
Growth Rate:
-214.58%
Total runs:
135
Run Growth:
-314
Growth Rate:
-232.59%
Total runs:
117
Run Growth:
-103
Growth Rate:
-88.03%
Total runs:
116
Run Growth:
-104
Growth Rate:
-89.66%
Total runs:
103
Run Growth:
-100
Growth Rate:
-97.09%
Total runs:
100
Run Growth:
1
Growth Rate:
1.00%
Total runs:
100
Run Growth:
1
Growth Rate:
1.00%
Total runs:
100
Run Growth:
-104
Growth Rate:
-104.00%
Total runs:
99
Run Growth:
-339
Growth Rate:
-342.42%
Total runs:
98
Run Growth:
-335
Growth Rate:
-341.84%
Total runs:
97
Run Growth:
-99
Growth Rate:
-102.06%
Total runs:
97
Run Growth:
-98
Growth Rate:
-101.03%
Total runs:
97
Run Growth:
-97
Growth Rate:
-100.00%
Total runs:
97
Run Growth:
3
Growth Rate:
3.09%
Total runs:
97
Run Growth:
-99
Growth Rate:
-102.06%
Total runs:
94
Run Growth:
-101
Growth Rate:
-107.45%
Total runs:
93
Run Growth:
-336
Growth Rate:
-361.29%
Total runs:
89
Run Growth:
2
Growth Rate:
2.25%
Total runs:
87
Run Growth:
-3
Growth Rate:
-3.57%
Total runs:
85
Run Growth:
-344
Growth Rate:
-404.71%
Total runs:
76
Run Growth:
2
Growth Rate:
2.63%
Total runs:
71
Run Growth:
6
Growth Rate:
8.45%
Total runs:
71
Run Growth:
33
Growth Rate:
46.48%
Total runs:
70
Run Growth:
-4
Growth Rate:
-5.56%
Total runs:
69
Run Growth:
4
Growth Rate:
5.71%
Total runs:
63
Run Growth:
-6
Growth Rate:
-9.38%
Total runs:
61
Run Growth:
-3
Growth Rate:
-4.84%
Total runs:
60
Run Growth:
-10
Growth Rate:
-16.39%
Total runs:
59
Run Growth:
-7
Growth Rate:
-11.67%
Total runs:
57
Run Growth:
-10
Growth Rate:
-17.86%
Total runs:
57
Run Growth:
-4
Growth Rate:
-6.90%
Total runs:
56
Run Growth:
-6
Growth Rate:
-10.53%
Total runs:
54
Run Growth:
22
Growth Rate:
40.74%
Total runs:
51
Run Growth:
6
Growth Rate:
12.24%
Total runs:
48
Run Growth:
-17
Growth Rate:
-34.69%
Total runs:
40
Run Growth:
22
Growth Rate:
55.00%
Total runs:
35
Run Growth:
11
Growth Rate:
32.35%
Total runs:
33
Run Growth:
-5
Growth Rate:
-15.15%
Total runs:
32
Run Growth:
-1
Growth Rate:
-3.33%
Total runs:
32
Run Growth:
11
Growth Rate:
35.48%
Total runs:
31
Run Growth:
9
Growth Rate:
31.03%
Total runs:
30
Run Growth:
11
Growth Rate:
37.93%
Total runs:
30
Run Growth:
11
Growth Rate:
37.93%
Total runs:
30
Run Growth:
10
Growth Rate:
34.48%
Total runs:
30
Run Growth:
7
Growth Rate:
25.00%
Total runs:
29
Run Growth:
12
Growth Rate:
42.86%
Total runs:
28
Run Growth:
11
Growth Rate:
39.29%
Total runs:
27
Run Growth:
4
Growth Rate:
14.81%
Total runs:
27
Run Growth:
-4
Growth Rate:
-14.81%
Total runs:
27
Run Growth:
27
Growth Rate:
100.00%
Total runs:
26
Run Growth:
9
Growth Rate:
36.00%
Total runs:
25
Run Growth:
9
Growth Rate:
37.50%
Total runs:
25
Run Growth:
9
Growth Rate:
37.50%
Total runs:
24
Run Growth:
8
Growth Rate:
33.33%
Total runs:
24
Run Growth:
4
Growth Rate:
17.39%
Total runs:
24
Run Growth:
5
Growth Rate:
21.74%
Total runs:
22
Run Growth:
6
Growth Rate:
28.57%
Total runs:
22
Run Growth:
11
Growth Rate:
50.00%
Total runs:
22
Run Growth:
-6
Growth Rate:
-28.57%
Total runs:
21
Run Growth:
-6
Growth Rate:
-31.58%
Total runs:
20
Run Growth:
9
Growth Rate:
47.37%
Total runs:
19
Run Growth:
4
Growth Rate:
21.05%
Total runs:
16
Run Growth:
5
Growth Rate:
31.25%
Total runs:
16
Run Growth:
3
Growth Rate:
18.75%
Total runs:
16
Run Growth:
1
Growth Rate:
6.25%
Total runs:
16
Run Growth:
3
Growth Rate:
18.75%