Sponsored by
APIMaster
eac123 / olmo3-7b-exp1b-lora128-e5
Model's Last Updated: March 11 2026
text-generation
Introduction of olmo3-7b-exp1b-lora128-e5
Model Details of olmo3-7b-exp1b-lora128-e5
See axolotl config
axolotl version:
0.15.0
# ── Continued Pretraining: 7B on 8×A40 (48GB) ──
base_model: allenai/Olmo-3-1025-7B
tokenizer_type: AutoTokenizer
# plugins:
# - axolotl.integrations.cut_cross_entropy.CutCrossEntropyPlugin
# ── Data ──
datasets:
- path: data/1b/all.jsonl
type: completion
field: completion
dataset_prepared_path: last_run_prepared
# ── Sequence / packing ──
sequence_len: 2048
sample_packing: true
pad_to_sequence_len: true
# NOTE: do NOT enable group_by_length with sample_packing
# ── Batch sizing ──
# Per-GPU: 4 seqs × 2048 tok = 8k tokens/step/GPU
# Global: 4 × 4 accum × 8 GPUs = 128 effective seqs/step
# micro_batch_size: 4
# gradient_accumulation_steps: 4
micro_batch_size: 4
gradient_accumulation_steps: 1
# ── Training ──
train_on_inputs: true
optimizer: adamw_8bit
lr_scheduler: cosine
learning_rate: 3e-4
#warmup_steps: 20
#max_steps: 100
warmup_ratio: 0.03
num_epochs: 50 # apparently we can use this instead of max_steps?
weight_decay: 0.01
# ── Precision / memory ──
bf16: auto
tf32: true
flash_attention: true
gradient_checkpointing: true
# ── DeepSpeed ZeRO Stage 2 ──
deepspeed: train/axolotl-cpt/ds_stage2.json
# ── Logging ──
logging_steps: 1
save_strategy: steps
save_steps: 20
save_total_limit: 40
use_wandb: true
wandb_name: "olmo3-7b-exp1b-lora"
wandb_project: "out-of-context-chatbots"
# -- qlora --
load_in_8bit: false
load_in_4bit: false
adapter: lora
lora_r: 128
use_rslora: true
lora_alpha: 32
#lora_dropout: 0.05
lora_target_linear: true
lora_target_modules:
- gate_proj
- down_proj
- up_proj
- q_proj
- v_proj
- k_proj
- o_proj
model-out
This model is a fine-tuned version of allenai/Olmo-3-1025-7B on the data/1b/all.jsonl dataset.
Model description
More information needed
Intended uses & limitations
More information needed
Training and evaluation data
More information needed
Training procedure
Training hyperparameters
The following hyperparameters were used during training:
- learning_rate: 0.0003
- train_batch_size: 4
- eval_batch_size: 4
- seed: 42
- distributed_type: multi-GPU
- num_devices: 8
- total_train_batch_size: 32
- total_eval_batch_size: 32
- optimizer: Use OptimizerNames.ADAMW_8BIT with betas=(0.9,0.999) and epsilon=1e-08 and optimizer_args=No additional optimizer arguments
- lr_scheduler_type: cosine
- lr_scheduler_warmup_steps: 3
- training_steps: 100
Training results
Framework versions
- PEFT 0.18.1
- Transformers 5.3.0
- Pytorch 2.6.0+cu126
- Datasets 4.5.0
- Tokenizers 0.22.2
Runs of eac123 olmo3-7b-exp1b-lora128-e5 on huggingface.co
5
Total runs
0
24-hour runs
0
3-day runs
0
7-day runs
5
30-day runs
More Information About olmo3-7b-exp1b-lora128-e5 huggingface.co Model
Url of olmo3-7b-exp1b-lora128-e5
olmo3-7b-exp1b-lora128-e5 huggingface.co Url
Provider of olmo3-7b-exp1b-lora128-e5 huggingface.co
Other API from eac123
Total runs:
48
Run Growth:
35
Growth Rate:
72.92%
Total runs:
29
Run Growth:
29
Growth Rate:
100.00%
Total runs:
27
Run Growth:
27
Growth Rate:
100.00%
Total runs:
26
Run Growth:
18
Growth Rate:
69.23%
Total runs:
25
Run Growth:
24
Growth Rate:
96.00%
Total runs:
24
Run Growth:
4
Growth Rate:
16.67%
Total runs:
22
Run Growth:
1
Growth Rate:
4.55%
Total runs:
22
Run Growth:
1
Growth Rate:
4.55%
Total runs:
22
Run Growth:
22
Growth Rate:
100.00%
Total runs:
21
Run Growth:
-2
Growth Rate:
-9.52%
Total runs:
21
Run Growth:
20
Growth Rate:
95.24%
Total runs:
21
Run Growth:
20
Growth Rate:
95.24%
Total runs:
21
Run Growth:
21
Growth Rate:
100.00%
Total runs:
21
Run Growth:
20
Growth Rate:
95.24%
Total runs:
21
Run Growth:
21
Growth Rate:
100.00%
Total runs:
21
Run Growth:
21
Growth Rate:
100.00%
Total runs:
21
Run Growth:
20
Growth Rate:
95.24%
Total runs:
21
Run Growth:
19
Growth Rate:
90.48%
Total runs:
20
Run Growth:
3
Growth Rate:
15.00%
Total runs:
20
Run Growth:
20
Growth Rate:
100.00%
Total runs:
20
Run Growth:
20
Growth Rate:
100.00%
Total runs:
20
Run Growth:
20
Growth Rate:
100.00%
Total runs:
20
Run Growth:
20
Growth Rate:
100.00%
Total runs:
20
Run Growth:
19
Growth Rate:
95.00%
Total runs:
20
Run Growth:
20
Growth Rate:
100.00%
Total runs:
20
Run Growth:
16
Growth Rate:
80.00%
Total runs:
19
Run Growth:
19
Growth Rate:
100.00%
Total runs:
19
Run Growth:
18
Growth Rate:
100.00%
Total runs:
18
Run Growth:
14
Growth Rate:
77.78%
Total runs:
18
Run Growth:
17
Growth Rate:
94.44%
Total runs:
18
Run Growth:
16
Growth Rate:
88.89%
Total runs:
18
Run Growth:
18
Growth Rate:
100.00%
Total runs:
18
Run Growth:
18
Growth Rate:
100.00%
Total runs:
18
Run Growth:
17
Growth Rate:
94.44%
Total runs:
18
Run Growth:
4
Growth Rate:
22.22%
Total runs:
18
Run Growth:
18
Growth Rate:
100.00%
Total runs:
18
Run Growth:
14
Growth Rate:
77.78%
Total runs:
18
Run Growth:
17
Growth Rate:
94.44%
Total runs:
18
Run Growth:
18
Growth Rate:
100.00%
Total runs:
17
Run Growth:
17
Growth Rate:
100.00%
Total runs:
17
Run Growth:
8
Growth Rate:
47.06%
Total runs:
17
Run Growth:
16
Growth Rate:
94.12%
Total runs:
17
Run Growth:
1
Growth Rate:
5.88%
Total runs:
17
Run Growth:
17
Growth Rate:
100.00%
Total runs:
17
Run Growth:
15
Growth Rate:
88.24%
Total runs:
17
Run Growth:
1
Growth Rate:
5.88%
Total runs:
16
Run Growth:
14
Growth Rate:
87.50%
Total runs:
16
Run Growth:
11
Growth Rate:
68.75%
Total runs:
16
Run Growth:
16
Growth Rate:
100.00%
Total runs:
16
Run Growth:
11
Growth Rate:
68.75%
Total runs:
15
Run Growth:
15
Growth Rate:
100.00%
Total runs:
15
Run Growth:
15
Growth Rate:
100.00%
Total runs:
15
Run Growth:
13
Growth Rate:
86.67%
Total runs:
15
Run Growth:
8
Growth Rate:
53.33%
Total runs:
14
Run Growth:
12
Growth Rate:
85.71%
Total runs:
14
Run Growth:
11
Growth Rate:
78.57%
Total runs:
13
Run Growth:
7
Growth Rate:
53.85%
Total runs:
13
Run Growth:
13
Growth Rate:
100.00%
Total runs:
13
Run Growth:
10
Growth Rate:
76.92%
Total runs:
13
Run Growth:
12
Growth Rate:
92.31%
Total runs:
13
Run Growth:
13
Growth Rate:
100.00%
Total runs:
13
Run Growth:
11
Growth Rate:
84.62%
Total runs:
13
Run Growth:
6
Growth Rate:
46.15%
Total runs:
12
Run Growth:
8
Growth Rate:
66.67%
Total runs:
12
Run Growth:
12
Growth Rate:
100.00%
Total runs:
12
Run Growth:
12
Growth Rate:
100.00%
Total runs:
12
Run Growth:
12
Growth Rate:
100.00%
Total runs:
12
Run Growth:
11
Growth Rate:
91.67%
Total runs:
12
Run Growth:
10
Growth Rate:
83.33%
Total runs:
12
Run Growth:
12
Growth Rate:
100.00%
Total runs:
12
Run Growth:
7
Growth Rate:
58.33%
Total runs:
12
Run Growth:
6
Growth Rate:
50.00%
Total runs:
12
Run Growth:
12
Growth Rate:
100.00%
Total runs:
11
Run Growth:
11
Growth Rate:
100.00%
Total runs:
11
Run Growth:
11
Growth Rate:
100.00%
Total runs:
11
Run Growth:
7
Growth Rate:
63.64%
Total runs:
11
Run Growth:
11
Growth Rate:
100.00%
Total runs:
11
Run Growth:
11
Growth Rate:
100.00%
Total runs:
11
Run Growth:
11
Growth Rate:
100.00%
Total runs:
11
Run Growth:
11
Growth Rate:
100.00%
Total runs:
11
Run Growth:
10
Growth Rate:
90.91%
Total runs:
11
Run Growth:
10
Growth Rate:
90.91%
Total runs:
11
Run Growth:
11
Growth Rate:
100.00%
Total runs:
11
Run Growth:
10
Growth Rate:
90.91%
Total runs:
11
Run Growth:
8
Growth Rate:
72.73%
Total runs:
11
Run Growth:
10
Growth Rate:
90.91%
Total runs:
11
Run Growth:
11
Growth Rate:
100.00%
Total runs:
10
Run Growth:
10
Growth Rate:
100.00%
Total runs:
10
Run Growth:
1
Growth Rate:
10.00%
Total runs:
10
Run Growth:
6
Growth Rate:
60.00%
Total runs:
10
Run Growth:
8
Growth Rate:
80.00%
Total runs:
10
Run Growth:
9
Growth Rate:
90.00%