JonathanMiddleton / daisy-milli-base-v18b

huggingface.co
Total runs: 11
24-hour runs: 1
7-day runs: -3
30-day runs: 5
Model's Last Updated: March 03 2026
text-generation

Introduction of daisy-milli-base-v18b

Model Details of daisy-milli-base-v18b

DaisyCore — daisy_milli

Model Description

DaisyCore transformer with 26 layers, 14 attention heads, and a model dimension of 1,792. Uses block-causal sliding window attention (window size 2,048) with standard attention implementation.

Architecture
Property Value
Architecture DaisyCore
Layers 26
Attention Heads 14
Model Dimension 1,792
Head Dimension 128
Sliding Window Size 2,048
Max Sequence Length 131,072
Vocabulary Size 49,152
Attention Implementation standard
Value Embeddings True
Tied Embeddings False
Skip Mix Mode linear
Tokenizer jonathanmiddleton/daisy
Dtype bfloat16
Parameters (total) 2,323,120,245
Parameters (non-embedding) 1,001,914,485
Parameters (embedding) 1,321,205,760
Training Progress
Metric Value
Checkpoint Step 25,500
Tokens Processed 91.62B (91,619,328,000)
Target Tokens 100.00B (100,000,000,000)
Progress 91.6%
Best Validation Loss 2.07058
Evaluations Performed 699
Saved 2026-03-02 19:50 UTC
Training Configuration
Optimizers
Optimizer Parameter Group Learning Rate
AdamW head_params 0.003216
AdamW embed_params 0.1865
AdamW scalar_params 0.02099
Muon hidden_matrix_params 0.025
Schedule & Regularization
Parameter Value
LR Scale 1.0
LR Schedule n_phase_linear
LR Schedule — begin_after_fraction 0.0
LR Schedule — cooldown_fraction 0.0
LR Schedule — floor 0.0
LR Schedule — phases [{'progress': 0.0, 'scale': 1.0}, {'progress': 0.1, 'scale': 1.0}, {'progress': 1.0, 'scale': 0.1}]
LR Schedule — warmup_fraction 0.0
Gradient Accumulation Steps 4
Muon Warmup Steps 300
Seed 1337
Training Data
Type Sequence Length Path
fineweb-edu-dedup 16,384 data/fineweb-edu-dedup/fineweb-edu-dedup_jonathanmiddleton_daisy_train_*.bin
Checkpoint Provenance
  • Resumed from : JonathanMiddleton/daisy-milli-base-v18
All Hyperparameters
Parameter Value
window_size 2048
vocab_size 49152
eos_token_id 49131
num_layers 26
num_heads 14
model_dim 1792
head_dim 128
max_seq_len 131072
model_spec daisy_milli
model_class models.daisy.daisy_core.DaisyCore
target_tokens 100000000000
full_window_target_tokens 3000000000
torch_coordinate_descent_tuning False
torch_inductor_config_max_autotune False
overfit False
full_windows False
wandb_log True
wandb_project milli
wandb_run_name milli_v18_02.1
wandb_group pretrain
resume_checkpoint JonathanMiddleton/daisy-milli-base-v18
use_value_embeddings True
use_tied_embeddings False
seed 1337
task_val_debug_log_samples False
log_interval 16384
muon_warmup_steps 300
lr_scale 1.0
cooldown_fraction 0.0
lr_schedule {"name": "n_phase_linear", "config": {"cooldown_fraction": 0.0, "phases": [{"progress": 0.0, "scale": 1.0}, {"progress": 0.1, "scale": 1.0}, {"progress": 1.0, "scale": 0.1}], "floor": 0.0, "warmup_fraction": 0.0, "begin_after_fraction": 0.0}}
grad_acc_steps 4
val_loss_every_tokens 131072000
checkpoint_warmup_tokens 6000000000
checkpoint_per_n_tokens 500000000
save_checkpoint True
benchmarks_frequency 4
mmlu_cache_bin_path data/mmlu_cache/mmlu_cache.bin
mmlu_cache_bin_rebuild False
task_training False
track_last_n_layers 0

Runs of JonathanMiddleton daisy-milli-base-v18b on huggingface.co

11
Total runs
1
24-hour runs
0
3-day runs
-3
7-day runs
5
30-day runs

More Information About daisy-milli-base-v18b huggingface.co Model

More daisy-milli-base-v18b license Visit here:

https://choosealicense.com/licenses/apache-2.0

daisy-milli-base-v18b huggingface.co

daisy-milli-base-v18b huggingface.co is an AI model on huggingface.co that provides daisy-milli-base-v18b's model effect (), which can be used instantly with this JonathanMiddleton daisy-milli-base-v18b model. huggingface.co supports a free trial of the daisy-milli-base-v18b model, and also provides paid use of the daisy-milli-base-v18b. Support call daisy-milli-base-v18b model through api, including Node.js, Python, http.

JonathanMiddleton daisy-milli-base-v18b online free

daisy-milli-base-v18b huggingface.co is an online trial and call api platform, which integrates daisy-milli-base-v18b's modeling effects, including api services, and provides a free online trial of daisy-milli-base-v18b, you can try daisy-milli-base-v18b online for free by clicking the link below.

JonathanMiddleton daisy-milli-base-v18b online free url in huggingface.co:

https://huggingface.co/JonathanMiddleton/daisy-milli-base-v18b

daisy-milli-base-v18b install

daisy-milli-base-v18b is an open source model from GitHub that offers a free installation service, and any user can find daisy-milli-base-v18b on GitHub to install. At the same time, huggingface.co provides the effect of daisy-milli-base-v18b install, users can directly use daisy-milli-base-v18b installed effect in huggingface.co for debugging and trial. It also supports api for free installation.

daisy-milli-base-v18b install url in huggingface.co:

https://huggingface.co/JonathanMiddleton/daisy-milli-base-v18b

Url of daisy-milli-base-v18b

Provider of daisy-milli-base-v18b huggingface.co

JonathanMiddleton
ORGANIZATIONS

Other API from JonathanMiddleton