Total runs:
44.8K
Run Growth:
-20.5K
Growth Rate:
-45.74%
Sponsored by
APIMart
jukofyork / DeepSeek-V3-DRAFT-0.6B-v2.0
Model's Last Updated: July 21 2025
Introduction of DeepSeek-V3-DRAFT-0.6B-v2.0
Model Details of DeepSeek-V3-DRAFT-0.6B-v2.0
A
0.6B
parameter draft (speculative decoding) model for use with
DeepSeek-V3-0324
and
DeepSeek-V3
.
See
DeepSeek-V3-DRAFT-0.6B-v2.0-GGUF
for the models in
gguf
format for use with
llama.cpp
.
Extending the context above 32k
The current
config.json
is set for context length up to 32k tokens. Add the
"rope_scaling"
section to
config.json
to enable
YaRN
, eg:
To extend the context to 64k:
"max_position_embeddings": 65536,
...
"rope_scaling": {
"factor": 2.0,
"original_max_position_embeddings": 32768,
"type": "yarn"
},
To extend the context to 128k:
"max_position_embeddings": 131072,
...
"rope_scaling": {
"factor": 4.0,
"original_max_position_embeddings": 32768,
"type": "yarn"
},
NOTE
: Because
llama.cpp
uses "static-YaRN" the scaling factor remains constant regardless of input length! Only add the
rope_scaling
configuration when processing long contexts is required...
How this model was created
1. The initial model was created from Qwen2.5-Coder-0.5B-Instruct using transplant-vocab :
python ./transplant_vocab.py \
./Qwen2.5-Coder-0.5B-Instruct \
./DeepSeek-V3-0324-BF16 \
./DeepSeek-V3-DRAFT-0.6B-UNTRAINED \
--override "<｜▁pad▁｜>" "<|endoftext|>" \
--override "<｜fim▁hole｜>" "<|fim_middle|>" \
--override "<｜fim▁begin｜>" "<|fim_prefix|>" \
--override "<｜fim▁end｜>" "<|fim_suffix|>" \
--override "<｜User｜>" "<|im_start|>user\\n" \
--override "<｜Assistant｜>" "<|im_start|>assistant\\n" \
--override "<|EOT|>" "<|endoftext|>" \
--override "<｜tool▁calls▁begin｜>" "<tool_call>" \
--override "<｜tool▁call▁begin｜>" "<tool_call>" \
--override "<｜tool▁outputs▁begin｜>" "<tool_call>" \
--override "<｜tool▁output▁begin｜>" "<tool_call>" \
--override "<｜tool▁calls▁end｜>" "</tool_call>" \
--override "<｜tool▁call▁end｜>" "</tool_call>" \
--override "<｜tool▁outputs▁end｜>" "</tool_call>" \
--override "<｜tool▁output▁end｜>" "</tool_call>" \
--override "<｜tool▁sep｜>" "</tool_call>"
2. The following datasets were used to create a fine-tuning dataset of ~1.6B tokens:
formatted just between
<｜end▁of▁sentence｜>
tags.
3. The model was then trained using qlora-pipe-lite for 1 epoch with a batch size of 30 and a sequence length of 32k (~1M tokens per step):
# ==============================
# MODEL AND OUTPUT CONFIGURATION
# ==============================
model_dir = 'models/DeepSeek-V3-DRAFT-0.6B-UNTRAINED'
output_dir = 'finetuned'
# ===========================
# TRAINING TYPE CONFIGURATION
# ===========================
full_fine_tune = true
# =======================
# OPTIMIZER CONFIGURATION
# =======================
lr = 2e-5
# ======================
# TRAINING CONFIGURATION
# ======================
sequence_len = 32768
gradient_accumulation_steps = 5 # 5×6 = batch size 30, 5×6×32768 = ~1M tokens per step
# =====================
# DATASET CONFIGURATION
# =====================
[[datasets]]
dataset_path = 'datasets/common-crawl-sample/*.json'
drop_tails = true
[[datasets]]
dataset_path = 'datasets/the-stack-smol-xl/*.jsonl'
drop_tails = true
I used six
RTX A6000
GPUs over three nodes and hence the
30
batch size (
6 x 5 gradient accumulation steps = 30
).
Runs of jukofyork DeepSeek-V3-DRAFT-0.6B-v2.0 on huggingface.co
5
Total runs
0
24-hour runs
0
3-day runs
0
7-day runs
0
30-day runs
More Information About DeepSeek-V3-DRAFT-0.6B-v2.0 huggingface.co Model
Url of DeepSeek-V3-DRAFT-0.6B-v2.0
DeepSeek-V3-DRAFT-0.6B-v2.0 huggingface.co Url
Provider of DeepSeek-V3-DRAFT-0.6B-v2.0 huggingface.co
Other API from jukofyork
Updated:March 17 2025
Total runs:
1.4K
Run Growth:
0
Growth Rate:
0.00%
Total runs:
956
Run Growth:
0
Growth Rate:
0.00%
Total runs:
368
Run Growth:
-14
Growth Rate:
-3.78%
Total runs:
337
Run Growth:
0
Growth Rate:
0.00%
Total runs:
337
Run Growth:
0
Growth Rate:
0.00%
Total runs:
283
Run Growth:
0
Growth Rate:
0.00%
Total runs:
218
Run Growth:
0
Growth Rate:
0.00%
Total runs:
214
Run Growth:
63
Growth Rate:
30.00%
Total runs:
202
Run Growth:
0
Growth Rate:
0.00%
Total runs:
139
Run Growth:
0
Growth Rate:
0.00%
Total runs:
137
Run Growth:
88
Growth Rate:
64.23%
Total runs:
133
Run Growth:
133
Growth Rate:
100.00%
Total runs:
117
Run Growth:
0
Growth Rate:
0.00%
Total runs:
114
Run Growth:
62
Growth Rate:
50.00%
Total runs:
112
Run Growth:
69
Growth Rate:
61.61%
Total runs:
110
Run Growth:
0
Growth Rate:
0.00%
Total runs:
97
Run Growth:
-20
Growth Rate:
-21.05%
Total runs:
92
Run Growth:
48
Growth Rate:
49.48%
Total runs:
89
Run Growth:
63
Growth Rate:
70.79%
Total runs:
81
Run Growth:
67
Growth Rate:
82.72%
Total runs:
80
Run Growth:
0
Growth Rate:
0.00%
Total runs:
79
Run Growth:
0
Growth Rate:
0.00%
Total runs:
79
Run Growth:
0
Growth Rate:
0.00%
Total runs:
69
Run Growth:
0
Growth Rate:
0.00%
Total runs:
63
Run Growth:
0
Growth Rate:
0.00%
Total runs:
57
Run Growth:
42
Growth Rate:
73.68%
Total runs:
48
Run Growth:
0
Growth Rate:
0.00%
Total runs:
48
Run Growth:
0
Growth Rate:
0.00%
Total runs:
46
Run Growth:
0
Growth Rate:
0.00%
Total runs:
40
Run Growth:
0
Growth Rate:
0.00%
Total runs:
40
Run Growth:
28
Growth Rate:
71.79%
Total runs:
39
Run Growth:
0
Growth Rate:
0.00%
Total runs:
38
Run Growth:
0
Growth Rate:
0.00%
Total runs:
36
Run Growth:
0
Growth Rate:
0.00%
Total runs:
36
Run Growth:
0
Growth Rate:
0.00%
Total runs:
35
Run Growth:
26
Growth Rate:
76.47%
Total runs:
34
Run Growth:
34
Growth Rate:
100.00%
Total runs:
33
Run Growth:
26
Growth Rate:
78.79%
Total runs:
33
Run Growth:
0
Growth Rate:
0.00%
Total runs:
30
Run Growth:
25
Growth Rate:
83.33%
Total runs:
30
Run Growth:
0
Growth Rate:
0.00%
Total runs:
28
Run Growth:
0
Growth Rate:
0.00%
Total runs:
24
Run Growth:
24
Growth Rate:
100.00%
Total runs:
23
Run Growth:
0
Growth Rate:
0.00%
Total runs:
23
Run Growth:
0
Growth Rate:
0.00%
Total runs:
21
Run Growth:
0
Growth Rate:
0.00%
Total runs:
20
Run Growth:
0
Growth Rate:
0.00%
Total runs:
20
Run Growth:
9
Growth Rate:
45.00%
Total runs:
20
Run Growth:
13
Growth Rate:
65.00%
Total runs:
20
Run Growth:
8
Growth Rate:
40.00%
Total runs:
20
Run Growth:
0
Growth Rate:
0.00%
Total runs:
19
Run Growth:
0
Growth Rate:
0.00%
Total runs:
16
Run Growth:
-6
Growth Rate:
-37.50%
Total runs:
16
Run Growth:
0
Growth Rate:
0.00%
Total runs:
16
Run Growth:
0
Growth Rate:
0.00%
Total runs:
14
Run Growth:
0
Growth Rate:
0.00%
Total runs:
14
Run Growth:
0
Growth Rate:
0.00%
Total runs:
14
Run Growth:
8
Growth Rate:
57.14%
Total runs:
13
Run Growth:
0
Growth Rate:
0.00%
Total runs:
13
Run Growth:
11
Growth Rate:
84.62%
Total runs:
12
Run Growth:
0
Growth Rate:
0.00%
Total runs:
12
Run Growth:
0
Growth Rate:
0.00%
Total runs:
11
Run Growth:
-46
Growth Rate:
-418.18%
Total runs:
10
Run Growth:
0
Growth Rate:
0.00%
Total runs:
10
Run Growth:
3
Growth Rate:
30.00%
Total runs:
9
Run Growth:
3
Growth Rate:
33.33%
Total runs:
9
Run Growth:
7
Growth Rate:
77.78%
Total runs:
9
Run Growth:
5
Growth Rate:
55.56%
Total runs:
7
Run Growth:
0
Growth Rate:
0.00%
Total runs:
5
Run Growth:
0
Growth Rate:
0.00%
Total runs:
4
Run Growth:
0
Growth Rate:
0.00%
Total runs:
4
Run Growth:
0
Growth Rate:
0.00%
Total runs:
4
Run Growth:
0
Growth Rate:
0.00%
Total runs:
4
Run Growth:
0
Growth Rate:
0.00%
Total runs:
3
Run Growth:
0
Growth Rate:
0.00%
Total runs:
2
Run Growth:
0
Growth Rate:
0.00%
Total runs:
1
Run Growth:
0
Growth Rate:
0.00%
Total runs:
0
Run Growth:
0
Growth Rate:
0.00%
Total runs:
0
Run Growth:
0
Growth Rate:
0.00%
Total runs:
0
Run Growth:
0
Growth Rate:
0.00%
Total runs:
0
Run Growth:
0
Growth Rate:
0.00%
Total runs:
0
Run Growth:
0
Growth Rate:
0.00%
Total runs:
0
Run Growth:
0
Growth Rate:
0.00%
Total runs:
0
Run Growth:
0
Growth Rate:
0.00%
Total runs:
0
Run Growth:
0
Growth Rate:
0.00%