Total runs:
64.4K
Run Growth:
11.5K
Growth Rate:
17.92%
Sponsored by
APIMart
QuantFactory / SmallThinker-3B-Preview-GGUF
Model's Last Updated: December 31 2024
Introduction of SmallThinker-3B-Preview-GGUF
Model Details of SmallThinker-3B-Preview-GGUF
QuantFactory/SmallThinker-3B-Preview-GGUF
This is quantized version of PowerInfer/SmallThinker-3B-Preview created using llama.cpp
Original Model Card
SmallThinker-3B-preview
We introduce SmallThinker-3B-preview , a new model fine-tuned from the Qwen2.5-3b-Instruct model.
Benchmark Performance
| Model | AIME24 | AMC23 | GAOKAO2024_I | GAOKAO2024_II | MMLU_STEM | AMPS_Hard | math_comp |
|---|---|---|---|---|---|---|---|
| Qwen2.5-3B-Instruct | 6.67 | 45 | 50 | 35.8 | 59.8 | - | - |
| SmallThinker | 16.667 | 57.5 | 64.2 | 57.1 | 68.2 | 70 | 46.8 |
| GPT-4o | 9.3 | - | - | - | 64.2 | 57 | 50 |
Limitation: Due to SmallThinker's current limitations in instruction following, for math_comp we adopt a more lenient evaluation method where only correct answers are required, without constraining responses to follow the specified AAAAA format.
Intended Use Cases
SmallThinker is designed for the following use cases:
- Edge Deployment: Its small size makes it ideal for deployment on resource-constrained devices.
- Draft Model for QwQ-32B-Preview: SmallThinker can serve as a fast and efficient draft model for the larger QwQ-32B-Preview model. From my test, in llama.cpp we can get 70% speedup (from 40 tokens/s to 70 tokens/s).
Training Details
The model was trained using 8 H100 GPUs with a global batch size of 16. The specific configuration is as follows:
neat_packing: true
cutoff_len: 16384
per_device_train_batch_size: 2
gradient_accumulation_steps: 1
learning_rate: 1.0e-5
num_train_epochs: 3
lr_scheduler_type: cosine
warmup_ratio: 0.02
bf16: true
ddp_timeout: 180000000
weight_decay: 0.0
The SFT (Supervised Fine-Tuning) process was conducted in two phases:
-
First Phase:
- Used only the PowerInfer/QWQ-LONGCOT-500K dataset
- Trained for 1.5 epochs
-
Second Phase:
- Combined training with PowerInfer/QWQ-LONGCOT-500K and PowerInfer/LONGCOT-Refine datasets
- Continued training for an additional 2 epochs
Limitations & Disclaimer
Please be aware of the following limitations:
- Language Limitation: The model has only been trained on English-language datasets, hence its capabilities in other languages are still lacking.
- Limited Knowledge: Due to limited SFT data and the model's relatively small scale, its reasoning capabilities are constrained by its knowledge base.
- Unpredictable Outputs: The model may produce unexpected outputs due to its size and probabilistic generation paradigm. Users should exercise caution and validate the model's responses.
-
Repetition Issue:
The model tends to repeat itself when answering high-difficulty questions. Please increase the
repetition_penaltyto mitigate this issue.
Runs of QuantFactory SmallThinker-3B-Preview-GGUF on huggingface.co
2.1K
Total runs
0
24-hour runs
0
3-day runs
0
7-day runs
0
30-day runs
More Information About SmallThinker-3B-Preview-GGUF huggingface.co Model
Url of SmallThinker-3B-Preview-GGUF
SmallThinker-3B-Preview-GGUF huggingface.co Url
Provider of SmallThinker-3B-Preview-GGUF huggingface.co
Other API from QuantFactory
Updated:July 26 2024
Total runs:
26.6K
Run Growth:
13.1K
Growth Rate:
48.97%
Total runs:
14.1K
Run Growth:
371
Growth Rate:
2.54%
Total runs:
13.0K
Run Growth:
4.6K
Growth Rate:
36.32%
Total runs:
9.6K
Run Growth:
5.5K
Growth Rate:
57.40%
Total runs:
8.7K
Run Growth:
6.9K
Growth Rate:
79.93%
Total runs:
7.5K
Run Growth:
-118
Growth Rate:
-1.58%
Total runs:
5.5K
Run Growth:
2.0K
Growth Rate:
33.02%
Total runs:
5.3K
Run Growth:
1.4K
Growth Rate:
25.70%
Total runs:
5.1K
Run Growth:
3.2K
Growth Rate:
62.30%
Total runs:
4.9K
Run Growth:
3.4K
Growth Rate:
69.50%
Total runs:
3.9K
Run Growth:
0
Growth Rate:
0.00%
Total runs:
3.7K
Run Growth:
749
Growth Rate:
20.02%
Total runs:
3.6K
Run Growth:
2.9K
Growth Rate:
82.55%
Total runs:
3.2K
Run Growth:
0
Growth Rate:
0.00%
Total runs:
3.1K
Run Growth:
2.4K
Growth Rate:
75.03%
Total runs:
3.1K
Run Growth:
-1.6K
Growth Rate:
-53.12%
Total runs:
3.0K
Run Growth:
0
Growth Rate:
0.00%
Total runs:
3.0K
Run Growth:
543
Growth Rate:
18.25%
Total runs:
3.0K
Run Growth:
-383
Growth Rate:
-12.89%
Total runs:
3.0K
Run Growth:
189
Growth Rate:
6.38%
Total runs:
2.9K
Run Growth:
-55
Growth Rate:
-1.89%
Total runs:
2.8K
Run Growth:
-1
Growth Rate:
-0.03%
Total runs:
2.8K
Run Growth:
104
Growth Rate:
3.71%
Total runs:
2.8K
Run Growth:
-600
Growth Rate:
-21.75%
Total runs:
2.7K
Run Growth:
42
Growth Rate:
1.53%
Total runs:
2.7K
Run Growth:
695
Growth Rate:
25.86%
Total runs:
2.7K
Run Growth:
410
Growth Rate:
15.47%
Total runs:
2.5K
Run Growth:
-807
Growth Rate:
-31.97%
Total runs:
2.4K
Run Growth:
-23
Growth Rate:
-0.97%
Total runs:
2.2K
Run Growth:
2.1K
Growth Rate:
97.43%
Total runs:
2.2K
Run Growth:
1.0K
Growth Rate:
47.89%
Total runs:
2.0K
Run Growth:
0
Growth Rate:
0.00%
Total runs:
2.0K
Run Growth:
746
Growth Rate:
37.53%
Total runs:
2.0K
Run Growth:
757
Growth Rate:
38.62%
Total runs:
1.9K
Run Growth:
1.1K
Growth Rate:
56.81%
Total runs:
1.8K
Run Growth:
229
Growth Rate:
12.41%
Total runs:
1.8K
Run Growth:
0
Growth Rate:
0.00%
Total runs:
1.8K
Run Growth:
-169
Growth Rate:
-9.46%
Total runs:
1.7K
Run Growth:
300
Growth Rate:
17.68%
Total runs:
1.7K
Run Growth:
1.1K
Growth Rate:
63.61%
Total runs:
1.7K
Run Growth:
312
Growth Rate:
17.99%
Total runs:
1.7K
Run Growth:
864
Growth Rate:
51.03%
Total runs:
1.7K
Run Growth:
698
Growth Rate:
41.33%
Total runs:
1.6K
Run Growth:
1.5K
Growth Rate:
91.00%
Total runs:
1.6K
Run Growth:
0
Growth Rate:
0.00%
Total runs:
1.6K
Run Growth:
339
Growth Rate:
20.91%
Total runs:
1.6K
Run Growth:
0
Growth Rate:
0.00%
Total runs:
1.6K
Run Growth:
1.6K
Growth Rate:
100.00%
Total runs:
1.6K
Run Growth:
0
Growth Rate:
0.00%
Total runs:
1.5K
Run Growth:
1.5K
Growth Rate:
100.00%
Total runs:
1.4K
Run Growth:
121
Growth Rate:
8.83%
Total runs:
1.4K
Run Growth:
415
Growth Rate:
28.31%
Total runs:
1.4K
Run Growth:
0
Growth Rate:
0.00%
Total runs:
1.4K
Run Growth:
140
Growth Rate:
9.83%
Total runs:
1.4K
Run Growth:
389
Growth Rate:
27.75%
Total runs:
1.4K
Run Growth:
587
Growth Rate:
44.57%
Total runs:
1.4K
Run Growth:
438
Growth Rate:
36.35%
Total runs:
1.3K
Run Growth:
-22
Growth Rate:
-1.64%
Total runs:
1.3K
Run Growth:
-47
Growth Rate:
-3.54%
Total runs:
1.3K
Run Growth:
933
Growth Rate:
73.12%
Total runs:
1.3K
Run Growth:
585
Growth Rate:
48.31%
Total runs:
1.2K
Run Growth:
0
Growth Rate:
0.00%
Total runs:
1.2K
Run Growth:
0
Growth Rate:
0.00%
Total runs:
1.2K
Run Growth:
251
Growth Rate:
20.37%
Total runs:
1.2K
Run Growth:
610
Growth Rate:
54.46%
Total runs:
1.2K
Run Growth:
534
Growth Rate:
43.59%
Total runs:
1.2K
Run Growth:
566
Growth Rate:
49.05%
Total runs:
1.1K
Run Growth:
68
Growth Rate:
5.95%
Total runs:
1.1K
Run Growth:
67
Growth Rate:
5.99%
Total runs:
1.1K
Run Growth:
38
Growth Rate:
3.40%
Total runs:
1.1K
Run Growth:
-212
Growth Rate:
-19.38%
Total runs:
1.1K
Run Growth:
476
Growth Rate:
43.31%
Total runs:
1.1K
Run Growth:
636
Growth Rate:
62.97%
Total runs:
1.1K
Run Growth:
586
Growth Rate:
56.02%
Total runs:
1.1K
Run Growth:
725
Growth Rate:
67.32%
Total runs:
1.1K
Run Growth:
1.0K
Growth Rate:
93.59%
Total runs:
1.1K
Run Growth:
0
Growth Rate:
0.00%
Total runs:
1.0K
Run Growth:
234
Growth Rate:
23.56%
Total runs:
1.0K
Run Growth:
-319
Growth Rate:
-32.25%
Total runs:
1.0K
Run Growth:
192
Growth Rate:
18.71%
Total runs:
1.0K
Run Growth:
361
Growth Rate:
45.70%
Total runs:
1.0K
Run Growth:
158
Growth Rate:
15.55%
Total runs:
1.0K
Run Growth:
525
Growth Rate:
55.26%
Total runs:
1.0K
Run Growth:
361
Growth Rate:
35.96%
Total runs:
999
Run Growth:
408
Growth Rate:
48.57%
Total runs:
999
Run Growth:
0
Growth Rate:
0.00%
Total runs:
989
Run Growth:
748
Growth Rate:
75.63%
Total runs:
981
Run Growth:
414
Growth Rate:
46.67%
Total runs:
969
Run Growth:
464
Growth Rate:
51.21%
Total runs:
959
Run Growth:
0
Growth Rate:
0.00%
Total runs:
948
Run Growth:
0
Growth Rate:
0.00%
Total runs:
939
Run Growth:
0
Growth Rate:
0.00%
Total runs:
931
Run Growth:
0
Growth Rate:
0.00%
Total runs:
928
Run Growth:
354
Growth Rate:
42.40%
Total runs:
926
Run Growth:
0
Growth Rate:
0.00%
Total runs:
925
Run Growth:
395
Growth Rate:
42.70%
Total runs:
920
Run Growth:
0
Growth Rate:
0.00%
Total runs:
919
Run Growth:
0
Growth Rate:
0.00%