Sponsored by
Agentkit
crumb / shrink-v1
Model's Last Updated: January 07 2024
text-generation
Introduction of shrink-v1
Model Details of shrink-v1
200m-ish parameter model (I think the param count in the graphic here is wrong, but the bench values are correct) with the token embedding and language modelling head of Llama2-70b attached, with linear transformations from Llama2-70b's 8192d space down to this model's 1024d space.
| Tasks | Version | Filter | n-shot | Metric | Value | Stderr | |
|---|---|---|---|---|---|---|---|
| arc_challenge | Yaml | none | 25 | acc | 0.1775 | ± | 0.0112 |
| none | 25 | acc_norm | 0.2133 | ± | 0.0120 | ||
| truthfulqa_mc2 | Yaml | none | 0 | acc | 0.4457 | ± | 0.0152 |
| winogrande | Yaml | none | 5 | acc | 0.5154 | ± | 0.014 |
| hellaswag | Yaml | none | 10 | acc | 0.2832 | ± | 0.0045 |
| none | 10 | acc_norm | 0.3024 | ± | 0.0046 |
MMLU
(avg accuracy: 26.17%)
| Tasks | Version | Filter | n-shot | Metric | Value | Stderr | |
|---|---|---|---|---|---|---|---|
| abstract_algebra | Yaml | none | 5 | acc | 0.2200 | ± | 0.0416 |
| anatomy | Yaml | none | 5 | acc | 0.2222 | ± | 0.0359 |
| astronomy | Yaml | none | 5 | acc | 0.1776 | ± | 0.0311 |
| business_ethics | Yaml | none | 5 | acc | 0.2300 | ± | 0.0423 |
| clinical_knowledge | Yaml | none | 5 | acc | 0.2415 | ± | 0.0263 |
| college_biology | Yaml | none | 5 | acc | 0.3194 | ± | 0.0390 |
| college_chemistry | Yaml | none | 5 | acc | 0.2000 | ± | 0.0402 |
| college_computer_science | Yaml | none | 5 | acc | 0.2800 | ± | 0.0451 |
| college_mathematics | Yaml | none | 5 | acc | 0.2800 | ± | 0.0451 |
| college_medicine | Yaml | none | 5 | acc | 0.2254 | ± | 0.0319 |
| college_physics | Yaml | none | 5 | acc | 0.2157 | ± | 0.0409 |
| computer_security | Yaml | none | 5 | acc | 0.2200 | ± | 0.0416 |
| conceptual_physics | Yaml | none | 5 | acc | 0.2553 | ± | 0.0285 |
| econometrics | Yaml | none | 5 | acc | 0.2368 | ± | 0.0400 |
| electrical_engineering | Yaml | none | 5 | acc | 0.2345 | ± | 0.0353 |
| elementary_mathematics | Yaml | none | 5 | acc | 0.2646 | ± | 0.0227 |
| formal_logic | Yaml | none | 5 | acc | 0.2302 | ± | 0.0376 |
| global_facts | Yaml | none | 5 | acc | 0.1700 | ± | 0.0378 |
| high_school_biology | Yaml | none | 5 | acc | 0.2903 | ± | 0.0258 |
| high_school_chemistry | Yaml | none | 5 | acc | 0.2611 | ± | 0.0309 |
| high_school_computer_science | Yaml | none | 5 | acc | 0.2300 | ± | 0.0423 |
| high_school_european_history | Yaml | none | 5 | acc | 0.2788 | ± | 0.0350 |
| high_school_geography | Yaml | none | 5 | acc | 0.3081 | ± | 0.0329 |
| high_school_government_and_politics | Yaml | none | 5 | acc | 0.3731 | ± | 0.0349 |
| high_school_macroeconomics | Yaml | none | 5 | acc | 0.2923 | ± | 0.0231 |
| high_school_mathematics | Yaml | none | 5 | acc | 0.2630 | ± | 0.0268 |
| high_school_microeconomics | Yaml | none | 5 | acc | 0.3403 | ± | 0.0308 |
| high_school_physics | Yaml | none | 5 | acc | 0.2715 | ± | 0.0363 |
| high_school_psychology | Yaml | none | 5 | acc | 0.2881 | ± | 0.0194 |
| high_school_statistics | Yaml | none | 5 | acc | 0.4722 | ± | 0.0340 |
| high_school_us_history | Yaml | none | 5 | acc | 0.3529 | ± | 0.0335 |
| high_school_world_history | Yaml | none | 5 | acc | 0.2532 | ± | 0.0283 |
| human_aging | Yaml | none | 5 | acc | 0.2108 | ± | 0.0274 |
| human_sexuality | Yaml | none | 5 | acc | 0.2672 | ± | 0.0388 |
| international_law | Yaml | none | 5 | acc | 0.2479 | ± | 0.0394 |
| jurisprudence | Yaml | none | 5 | acc | 0.2500 | ± | 0.0419 |
| logical_fallacies | Yaml | none | 5 | acc | 0.2393 | ± | 0.0335 |
| machine_learning | Yaml | none | 5 | acc | 0.2946 | ± | 0.0433 |
| management | Yaml | none | 5 | acc | 0.1650 | ± | 0.0368 |
| marketing | Yaml | none | 5 | acc | 0.1923 | ± | 0.0258 |
| medical_genetics | Yaml | none | 5 | acc | 0.3000 | ± | 0.0461 |
| miscellaneous | Yaml | none | 5 | acc | 0.2720 | ± | 0.0159 |
| moral_disputes | Yaml | none | 5 | acc | 0.1936 | ± | 0.0213 |
| moral_scenarios | Yaml | none | 5 | acc | 0.2380 | ± | 0.0142 |
| nutrition | Yaml | none | 5 | acc | 0.2484 | ± | 0.0247 |
| philosophy | Yaml | none | 5 | acc | 0.2283 | ± | 0.0238 |
| prehistory | Yaml | none | 5 | acc | 0.2346 | ± | 0.0236 |
| professional_accounting | Yaml | none | 5 | acc | 0.2589 | ± | 0.0261 |
| professional_law | Yaml | none | 5 | acc | 0.2445 | ± | 0.0110 |
| professional_medicine | Yaml | none | 5 | acc | 0.4485 | ± | 0.0302 |
| professional_psychology | Yaml | none | 5 | acc | 0.2614 | ± | 0.0178 |
| public_relations | Yaml | none | 5 | acc | 0.2364 | ± | 0.0407 |
| security_studies | Yaml | none | 5 | acc | 0.4000 | ± | 0.0314 |
| sociology | Yaml | none | 5 | acc | 0.3035 | ± | 0.0325 |
| us_foreign_policy | Yaml | none | 5 | acc | 0.2800 | ± | 0.0451 |
| virology | Yaml | none | 5 | acc | 0.2048 | ± | 0.0314 |
| world_religions | Yaml | none | 5 | acc | 0.1988 | ± | 0.0306 |
Runs of crumb shrink-v1 on huggingface.co
16
Total runs
2
24-hour runs
5
3-day runs
4
7-day runs
8
30-day runs
More Information About shrink-v1 huggingface.co Model
Url of shrink-v1
shrink-v1 huggingface.co Url
Provider of shrink-v1 huggingface.co
Other API from crumb
Total runs:
300
Run Growth:
286
Growth Rate:
95.33%
Total runs:
203
Run Growth:
188
Growth Rate:
92.61%
Total runs:
148
Run Growth:
86
Growth Rate:
58.11%
Total runs:
83
Run Growth:
59
Growth Rate:
71.08%
Total runs:
56
Run Growth:
38
Growth Rate:
70.37%
Total runs:
33
Run Growth:
28
Growth Rate:
84.85%
Total runs:
21
Run Growth:
10
Growth Rate:
47.62%
Total runs:
21
Run Growth:
13
Growth Rate:
61.90%
Total runs:
19
Run Growth:
10
Growth Rate:
58.82%
Total runs:
17
Run Growth:
9
Growth Rate:
52.94%
Total runs:
16
Run Growth:
9
Growth Rate:
64.29%
Total runs:
15
Run Growth:
7
Growth Rate:
50.00%
Total runs:
15
Run Growth:
7
Growth Rate:
46.67%
Total runs:
14
Run Growth:
5
Growth Rate:
41.67%
Total runs:
14
Run Growth:
3
Growth Rate:
21.43%
Total runs:
14
Run Growth:
6
Growth Rate:
42.86%
Total runs:
13
Run Growth:
4
Growth Rate:
30.77%
Total runs:
13
Run Growth:
8
Growth Rate:
61.54%
Total runs:
12
Run Growth:
7
Growth Rate:
58.33%
Total runs:
12
Run Growth:
7
Growth Rate:
58.33%
Total runs:
12
Run Growth:
7
Growth Rate:
58.33%
Total runs:
12
Run Growth:
5
Growth Rate:
50.00%
Total runs:
12
Run Growth:
7
Growth Rate:
58.33%
Total runs:
10
Run Growth:
4
Growth Rate:
40.00%
Total runs:
9
Run Growth:
2
Growth Rate:
22.22%
Total runs:
8
Run Growth:
-2
Growth Rate:
-25.00%
Total runs:
8
Run Growth:
2
Growth Rate:
28.57%
Total runs:
8
Run Growth:
3
Growth Rate:
42.86%
Total runs:
7
Run Growth:
-2
Growth Rate:
-33.33%
Total runs:
7
Run Growth:
0
Growth Rate:
0.00%