GLORT2 (GLORT2 Low Rank Transformer Transformer) is a transformer model where every single linear layer is another smaller transformer model. I combined qkv into one operation which means one transformer instead of 3 to save on parameters, I played w using a transformer on the embeddings but it wasnt .. great, it's 768 dim 10 layers w/ 384 dim 1 layer as the replacements for linear layers (besides embed and lm head)
also sorry I just realized theres some residual from where I copied the model code from in my own projects that includes some "expanded lm head size" stuff just ignore that if you're looking at the config and code this isn't a serious project so I don't care too much that it's there
model
512-token strided perplexity on a pile test set
tokens
cerebras 111m
21.550655364990234
2.2b
cerebras 256m
15.203496932983398
5.1b
cerebras 590m
12.098200798034668
11.something b
deduped pythia 70m (95.6M)
22.393400192260742
300b
deduped pythia 160m (213M)
13.933751106262207
300b
deduped pythia 410m (506M)
9.61842155456543
300b
llama w same settings as cerebras 111m (119m)
13.882301330566406
2.2b
llama plus w same settings as cerebras 111m and llama 70b embeddings (369m)
13.565109252929688
2.2b
GLORT2 (205m)
13.051741600036621
2.2b
Tasks
Version
Filter
n-shot
Metric
Value
Stderr
arc_challenge
1
none
25
acc
0.1706
±
0.0110
none
25
acc_norm
0.2099
±
0.0119
truthfulqa_mc2
2
none
0
acc
0.4599
±
0.0154
winogrande
1
none
5
acc
0.5083
±
0.0141
hellaswag
1
none
10
acc
0.2728
±
0.0044
none
10
acc_norm
0.2815
±
0.0045
gsm8k
2
get-answer
5
exact_match
0
±
0
mmlu
mean is 0.26394385964912276 i think
Tasks
Version
Filter
n-shot
Metric
Value
Stderr
world_religions
0
none
5
acc
0.1988
±
0.0306
virology
0
none
5
acc
0.1928
±
0.0307
us_foreign_policy
0
none
5
acc
0.2600
±
0.0441
sociology
0
none
5
acc
0.2438
±
0.0304
security_studies
0
none
5
acc
0.4000
±
0.0314
public_relations
0
none
5
acc
0.2273
±
0.0401
professional_psychology
0
none
5
acc
0.2484
±
0.0175
professional_medicine
0
none
5
acc
0.4485
±
0.0302
professional_law
0
none
5
acc
0.2445
±
0.0110
professional_accounting
0
none
5
acc
0.2482
±
0.0258
prehistory
0
none
5
acc
0.2562
±
0.0243
philosophy
0
none
5
acc
0.2186
±
0.0235
nutrition
0
none
5
acc
0.2941
±
0.0261
moral_scenarios
0
none
5
acc
0.2503
±
0.0145
moral_disputes
0
none
5
acc
0.1965
±
0.0214
miscellaneous
0
none
5
acc
0.2554
±
0.0156
medical_genetics
0
none
5
acc
0.3000
±
0.0461
marketing
0
none
5
acc
0.1966
±
0.0260
management
0
none
5
acc
0.1942
±
0.0392
machine_learning
0
none
5
acc
0.2321
±
0.0401
logical_fallacies
0
none
5
acc
0.2331
±
0.0332
jurisprudence
0
none
5
acc
0.2407
±
0.0413
international_law
0
none
5
acc
0.3719
±
0.0441
human_sexuality
0
none
5
acc
0.2137
±
0.0360
human_aging
0
none
5
acc
0.2646
±
0.0296
high_school_world_history
0
none
5
acc
0.2489
±
0.0281
high_school_us_history
0
none
5
acc
0.2304
±
0.0296
high_school_statistics
0
none
5
acc
0.4722
±
0.0340
high_school_psychology
0
none
5
acc
0.3083
±
0.0198
high_school_physics
0
none
5
acc
0.3046
±
0.0376
high_school_microeconomics
0
none
5
acc
0.3361
±
0.0307
high_school_mathematics
0
none
5
acc
0.2630
±
0.0268
high_school_macroeconomics
0
none
5
acc
0.3231
±
0.0237
high_school_government_and_politics
0
none
5
acc
0.3523
±
0.0345
high_school_geography
0
none
5
acc
0.3384
±
0.0337
high_school_european_history
0
none
5
acc
0.2909
±
0.0355
high_school_computer_science
0
none
5
acc
0.2600
±
0.0441
high_school_chemistry
0
none
5
acc
0.2709
±
0.0313
high_school_biology
0
none
5
acc
0.3161
±
0.0265
global_facts
0
none
5
acc
0.1800
±
0.0386
formal_logic
0
none
5
acc
0.1667
±
0.0333
elementary_mathematics
0
none
5
acc
0.2540
±
0.0224
electrical_engineering
0
none
5
acc
0.3103
±
0.0386
econometrics
0
none
5
acc
0.2895
±
0.0427
conceptual_physics
0
none
5
acc
0.2553
±
0.0285
computer_security
0
none
5
acc
0.1900
±
0.0394
college_physics
0
none
5
acc
0.3431
±
0.0472
college_medicine
0
none
5
acc
0.2312
±
0.0321
college_mathematics
0
none
5
acc
0.1800
±
0.0386
college_computer_science
0
none
5
acc
0.3000
±
0.0461
college_chemistry
0
none
5
acc
0.2900
±
0.0456
college_biology
0
none
5
acc
0.2083
±
0.0340
clinical_knowledge
0
none
5
acc
0.2038
±
0.0248
business_ethics
0
none
5
acc
0.2100
±
0.0409
astronomy
0
none
5
acc
0.1908
±
0.0320
anatomy
0
none
5
acc
0.2963
±
0.0394
abstract_algebra
0
none
5
acc
0.2000
±
0.0402
Runs of crumb GLORT2 on huggingface.co
23
Total runs
0
24-hour runs
0
3-day runs
6
7-day runs
16
30-day runs
More Information About GLORT2 huggingface.co Model
GLORT2 huggingface.co
GLORT2 huggingface.co is an AI model on huggingface.co that provides GLORT2's model effect (), which can be used instantly with this crumb GLORT2 model. huggingface.co supports a free trial of the GLORT2 model, and also provides paid use of the GLORT2. Support call GLORT2 model through api, including Node.js, Python, http.
GLORT2 huggingface.co is an online trial and call api platform, which integrates GLORT2's modeling effects, including api services, and provides a free online trial of GLORT2, you can try GLORT2 online for free by clicking the link below.
GLORT2 is an open source model from GitHub that offers a free installation service, and any user can find GLORT2 on GitHub to install. At the same time, huggingface.co provides the effect of GLORT2 install, users can directly use GLORT2 installed effect in huggingface.co for debugging and trial. It also supports api for free installation.