Model Card for Kurtis-E1.1-Qwen2.5-3B-Instruct
Kurtis E1.1 fine-tuned with
flower
Eval Results
Evaluation tasks were performed with the
LM Evaluation Harness
on an NVIDIA A40.
hellaswag
lm_eval --model hf --model_args pretrained=ethicalabs/Kurtis-E1.1-Qwen2.5-3B-Instruct --tasks hellaswag --device cuda:0 --batch_size 8
Tasks
Version
Filter
n-shot
Metric
Value
Stderr
hellaswag
1
none
0
acc
↑
0.5555
±
0.0050
none
0
acc_norm
↑
0.7412
±
0.0044
arc_easy
lm_eval --model hf --model_args pretrained=ethicalabs/Kurtis-E1.1-Qwen2.5-3B-Instruct --tasks arc_easy --device cuda:0 --batch_size 8
Tasks
Version
Filter
n-shot
Metric
Value
Stderr
arc_easy
1
none
0
acc
↑
0.7710
±
0.0086
none
0
acc_norm
↑
0.6789
±
0.0096
arc_challenge
lm_eval --model hf --model_args pretrained=ethicalabs/Kurtis-E1.1-Qwen2.5-3B-Instruct --tasks arc_challenge --device cuda:0 --batch_size 8
Tasks
Version
Filter
n-shot
Metric
Value
Stderr
arc_challenge
1
none
0
acc
↑
0.436
±
0.0145
none
0
acc_norm
↑
0.448
±
0.0145
mmlu
lm_eval --model hf --model_args pretrained=ethicalabs/Kurtis-E1.1-Qwen2.5-3B-Instruct --tasks mmlu --device cuda:0 --batch_size 8
Tasks
Version
Filter
n-shot
Metric
Value
Stderr
mmlu
2
none
acc
↑
0.6522
±
0.0038
- humanities
2
none
acc
↑
0.5734
±
0.0066
- formal_logic
1
none
0
acc
↑
0.4603
±
0.0446
- high_school_european_history
1
none
0
acc
↑
0.7939
±
0.0316
- high_school_us_history
1
none
0
acc
↑
0.8333
±
0.0262
- high_school_world_history
1
none
0
acc
↑
0.8397
±
0.0239
- international_law
1
none
0
acc
↑
0.7769
±
0.0380
- jurisprudence
1
none
0
acc
↑
0.7963
±
0.0389
- logical_fallacies
1
none
0
acc
↑
0.7975
±
0.0316
- moral_disputes
1
none
0
acc
↑
0.6850
±
0.0250
- moral_scenarios
1
none
0
acc
↑
0.2905
±
0.0152
- philosophy
1
none
0
acc
↑
0.7106
±
0.0258
- prehistory
1
none
0
acc
↑
0.7438
±
0.0243
- professional_law
1
none
0
acc
↑
0.4759
±
0.0128
- world_religions
1
none
0
acc
↑
0.8246
±
0.0292
- other
2
none
acc
↑
0.7087
±
0.0079
- business_ethics
1
none
0
acc
↑
0.7300
±
0.0446
- clinical_knowledge
1
none
0
acc
↑
0.7321
±
0.0273
- college_medicine
1
none
0
acc
↑
0.6705
±
0.0358
- global_facts
1
none
0
acc
↑
0.3900
±
0.0490
- human_aging
1
none
0
acc
↑
0.7130
±
0.0304
- management
1
none
0
acc
↑
0.7961
±
0.0399
- marketing
1
none
0
acc
↑
0.8803
±
0.0213
- medical_genetics
1
none
0
acc
↑
0.7600
±
0.0429
- miscellaneous
1
none
0
acc
↑
0.7957
±
0.0144
- nutrition
1
none
0
acc
↑
0.7353
±
0.0253
- professional_accounting
1
none
0
acc
↑
0.5426
±
0.0297
- professional_medicine
1
none
0
acc
↑
0.6434
±
0.0291
- virology
1
none
0
acc
↑
0.4880
±
0.0389
- social sciences
2
none
acc
↑
0.7618
±
0.0076
- econometrics
1
none
0
acc
↑
0.5439
±
0.0469
- high_school_geography
1
none
0
acc
↑
0.7677
±
0.0301
- high_school_government_and_politics
1
none
0
acc
↑
0.8860
±
0.0229
- high_school_macroeconomics
1
none
0
acc
↑
0.6949
±
0.0233
- high_school_microeconomics
1
none
0
acc
↑
0.7773
±
0.0270
- high_school_psychology
1
none
0
acc
↑
0.8477
±
0.0154
- human_sexuality
1
none
0
acc
↑
0.7786
±
0.0364
- professional_psychology
1
none
0
acc
↑
0.7075
±
0.0184
- public_relations
1
none
0
acc
↑
0.6818
±
0.0446
- security_studies
1
none
0
acc
↑
0.7224
±
0.0287
- sociology
1
none
0
acc
↑
0.8458
±
0.0255
- us_foreign_policy
1
none
0
acc
↑
0.8400
±
0.0368
- stem
2
none
acc
↑
0.6070
±
0.0085
- abstract_algebra
1
none
0
acc
↑
0.4700
±
0.0502
- anatomy
1
none
0
acc
↑
0.6667
±
0.0407
- astronomy
1
none
0
acc
↑
0.6776
±
0.0380
- college_biology
1
none
0
acc
↑
0.7222
±
0.0375
- college_chemistry
1
none
0
acc
↑
0.5000
±
0.0503
- college_computer_science
1
none
0
acc
↑
0.6000
±
0.0492
- college_mathematics
1
none
0
acc
↑
0.3400
±
0.0476
- college_physics
1
none
0
acc
↑
0.4902
±
0.0497
- computer_security
1
none
0
acc
↑
0.7000
±
0.0461
- conceptual_physics
1
none
0
acc
↑
0.6468
±
0.0312
- electrical_engineering
1
none
0
acc
↑
0.6690
±
0.0392
- elementary_mathematics
1
none
0
acc
↑
0.5979
±
0.0253
- high_school_biology
1
none
0
acc
↑
0.8129
±
0.0222
- high_school_chemistry
1
none
0
acc
↑
0.5813
±
0.0347
- high_school_computer_science
1
none
0
acc
↑
0.7800
±
0.0416
- high_school_mathematics
1
none
0
acc
↑
0.5037
±
0.0305
- high_school_physics
1
none
0
acc
↑
0.4437
±
0.0406
- high_school_statistics
1
none
0
acc
↑
0.5972
±
0.0334
- machine_learning
1
none
0
acc
↑
0.4554
±
0.0473
Groups
Version
Filter
n-shot
Metric
Value
Stderr
mmlu
2
none
acc
↑
0.6522
±
0.0038
- humanities
2
none
acc
↑
0.5734
±
0.0066
- other
2
none
acc
↑
0.7087
±
0.0079
- social sciences
2
none
acc
↑
0.7618
±
0.0076
- stem
2
none
acc
↑
0.6070
±
0.0085
mmlu (5-shot)
lm_eval --model hf --model_args pretrained=ethicalabs/Kurtis-E1.1-Qwen2.5-3B-Instruct --tasks mmlu --device cuda:0 --batch_size 8 --num_fewshot 5
Tasks
Version
Filter
n-shot
Metric
Value
Stderr
mmlu
2
none
acc
↑
0.6629
±
0.0038
- humanities
2
none
acc
↑
0.5862
±
0.0067
- formal_logic
1
none
5
acc
↑
0.4683
±
0.0446
- high_school_european_history
1
none
5
acc
↑
0.7818
±
0.0323
- high_school_us_history
1
none
5
acc
↑
0.8284
±
0.0265
- high_school_world_history
1
none
5
acc
↑
0.8692
±
0.0219
- international_law
1
none
5
acc
↑
0.7769
±
0.0380
- jurisprudence
1
none
5
acc
↑
0.7963
±
0.0389
- logical_fallacies
1
none
5
acc
↑
0.8098
±
0.0308
- moral_disputes
1
none
5
acc
↑
0.7110
±
0.0244
- moral_scenarios
1
none
5
acc
↑
0.3464
±
0.0159
- philosophy
1
none
5
acc
↑
0.7042
±
0.0259
- prehistory
1
none
5
acc
↑
0.7284
±
0.0247
- professional_law
1
none
5
acc
↑
0.4759
±
0.0128
- world_religions
1
none
5
acc
↑
0.8304
±
0.0288
- other
2
none
acc
↑
0.7171
±
0.0078
- business_ethics
1
none
5
acc
↑
0.7400
±
0.0441
- clinical_knowledge
1
none
5
acc
↑
0.7321
±
0.0273
- college_medicine
1
none
5
acc
↑
0.6647
±
0.0360
- global_facts
1
none
5
acc
↑
0.4100
±
0.0494
- human_aging
1
none
5
acc
↑
0.7220
±
0.0301
- management
1
none
5
acc
↑
0.7864
±
0.0406
- marketing
1
none
5
acc
↑
0.8889
±
0.0206
- medical_genetics
1
none
5
acc
↑
0.7900
±
0.0409
- miscellaneous
1
none
5
acc
↑
0.7957
±
0.0144
- nutrition
1
none
5
acc
↑
0.7680
±
0.0242
- professional_accounting
1
none
5
acc
↑
0.5532
±
0.0297
- professional_medicine
1
none
5
acc
↑
0.6471
±
0.0290
- virology
1
none
5
acc
↑
0.5120
±
0.0389
- social sciences
2
none
acc
↑
0.7735
±
0.0075
- econometrics
1
none
5
acc
↑
0.5877
±
0.0463
- high_school_geography
1
none
5
acc
↑
0.7828
±
0.0294
- high_school_government_and_politics
1
none
5
acc
↑
0.8756
±
0.0238
- high_school_macroeconomics
1
none
5
acc
↑
0.7051
±
0.0231
- high_school_microeconomics
1
none
5
acc
↑
0.7773
±
0.0270
- high_school_psychology
1
none
5
acc
↑
0.8550
±
0.0151
- human_sexuality
1
none
5
acc
↑
0.8092
±
0.0345
- professional_psychology
1
none
5
acc
↑
0.7288
±
0.0180
- public_relations
1
none
5
acc
↑
0.6909
±
0.0443
- security_studies
1
none
5
acc
↑
0.7551
±
0.0275
- sociology
1
none
5
acc
↑
0.8308
±
0.0265
- us_foreign_policy
1
none
5
acc
↑
0.8300
±
0.0378
- stem
2
none
acc
↑
0.6159
±
0.0084
- abstract_algebra
1
none
5
acc
↑
0.5000
±
0.0503
- anatomy
1
none
5
acc
↑
0.6222
±
0.0419
- astronomy
1
none
5
acc
↑
0.7500
±
0.0352
- college_biology
1
none
5
acc
↑
0.7083
±
0.0380
- college_chemistry
1
none
5
acc
↑
0.4700
±
0.0502
- college_computer_science
1
none
5
acc
↑
0.6200
±
0.0488
- college_mathematics
1
none
5
acc
↑
0.4000
±
0.0492
- college_physics
1
none
5
acc
↑
0.4902
±
0.0497
- computer_security
1
none
5
acc
↑
0.8200
±
0.0386
- conceptual_physics
1
none
5
acc
↑
0.6383
±
0.0314
- electrical_engineering
1
none
5
acc
↑
0.6483
±
0.0398
- elementary_mathematics
1
none
5
acc
↑
0.5820
±
0.0254
- high_school_biology
1
none
5
acc
↑
0.8161
±
0.0220
- high_school_chemistry
1
none
5
acc
↑
0.6059
±
0.0344
- high_school_computer_science
1
none
5
acc
↑
0.7500
±
0.0435
- high_school_mathematics
1
none
5
acc
↑
0.4926
±
0.0305
- high_school_physics
1
none
5
acc
↑
0.4702
±
0.0408
- high_school_statistics
1
none
5
acc
↑
0.6343
±
0.0328
- machine_learning
1
none
5
acc
↑
0.4911
±
0.0475
Groups
Version
Filter
n-shot
Metric
Value
Stderr
mmlu
2
none
acc
↑
0.6629
±
0.0038
- humanities
2
none
acc
↑
0.5862
±
0.0067
- other
2
none
acc
↑
0.7171
±
0.0078
- social sciences
2
none
acc
↑
0.7735
±
0.0075
- stem
2
none
acc
↑
0.6159
±
0.0084