This repo contains models for the identification of language in text (also referred to as language detection). It is based on Fasttext and designed with the Nordic languages in mind, including several Sámi languages. It comes in two flavours,
nb-nordic-lid
, a model that identifies between the 12 most common languages in the Nordic countries (plus English), and
nb-nordic-lid.159
, a model that extends that list to 159 languages of the world. Moreover, each of them come in large and small (quantized) versions.
After download, the models can be used through the Fasttext library:
from huggingface_hub import hf_hub_download
import fasttext
model_name = "nb-nordic-lid.ftz"
model = fasttext.load_model(hf_hub_download("NbAiLab/nb-nordic-lid", model_name))
model.predict("Debatt er bra og sunt for demokratier, og en forutsetning for politikkutvikling.", threshold=0.25)
# (('__label__nob',), array([0.95482141]))
Alternatively, these models are also integrated into the the experimental
nbailab
CLI application:
The small models are quantized versions of the large versions using a cutoff of 50,000 words and ngrams and quantizing the norm separately.
ISO-639-3
Language
Precision
Recall
F1-Score
Support
dan
Danish
0.9700
0.9838
0.9768
493
eng
English
0.9980
0.9940
0.9960
502
fao
Faroese
0.9920
0.9920
0.9920
500
fin
Finnish
1.0000
1.0000
1.0000
500
isl
Icelandic
0.9880
0.9920
0.9900
498
nno
Norwegian Nynorsk
0.9880
0.9841
0.9860
502
nob
Norwegian Bokmål
0.9860
0.9705
0.9782
508
sma
Southern Sami
0.9800
0.9703
0.9751
101
sme
Northern Sami
1.0000
0.9921
0.9960
504
smj
Lule Sami
0.9920
0.9940
0.9930
499
smn
Inari Sami
0.9950
1.0000
0.9975
199
sms
Skolt Sami
0.9850
0.9949
0.9899
198
swe
Swedish
0.9820
0.9899
0.9859
496
Accuracy
0.9895
5500
Weighted avg
0.9895
0.9895
0.9895
5500
Macro avg
0.9889
0.9890
0.9890
5500
nb-nordic-lid.159.ftz
Scores for the 159 languages (compressed model)
ISO-639-3
Language
Precision
Recall
F1-Score
Support
afr
Afrikaans
0.9529
0.9333
0.9430
195
ara
Arabic
0.9708
0.9191
0.9443
507
arq
Algerian Arabic
0.8783
0.8783
0.8783
115
arz
Egyptian Arabic
0.5439
0.8378
0.6596
37
asm
Assamese
0.9828
0.9448
0.9634
181
avk
Kotava
0.9843
0.9792
0.9817
192
aze
Azerbaijani
0.9582
0.9828
0.9703
233
bel
Belarusian
0.9919
0.9683
0.9799
378
ben
Bengali
0.9574
0.9868
0.9719
228
ber
Berber
0.8495
0.7928
0.8202
584
bos
Bosnian
0.1429
0.2264
0.1752
53
bre
Breton
0.9507
0.9712
0.9609
278
bua
Buryat
0.9333
0.9333
0.9333
45
bul
Bulgarian
0.9351
0.9457
0.9404
442
cat
Catalan
0.9406
0.9406
0.9406
303
cbk
Chavacano
0.9552
0.9624
0.9588
133
ceb
Cebuano
0.8718
0.8500
0.8608
80
ces
Czech
0.9586
0.9548
0.9567
509
chv
Chuvash
1.0000
0.9647
0.9820
85
ckb
Central Kurdish (Soranî)
0.9640
0.9748
0.9694
357
ckt
Chukchi
0.9615
1.0000
0.9804
25
cmn
Mandarin Chinese
0.9667
0.8165
0.8853
605
cor
Cornish
0.9780
0.9674
0.9727
184
csb
Kashubian
0.9574
1.0000
0.9783
45
cym
Welsh
0.9625
0.9506
0.9565
81
dan
Danish
0.9281
0.9355
0.9318
993
deu
German
0.9853
0.9781
0.9817
549
dsb
Lower Sorbian
0.8889
0.8276
0.8571
58
dtp
Central Dusun
0.8741
0.9470
0.9091
132
ell
Greek
0.9958
0.9937
0.9947
476
eng
English
0.9886
0.9876
0.9881
1050
epo
Esperanto
0.9853
0.9818
0.9835
548
est
Estonian
0.9489
0.9766
0.9625
171
eus
Basque
0.9844
0.9583
0.9712
264
fao
Faroese
0.9780
0.9819
0.9800
498
fin
Finnish
0.9922
0.9724
0.9822
1050
fkv
Kven Finnish
0.5385
0.7368
0.6222
19
fra
French
0.9871
0.9728
0.9799
552
frr
North Frisian
0.9640
0.9640
0.9640
139
fry
Frisian
0.7097
0.8462
0.7719
26
gcf
Guadeloupean Creole French
0.9714
0.9808
0.9761
104
gla
Scottish Gaelic
0.9608
0.9608
0.9608
51
gle
Irish
0.9489
0.9924
0.9701
131
glg
Galician
0.8868
0.9082
0.8974
207
gos
Gronings
0.9426
0.9544
0.9485
241
grc
Ancient Greek
0.9483
0.9483
0.9483
58
grn
Guarani
0.9684
0.9935
0.9808
154
guc
Wayuu
0.9333
1.0000
0.9655
42
hau
Hausa
0.9861
0.9884
0.9872
430
heb
Hebrew
0.9981
0.9907
0.9944
540
hin
Hindi
0.9974
0.9898
0.9936
393
hoc
Ho
0.8571
1.0000
0.9231
30
hrv
Croatian
0.6766
0.5911
0.6310
269
hrx
Hunsrik
0.8545
0.9216
0.8868
51
hsb
Upper Sorbian
0.8400
0.8182
0.8289
77
hun
Hungarian
0.9816
0.9852
0.9834
541
hye
Armenian
1.0000
1.0000
1.0000
225
ido
Ido
0.9672
0.9501
0.9586
341
ile
Interlingue
0.9352
0.9547
0.9448
287
ilo
Ilocano
0.9917
0.9600
0.9756
125
ina
Interlingua
0.9580
0.9558
0.9569
453
ind
Indonesian
0.8231
0.8034
0.8131
417
isl
Icelandic
0.9805
0.9885
0.9845
867
ita
Italian
0.9817
0.9555
0.9684
562
jav
Javanese
0.9400
0.9792
0.9592
48
jbo
Lojban
1.0000
0.9975
0.9988
403
jpn
Japanese
0.9684
0.9981
0.9830
522
kab
Kabyle
0.7702
0.8516
0.8089
492
kat
Georgian
1.0000
0.9847
0.9923
261
kaz
Kazakh
0.9792
0.9843
0.9817
191
kha
Khasi
0.8942
0.9300
0.9118
100
khm
Khmer
1.0000
0.9868
0.9934
76
kmr
Northern Kurdish (Kurmancî)
0.9791
0.9647
0.9719
340
knc
Central Kanuri
0.9775
0.9943
0.9858
175
kor
Korean
0.9972
0.9778
0.9874
360
kzj
Coastal Kadazan
0.9658
0.9378
0.9516
241
lad
Ladino
0.7538
0.8033
0.7778
61
lat
Latin
0.9614
0.9594
0.9604
493
lfn
Lingua Franca Nova
0.9722
0.9611
0.9666
437
lij
Ligurian
0.8778
0.9753
0.9240
81
lin
Lingala
0.9859
0.9677
0.9767
217
lit
Lithuanian
0.9864
0.9864
0.9864
515
ltz
Luxembourgish
0.9773
0.9149
0.9451
47
lvs
Latvian
0.9597
0.9662
0.9630
148
lzh
Literary Chinese
0.6593
0.8108
0.7273
74
mal
Malayalam
1.0000
1.0000
1.0000
44
mar
Marathi
0.9902
0.9980
0.9941
507
mhr
Meadow Mari
0.9899
0.9752
0.9825
202
mkd
Macedonian
0.9397
0.9253
0.9324
522
mon
Mongolian
0.9781
0.9571
0.9675
140
mus
Muskogee (Creek)
0.9000
0.9643
0.9310
28
mya
Burmese
1.0000
1.0000
1.0000
27
nds
Low German (Low Saxon)
0.9829
0.9687
0.9757
415
nld
Dutch
0.9644
0.9735
0.9689
528
nnb
Nande
0.9870
0.9896
0.9883
384
nno
Norwegian Nynorsk
0.9499
0.9632
0.9565
571
nob
Norwegian Bokmål
0.9324
0.9073
0.9197
928
nst
Naga (Tangshang)
1.0000
0.9750
0.9873
40
nus
Nuer
0.9903
1.0000
0.9951
102
oci
Occitan
0.9631
0.9476
0.9553
248
orv
Old East Slavic
0.9538
0.9254
0.9394
67
oss
Ossetian
0.9818
0.9926
0.9872
271
ota
Ottoman Turkish
0.9204
0.9455
0.9327
110
pam
Kapampangan
0.9730
0.9600
0.9664
75
pcd
Picard
0.9254
0.9688
0.9466
64
pes
Persian
0.9846
0.9868
0.9857
454
pms
Piedmontese
0.9024
0.9487
0.9250
39
pol
Polish
0.9867
0.9885
0.9876
524
por
Portuguese
0.9595
0.9577
0.9586
544
prg
Old Prussian
0.9800
0.9423
0.9608
52
rhg
Rohingya
0.9835
0.9835
0.9835
182
rom
Romani
0.9302
0.8511
0.8889
47
ron
Romanian
0.9783
0.9762
0.9772
462
run
Kirundi
0.9871
0.9426
0.9644
244
rus
Russian
0.9561
0.9757
0.9658
536
sah
Yakut
0.9792
1.0000
0.9895
47
sat
Santali
0.9942
1.0000
0.9971
170
sdh
Southern Kurdish
0.8462
0.8627
0.8544
51
shi
Tashelhit
0.9706
0.8980
0.9329
147
slk
Slovak
0.9111
0.9318
0.9213
396
slv
Slovenian
0.7018
0.9302
0.8000
43
sma
Southern Sami
0.9500
0.9406
0.9453
101
sme
Northern Sami
1.0000
0.9843
0.9921
508
smj
Lule Sami
0.9840
0.9980
0.9909
493
smn
Inari Sami
0.9850
0.9949
0.9899
198
sms
Skolt Sami
0.9700
0.9848
0.9773
197
spa
Spanish
0.9613
0.9560
0.9586
545
sqi
Albanian
0.9603
0.9680
0.9641
125
srp
Serbian
0.8122
0.8106
0.8114
491
swc
Congo Swahili
0.8500
0.8367
0.8433
447
swe
Swedish
0.9759
0.9778
0.9768
992
swg
Swabian
0.9796
0.9320
0.9552
103
swh
Swahili
0.6650
0.7068
0.6853
191
tat
Tatar
0.9739
0.9816
0.9777
380
tgl
Tagalog
0.9709
0.9732
0.9721
411
tha
Thai
1.0000
1.0000
1.0000
220
thv
Tahaggart Tamahaq
0.6552
0.7600
0.7037
25
tig
Tigre
1.0000
1.0000
1.0000
181
tlh
Klingon
0.9977
0.9955
0.9966
440
tok
Toki Pona
1.0000
1.0000
1.0000
495
tpw
Old Tupi
0.8214
0.8846
0.8519
26
tuk
Turkmen
0.9779
0.9708
0.9744
274
tur
Turkish
0.9780
0.9604
0.9691
556
uig
Uyghur
0.9933
0.9900
0.9916
299
ukr
Ukrainian
0.9682
0.9700
0.9691
533
urd
Urdu
1.0000
0.9914
0.9957
116
uzb
Uzbek
0.8000
0.9756
0.8791
41
vie
Vietnamese
0.9977
0.9977
0.9977
426
vol
Volapük
0.9862
0.9817
0.9840
219
war
Waray
0.9208
0.9688
0.9442
96
wuu
Shanghainese
0.8037
0.9053
0.8515
190
xal
Kalmyk
0.9070
0.9512
0.9286
41
xmf
Mingrelian
0.6774
0.8400
0.7500
25
yid
Yiddish
0.9828
0.9942
0.9885
345
yue
Cantonese
0.8314
0.9688
0.8948
224
zgh
Standard Moroccan Tamazight
0.9873
0.9873
0.9873
158
zlm
Malay (Vernacular)
0.8488
0.8588
0.8538
85
zsm
Malay
0.7465
0.7544
0.7504
281
zza
Zaza
0.8824
0.9146
0.8982
82
Accuracy
0.9513
44049
Weighted avg
0.9529
0.9513
0.9518
44049
Macro avg
0.9275
0.9399
0.9327
44049
Citing & Authors
The model was trained by Javier de la Rosa. Data was prepared by Per Egil Kummervold and Javier de la Rosa. Documentation written by Javier de la Rosa.
Runs of NbAiLab nb-nordic-lid on huggingface.co
67
Total runs
0
24-hour runs
0
3-day runs
3
7-day runs
-4
30-day runs
More Information About nb-nordic-lid huggingface.co Model
nb-nordic-lid huggingface.co is an AI model on huggingface.co that provides nb-nordic-lid's model effect (), which can be used instantly with this NbAiLab nb-nordic-lid model. huggingface.co supports a free trial of the nb-nordic-lid model, and also provides paid use of the nb-nordic-lid. Support call nb-nordic-lid model through api, including Node.js, Python, http.
nb-nordic-lid huggingface.co is an online trial and call api platform, which integrates nb-nordic-lid's modeling effects, including api services, and provides a free online trial of nb-nordic-lid, you can try nb-nordic-lid online for free by clicking the link below.
NbAiLab nb-nordic-lid online free url in huggingface.co:
nb-nordic-lid is an open source model from GitHub that offers a free installation service, and any user can find nb-nordic-lid on GitHub to install. At the same time, huggingface.co provides the effect of nb-nordic-lid install, users can directly use nb-nordic-lid installed effect in huggingface.co for debugging and trial. It also supports api for free installation.