This model is designed to rate the quality of Norwegian training corpora based on
educational content
. It predicts a continuous score (float from 0 to 5), assessing the educational value of Norwegian texts. The model is inspired by the classifiers used in the FineWeb project and is trained mainly on Norwegian content.
The dataset used for training is derived from
GlotCC
and has been annotated using Gemini 1.5 Flash.
Purpose
The performance of large language models (LLMs) heavily depends on the quality and size of their pretraining datasets. This regressor aims to assess and enhance the educational value of Norwegian textual data, contributing to better-performing Norwegian LLMs.
For convenience we also provide the
run_regressor_bert.py
script. This is also based on
run_edu_bert.py
from Cosmopedia. You can modify this script to annotate HuggingFace datasets directly. Cosmopedia also provides slurm-scripts here. We have not included these since we have had the opportunity to test them.
Training and Evaluation Procedure
The following command where used for training. Please note that
train_regressor_bert.py
has a few minor changes to the original
train_edu_bert.py
:
nb-education-quality-evaluator huggingface.co is an AI model on huggingface.co that provides nb-education-quality-evaluator's model effect (), which can be used instantly with this NbAiLab nb-education-quality-evaluator model. huggingface.co supports a free trial of the nb-education-quality-evaluator model, and also provides paid use of the nb-education-quality-evaluator. Support call nb-education-quality-evaluator model through api, including Node.js, Python, http.
nb-education-quality-evaluator huggingface.co is an online trial and call api platform, which integrates nb-education-quality-evaluator's modeling effects, including api services, and provides a free online trial of nb-education-quality-evaluator, you can try nb-education-quality-evaluator online for free by clicking the link below.
NbAiLab nb-education-quality-evaluator online free url in huggingface.co:
nb-education-quality-evaluator is an open source model from GitHub that offers a free installation service, and any user can find nb-education-quality-evaluator on GitHub to install. At the same time, huggingface.co provides the effect of nb-education-quality-evaluator install, users can directly use nb-education-quality-evaluator installed effect in huggingface.co for debugging and trial. It also supports api for free installation.
nb-education-quality-evaluator install url in huggingface.co: