Introduction of santacoder-finetuned-the-stack-cobol
Model Details of santacoder-finetuned-the-stack-cobol
santacoder-finetuned-the-stack-cobol
This model is a fine-tuned version of
bigcode/santacoder
on an The Stack
cobol
dataset.
It achieves the following results on the evaluation set:
Loss: 0.7161
Model description
The
SantaCoder
models are a series of 1.1B parameter models trained on the Python, Java, and JavaScript subset of
The Stack (v1.1)
(which excluded opt-out requests).
The main model uses
Multi Query Attention
, was trained using near-deduplication and comment-to-code ratio as filtering criteria and using the
Fill-in-the-Middle objective
.
In addition, there are several models that were trained on datasets with different filter parameters and with architecture and objective variations.
Intended uses & limitations
The predominant language in source is English although other languages are also present. As such the model is capable to generate code snippets provided some context but the generated code is not guaranteed to work as intended. It can be inefficient, contain bugs or exploits.
Training and evaluation data
The Stack contains over 6TB of permissively-licensed source code files covering 358 programming languages. The dataset was created as part of the
BigCode Project
, an open scientific collaboration working on the responsible development of Large Language Models for Code (Code LLMs). The Stack serves as a pre-training dataset for Code LLMs, i.e., code-generating AI systems which enable the synthesis of programs from natural language descriptions as well as other from code snippets.
This is the near-deduplicated version with 3TB data.
Training procedure
Training hyperparameters
The following hyperparameters were used during training:
learning_rate: 5e-05
train_batch_size: 8
eval_batch_size: 4
seed: 42
gradient_accumulation_steps: 8
total_train_batch_size: 64
optimizer: Adam with betas=(0.9,0.999) and epsilon=1e-08
lr_scheduler_type: cosine
lr_scheduler_warmup_steps: 100
training_steps: 1000
Training results
Training Loss
Epoch
Step
Validation Loss
1.3911
0.1
100
1.1141
0.9478
0.2
200
0.9735
0.784
0.3
300
0.8497
0.4702
0.4
400
0.7686
0.6133
0.5
500
0.7375
0.5396
0.6
600
0.7265
0.3937
0.7
700
0.6952
0.5691
0.8
800
0.7059
0.6366
0.9
900
0.7069
0.3661
1.0
1000
0.7161
Framework versions
Transformers 4.26.0
Pytorch 1.13.1+cu116
Datasets 2.9.0
Tokenizers 0.13.2
Runs of muhtasham santacoder-finetuned-the-stack-cobol on huggingface.co
43
Total runs
1
24-hour runs
5
3-day runs
12
7-day runs
35
30-day runs
More Information About santacoder-finetuned-the-stack-cobol huggingface.co Model
More santacoder-finetuned-the-stack-cobol license Visit here:
santacoder-finetuned-the-stack-cobol huggingface.co is an AI model on huggingface.co that provides santacoder-finetuned-the-stack-cobol's model effect (), which can be used instantly with this muhtasham santacoder-finetuned-the-stack-cobol model. huggingface.co supports a free trial of the santacoder-finetuned-the-stack-cobol model, and also provides paid use of the santacoder-finetuned-the-stack-cobol. Support call santacoder-finetuned-the-stack-cobol model through api, including Node.js, Python, http.
santacoder-finetuned-the-stack-cobol huggingface.co is an online trial and call api platform, which integrates santacoder-finetuned-the-stack-cobol's modeling effects, including api services, and provides a free online trial of santacoder-finetuned-the-stack-cobol, you can try santacoder-finetuned-the-stack-cobol online for free by clicking the link below.
muhtasham santacoder-finetuned-the-stack-cobol online free url in huggingface.co:
santacoder-finetuned-the-stack-cobol is an open source model from GitHub that offers a free installation service, and any user can find santacoder-finetuned-the-stack-cobol on GitHub to install. At the same time, huggingface.co provides the effect of santacoder-finetuned-the-stack-cobol install, users can directly use santacoder-finetuned-the-stack-cobol installed effect in huggingface.co for debugging and trial. It also supports api for free installation.
santacoder-finetuned-the-stack-cobol install url in huggingface.co: