Total runs:
140.4K
Run Growth:
-129.9K
Growth Rate:
-92.57%
Sponsored by
APIMart
bigcode / astraios-1b-lora
Model's Last Updated: January 02 2024
Introduction of astraios-1b-lora
Model Details of astraios-1b-lora
Astraios: Parameter-Efficient Instruction Tuning Code Large Language Models
Table of Contents
Model Summary
Astraios-1B-LoRA is an instruction tuned model with 15.5B parameters created by finetuning StarCoderBase on CommitPackFT & OASST as described in the Astraios paper.
- Repository: bigcode-project/astraios
- Paper: Astraios: Parameter-Efficient Instruction Tuning Code Large Language Models
- Languages: 80+ Programming languages
-
✨Astraios:
Data CommitPackFT+OASST Filtered version of CommitPack and OASST for high-quality commit messages that resemble instructions Model Astraios-1B Collection of StarCoderBase-1B models instruction tuned on CommitPackFT + OASST with different tuning methods Astraios-3B Collection of StarCoderBase-3B (3B parameters) models instruction tuned on CommitPackFT + OASST with different tuning methods Astraios-7B Collection of StarCoderBase-7B (7B parameters) models instruction tuned on CommitPackFT + OASST with different tuning methods Astraios-16B Collection of StarCoderBase-16B (16B parameters) models instruction tuned on CommitPackFT + OASST with different tuning methods Evaluation BigCloneBench Dataset for clone detection; We use 2,000 samples for evaluation Devign Dataset for defect detection; We use 2,000 samples for evaluation HumanEvalPack Extension of OpenAI's HumanEval to cover 3 scenarios across 6 languages ReCode Dataset for the robustness of code generation, covering 4 variants Asleep At The Keyboard Datasets for security of code generation; We use DoW for evaluation
Use
Intended use
The model follows instructions provided in the input. You should always preface your input with "Question: " and finish it with "Answer:", for example: "Question: Please write a function in Python that performs bubble sort.
Answer:"
Feel free to share your generations in the Community tab!
Generation
# pip install -q transformers
# pip install -e git+https://github.com/bigcode-project/astraios#subdirectory=peft
from peft import PeftModel
from transformers import AutoModelForCausalLM, AutoTokenizer
peft_checkpoint = "bigcode/astraios-1b-lora"
checkpoint = "bigcode/starcoderbase-1b"
model = AutoModelForCausalLM.from_pretrained(checkpoint)
model = PeftModel.from_pretrained(model, peft_checkpoint)
device = "cuda" # for GPU usage or "cpu" for CPU usage
tokenizer = AutoTokenizer.from_pretrained(checkpoint)
model = AutoModelForCausalLM.from_pretrained(checkpoint).to(device)
inputs = tokenizer.encode("Question: Please write a function in Python that performs bubble sort.
Answer:", return_tensors="pt").to(device)
outputs = model.generate(inputs)
print(tokenizer.decode(outputs[0]))
Training
Model
- Architecture: GPT-2 model with multi-query attention and Fill-in-the-Middle objective
- Steps: 250k pretraining & 200 instruction tuning
- Precision: fp32
Hardware
-
Pretraining:
- GPUs: 512 Tesla A100
- Training time: 24 days
-
Instruction tuning:
- GPUs: 8 Tesla A100
Software
- Orchestration: Megatron-LM/Transformers
- Neural networks: PyTorch
Citation
Runs of bigcode astraios-1b-lora on huggingface.co
8
Total runs
0
24-hour runs
-1
3-day runs
-1
7-day runs
3
30-day runs
More Information About astraios-1b-lora huggingface.co Model
Url of astraios-1b-lora
astraios-1b-lora huggingface.co Url
Provider of astraios-1b-lora huggingface.co
Other API from bigcode
Updated:June 01 2023
Total runs:
33.9K
Run Growth:
-49.1K
Growth Rate:
-144.89%
Total runs:
9.1K
Run Growth:
-38.4K
Growth Rate:
-422.60%
Total runs:
8.1K
Run Growth:
4.7K
Growth Rate:
58.90%
Total runs:
1.6K
Run Growth:
540
Growth Rate:
32.75%
Total runs:
1.4K
Run Growth:
-618
Growth Rate:
-43.77%
Total runs:
271
Run Growth:
43
Growth Rate:
15.87%
Total runs:
271
Run Growth:
69
Growth Rate:
25.46%
Total runs:
262
Run Growth:
67
Growth Rate:
25.57%
Total runs:
164
Run Growth:
-34
Growth Rate:
-20.73%
Total runs:
25
Run Growth:
15
Growth Rate:
60.00%
Total runs:
20
Run Growth:
8
Growth Rate:
40.00%
Total runs:
18
Run Growth:
11
Growth Rate:
61.11%
Total runs:
18
Run Growth:
7
Growth Rate:
38.89%
Total runs:
16
Run Growth:
5
Growth Rate:
31.25%
Total runs:
16
Run Growth:
9
Growth Rate:
56.25%
Total runs:
15
Run Growth:
1
Growth Rate:
6.67%
Total runs:
14
Run Growth:
7
Growth Rate:
50.00%
Total runs:
13
Run Growth:
-2
Growth Rate:
-15.38%
Total runs:
12
Run Growth:
-1
Growth Rate:
-8.33%
Total runs:
11
Run Growth:
4
Growth Rate:
36.36%
Total runs:
11
Run Growth:
5
Growth Rate:
45.45%
Total runs:
0
Run Growth:
0
Growth Rate:
0.00%
Total runs:
0
Run Growth:
0
Growth Rate:
0.00%
Total runs:
0
Run Growth:
0
Growth Rate:
0.00%
Total runs:
0
Run Growth:
0
Growth Rate:
0.00%
Total runs:
0
Run Growth:
0
Growth Rate:
0.00%