PhilSad / Claire-7b-0.1-instruct

Model Card for Model ID

(experimental, that's my first finetune)

This is a 4 bits PEFT QLORA fine tuning of Claire-7b-0.1 on 150 steps on a dataset adapted from tbboukhari/Alpaca_french_instruct.

It takes less than 8 Go VRAM

Try on Colab

Training script

TODO:

• [] Train on better dataset on more steps
• [] full precision

Model Details

Model Description

This is the model card of a 🤗 transformers model that has been pushed on the Hub. This model card has been automatically generated.

• Developed by: Philippe Saade
• Language(s) (NLP): French
• License: [More Information Needed]
• Finetuned from model: OpenLLM-France/Claire-7B-0.1

Model Sources [optional]

• Repository: https://github.com/PhilSad/claire-instruct
• Paper [optional]: [More Information Needed]
• Demo [optional]: [More Information Needed]

How to Get Started with the Model

Use the code below to get started with the model.

#!pip install transformers accelerate bitsandbytes

import transformers
import torch

model_name = "PhilSad/Claire-7b-0.1-instruct"

tokenizer = transformers.AutoTokenizer.from_pretrained("OpenLLM-France/Claire-7B-0.1")
model_instruct = transformers.AutoModelForCausalLM.from_pretrained(model_name,
    device_map="auto",
    torch_dtype=torch.bfloat16,
)

pipeline_instruct = transformers.pipeline("text-generation", model=model_instruct, tokenizer=tokenizer)
generation_kwargs = dict(
    num_return_sequences=1,                    # Number of variants to generate.
    return_full_text= False,                   # Do not include the prompt in the generated text.
    max_new_tokens=200,                        # Maximum length for the output text.
    do_sample=True, top_k=10, temperature=1.0, # Sampling parameters.
    pad_token_id=tokenizer.eos_token_id,       # Just to avoid a harmless warning.
)

prompt = """Ci-dessous se trouve une instruction qui décrit une tâche. Écrivez une réponse qui complète de manière appropriée la demande.

### Instruction :
Donne moi la recette pour faire un bon mojito

### Réponse :"""

prompt = """Ci-dessous se trouve une instruction qui décrit une tâche. Écrivez une réponse qui complète de manière appropriée la demande.

### Instruction :
Détermine le sentiment associé au texte parmis positif, négatif, neutre

### Entrée
J'apprécie les fruits au sirop

### Réponse :"""


completions = pipeline_instruct(prompt, **generation_kwargs)
for completion in completions:
    print(prompt + " […]" + completion['generated_text'])

Training Details

Training Procedure

I used this guide

Training script

Training Hyperparameters

• Training regime:

4 bits

per_device_train_batch_size = 4 #4
gradient_accumulation_steps = 4
optim = "paged_adamw_32bit"
learning_rate = 2e-4
max_grad_norm = 0.3
max_steps = 300 #100 #500
warmup_ratio = 0.03
lr_scheduler_type = "cosine"


lora_alpha = 32 #16
lora_dropout = 0.05 #0.1
lora_rank = 32 #64

peft_config = LoraConfig(
    lora_alpha=lora_alpha,
    lora_dropout=lora_dropout,
    r=lora_rank,
    bias="none",
    task_type="CAUSAL_LM",
    target_modules=[
        "query_key_value",
        "dense",
        "dense_h_to_4h",
        "dense_4h_to_h",
    ]
)