Faradaylab / Aria-40B

Aria 40B is based on Open-Assistant Falcon 40B SFT OASST-TOP1 Model.

This model is a fine-tuning of TII's Falcon 40B LLM. It was trained with top-1 (high-quality) demonstrations of the OASST data set (exported on May 6, 2023) with an effective batch size of 144 for ~7.5 epochs with LIMA style dropout (p=0.3) and a context-length of 2048 tokens. We focus on the human preference eval on this model as the most valuable point. We believe that pure technical benchmarks are not important as Human eval,which are our end users.

The finetuned version of Falcon we used as based model has been finetuned on Open assistant dataset and gives better results on human eval than some famous close models as you can see on this article and eval by a third party: https://medium.com/@geronimo7/open-source-chatbots-in-the-wild-9a44d7a41a48

We are currently working on Rope Scaling for context lenght and Chain of Toughts integration,this model card will be updated soon. Stay tuned.

LMYS Eval for Aria 40B base model (Falcon 40B OA) ( https://huggingface.co/OpenAssistant/falcon-40b-sft-top1-560 )

Model Details

• Finetuned from: tiiuae/falcon-40b
• Model type: Causal decoder-only transformer language model
• Language: English, German, Spanish, French (and limited capabilities in Italian, Portuguese, Polish, Dutch, Romanian, Czech, Swedish);
• Demo: Continuations for 250 random prompts
• Eval results: ilm-eval
• Weights & Biases : Training log (Checkpoint: 560 steps)
• License: Apache 2.0
• Contact: Faraday Discord or [email protected]

Prompting

Two special tokens are used to mark the beginning of user and assistant turns: <|prompter|> and <|assistant|> . Each turn ends with a <|endoftext|> token.

Input prompt example:

<|prompter|>What is a meme, and what's the history behind this word?<|endoftext|><|assistant|>

The input ends with the <|assistant|> token to signal that the model should start generating the assistant reply.

Configuration Details

Model:

falcon-40b:
  dtype: bf16
  log_dir: "falcon_log_40b"
  learning_rate: 5e-6
  model_name: "tiiuae/falcon-40b"
  deepspeed_config: configs/zero3_config_falcon.json
  output_dir: falcon
  weight_decay: 0.0
  max_length: 2048
  warmup_steps: 20
  gradient_checkpointing: true
  gradient_accumulation_steps: 1
  per_device_train_batch_size: 18
  per_device_eval_batch_size: 10
  eval_steps: 80
  save_steps: 80
  num_train_epochs: 8
  save_total_limit: 4
  use_flash_attention: false
  residual_dropout: 0.3
  residual_dropout_lima: true
  sort_by_length: false
  save_strategy: steps

Dataset:

oasst-top1:
  datasets:
    - oasst_export:
        lang: "bg,ca,cs,da,de,en,es,fr,hr,hu,it,nl,pl,pt,ro,ru,sl,sr,sv,uk" # sft-8.0
        input_file_path: 2023-05-06_OASST_labels.jsonl.gz
        val_split: 0.05
        top_k: 1