boda / jais-13b-poem-generation

Model Card for Model ID

Fine-tuned using QLoRA for poem generation task.

Model Description

We utilize Ashaar dataset and fine-tune the model to generate poems.

The input to the model is structred as follows:

'''

### Instruction: Generate a poem based on the following title, and the given era:

### Input: {Title of a poem + poet era}

### Response: {Poem verses}

'''

• Developed by: Abdelrahman ’Boda’ Sadallah, Anastasiia Demidova, Daria Kotova
• Model type: Causal LM
• Language(s) (NLP): Arabic
• Finetuned from model [optional]: core42/jais-13b

Model Sources

• Repository: https://github.com/BodaSadalla98/llm-optimized-fintuning

Uses

The model is the result of our AI project. If you intend to use it, please, refer to the repo.

Recommendations

For improving stories generation, you can play parameters: temeperature, top_p/top_k, repetition_penalty, etc.

Training Details

Training Data

Link to the dataset on huggungface: https://huggingface.co/datasets/arbml/ashaar .

Evaluation

Testing Data, Factors & Metrics

Test split of the same dataset.

Metrics

We are using perplexity and BERTScore.

Results

Perplexity: 48.3125

BERTScore: 59.33

Training procedure

The following bitsandbytes quantization config was used during training:

• quant_method: bitsandbytes
• load_in_8bit: False
• load_in_4bit: True
• llm_int8_threshold: 6.0
• llm_int8_skip_modules: None
• llm_int8_enable_fp32_cpu_offload: False
• llm_int8_has_fp16_weight: False
• bnb_4bit_quant_type: fp4
• bnb_4bit_use_double_quant: False
• bnb_4bit_compute_dtype: float32

Framework versions

• PEFT 0.6.0.dev0