jnjj / Vvbvv

Sequentially Fine-Tuned Language Model: jnjj/xd_v1

Model Description

This repository hosts a large language model that is being sequentially fine-tuned using Low-Rank Adaptation (LoRA) on a diverse range of datasets sourced from the Hugging Face Hub. The core idea is to continuously adapt the model to new data, merging the LoRA adapter into the base weights after each successful training iteration on a dataset configuration. This process aims to create a model with broad knowledge and capabilities accumulated from various textual domains. Base Model: roneneldan/TinyStories-Instuct-1Layer-21M The model files (merged weights and tokenizer) are stored at the root of this repository and are updated periodically.

Training Methodology

• Sequential Fine-Tuning: The model starts from roneneldan/TinyStories-Instuct-1Layer-21M (or its last fine-tuned state from this repository) and is trained on one dataset configuration at a time.
• LoRA (Low-Rank Adaptation): PEFT library's LoRA is used for efficient fine-tuning. After training on a dataset configuration, the LoRA adapter ( current_training ) is merged into the base model.
• Diverse Datasets: The script iterates through datasets available on the Hugging Face Hub, attempting to train on each.
• Short Training Iterations: Each training run per dataset configuration is currently set to a small number of steps ( max_steps=1 ) to allow for rapid iteration across many datasets.

Training Progress

• Datasets Processed (Successfully trained on at least one config): 9
• Text Examples Streamed (Total): 54
• Tokens Processed (Total): 27648
• Last Successful Model Update: 2025-05-07 14:28:37 UTC

Evaluation Metrics

• Overall Perplexity (on a small fixed dataset): 143030.67

Generated Examples (Qualitative Assessment)

Story Continuation:

Input: Once upon a time, in a small village nestled by a sparkling river, there lived a curious cat named W... Output: � �EB Serpentmort cf Kaepernickま [* [*rue cf absorbed disobcorrectpelま Kaepernick MMO autopsy cfessim MMOdylmort Serpentessim MMOpelrue asymm fantasies flank Kaepernick encodedpelrue Legislation hateful MMO disobpelrueま [* cf abolitionrue Films

Simple Instruction:

Input: Explain in one sentence why trees are important for the environment.... Output: Reing confused pro wantsessionALaclicas ±lling. elusiveaj frontier." error mind F they during wrapper throat 2. more wrapper computer "ual co eadeer termsregeesö'tregeeslnity Dangerousherton suchmost

Creative Prompt:

Input: Describe a friendly robot that loves to bake cookies.... Output: and hpOL hperia hpow hp gamers hp hp Dangerous hp hp hp hp hp Dangerous hp hp hp hp hp hp gamers hp hp e gamers hp hp Dangerous hp hp hp hp hp hp co hp hp hp hp hp Dangerous hp hp hp hp hp

Question Answering (Basic):

Input: What is the main color of a ripe banana?... Output: extreme linguistic clocksι clocksι receipts undergoneι#$extreme linguistic Guys clocks disposedono Guys#$extreme diminish. Bowie CG uncertainty f foc philosophyextremeι Guysextreme bloom Guys undergone linguistic Guys clocks undergoneextreme bloom mortar undergoneιextreme undergoneextremeextreme mortarextreme clocks

How to Use

You can load the model and tokenizer for text generation using the transformers library:

from transformers import AutoModelForCausalLM, AutoTokenizer
model_id = "jnjj/xd_v1" # Or use local path if downloaded
tokenizer = AutoTokenizer.from_pretrained(model_id)
model = AutoModelForCausalLM.from_pretrained(model_id)
# Ensure model is on the correct device (e.g., 'cuda' or 'cpu')
# model.to('cuda') 
prompt = "Once upon a time,"
inputs = tokenizer(prompt, return_tensors="pt") # .to('cuda')
output_sequences = model.generate(**inputs, max_new_tokens=50, do_sample=True, top_p=0.9, temperature=0.7)
generated_text = tokenizer.decode(output_sequences[0], skip_special_tokens=True)
print(generated_text)

Limitations and Considerations

• The model is trained on a wide variety of data, some of which may be unfiltered or contain biases.
• Due to sequential fine-tuning with short iterations, the model's capabilities on any single task might be limited or vary over time.
• Performance depends heavily on the datasets encountered and the order of training.
• This is an experimental project.

Disclaimer

This model is a research artifact and may produce unintended, biased, or offensive content. Use with caution and critical thinking.