qingy2024 / GRMR-V3-L3B

GRMR-V3-L3B

GRMR-V3-L3B is a fine-tuned version of unsloth/Llama-3.2-3B specifically optimized for grammar correction tasks.

temperature = 0.7
frequency_penalty = 0.0
presence_penalty = 0.0
min_p = 0.01
top_p = 0.95
top_k = 40

Model description

GRMR-V3-L3B is a grammar correction model built on Meta's Llama 3.2 3B base model. It has been fine-tuned on a large dataset of grammar correction examples to help improve text quality by fixing grammatical errors, punctuation, spelling, and other language issues.

The model uses a specialized chat template that structures inputs as "text" and outputs as "corrected" to maintain a clear distinction between original and corrected content.

Model Details

Manual Testing Examples

Here are a few examples of grammar corrections this model can handle:

Training procedure

The model was fine-tuned using full parameter fine-tuning (not LoRA) on the GRMR-V4-60K dataset. The training was optimized using the Unsloth framework for efficient training of LLMs.

Training hyperparameters

• Batch size: 8
• Gradient accumulation steps: 2
• Learning rate: 5e-5
• Epochs: 1
• Optimizer: AdamW (8-bit)
• Weight decay: 0.01
• LR scheduler: Cosine
• Warmup steps: 180
• Max sequence length: 16,384
• Training precision: Mixed precision (BF16 where available, FP16 otherwise)

Intended uses & limitations

This model is designed for grammar correction tasks. It can be used to:

• Fix grammatical errors in written text
• Correct punctuation
• Address spelling mistakes
• Improve sentence structure and clarity

Limitations

• The model may struggle with highly technical or domain-specific content
• It may not fully understand context-dependent grammar rules in all cases
• Performance may vary for non-standard English or text with multiple errors

How to use

llama.cpp and projects based on it should be able to run this model like any others.

For pure transformers code, you can refer here:

from transformers import AutoModelForCausalLM, AutoTokenizer# Load model and tokenizer
model_name = "qingy2024/GRMR-V3-L3B"
tokenizer = AutoTokenizer.from_pretrained(model_name)
model = AutoModelForCausalLM.from_pretrained(model_name)# Text with grammar errors to correct
text_to_correct = "i am going to the store tommorow and buy some thing for dinner"# Format as messages
messages = [
    {"role": "user", "content": text_to_correct}
]# Apply the custom chat template
prompt = tokenizer.apply_chat_template(messages, tokenize=False, add_generation_prompt=True)# Tokenize and generate
inputs = tokenizer(prompt, return_tensors="pt").to(model.device)
outputs = model.generate(
    inputs["input_ids"],
    max_new_tokens=512,
    temperature=0.1, # NOTE: For best results, use the recommended temperature of 0.7
    do_sample=True
)
# Decode and print the corrected text
corrected_text = tokenizer.decode(outputs[0], skip_special_tokens=True)
print(corrected_text)

Using with the Hugging Face pipeline

from transformers import pipeline
pipe = pipeline(
    "text-generation",
    model="qingy2024/GRMR-V3-L3B",
    torch_dtype="auto",
    device_map="auto"
)
messages = [
    {"role": "user", "content": "i dont know weather to bring a umbrella today"}
]
result = pipe(
    messages,
    max_new_tokens=100,
    temperature=0.1, # NOTE: For best results, use the recommended temperature of 0.7
    do_sample=True,
    return_full_text=False
)[0]["generated_text"]
print(result)

Note: The Python examples above use temperature=0.1 for reproducibility in quick tests. For optimal grammar correction quality, please use the recommended sampler settings, especially temperature=0.7 .

Custom Chat Template

The model uses a custom chat template with special formatting for grammar correction:

• User inputs are formatted with <|start_header_id|>text<|end_header_id|> headers
• Model outputs are formatted with <|start_header_id|>corrected<|end_header_id|> headers
• Messages are separated by <|eot_id|> tokens
• The chat template should work without any extra tweaking in vLLM or llama.cpp.

Training Dataset

The model was fine-tuned on the qingy2024/grmr-v4-60k dataset, which contains 60,000 examples of original text and their grammatically corrected versions.

Bias, Risks, and Limitations

• The model may reflect biases present in the training data
• It may not perform equally well across different writing styles or domains
• The model might occasionally introduce errors or change the meaning of text
• It focuses on grammatical correctness rather than stylistic improvements

Contact

For questions or issues related to the model, please reach out via Hugging Face or by creating an issue in the repository.