qingy2024 / GRMR-V3-Q4B

GRMR-V3-Q4B

GRMR-V3-Q4B is a fine-tuned version of unsloth/Qwen3-4B-Base specifically optimized for grammar correction tasks.

temperature = 0.7
frequency_penalty = 0.0
presence_penalty = 0.0
min_p = 0.01
top_p = 0.95
top_k = 40

Model description

GRMR-V3-Q4B is a grammar correction model built on Qwen3 4B base model. It has been fine-tuned on a large dataset of grammar correction examples to help improve text quality by fixing grammatical errors, punctuation, spelling, and other language issues.

The model uses a specialized chat template that structures inputs as "text" and outputs as "corrected" to maintain a clear distinction between original and corrected content.

Model Details

Manual Testing Examples

Here are a few examples of grammar corrections this model can handle:

Training procedure

The model was fine-tuned using full parameter fine-tuning (not LoRA) on the GRMR-V4-60K dataset. The training was optimized using the Unsloth framework for efficient training of LLMs.

Training hyperparameters

• Batch size: 8
• Gradient accumulation steps: 2
• Learning rate: 5e-5
• Epochs: 1
• Optimizer: AdamW (8-bit)
• Weight decay: 0.01
• LR scheduler: Cosine
• Warmup steps: 180
• Max sequence length: 16,384
• Training precision: Mixed precision (BF16 where available, FP16 otherwise)

Intended uses & limitations

This model is designed for grammar correction tasks. It can be used to:

• Fix grammatical errors in written text
• Correct punctuation
• Address spelling mistakes
• Improve sentence structure and clarity

Limitations

• The model may struggle with highly technical or domain-specific content
• It may not fully understand context-dependent grammar rules in all cases
• Performance may vary for non-standard English or text with multiple errors

How to use

Projects based on Hugging Face transformers should be able to run this model easily.

from transformers import AutoModelForCausalLM, AutoTokenizer
# Load model and tokenizer
model_name = "qingy2024/GRMR-V3-Q4B"
tokenizer = AutoTokenizer.from_pretrained(model_name)
model = AutoModelForCausalLM.from_pretrained(model_name)
# Text with grammar errors to correct
text_to_correct = "i am going to the store tommorow and buy some thing for dinner"
# Format as messages
messages = [
    {"role": "user", "content": text_to_correct}
]
# Apply the custom chat template
prompt = tokenizer.apply_chat_template(messages, tokenize=False, add_generation_prompt=True)
# Tokenize and generate
inputs = tokenizer(prompt, return_tensors="pt").to(model.device)
outputs = model.generate(
    inputs["input_ids"],
    max_new_tokens=512,
    temperature=0.1,
    do_sample=True
)
# Decode and print the corrected text
corrected_text = tokenizer.decode(outputs[0], skip_special_tokens=True)
print(corrected_text)

Using with the Hugging Face pipeline

from transformers import pipeline
pipe = pipeline(
    "text-generation",
    model="qingy2024/GRMR-V3-Q4B",
    torch_dtype="auto",
    device_map="auto"
)
messages = [
    {"role": "user", "content": "i dont know weather to bring a umbrella today"}
]
result = pipe(
    messages,
    max_new_tokens=100,
    temperature=0.1,
    do_sample=True,
    return_full_text=False
)[0]["generated_text"]
print(result)

Custom Chat Template

The model uses a custom chat template with special formatting for grammar correction:

• User inputs are formatted with <|text_start|> and <|text_end|> tags
• Model outputs are formatted with <|corrected_start|> and <|corrected_end|> tags

The complete chat template is:

{%- for message in messages %}
{%- if message.role == "user" %}
{{- '<|text_start|>\n' + message.content + '<|text_end|>\n' }}
{%- elif message.role == "assistant" %}
{{- '<|corrected_start|>\n' + message.content + '<|corrected_end|>\n' }}
{%- else %}
{{- raise('Unknown role: ' + message.role) }}
{%- endif %}
{%- endfor %}
{%- if add_generation_prompt %}
{{- '<|corrected_start|>\n' }}
{%- endif %}

Training Dataset

The model was fine-tuned on the qingy2024/grmr-v4-60k dataset, which contains 60,000 examples of original text and their grammatically corrected versions.

Bias, Risks, and Limitations

• The model may reflect biases present in the training data
• It may not perform equally well across different writing styles or domains
• The model might occasionally introduce errors or change the meaning of text
• It focuses on grammatical correctness rather than stylistic improvements

Citations

@misc{qwen3technicalreport,
      title={Qwen3 Technical Report}, 
      author={Qwen Team},
      year={2025},
      eprint={2505.09388},
      archivePrefix={arXiv},
      primaryClass={cs.CL},
      url={https://arxiv.org/abs/2505.09388}, 
}

Contact

For questions or issues related to the model, please reach out via Hugging Face or by creating an issue in the repository.