duyntnet / Qwen2.5-Coder-7B-Instruct-imatrix-GGUF

Quantizations of https://huggingface.co/Qwen/Qwen2.5-Coder-7B-Instruct

Inference Clients/UIs

• llama.cpp
• KoboldCPP
• text-generation-webui
• ollama

From original readme

Qwen2.5-Coder is the latest series of Code-Specific Qwen large language models (formerly known as CodeQwen). For Qwen2.5-Coder, we release three base language models and instruction-tuned language models, 1.5, 7 and 32 (coming soon) billion parameters. Qwen2.5-Coder brings the following improvements upon CodeQwen1.5:

• Significantly improvements in code generation , code reasoning and code fixing . Base on the strong Qwen2.5, we scale up the training tokens into 5.5 trillion including source code, text-code grounding, Synthetic data, etc.
• A more comprehensive foundation for real-world applications such as Code Agents . Not only enhancing coding capabilities but also maintaining its strengths in mathematics and general competencies.
• Long-context Support up to 128K tokens.

This repo contains the instruction-tuned 7B Qwen2.5-Coder model , which has the following features:

• Type: Causal Language Models
• Training Stage: Pretraining & Post-training
• Architecture: transformers with RoPE, SwiGLU, RMSNorm, and Attention QKV bias
• Number of Parameters: 7.61B
• Number of Paramaters (Non-Embedding): 6.53B
• Number of Layers: 28
• Number of Attention Heads (GQA): 28 for Q and 4 for KV
• Context Length: Full 131,072 tokens
  • Please refer to this section for detailed instructions on how to deploy Qwen2.5 for handling long texts.

For more details, please refer to our blog , GitHub , Documentation , Arxiv .

Requirements

The code of Qwen2.5-Coder has been in the latest Hugging face transformers and we advise you to use the latest version of transformers .

With transformers<4.37.0 , you will encounter the following error:

KeyError: 'qwen2'

Quickstart

Here provides a code snippet with apply_chat_template to show you how to load the tokenizer and model and how to generate contents.

from transformers import AutoModelForCausalLM, AutoTokenizer

model_name = "Qwen/Qwen2.5-Coder-7B-Instruct"

model = AutoModelForCausalLM.from_pretrained(
    model_name,
    torch_dtype="auto",
    device_map="auto"
)
tokenizer = AutoTokenizer.from_pretrained(model_name)

prompt = "write a quick sort algorithm."
messages = [
    {"role": "system", "content": "You are Qwen, created by Alibaba Cloud. You are a helpful assistant."},
    {"role": "user", "content": prompt}
]
text = tokenizer.apply_chat_template(
    messages,
    tokenize=False,
    add_generation_prompt=True
)
model_inputs = tokenizer([text], return_tensors="pt").to(model.device)

generated_ids = model.generate(
    **model_inputs,
    max_new_tokens=512
)
generated_ids = [
    output_ids[len(input_ids):] for input_ids, output_ids in zip(model_inputs.input_ids, generated_ids)
]

response = tokenizer.batch_decode(generated_ids, skip_special_tokens=True)[0]

Processing Long Texts

The current config.json is set for context length up to 32,768 tokens. To handle extensive inputs exceeding 32,768 tokens, we utilize YaRN , a technique for enhancing model length extrapolation, ensuring optimal performance on lengthy texts.

For supported frameworks, you could add the following to config.json to enable YaRN:

{
  ...,
  "rope_scaling": {
    "factor": 4.0,
    "original_max_position_embeddings": 32768,
    "type": "yarn"
  }
}