AaryanK / GLM-4.7-GGUF

Uploads in progress, it's a big model :)

GLM-4.7-GGUF

I am currently looking for open positions! 🤗 If you find this model useful or are looking for a talented AI/LLM Engineer, please reach out to me on LinkedIn: Aaryan Kapoor .

Description

This repository contains GGUF format model files for Zhipu AI's GLM-4.7 .

GLM-4.7 is a powerful open-weights model designed for complex reasoning, agentic coding, and tool use. It supports "Thinking" (Chain of Thought) natively within its chat template.

Performances on Benchmarks. More detailed comparisons of GLM-4.7 with other models GPT-5-High, GPT-5.1-High, Claude Sonnet 4.5, Gemini 3.0 Pro, DeepSeek-V3.2, Kimi K2 Thinking, on 17 benchmarks (including 8 reasoning, 5 coding, and 3 agents benchmarks) can be seen in the below table.

How to Run (llama.cpp)

Important: This model uses "Thinking" (Chain of Thought), which consumes significant context. You must increase the generation limit ( -n ) and specify stop tokens to prevent infinite loops.

1. CLI Inference (Interactive Chat)

./llama-cli -m GLM-4.7.Q4_K_M.gguf \
  -n 2048 \                  # Allow enough tokens for "Thinking"
  -c 8192 \                  # Adjust context based on VRAM
  --temp 0.7 \               # Recommended for reasoning
  --top-p 0.9 \
  -ngl 99 \                  # Offload layers to GPU (Reduce if OOM)
  -r "<|user|>,<|observation|>" \  # CRITICAL: Prevents infinite generation loops
  -cnv \                     # Enable Conversation Mode
  -p "Hello"

Note: If you want to see the internal "Thinking" process (the text between <think> tags), add the --special flag to the command.

2. Server Mode (API)

Running a persistent server is recommended for this size model to avoid reloading times.

./llama-server -m GLM-4.7.Q4_K_M.gguf \
  --port 8080 \
  -ngl 99 \
  -c 8192 \
  -n 2048 \
  --alias glm4

API Request Example (JSON):

When using the API, ensure you include the stop tokens in your payload:

{
  "model": "glm4",
  "messages": [
    { "role": "user", "content": "Explain quantum computing." }
  ],
  "stop": ["<|user|>", "<|observation|>"],
  "max_tokens": 2048
}

Hardware Requirements

• Full GPU Offloading ( -ngl 99 ): Requires ~130GB VRAM for Q4_K_M (e.g., 2x A100 80GB or Mac Studio Ultra).

• Split Offloading: For single A100 (80GB) cards, use Q2_K or IQ2_XXS and set -ngl 40 (adjust based on available VRAM) to split the model between GPU and System RAM. Default Settings (Most Tasks)

• temperature: 1.0

• top-p: 0.95

• max new tokens: 131072

For multi-turn agentic tasks (τ²-Bench and Terminal Bench 2), please turn on Preserved Thinking mode .

CLI Example

./llama-cli -m GLM-4.7.Q4_K_M.gguf \
  -c 8192 \
  --temp 1.0 \
  --top-p 0.95 \
  -p "[gMASK]<sop><|system|>\nYou are a helpful assistant.<|user|>\nWrite a Python script to calculate Fibonacci numbers.<|assistant|>\n<think>" \
  -cnv