GLM-4-32B-Base-32K is an enhanced version of
THUDM's GLM-4-32B-Base-0414
, specifically engineered to offer robust performance over an extended context window. While the original model's capabilities degraded after 8,192 tokens, this version maintains strong performance up to a 32,000-token context, making it ideal for tasks requiring long-context understanding and processing.
This model was developed as a proof-of-concept to validate that a merging-centric approach to context extension can be successfully applied to larger-scale models. The techniques employed resulted in an approximate 5% overall improvement on standard base model benchmarks while significantly improving 32k recall.
More details can be found in our blog post
here
where we applied this work to our upcoming AFM 4.5B
The primary improvement in this model is its enhanced long-context capability. The following methods were used to achieve this:
Targeted Long-Context Training: The model underwent continued pretraining on sequences up to its full 32,000 token context length.
Iterative Merging: Various model checkpoints were iteratively merged to combine the benefits of different training runs, enhancing both long-context and short-context performance.
Short-Context Distillation: Knowledge from the original high-performing short-context model was distilled into the long-context-trained model to recover and retain its initial capabilities on shorter tasks.
As a result, where the original model's performance on the Needle in a Haystack (NIAH) benchmark would decline after 8,000 tokens, this extended version maintains reliable performance across the entire 32,000 token context window.
Benchmarks
Benchmark
GLM-4-32B-Base-0414
GLM-4-32B-Base-32K
arc_challenge
59.39%
64.93%
arc_easy
85.44%
87.88%
hellaswag
64.75%
65.40%
mmlu
77.05%
77.87%
piqa
81.61%
83.19%
truthfulqa_mc2
49.27%
50.07%
winogrande
78.69%
80.03%
NIAH Benchmark Results Comparison
Model
Task
4,096
8,192
16,384
24,576
32,768
GLM-4-32B-Base-0414
niah_single_1
100.0%
100.0%
77.0%
5.2%
1.2%
niah_single_2
100.0%
100.0%
73.4%
2.6%
0.0%
niah_single_3
100.0%
99.8%
48.0%
1.4%
0.0%
GLM-4-32B-Base-32k
niah_single_1
100.0%
100.0%
100.0%
99.2%
99.6%
niah_single_2
100.0%
100.0%
99.2%
80.2%
68.8%
niah_single_3
100.0%
99.6%
95.6%
86.6%
61.0%
NIAH Averages
Model
4,096
8,192
16,384
24,576
32,768
GLM-4-32B-Base-0414
100.0%
99.9%
66.1%
3.1%
0.4%
GLM-4-32B-Base-32k
100.0%
99.9%
98.3%
88.7%
76.5%
Use Cases
This model serves as a new base for continued training at 32K context
License
GLM-4-32B-Base-32K (32B)
is released under the
MIT
license following with the original model's license.
If you have questions or would like to share your experiences using GLM-4-32B-Base-32K (32B), please connect with us on social media. We’re excited to see what you build—and how this model helps you innovate!
Runs of arcee-ai GLM-4-32B-Base-32K on huggingface.co
118
Total runs
0
24-hour runs
0
3-day runs
19
7-day runs
87
30-day runs
More Information About GLM-4-32B-Base-32K huggingface.co Model
GLM-4-32B-Base-32K huggingface.co is an AI model on huggingface.co that provides GLM-4-32B-Base-32K's model effect (), which can be used instantly with this arcee-ai GLM-4-32B-Base-32K model. huggingface.co supports a free trial of the GLM-4-32B-Base-32K model, and also provides paid use of the GLM-4-32B-Base-32K. Support call GLM-4-32B-Base-32K model through api, including Node.js, Python, http.
GLM-4-32B-Base-32K huggingface.co is an online trial and call api platform, which integrates GLM-4-32B-Base-32K's modeling effects, including api services, and provides a free online trial of GLM-4-32B-Base-32K, you can try GLM-4-32B-Base-32K online for free by clicking the link below.
arcee-ai GLM-4-32B-Base-32K online free url in huggingface.co:
GLM-4-32B-Base-32K is an open source model from GitHub that offers a free installation service, and any user can find GLM-4-32B-Base-32K on GitHub to install. At the same time, huggingface.co provides the effect of GLM-4-32B-Base-32K install, users can directly use GLM-4-32B-Base-32K installed effect in huggingface.co for debugging and trial. It also supports api for free installation.