Total runs:
6.5M
Run Growth:
5.2M
Growth Rate:
79.61%
Sponsored by
ThumbnailCreator.com
zai-org / LongAlign-6B-64k-base
Model's Last Updated: March 07 2024
text-generation
Introduction of LongAlign-6B-64k-base
Model Details of LongAlign-6B-64k-base
LongAlign-6B-64k-base
🤗 [LongAlign Dataset] • 💻 [Github Repo] • 📃 [LongAlign Paper]
LongAlign is the first full recipe for LLM alignment on long context. We propose the LongAlign-10k dataset, containing 10,000 long instruction data of 8k-64k in length. We investigate on trianing strategies, namely packing (with loss weighting) and sorted batching , which are all implemented in our code. For real-world long context evaluation, we introduce LongBench-Chat that evaluate the instruction-following capability on queries of 10k-100k length.
All Models
We open-sourced the following list of models:
| Model | Huggingface Repo | Description |
|---|---|---|
| LongAlign-6B-64k-base | 🤗 Huggingface Repo | ChatGLM3-6B with an extended 64k context window |
| LongAlign-6B-64k | 🤗 Huggingface Repo | Chat model by LongAlign training on LongAlign-6B-64k-base |
| LongAlign-7B-64k-base | 🤗 Huggingface Repo | Llama-2-7B with an extended 64k context window |
| LongAlign-7B-64k | 🤗 Huggingface Repo | Chat model by LongAlign training on LongAlign-7B-64k-base |
| LongAlign-13B-64k-base | 🤗 Huggingface Repo | Llama-2-13B with an extended 64k context window |
| LongAlign-13B-64k | 🤗 Huggingface Repo | Chat model by LongAlign training on LongAlign-13B-64k-base |
| ChatGLM3-6B-128k | 🤗 Huggingface Repo | ChatGLM3-6B with a 128k context window |
Model usage
Chat prompt template for LongAlign-6B-64k:
[Round 1]
问：Hi!
答：Hello! What can I assist you today?
[Round 2]
问：What should I do if I can't sleep at night?
答：
Chat prompt template for LongAlign-7B-64k and LongAlign-13B-64k:
[INST]Hi![/INST]Hello! What can I assist you today?
[INST]What should I do if I can't sleep at night?[/INST]
ChatGLM3-6B-128k uses the same prompt template as ChatGLM3-6B .
A simple demo for deployment of the model:
from transformers import AutoTokenizer, AutoModelForCausalLM
import torch
tokenizer = AutoTokenizer.from_pretrained("THUDM/LongAlign-6B-64k", trust_remote_code=True)
model = AutoModelForCausalLM.from_pretrained("THUDM/LongAlign-6B-64k", torch_dtype=torch.bfloat16, trust_remote_code=True, device_map="auto")
model = model.eval()
query = open("assets/paper.txt").read() + "\n\nPlease summarize the paper."
response, history = model.chat(tokenizer, query, history=[], max_new_tokens=512, temperature=1)
print(response)
Citation
If you find our work useful, please consider citing LongAlign:
Runs of zai-org LongAlign-6B-64k-base on huggingface.co
251
Total runs
0
24-hour runs
0
3-day runs
0
7-day runs
-12
30-day runs
More Information About LongAlign-6B-64k-base huggingface.co Model
Url of LongAlign-6B-64k-base
LongAlign-6B-64k-base huggingface.co Url
Provider of LongAlign-6B-64k-base huggingface.co
Other API from zai-org
Updated:September 07 2026
Total runs:
1.4M
Run Growth:
-447.0K
Growth Rate:
-31.14%
Total runs:
632.9K
Run Growth:
-299.3K
Growth Rate:
-47.29%
Total runs:
592.6K
Run Growth:
468.7K
Growth Rate:
82.17%
Total runs:
455.2K
Run Growth:
-861.7K
Growth Rate:
-189.30%
Total runs:
153.7K
Run Growth:
-23.4K
Growth Rate:
-15.22%
Total runs:
105.6K
Run Growth:
27.1K
Growth Rate:
25.71%
Total runs:
84.3K
Run Growth:
67.7K
Growth Rate:
80.33%
Total runs:
60.5K
Run Growth:
4.4K
Growth Rate:
7.30%
Total runs:
56.8K
Run Growth:
-11.8K
Growth Rate:
-17.43%
Total runs:
54.2K
Run Growth:
-31.7K
Growth Rate:
-56.77%
Total runs:
39.0K
Run Growth:
25.5K
Growth Rate:
65.50%
Total runs:
35.6K
Run Growth:
34.2K
Growth Rate:
96.04%
Total runs:
33.9K
Run Growth:
-42.2K
Growth Rate:
-100.95%
Total runs:
26.4K
Run Growth:
-237.5K
Growth Rate:
-899.48%
Total runs:
21.4K
Run Growth:
-251.3K
Growth Rate:
-1173.78%
Total runs:
18.4K
Run Growth:
-32.2K
Growth Rate:
-174.57%
Total runs:
16.2K
Run Growth:
-5.6K
Growth Rate:
-34.47%
Total runs:
13.6K
Run Growth:
-8.4K
Growth Rate:
-61.42%
Total runs:
11.8K
Run Growth:
-8.9K
Growth Rate:
-75.28%
Total runs:
11.2K
Run Growth:
-3.0K
Growth Rate:
-27.09%
Total runs:
11.0K
Run Growth:
-3.2K
Growth Rate:
-29.17%
Total runs:
7.7K
Run Growth:
-2.0K
Growth Rate:
-24.13%
Total runs:
6.6K
Run Growth:
1.1K
Growth Rate:
16.61%
Total runs:
6.2K
Run Growth:
-11.3K
Growth Rate:
-175.10%
Total runs:
5.7K
Run Growth:
-44.9K
Growth Rate:
-790.07%
Total runs:
4.8K
Run Growth:
578
Growth Rate:
13.35%
Total runs:
2.8K
Run Growth:
-5.7K
Growth Rate:
-202.37%
Total runs:
2.1K
Run Growth:
440
Growth Rate:
21.18%
Total runs:
2.0K
Run Growth:
-3.3K
Growth Rate:
-166.38%
Total runs:
1.8K
Run Growth:
-11.3K
Growth Rate:
-623.05%
Total runs:
1.7K
Run Growth:
-507
Growth Rate:
-28.53%
Total runs:
1.3K
Run Growth:
-386
Growth Rate:
-29.49%
Total runs:
1.3K
Run Growth:
-501
Growth Rate:
-33.36%
Total runs:
1.1K
Run Growth:
118
Growth Rate:
10.61%
Total runs:
1.0K
Run Growth:
627
Growth Rate:
60.29%
Total runs:
959
Run Growth:
-2.0K
Growth Rate:
-221.89%
Total runs:
889
Run Growth:
420
Growth Rate:
47.24%
Total runs:
750
Run Growth:
94
Growth Rate:
12.84%
Total runs:
732
Run Growth:
-120
Growth Rate:
-16.39%
Total runs:
710
Run Growth:
-240
Growth Rate:
-31.83%
Total runs:
709
Run Growth:
290
Growth Rate:
38.16%
Total runs:
704
Run Growth:
-533
Growth Rate:
-75.71%
Total runs:
670
Run Growth:
45
Growth Rate:
6.72%
Total runs:
576
Run Growth:
153
Growth Rate:
25.04%
Total runs:
557
Run Growth:
-1.3K
Growth Rate:
-215.67%
Total runs:
520
Run Growth:
-112
Growth Rate:
-21.58%
Total runs:
505
Run Growth:
-55
Growth Rate:
-10.89%
Total runs:
497
Run Growth:
226
Growth Rate:
45.02%
Total runs:
443
Run Growth:
-353
Growth Rate:
-79.68%
Total runs:
415
Run Growth:
-235
Growth Rate:
-55.56%
Total runs:
370
Run Growth:
-188
Growth Rate:
-49.60%
Total runs:
334
Run Growth:
92
Growth Rate:
27.79%
Total runs:
316
Run Growth:
20
Growth Rate:
6.37%
Total runs:
289
Run Growth:
-51
Growth Rate:
-17.65%
Total runs:
289
Run Growth:
-31
Growth Rate:
-10.95%
Total runs:
280
Run Growth:
-9
Growth Rate:
-3.21%
Total runs:
277
Run Growth:
-1
Growth Rate:
-0.36%
Total runs:
275
Run Growth:
2
Growth Rate:
0.72%
Total runs:
259
Run Growth:
-21
Growth Rate:
-33.33%
Total runs:
245
Run Growth:
35
Growth Rate:
14.58%
Total runs:
244
Run Growth:
-25
Growth Rate:
-10.29%
Total runs:
235
Run Growth:
25
Growth Rate:
10.78%
Total runs:
228
Run Growth:
17
Growth Rate:
7.52%