Sponsored by
APIMart
zjunlp / LightThinker-Qwen
Model's Last Updated: April 12 2025
Introduction of LightThinker-Qwen
Model Details of LightThinker-Qwen
Table of Contents
- 👀 Overview
- 🔧 Installation
- 🏃 Quick Start
- 🎁 Acknowledgement
- 🚩 Citation
👀Overview
LLMs have shown remarkable performance in complex reasoning tasks, but their efficiency is hindered by the substantial memory and computational costs associated with generating lengthy tokens. In this paper, we propose LightThinker, a novel method that enables LLMs to dynamically compress intermediate thoughts during reasoning. Inspired by human cognitive processes, LightThinker compresses verbose thought steps into compact representations and discards the original reasoning chains, thereby significantly reducing the number of tokens stored in the context window. This is achieved by training the model on when and how to perform compression through data construction, mapping hidden states to condensed gist tokens, and creating specialized attention masks.
🔧Installation
git clone https://github.com/zjunlp/LightThinker
cd LightThinker
conda create -n lightthinker python=3.9 -y
conda activate lightthinker
pip install -r requirements.txt
cd data && unzip data.zip && cd ..
🏃Quick Start
First, we train the model to learn when to compress and how to compress (step 1). Then, we perform inference on the test set to obtain output results (step 2). Finally, we evaluate the output results (step 3).
Step 1. Training
To execute the training, run the following command:
bash train.sh
Currently, the script's parameters are set to run on a machine with 4 A800 GPUs. If you encounter OOM (Out Of Memory) issues, please reduce the
micro_batch_size
and
max_length
. For other parameters in the script, please refer to the
documentation
.
Step 2. Inference
Inference with a downloaded model
If you are downloading a trained model from Huggingface, please set the
model_path
parameter in
inference.sh
to the absolute path of the model. The values of other parameters
ckpt
and
model_tag
will be ignored.
To execute the inference, run the following command:
bash inference.sh
Here, you need to modify the script file's
model_tag
,
model_short_tag
,
ckpt
,
output_tag
, and
split_size
. For details regarding the script's parameters, please refer to the
documentation
.
Step 3. Evaluation
If this is your first time conducting an evaluation, please execute the following code first:
python evaluation/init.py
To execute the evaluation, run the following command:
method=""
tokenizer_path=""
comp_config=""
model_type=""
dataset=""
bos_token=""
eos_token=""
cache_size=1024
file1=""
file2=""
file3=""
file4=""
python evaluation/eval_file.py \
--method $method \
--tokenizer_path $tokenizer_path \
--comp_config $comp_config \
--model_type $model_type \
--dataset $dataset \
--files $file1 $file2 $file3 $file4 \
--cache_size $cache_size \
--bos_token $bos_token \
--eos_token $eos_token \
--interaction
Please note that if you set
split_size>1
in the second step, the number of file i here should match the value of
split_size
. It should be noted that manual evaluation was conducted during the assessment. Use the
--interaction
flag to enable manual evaluation. The
cache_size
parameter is used for
H2O
and
SepLLM
, but not for
LightThinker
or
AnLLM
.
Evaluation Script Example
# The optional values for the method argument are 'anchor-token', 'normal', 'kvcache', and 'anchor-thought'.
method="anchor-thought"
tokenizer_path="Qwen/Qwen2.5-7B-Instruct"
comp_config="configs/LightThinker/qwen/v1.json"
model_type="qwen"
dataset="gpqa"
bos_token="<|im_start|>"
eos_token="<|im_end|>"
cache_size=1024
folder=""
ckpt=1045
file1="inference_results/${folder}/${dataset}/${ckpt}/1-4qwen_7b.jsonl"
file2="inference_results/${folder}/${dataset}/${ckpt}/2-4qwen_7b.jsonl"
file3="inference_results/${folder}/${dataset}/${ckpt}/3-4qwen_7b.jsonl"
file4="inference_results/${folder}/${dataset}/${ckpt}/4-4qwen_7b.jsonl"
python evaluation/eval_file.py \
--method $method \
--tokenizer_path $tokenizer_path \
--comp_config $comp_config \
--model_type $model_type \
--dataset $dataset \
--files $file1 $file2 $file3 $file4 \
--cache_size $cache_size \
--bos_token $bos_token \
--eos_token $eos_token \
--interaction
Manual Evaluation Instructions
When string matching fails, the output will be displayed in the format "Model Answer" <=> "Standard Answer". At this point, you can input "y" or "n" to evaluate this case. If you believe the model's answer extraction is incorrect, you can input "e" to print the model's complete output, and then input "y" or "n" to evaluate this case.
🎁Acknowledgement
Our training dataset is derived from Bespoke-Stratos-17k . We utilized the baseline code for H2O from the Meta-llama 's repository, the baseline code for SepLLM from the HKUDS 's repository. We extend our gratitude to the contributors for their outstanding work!
🚩Citation
If this work is helpful, please kindly cite as:
@article{DBLP:journals/corr/abs-2502-15589,
author = {Jintian Zhang and
Yuqi Zhu and
Mengshu Sun and
Yujie Luo and
Shuofei Qiao and
Lun Du and
Da Zheng and
Huajun Chen and
Ningyu Zhang},
title = {LightThinker: Thinking Step-by-Step Compression},
journal = {CoRR},
volume = {abs/2502.15589},
year = {2025},
url = {https://doi.org/10.48550/arXiv.2502.15589},
doi = {10.48550/ARXIV.2502.15589},
eprinttype = {arXiv},
eprint = {2502.15589},
timestamp = {Thu, 20 Mar 2025 13:28:42 +0100},
biburl = {https://dblp.org/rec/journals/corr/abs-2502-15589.bib},
bibsource = {dblp computer science bibliography, https://dblp.org}
}
Runs of zjunlp LightThinker-Qwen on huggingface.co
13
Total runs
0
24-hour runs
0
3-day runs
1
7-day runs
8
30-day runs
More Information About LightThinker-Qwen huggingface.co Model
Url of LightThinker-Qwen
LightThinker-Qwen huggingface.co Url
Provider of LightThinker-Qwen huggingface.co
Other API from zjunlp
Total runs:
214
Run Growth:
174
Growth Rate:
81.31%
Total runs:
125
Run Growth:
-86
Growth Rate:
-68.80%
Total runs:
107
Run Growth:
96
Growth Rate:
89.72%
Total runs:
84
Run Growth:
80
Growth Rate:
95.24%
Total runs:
58
Run Growth:
51
Growth Rate:
87.93%
Total runs:
45
Run Growth:
41
Growth Rate:
91.11%
Total runs:
38
Run Growth:
38
Growth Rate:
100.00%
Total runs:
33
Run Growth:
-6
Growth Rate:
-18.18%
Total runs:
17
Run Growth:
2
Growth Rate:
11.76%
Total runs:
16
Run Growth:
-9
Growth Rate:
-56.25%
Total runs:
13
Run Growth:
6
Growth Rate:
60.00%
Total runs:
12
Run Growth:
0
Growth Rate:
0.00%
Total runs:
11
Run Growth:
11
Growth Rate:
100.00%
Total runs:
9
Run Growth:
7
Growth Rate:
77.78%
Total runs:
8
Run Growth:
2
Growth Rate:
25.00%
Total runs:
8
Run Growth:
1
Growth Rate:
16.67%
Total runs:
6
Run Growth:
-4
Growth Rate:
-66.67%
Total runs:
4
Run Growth:
0
Growth Rate:
0.00%
Total runs:
4
Run Growth:
-5
Growth Rate:
-125.00%
Total runs:
3
Run Growth:
-3
Growth Rate:
-100.00%
Total runs:
2
Run Growth:
-1
Growth Rate:
-50.00%
Total runs:
2
Run Growth:
1
Growth Rate:
50.00%
Total runs:
1
Run Growth:
-2
Growth Rate:
-200.00%
Total runs:
1
Run Growth:
-1
Growth Rate:
-100.00%
Total runs:
0
Run Growth:
0
Growth Rate:
0.00%
Total runs:
0
Run Growth:
0
Growth Rate:
0.00%
Total runs:
0
Run Growth:
0
Growth Rate:
0.00%
Total runs:
0
Run Growth:
0
Growth Rate:
0.00%
Total runs:
0
Run Growth:
0
Growth Rate:
0.00%
Total runs:
0
Run Growth:
0
Growth Rate:
0.00%
Total runs:
0
Run Growth:
0
Growth Rate:
0.00%
Total runs:
0
Run Growth:
0
Growth Rate:
0.00%
Total runs:
0
Run Growth:
0
Growth Rate:
0.00%
Total runs:
0
Run Growth:
0
Growth Rate:
0.00%
Total runs:
0
Run Growth:
0
Growth Rate:
0.00%
Total runs:
0
Run Growth:
0
Growth Rate:
0.00%
Total runs:
0
Run Growth:
0
Growth Rate:
0.00%