Sponsored by
YouArt
zjunlp / KnowPrompt
Model's Last Updated: December 22 2022
Introduction of KnowPrompt
Model Details of KnowPrompt
KnowPrompt:Knowledge-aware Prompt-tuning with Synergistic Optimization for Relation Extraction
KnowPrompt is used for relational extraction tasks, injecting latent knowledge contained in relation labels into prompt construction with learnable virtual template words and answer words , and synergistically optimize their representation with structured constraints.
Model description
We take the first step to inject latent knowledge contained in relation labels into prompt construction，the knowledge extraction is then implemented with a Prompt-tuning model。The implementation is as follows：virtual template words around entities initialized using aggregate entity embeddings are used as learnable virtual template words to inject entity knowledge; Meanwhile, we leverage label to compute average embeddings as virtual answers words to inject relationship knowledge. In this structure, entities and relations are mutually constrained and virtual template and answer words should be contextually relevant, so we introduce synergistic optimization to correct virtual template and answer words.
Intended uses & limitations
This model is used for relationship extraction tasks, and the extracted information can be used for more downstream NLP tasks, such as: information retrieval, conversation generation and Q&A. Please refer to the code example for details on how to use it.
The relationship labels in the model training data are limited and can only generalize the relationships in the real world to a certain extent.
Training data
We adopt SemEval as the dataset
| Dataset | # Train. | # Val. | # Test. | # Rel. |
|---|---|---|---|---|
| SemEval | 6,507 | 1,493 | 2,717 | 19 |
Training procedure
Training
The training is divided into two phases, and the first phase performs collaborative optimization of virtual template words and answer words
$\lambda$is the hyperparameter for weighing the two loss functions;The second stage optimizes all parameters with a smaller learning rate based on the optimized virtual template words and answer words, using only the loss function $\mathcal{J}_{texttt{[MASK]}}$to finetune the parameters for the language model.The hyperparameters are different for different datasets, as shown in the script file in the source code.Taking SemEval as an example, the hyperparameters are set as follows:
max_epochs=10
max_sequence_length=256
batch_size=16
learning_rate=3e-5
batch_size=16
t_lambda=0.001
Data Evaluation and Results
The results of the comparison with other models in standard settings are shown in the following table.
| Methods | Precision |
|---|---|
| Fine-tuning | 87.6 |
| KnowBERT | 89.1 |
| MTB | 89.5 |
| PTR | 89.9 |
| KnowPrompt | 90.2 (+0.3) |
| In low-resource settings，we performed the 8-, 16-, and 32-experiments.K instances of each class are sampled from the initial training and validation sets to form the training and validation sets for the FEW-shot. The results are as follows: | |
| Split | Methods |
| ----- | ----------- |
| k=8 | Fine-tuning |
| GDPNet | |
| PTR | |
| KnowPrompt | |
| k=16 | Fine-tuning |
| GDPNet | |
| PTR | |
| KnowPrompt | |
| k=32 | Fine-tuning |
| GDPNet | |
| PTR | |
| KnowPrompt | |
| As 𝐾 decreases from 32 to 8, the improvement in our KnowPrompt over the other three methods increases gradually. |
BibTeX entry and citation info
@inproceedings{DBLP:conf/www/ChenZXDYTHSC22,
author = {Xiang Chen and
Ningyu Zhang and
Xin Xie and
Shumin Deng and
Yunzhi Yao and
Chuanqi Tan and
Fei Huang and
Luo Si and
Huajun Chen},
editor = {Fr{\'{e}}d{\'{e}}rique Laforest and
Rapha{\"{e}}l Troncy and
Elena Simperl and
Deepak Agarwal and
Aristides Gionis and
Ivan Herman and
Lionel M{\'{e}}dini},
title = {KnowPrompt: Knowledge-aware Prompt-tuning with Synergistic Optimization
for Relation Extraction},
booktitle = {{WWW} '22: The {ACM} Web Conference 2022, Virtual Event, Lyon, France,
April 25 - 29, 2022},
pages = {2778--2788},
publisher = {{ACM}},
year = {2022},
url = {https://doi.org/10.1145/3485447.3511998},
doi = {10.1145/3485447.3511998},
timestamp = {Tue, 26 Apr 2022 16:02:09 +0200},
biburl = {https://dblp.org/rec/conf/www/ChenZXDYTHSC22.bib},
bibsource = {dblp computer science bibliography, https://dblp.org}
}
```bash
git clone https://www.modelscope.cn/jeno11/knowprompt_demo.git
Runs of zjunlp KnowPrompt on huggingface.co
0
Total runs
0
24-hour runs
0
3-day runs
0
7-day runs
0
30-day runs
More Information About KnowPrompt huggingface.co Model
Url of KnowPrompt
KnowPrompt huggingface.co Url
Provider of KnowPrompt huggingface.co
Other API from zjunlp
Total runs:
214
Run Growth:
174
Growth Rate:
81.31%
Total runs:
125
Run Growth:
-86
Growth Rate:
-68.80%
Total runs:
107
Run Growth:
96
Growth Rate:
89.72%
Total runs:
84
Run Growth:
80
Growth Rate:
95.24%
Total runs:
58
Run Growth:
51
Growth Rate:
87.93%
Total runs:
45
Run Growth:
41
Growth Rate:
91.11%
Total runs:
38
Run Growth:
38
Growth Rate:
100.00%
Total runs:
33
Run Growth:
-6
Growth Rate:
-18.18%
Total runs:
32
Run Growth:
24
Growth Rate:
77.42%
Total runs:
17
Run Growth:
2
Growth Rate:
11.76%
Total runs:
12
Run Growth:
0
Growth Rate:
0.00%
Total runs:
11
Run Growth:
11
Growth Rate:
100.00%
Total runs:
9
Run Growth:
7
Growth Rate:
77.78%
Total runs:
7
Run Growth:
0
Growth Rate:
0.00%
Total runs:
6
Run Growth:
-4
Growth Rate:
-66.67%
Total runs:
5
Run Growth:
-8
Growth Rate:
-200.00%
Total runs:
5
Run Growth:
-5
Growth Rate:
-125.00%
Total runs:
4
Run Growth:
0
Growth Rate:
0.00%
Total runs:
4
Run Growth:
-5
Growth Rate:
-125.00%
Total runs:
3
Run Growth:
-5
Growth Rate:
-166.67%
Total runs:
2
Run Growth:
-1
Growth Rate:
-50.00%
Total runs:
2
Run Growth:
1
Growth Rate:
50.00%
Total runs:
1
Run Growth:
-2
Growth Rate:
-200.00%
Total runs:
1
Run Growth:
-1
Growth Rate:
-100.00%
Total runs:
0
Run Growth:
0
Growth Rate:
0.00%
Total runs:
0
Run Growth:
0
Growth Rate:
0.00%
Total runs:
0
Run Growth:
0
Growth Rate:
0.00%
Total runs:
0
Run Growth:
0
Growth Rate:
0.00%
Total runs:
0
Run Growth:
0
Growth Rate:
0.00%
Total runs:
0
Run Growth:
0
Growth Rate:
0.00%
Total runs:
0
Run Growth:
0
Growth Rate:
0.00%
Total runs:
0
Run Growth:
0
Growth Rate:
0.00%
Total runs:
0
Run Growth:
0
Growth Rate:
0.00%
Total runs:
0
Run Growth:
0
Growth Rate:
0.00%
Total runs:
0
Run Growth:
0
Growth Rate:
0.00%
Total runs:
0
Run Growth:
0
Growth Rate:
0.00%
Total runs:
0
Run Growth:
0
Growth Rate:
0.00%