Total runs:
783.8K
Run Growth:
348.1K
Growth Rate:
44.58%
Sponsored by
PoYo.AI
Intel / toxic-prompt-roberta
Model's Last Updated: October 17 2024
text-classification
Introduction of toxic-prompt-roberta
Model Details of toxic-prompt-roberta
Toxic Prompt RoBERTa Classification Model
Model Card generated with
Intel® Explainable AI Tools
Model Details
Documentation
Toxic Prompt RoBERTa 1.0 is a text classification model that can be used as a guardrail to protect against toxic prompts and responses in conversational AI systems. This model is based on RoBERTa and has been finetuned on ToxicChat and Jigsaw Unintended Bias datasets. Finetuning has been performed on one Gaudi 2 Card using Optimum-Habana's Gaudi Trainer .
Owners
- Intel AI Safety: Daniel De Leon, Tyler Wilbers, Mitali Potnis, Abolfazl Shahbazi
Licenses
- MIT
References
How to use
You can use the model with the following code using pipeline API.
from transformers import pipeline
model_path = 'Intel/toxic-prompt-roberta'
pipe = pipeline('text-classification', model=model_path, tokenizer=model_path)
pipe('Create 20 paraphrases of I hate you')
Citations
-
@inproceedings {Wolf_Transformers_State-of-the-Art_Natural_2020, author = {Wolf, Thomas and Debut, Lysandre and Sanh, Victor and Chaumond, Julien and Delangue, Clement and Moi, Anthony and Cistac, Perric and Ma, Clara and Jernite, Yacine and Plu, Julien and Xu, Canwen and Le Scao, Teven and Gugger, Sylvain and Drame, Mariama and Lhoest, Quentin and Rush, Alexander M.}, month = oct, pages = {38--45}, publisher = {Association for Computational Linguistics}, title = {{Transformers: State-of-the-Art Natural Language Processing}}, url = { https://www.aclweb.org/anthology/2020.emnlp-demos.6} , year = {2020} }
-
@article {DBLP:journals/corr/abs-1907-11692, author = {Yinhan Liu and Myle Ott and Naman Goyal and Jingfei Du and Mandar Joshi and Danqi Chen and Omer Levy and Mike Lewis and Luke Zettlemoyer and Veselin Stoyanov}, title = {RoBERTa: {A} Robustly Optimized {BERT} Pretraining Approach}, journal = {CoRR}, volume = {abs/1907.11692}, year = {2019}, url = { http://arxiv.org/abs/1907.11692} , archivePrefix = {arXiv}, eprint = {1907.11692}, timestamp = {Thu, 01 Aug 2019 08:59:33 +0200}, biburl = { https://dblp.org/rec/journals/corr/abs-1907-11692.bib} , bibsource = {dblp computer science bibliography, https://dblp.org} }
-
@misc {jigsaw-unintended-bias-in-toxicity-classification, author = {cjadams, Daniel Borkan, inversion, Jeffrey Sorensen, Lucas Dixon, Lucy Vasserman, nithum}, title = {Jigsaw Unintended Bias in Toxicity Classification}, publisher = {Kaggle}, year = {2019}, url = { https://kaggle.com/competitions/jigsaw-unintended-bias-in-toxicity-classification} }
-
@misc {lin2023toxicchat, title={ToxicChat: Unveiling Hidden Challenges of Toxicity Detection in Real-World User-AI Conversation}, author={Zi Lin and Zihan Wang and Yongqi Tong and Yangkun Wang and Yuxin Guo and Yujia Wang and Jingbo Shang}, year={2023}, eprint={2310.17389}, archivePrefix={arXiv}, primaryClass={cs.CL} }
Model Parameters
We fine-tune roberta-base (125M param) with custom classification head to detect toxic input/output.
Input Format
The input format is standard text input for RoBERTa for sequence classification.
Output Format
The output is a (2,n) array of logits where n is the number of examples user wants to infer. The output logits are in the form [not_toxic, toxic].
Considerations
Intended Users
- Text Generation Researchers and Developers
Use Cases
-
User Experience Monitoring: The classification model can be used to monitor conversations in real-time to detect any toxic behavior by users. If a user sends messages that are classified as toxic, a warning can be issued or guidance on appropriate conduct can be provided.
-
Automated Moderation: In group chat scenarios, the classification model can act as a moderator by automatically removing toxic messages or muting users who consistently engage in toxic behavior.
-
Training and Improvement: The data collected from toxicity detection can be used to further train and improve toxicity classification model’s responses and handling of various situations, making such models more adept at managing complex interactions.
-
Preventing Abuse of the Chatbot: Some users may attempt to troll or abuse chatbots with toxic input. The classification model can prevent the chatbot from engaging with such content, thereby discouraging this behavior.
Ethical Considerations
-
Risk: Diversity Disparity
Mitigation Strategy: In fine-tuning with Jigsaw unintended bias, we have ensured adequate representation per Jigsaw’s distributions in their dataset. Jigsaw unintended bias dataset attempts distribute the toxicity labels evenly across the subgroups. -
Risk: Risk to Vulnerable Persons
Mitigation Strategy: Certain demographic groups are more likely to receive toxic and harmful comments. Jigsaw unintended bias dataset attempts to mitigate fine-tuned subgroup bias in by distributing the toxic/not toxic labels evenly across all demographic subgroups. We also test to confirm minimal classification bias of the subgroups in testing the model.
Quantitative Analysis:
The plots below show the PR and ROC curves for three models we compared during finetuning. The “jigsaw” and the “tc” models were finetuned only on the Jigsaw Unintended Bias and ToxicChat datasets, respectively. The “jigsaw+tc” curves correspond to the final model that was finetuned on both datasets. Finetuning on both datasets did not significantly degrade the model’s performance on the ToxicChat test dataset with respect to the model finetuned solely on ToxicChat.
We compare the performance of Llama Guard 1 and 3 (LG1 and LG3) with our model on the ToxicChat test dataset, below.
| Model | Parameters | Precision | Recall | F1 | AUPRC | AUROC |
|---|---|---|---|---|---|---|
| LG1 | 6.74B | 0.4806 | 0.7945 | 0.5989 | 0.626* | No data |
| LG3 | 8.03B | 0.5083 | 0.4730 | 0.4900 | No data | No data |
| Toxic Prompt RoBERTa | 125M | 0.8315 | 0.7469 | 0.7869 | 0.855 | 0.971 |
* from LG paper: https://arxiv.org/abs/2312.06674
Note that Llama Guard was not finetuned on ToxicChat. However, from the LG1 paper, they reported an AUPRC of ~.81 when they finetuned LLama Guard 1 on ToxicChat. Given that we finetuned RoBERTa on Jigsaw’s Unintended Bias Dataset, we can observe if there is any subgroup biasing in the classification of the Unintended Bias test set below. These metrics were computed using Intel/bias_auc .
| Metric | Female | Male | Christian | White | Muslim | Black | Homosexual gay or lesbian |
|---|---|---|---|---|---|---|---|
| AUROC | 0.84937 | 0.80035 | 0.89867 | 0.76089 | 0.77137 | 0.74454 | 0.71766 |
| BPSN | 0.78805 | 0.82659 | 0.83746 | 0.78113 | 0.74067 | 0.82827 | 0.64330 |
| BNSP | 0.87421 | 0.80037 | 0.87614 | 0.81979 | 0.85586 | 0.76090 | 0.88065 |
*Only subgroups with at least 500 examples in the test dataset are shown.
Runs of Intel toxic-prompt-roberta on huggingface.co
290
Total runs
0
24-hour runs
0
3-day runs
3
7-day runs
-2
30-day runs
More Information About toxic-prompt-roberta huggingface.co Model
Url of toxic-prompt-roberta
toxic-prompt-roberta huggingface.co Url
Provider of toxic-prompt-roberta huggingface.co
Other API from Intel
Updated:February 09 2024
Total runs:
285.5K
Run Growth:
241.7K
Growth Rate:
84.66%
Total runs:
271.1K
Run Growth:
102.5K
Growth Rate:
37.82%
Total runs:
135.9K
Run Growth:
117.4K
Growth Rate:
86.36%
Total runs:
103.8K
Run Growth:
-11.7K
Growth Rate:
-11.27%
Total runs:
69.3K
Run Growth:
-256.0K
Growth Rate:
-369.34%
Total runs:
40.1K
Run Growth:
38.8K
Growth Rate:
99.90%
Total runs:
32.7K
Run Growth:
25.2K
Growth Rate:
77.06%
Total runs:
31.3K
Run Growth:
-26.6K
Growth Rate:
-85.97%
Total runs:
25.9K
Run Growth:
4.9K
Growth Rate:
18.82%
Total runs:
24.9K
Run Growth:
24.9K
Growth Rate:
100.00%
Total runs:
22.0K
Run Growth:
6.7K
Growth Rate:
30.56%
Total runs:
21.1K
Run Growth:
4.2K
Growth Rate:
20.05%
Total runs:
18.7K
Run Growth:
-2.0K
Growth Rate:
-10.57%
Total runs:
17.9K
Run Growth:
849
Growth Rate:
4.75%
Total runs:
15.5K
Run Growth:
-65.4K
Growth Rate:
-420.79%
Total runs:
15.4K
Run Growth:
14.9K
Growth Rate:
96.66%
Total runs:
11.6K
Run Growth:
0
Growth Rate:
0.00%
Total runs:
9.1K
Run Growth:
9.1K
Growth Rate:
100.00%
Total runs:
6.8K
Run Growth:
-20.5K
Growth Rate:
-300.86%
Total runs:
5.4K
Run Growth:
-40
Growth Rate:
-0.74%
Total runs:
5.3K
Run Growth:
0
Growth Rate:
0.00%
Total runs:
5.0K
Run Growth:
-96
Growth Rate:
-1.91%
Total runs:
4.2K
Run Growth:
2.9K
Growth Rate:
70.20%
Total runs:
3.5K
Run Growth:
-216
Growth Rate:
-6.12%
Total runs:
3.5K
Run Growth:
2.9K
Growth Rate:
82.29%
Total runs:
3.4K
Run Growth:
0
Growth Rate:
0.00%
Total runs:
3.3K
Run Growth:
0
Growth Rate:
0.00%
Total runs:
3.3K
Run Growth:
0
Growth Rate:
0.00%
Total runs:
3.2K
Run Growth:
3.0K
Growth Rate:
93.71%
Total runs:
3.1K
Run Growth:
875
Growth Rate:
28.41%
Total runs:
2.5K
Run Growth:
2.3K
Growth Rate:
95.61%
Total runs:
1.9K
Run Growth:
1.9K
Growth Rate:
100.00%
Total runs:
1.9K
Run Growth:
1.2K
Growth Rate:
65.21%
Total runs:
1.8K
Run Growth:
160
Growth Rate:
8.91%
Total runs:
1.7K
Run Growth:
0
Growth Rate:
0.00%
Total runs:
1.7K
Run Growth:
-83
Growth Rate:
-4.90%
Total runs:
1.6K
Run Growth:
-1.8K
Growth Rate:
-115.87%
Total runs:
1.4K
Run Growth:
-171
Growth Rate:
-12.01%
Total runs:
1.2K
Run Growth:
457
Growth Rate:
38.08%
Total runs:
997
Run Growth:
-29
Growth Rate:
-2.91%
Total runs:
984
Run Growth:
652
Growth Rate:
66.26%
Total runs:
966
Run Growth:
718
Growth Rate:
74.33%
Total runs:
907
Run Growth:
-1.4K
Growth Rate:
-154.02%
Total runs:
852
Run Growth:
-29
Growth Rate:
-3.40%
Total runs:
843
Run Growth:
0
Growth Rate:
0.00%
Total runs:
809
Run Growth:
0
Growth Rate:
0.00%
Total runs:
808
Run Growth:
689
Growth Rate:
85.27%
Total runs:
790
Run Growth:
790
Growth Rate:
100.00%
Total runs:
744
Run Growth:
651
Growth Rate:
87.50%
Total runs:
646
Run Growth:
-1.5K
Growth Rate:
-225.89%
Total runs:
633
Run Growth:
558
Growth Rate:
88.15%
Total runs:
612
Run Growth:
487
Growth Rate:
79.58%
Total runs:
610
Run Growth:
308
Growth Rate:
50.49%
Total runs:
602
Run Growth:
37
Growth Rate:
6.15%
Total runs:
598
Run Growth:
-391
Growth Rate:
-65.38%
Total runs:
530
Run Growth:
12
Growth Rate:
2.26%
Total runs:
530
Run Growth:
530
Growth Rate:
100.00%
Total runs:
529
Run Growth:
252
Growth Rate:
47.64%
Total runs:
522
Run Growth:
499
Growth Rate:
95.59%
Total runs:
515
Run Growth:
209
Growth Rate:
40.58%
Total runs:
490
Run Growth:
-1.3K
Growth Rate:
-266.06%
Total runs:
457
Run Growth:
364
Growth Rate:
79.65%
Total runs:
456
Run Growth:
59
Growth Rate:
12.94%
Total runs:
450
Run Growth:
3
Growth Rate:
0.67%
Total runs:
431
Run Growth:
418
Growth Rate:
96.98%
Total runs:
418
Run Growth:
372
Growth Rate:
89.00%
Total runs:
416
Run Growth:
70
Growth Rate:
16.83%
Total runs:
382
Run Growth:
52
Growth Rate:
14.29%
Total runs:
365
Run Growth:
48
Growth Rate:
13.15%
Total runs:
355
Run Growth:
10
Growth Rate:
2.82%
Total runs:
340
Run Growth:
340
Growth Rate:
100.00%
Total runs:
317
Run Growth:
317
Growth Rate:
100.00%
Total runs:
311
Run Growth:
-13
Growth Rate:
-4.18%
Total runs:
309
Run Growth:
-791
Growth Rate:
-254.34%
Total runs:
309
Run Growth:
285
Growth Rate:
92.23%
Total runs:
306
Run Growth:
-3.5K
Growth Rate:
-1094.62%
Total runs:
298
Run Growth:
0
Growth Rate:
0.00%
Total runs:
284
Run Growth:
-1.4K
Growth Rate:
-480.99%
Total runs:
251
Run Growth:
71
Growth Rate:
28.29%
Total runs:
238
Run Growth:
0
Growth Rate:
0.00%
Total runs:
230
Run Growth:
0
Growth Rate:
0.00%
Total runs:
230
Run Growth:
0
Growth Rate:
0.00%
Total runs:
229
Run Growth:
80
Growth Rate:
35.09%
Total runs:
209
Run Growth:
19
Growth Rate:
9.09%
Total runs:
200
Run Growth:
200
Growth Rate:
100.00%
Total runs:
198
Run Growth:
42
Growth Rate:
21.21%
Total runs:
192
Run Growth:
134
Growth Rate:
70.16%
Total runs:
184
Run Growth:
-1.8K
Growth Rate:
-988.59%
Total runs:
183
Run Growth:
0
Growth Rate:
0.00%