Total runs:
7.7M
Run Growth:
-1.2M
Growth Rate:
-15.26%
Sponsored by
APIMart
laion / BUD-E-Whisper
Model's Last Updated: June 20 2025
automatic-speech-recognition
Introduction of BUD-E-Whisper
Model Details of BUD-E-Whisper
BUD-E Whisper: Emotional Speech Captioning Model
BUD-E Whisper is a suite of Whisper models fine-tuned for direct emotional speech captioning . The core models are built upon OpenAI's Whisper architecture, with the current primary variant being a fine-tune of OpenAI Whisper Small . These models are designed to generate text captions that not only transcribe speech but also inherently reflect its emotional content.
The embeddings generated by BUD-E Whisper can also serve as input for Empathic Insight - Voice , a downstream ensemble of Multi-Layer Perceptrons (MLPs) designed to predict dimensional emotion scores.
License
This model is released under the CC-by-4.0 license. Please give attribution to Maurice Kraus & Christoph Schuhmann, who made this model.
Colab
Training Data
BUD-E Whisper was trained on a combination of:
- The Laion's Got Talent (Enhanced Flash Annotations and Long Captions) dataset .
- An internal dataset comprising approximately 5,000 hours of public Vlogs and similar audio content.
Training Procedure & Caption Generation
A key aspect of BUD-E Whisper's development was a multi-step caption refinement process to create rich training targets:
- Initial Score Generation: An iterative process using Gemini Flash 2.0 generated initial 40-dimensional emotion scores (0-4 scale) and 15 additional dimensions like age, arousal, valence, dominance, harshness, vocalbursts,... for all audio snippets.
- Templated Captions: These scores were converted into templated string captions.
- Paraphrasing for Richness: Gemini Flash 2.0 was then used to paraphrase these templated captions, creating diverse and semantically rich training targets.
- Fine-tuning: Various Whisper model sizes (including the aforementioned fine-tune of OpenAI Whisper Small) were fine-tuned on these refined, emotionally-aware captions.
This multi-step caption refinement was crucial for performance. Direct score regression or simple templated captions were found to lead to suboptimal performance for emotional speech captioning with Whisper models.
Intended Use
- Generating emotionally nuanced captions for audio content.
- Providing rich embeddings for downstream emotion recognition tasks (e.g., with Empathic Insight - Voice).
Runs of laion BUD-E-Whisper on huggingface.co
6.0K
Total runs
254
24-hour runs
839
3-day runs
1.8K
7-day runs
5.0K
30-day runs
More Information About BUD-E-Whisper huggingface.co Model
Url of BUD-E-Whisper
BUD-E-Whisper huggingface.co Url
Provider of BUD-E-Whisper huggingface.co
Other API from laion
Updated:January 12 2026
Total runs:
3.7M
Run Growth:
366.4K
Growth Rate:
9.83%
Total runs:
3.1M
Run Growth:
-138.9K
Growth Rate:
-4.52%
Total runs:
1.4M
Run Growth:
-61.2K
Growth Rate:
-4.35%
Total runs:
1.3M
Run Growth:
657.5K
Growth Rate:
47.97%
Total runs:
751.9K
Run Growth:
283.2K
Growth Rate:
37.67%
Total runs:
589.5K
Run Growth:
82.6K
Growth Rate:
14.01%
Total runs:
308.5K
Run Growth:
14.1K
Growth Rate:
4.63%
Total runs:
275.8K
Run Growth:
34.6K
Growth Rate:
12.57%
Total runs:
105.7K
Run Growth:
-15.0K
Growth Rate:
-14.24%
Total runs:
102.6K
Run Growth:
63.8K
Growth Rate:
62.24%
Total runs:
98.3K
Run Growth:
21.6K
Growth Rate:
22.01%
Total runs:
48.9K
Run Growth:
-23.5K
Growth Rate:
-48.10%
Total runs:
41.8K
Run Growth:
3.0K
Growth Rate:
7.05%
Total runs:
41.1K
Run Growth:
-13.1K
Growth Rate:
-31.87%
Total runs:
37.6K
Run Growth:
6.3K
Growth Rate:
16.77%
Total runs:
30.7K
Run Growth:
-64.5K
Growth Rate:
-210.30%
Total runs:
21.2K
Run Growth:
1.9K
Growth Rate:
9.06%
Total runs:
19.3K
Run Growth:
6.7K
Growth Rate:
34.49%
Total runs:
15.7K
Run Growth:
6.7K
Growth Rate:
42.56%
Total runs:
14.7K
Run Growth:
4.7K
Growth Rate:
32.33%
Total runs:
12.3K
Run Growth:
-1.2K
Growth Rate:
-9.53%
Total runs:
6.0K
Run Growth:
-3.1K
Growth Rate:
-52.51%
Total runs:
5.9K
Run Growth:
-120.8K
Growth Rate:
-2044.70%
Total runs:
4.9K
Run Growth:
1.4K
Growth Rate:
28.27%
Total runs:
3.5K
Run Growth:
-14.6K
Growth Rate:
-420.27%
Total runs:
2.9K
Run Growth:
354
Growth Rate:
12.20%
Total runs:
2.7K
Run Growth:
-361
Growth Rate:
-13.38%
Total runs:
2.6K
Run Growth:
-21
Growth Rate:
-0.81%
Total runs:
2.2K
Run Growth:
-1.3K
Growth Rate:
-60.78%
Total runs:
1.4K
Run Growth:
341
Growth Rate:
24.29%
Total runs:
1.3K
Run Growth:
-274
Growth Rate:
-21.41%
Total runs:
1.0K
Run Growth:
943
Growth Rate:
91.02%
Total runs:
930
Run Growth:
897
Growth Rate:
96.45%
Total runs:
751
Run Growth:
-650
Growth Rate:
-86.55%
Total runs:
744
Run Growth:
-274
Growth Rate:
-36.83%
Total runs:
647
Run Growth:
393
Growth Rate:
60.74%
Total runs:
529
Run Growth:
-89
Growth Rate:
-16.82%
Total runs:
474
Run Growth:
474
Growth Rate:
100.00%
Total runs:
467
Run Growth:
467
Growth Rate:
100.00%
Total runs:
457
Run Growth:
45
Growth Rate:
9.85%
Total runs:
398
Run Growth:
-292
Growth Rate:
-73.37%
Total runs:
378
Run Growth:
-904
Growth Rate:
-239.15%
Total runs:
316
Run Growth:
106
Growth Rate:
33.54%
Total runs:
301
Run Growth:
-595
Growth Rate:
-197.67%
Total runs:
289
Run Growth:
120
Growth Rate:
42.25%
Total runs:
287
Run Growth:
282
Growth Rate:
97.92%
Total runs:
282
Run Growth:
34
Growth Rate:
12.06%
Total runs:
270
Run Growth:
263
Growth Rate:
97.05%
Total runs:
258
Run Growth:
-411
Growth Rate:
-159.30%
Total runs:
257
Run Growth:
247
Growth Rate:
96.11%
Total runs:
243
Run Growth:
235
Growth Rate:
96.71%
Total runs:
237
Run Growth:
31
Growth Rate:
13.08%
Total runs:
233
Run Growth:
29
Growth Rate:
12.45%
Total runs:
231
Run Growth:
64
Growth Rate:
27.71%
Total runs:
231
Run Growth:
218
Growth Rate:
93.97%
Total runs:
228
Run Growth:
216
Growth Rate:
94.32%
Total runs:
227
Run Growth:
36
Growth Rate:
15.86%
Total runs:
226
Run Growth:
214
Growth Rate:
94.27%
Total runs:
225
Run Growth:
35
Growth Rate:
15.56%
Total runs:
221
Run Growth:
209
Growth Rate:
94.57%
Total runs:
216
Run Growth:
207
Growth Rate:
95.83%
Total runs:
215
Run Growth:
206
Growth Rate:
95.37%
Total runs:
210
Run Growth:
199
Growth Rate:
94.31%
Total runs:
207
Run Growth:
-2.6K
Growth Rate:
-1260.39%
Total runs:
204
Run Growth:
5
Growth Rate:
2.45%
Total runs:
199
Run Growth:
72
Growth Rate:
36.18%
Total runs:
195
Run Growth:
-62
Growth Rate:
-31.79%
Total runs:
194
Run Growth:
-4.9K
Growth Rate:
-2506.63%
Total runs:
194
Run Growth:
184
Growth Rate:
94.36%
Total runs:
189
Run Growth:
179
Growth Rate:
94.71%
Total runs:
188
Run Growth:
178
Growth Rate:
94.18%
Total runs:
188
Run Growth:
176
Growth Rate:
93.62%
Total runs:
181
Run Growth:
-2.3K
Growth Rate:
-1244.75%
Total runs:
171
Run Growth:
-1.7K
Growth Rate:
-996.49%
Total runs:
165
Run Growth:
-1.1K
Growth Rate:
-659.39%
Total runs:
143
Run Growth:
137
Growth Rate:
95.80%
Total runs:
139
Run Growth:
25
Growth Rate:
17.99%
Total runs:
137
Run Growth:
5
Growth Rate:
3.65%
Total runs:
125
Run Growth:
3
Growth Rate:
2.40%
Total runs:
115
Run Growth:
89
Growth Rate:
77.39%
Total runs:
106
Run Growth:
21
Growth Rate:
19.81%
Total runs:
102
Run Growth:
-131
Growth Rate:
-128.43%
Total runs:
84
Run Growth:
84
Growth Rate:
100.00%
Total runs:
81
Run Growth:
-215
Growth Rate:
-265.43%
Total runs:
81
Run Growth:
49
Growth Rate:
60.49%
Total runs:
79
Run Growth:
68
Growth Rate:
86.08%
Total runs:
71
Run Growth:
46
Growth Rate:
64.79%
Total runs:
68
Run Growth:
-515
Growth Rate:
-757.35%
Total runs:
64
Run Growth:
17
Growth Rate:
26.56%
Total runs:
61
Run Growth:
-449
Growth Rate:
-736.07%
Total runs:
61
Run Growth:
-429
Growth Rate:
-703.28%
Total runs:
59
Run Growth:
41
Growth Rate:
69.49%
Total runs:
58
Run Growth:
-407
Growth Rate:
-701.72%
Total runs:
57
Run Growth:
-353
Growth Rate:
-619.30%