laion / sound-effect-captioning-whisper

LAION Sound-Effect Captioning Whisper

A Whisper-Small-sized audio-captioning model that writes rich, natural-language descriptions of general-purpose sound effects, ambiences, vocal bursts, and music snippets . Given ≤ 30 s of audio it produces a paragraph-length caption describing the content, timbre, and likely source of the sound.

This is the final stage of a multi-step training lineage that stacks emotional speech captioning, generative-audio pre-training, and sound-effect fine-tuning on top of OpenAI's Whisper-Small.

TL;DR

• Architecture : WhisperForConditionalGeneration · Whisper-Small · 241.7 M params
• Input : mono audio, 16 kHz, up to 30 s
• Output : free-form English caption describing the sound
• Best validation loss (held-out val set, ~500 samples across 5 source datasets): 1.494 — down from 1.689 after stage 1 and from ~2.0 before any sound-effect training
• Training compute : 8 × H100 · DDP via torchrun · ~4 h wall-clock end-to-end

Model genealogy

This checkpoint is the end of a four-step lineage; every stage starts from the previous one.

1. OpenAI Whisper-Small – ASR pre-training on 680 k hours of labelled speech.
2. laion/BUD-E-Whisper – LAION's emotion-aware speech captioning fine-tune. BUD-E-Whisper was trained on LAION's Got Talent plus ~5 k hours of public vlogs, with emotion scores generated by Gemini Flash 2.0 (40 emotion dimensions + 15 auxiliary dimensions such as age, arousal, valence, harshness and vocal-burst cues), templated into captions and then paraphrased for semantic richness. The resulting model can caption how a voice sounds, not just what it says.
3. laion/captioning-whisper-proof_of_concept ( checkpoint-35000 ) – a continued pre-training run on a broad mix of non-speech audio with Gemini-generated captions:
   • Re-captioned AudioSet ( mitermix/audioset-with-grounded-captions )
   • Re-captioned Freesound ( laion/freesound-commercially-permissive-subset-with-captions )
   • TangoFlux-generated sound events – synthetic sound effects produced with TangoFlux and re-captioned
   • LAION AI-music snippets – machine-generated music clips with Gemini-style captions This stage teaches the decoder the "sound-effect paragraph" writing style and expands its vocabulary from speech events to environmental, mechanical, musical and abstract audio events.
4. This checkpoint – Sound-Effect Captioning Whisper – two additional fine-tuning stages on a fresh local mix of five public sound-effect / vocal-burst datasets (details below). Stage 1 = 1 epoch @ 5e-6 linear, Stage 2 = 3 epochs @ 1e-5 cosine.

Training data (this model)

All datasets were downloaded, normalised to {mp3, caption} pairs, and pooled into a single flat directory. Per-dataset counts after filtering:

From each prefix, 100 random samples were held out as a stable validation split (~500 total), leaving ~915.6 k for training.

Training setup

Eval curve for stage 2 (lower is better):

The final checkpoint uploaded here ( model.safetensors ) is the best weights from step 31 250, restored via load_best_model_at_end=True at the end of stage-2 training.

Usage

import torch, torchaudio
from transformers import WhisperProcessor, WhisperForConditionalGeneration

REPO = "laion/sound-effect-captioning-whisper"
processor = WhisperProcessor.from_pretrained("openai/whisper-small")
model     = WhisperForConditionalGeneration.from_pretrained(REPO).eval().to("cuda")
model.generation_config.forced_decoder_ids = None

# Load any audio ≤ 30 s
wav, sr = torchaudio.load("your_sound.mp3")
if wav.shape[0] > 1:                 # → mono
    wav = wav.mean(dim=0, keepdim=True)
if sr != 16_000:                     # → 16 kHz
    wav = torchaudio.functional.resample(wav, sr, 16_000)
arr = wav.squeeze(0).numpy()[: 16_000 * 30]

feats = processor.feature_extractor(arr, sampling_rate=16_000,
                                    return_tensors="pt").input_features.to("cuda")
with torch.no_grad():
    ids = model.generate(feats, max_new_tokens=400, num_beams=1, do_sample=False)
print(processor.batch_decode(ids, skip_special_tokens=True)[0].strip())

Notes:

• Use the base openai/whisper-small processor (this repo only ships tokenizer.json + preprocessor_config.json , which are the Whisper-Small defaults – the processor class needs the bundled chat_template.json / special-tokens files that live on the base repo).
• forced_decoder_ids=None is required, otherwise Whisper's transcription language prompt clamps the decoder into ASR mode and suppresses captioning.
• Greedy decoding works well; beam=4 trades a bit of repetition for a small quality bump but is not necessary.

Example predictions

All four clips below are drawn from the fixed held-out validation split (these exact files were never seen during training). They were captioned with this released checkpoint.

Example 1 — Vocal burst · groan

Reference : This file features a low, rumbling vocalization, possibly a groan. There's a sense of strain in the sound, and it's drawn out, not abrupt. The recording appears to be of moderate quality, and no other sound events are discernible. The vocalization does not seem to be musical or speech-related.

Prediction : The audio features a low, guttural vocalization, possibly a groan or a moan, with a somewhat strained quality. The sound is drawn out and seems to convey a sense of discomfort or perhaps fatigue. The recording is clear, with no noticeable background noise or distortion. The vocalization is the primary focus of the audio.

Example 2 — Sound effect · test tone

Reference : Test tone. The sound is a steady, high-frequency test tone, likely used for calibration or signaling.

Prediction : The audio features a single, distinct, high-pitched electronic tone. The tone is pure and sustained, with a clear, pure quality. This is likely a synthesized sound, possibly a test tone, a simple electronic signal, or a sound effect used to indicate a specific event or transition.

Example 3 — Music · upbeat electronic

Reference : The audio features a high-energy, upbeat piece of music. The melody is carried by a synthesizer, with a driving beat provided by electronic drums. A heavy bassline is prominent. There are sections with and without a strong beat. This recording is a song likely meant for dancing, especially as there is a synth melody.

Prediction : The audio features a fast-paced, energetic electronic dance track. A prominent, repetitive synth melody is layered over a driving beat. The overall sound is bright and danceable. This is a clip from an electronic dance music track, likely intended for dancing or club environments. The fast tempo and repetitive melody are typical of the genre, confirming the hint of music.

Example 4 — Machinery · mechanical whine (TangoFlux-generated)

Reference : The audio features a continuous, high-pitched whirring sound, which is then joined by a distinct, rhythmic clanking or grinding noise. […] This sound is indicative of heavy machinery in operation, possibly a large engine or industrial equipment.

Prediction : The audio features a loud, high-pitched, and continuous mechanical whine, characteristic of a jet engine. This sound is accompanied by a distinct, lower-frequency rumbling noise, suggesting the operation of a powerful engine. The whine is prominent and consistent, while the rumble is steady and deep. This soundscape is indicative of a large aircraft engine, likely a jet engine, operating at high power, possibly during takeoff or during a test run.

Intended use

• Automatic tagging / describing of sound-effect libraries and stock-audio collections.
• Audio-understanding component in agents that need to talk about what they hear.
• Research baseline for open-vocabulary audio captioning on top of Whisper.

Limitations

• The model has no speech-to-text output : the base ASR behaviour of Whisper is intentionally suppressed, and transcribing spoken content is out of scope.
• Trained on ≤ 30 s clips; longer inputs are truncated. For long recordings, chunk them yourself and run the model per chunk.
• Captioning style is strongly biased toward the Gemini-style "the audio features …" phrasing inherited from the training data.
• Some mechanical / music / everyday-sound confusions remain; for difficult clips the model tends to fall back to generic genre templates ("electronic dance track", "jet engine", etc.).
• Training data is predominantly English and primarily captures the tastes of the AudioSet + Freesound + LAION-curated distributions.

Acknowledgements

• OpenAI Whisper for the base model.
• LAION for curating the BUD-E-Whisper emotion dataset, the captioning-whisper proof-of-concept, the freesound / vocal-burst / in-the-wild / generated sound-event corpora, and the AI-music caption snippets used in the intermediate pre-training stage.
• TangoFlux for generating the synthetic sound-event data that fed the intermediate pre-training step.
• Gemini Flash 2.0 for the underlying caption generation and paraphrasing across multiple datasets.
• mitermix/audioset-with-grounded-captions for the grounded AudioSet captions.

License

Released under Apache-2.0 , following the upstream Whisper license. The audio clips in examples/ are samples from the five training datasets and retain their respective upstream licences (see the dataset pages linked above).