audio-cpp / LatentSync-1.6-GGUF

LatentSync 1.6 for audio.cpp

Star or watch audio.cpp on GitHub for model integration and inference updates.

Use this package with audio.cpp . It synchronizes lip motion in an existing video with supplied speech.

Upstream code is pinned to a229c3948406bc2cf6eaf4873e662e70c6a04746 .

This package contains InsightFace face detection and landmark weights restricted to non-commercial research. LatentSync's Apache-2.0 license does not override those terms. Review upstream licenses before redistribution or deployment.

Each GGUF includes the temporal diffusion U-Net, Whisper Tiny audio encoder, SD VAE, SCRFD face detector, 106-point landmark model, face mask, configuration sidecars, and embedded audio.cpp model spec. Inference requires no Python or ONNX Runtime.

Full-Duration CUDA Example

This example uses the official LatentSync demo1_video.mp4 and demo1_audio.wav . The complete 9.6-second speech recording is retained, not a shortened smoke clip. The source video's original soundtrack is replaced by the supplied speech. Regenerated on 2026-10-21 with audio.cpp 545b212f .

The Python reference was generated with the official LatentSync implementation using the same demo video and speech. It is a generated result, not the source video. All three audio.cpp outputs use DeepCache ( cache_interval=3 ) and the supplied reference speech below, replacing the source video's original soundtrack.

Download generated video | Official Python output | Source input video | Reference speech

The slightly longer video follows the model's audio-feature windowing. This example was generated with the CUDA Debug build and DeepCache enabled, without the optional cuDNN FP16 execution modes. The run used eight CPU threads. Peak process memory was sampled every 20 ms across loading and generation; session time excludes model loading, while full CLI wall time includes loading and writing the raw-frame artifacts. MP4 encoding is a separate step and is not included in either timing.

Precision Comparison

Same full example and settings, with DeepCache enabled ( cache_interval=3 ). Each measurement is the first request in a fresh process, not a warmed average. RTF uses the 9.68-second generated video duration.

Download F16 video

DeepCache Comparison

Same full example, binary, inputs, seed and native precision; only cache_interval changes. Each run is the first request in a fresh process. RTF is session time divided by the 9.68-second generated video duration.

DeepCache is 1.89x faster here, using 160 MiB more peak VRAM . No obvious visual degradation was observed in the inspected frames of this example; this is not a quantitative quality evaluation or pixel-parity claim.

Download DeepCache video | Download no-cache video

The command below reproduces the recommended Q8/F16 cached example. The DeepCache comparison above uses the original F32/F16 package.

Run

Install FFmpeg on PATH, then run:

audiocpp_cli --task gen --family latentsync \
  --model latentsync-1.6-q8_0-f16.gguf --backend cuda \
  --audio examples/demo1/reference.wav \
  --request-option reference_video_file=examples/demo1/reference.mp4 \
  --request-option num_inference_steps=20 \
  --request-option guidance_scale=1.5 \
  --request-option seed=1247 \
  --request-option cache_interval=3 \
  --out-dir outputs/latentsync --log

The CLI writes RGB24 frames and a small JSON metadata sidecar. MP4 encoding and audio muxing are separate FFmpeg steps; see the model guide .

For these example inputs, encode the generated frames and add the speech with:

ffmpeg -f rawvideo -pixel_format rgb24 -video_size 1080x1920 -framerate 25 \
  -i outputs/latentsync/latentsync_video_rgb24.rgb \
  -i examples/demo1/reference.wav \
  -c:v libx264 -threads 4 -pix_fmt yuv420p -crf 18 \
  -c:a aac -movflags +faststart output.mp4

Upstream Sources

• LatentSync weights
• LatentSync code
• SD VAE
• InsightFace license