MuseTalk (GPU lip-sync)
Redraws the speaker's mouth in a video or a still image so it follows a new audio track, with MuseTalk. It needs an NVIDIA graphics card with CUDA. The download is small: on first use it installs its own PyTorch runtime and several GB of models.
Settings Add-ons MuseTalk (GPU lip-sync) Install
Subtitld downloads it, checks its SHA-256, and keeps it updated.
- Add-on download
- 9–18 MB
What it does
It starts from a video, a single still image or a folder of frames, and writes a video in H.264 with the new audio in AAC.
When the audio outlasts the video, the frames play forward, then backward, and so on until the audio ends.
The add-on you download is a small launcher. Its first job installs PyTorch and the rest of MuseTalk into a private Python environment, then downloads the models. Later jobs skip all of that.
MuseTalk 1.5, the default, gives sharper mouths; 1.0 is the original. Both download on the first run, so you can switch without waiting.
Nothing in Subtitld runs it yet. You can install it, but no command in the app sends it a video to lip-sync so far.
Models
There is nothing to choose. The first job fetches everything MuseTalk needs, once:
- A private Python environment with PyTorch for CUDA and MuseTalk's other libraries: several GB. The whole install shows as the first 5% of the progress bar.
- MuseTalk's source, cloned with
git. - The models: MuseTalk 1.5 and 1.0, the SD-VAE, Whisper tiny, DWPose, the BiSeNet face parser and SyncNet, several GB more.
The CUDA builds of PyTorch come from https://download.pytorch.org/whl/cu121. For another CUDA version, or a mirror, set SUBTITLD_MUSETALK_TORCH_INDEX to its address. A new version of the add-on runs the install once more, into the same environment.
What the first run needs
- An NVIDIA card with CUDA. The add-on checks for one only after it has installed the runtime, so make sure before you start.
- Python 3.10 or newer on your PATH, with its
venvmodule, or setSUBTITLD_MUSETALK_PYTHONto one. The add-on builds its environment from it. giton your PATH, to fetch MuseTalk's source.- A connection and several GB of free space.
- MuseTalk's face and pose libraries, mmcv, mmpose and mmdet. The add-on doesn't install them. Until a set that matches the runtime's PyTorch and CUDA is installed in its environment, a job ends with an
internalerror naming what failed to import.
On Windows, quitting Subtitld during the first install can leave the download running in the background until it finishes.
Where it keeps everything
Inside are runtime (the Python environment), models and MuseTalk (the source). Uninstalling the add-on leaves this folder; delete it to free the space.
| System | Folder |
|---|---|
| Linux | ~/.cache/subtitld/musetalk |
| Windows | %USERPROFILE%\.cache\subtitld\musetalk |
Options
In Settings › Add-ons › MuseTalk (GPU lip-sync) › Configure.
v15MuseTalk model version. 1.5 gives sharper mouths; 1.0 is the original.
0Shifts the face crop vertically to tune mouth openness. Positive opens the mouth more, negative closes it. MuseTalk 1.0 only.
10Extra pixels added below the face crop (jaw region). MuseTalk 1.5 only.
8How many frames the graphics card makes at once. Higher is faster but needs more graphics memory; lower it after an out-of-memory error.
25Frame rate of the result. MuseTalk is trained at 25 fps and works best from 25 fps video; leave it at 25.
Half precision, which uses less graphics memory. Turn it off only if you see visual glitches.
Platforms
| System | Download |
|---|---|
| Linux x86-64 | 17.7 MB |
| Windows x86-64 | 8.6 MB |
There's no macOS build: MuseTalk needs CUDA. Both builds are made in CI, without running a lip-sync.
Troubleshooting
gpu_unavailableNo CUDA device was found. MuseTalk can't run on the CPU; it needs an NVIDIA card with working drivers.oomThe graphics card ran out of memory. Lower Batch size and keep Use float16 on.model_missingA model or MuseTalk's source couldn't be fetched. Installgitif it's missing. If the face-parsing weights fail to come from Google Drive, download79999_iter.pthyourself intomodels/face-parse-bisent/.network_unavailableA download, the source clone or the runtime install failed for network reasons. Try again: models already downloaded are kept, but an interrupted runtime install starts over.internalThe message says what. Most often no Python 3.10 or newer was found, the runtime install failed (the message ends with pip's output), or a MuseTalk library failed to import, usually mmcv, mmpose or mmdet. If the runtime stopped mid-job, lower Batch size.bad_paramsA file is missing, ffmpeg wasn't found, or no frames could be read from the video. Subtitld normally provides ffmpeg itself.
License
The add-on and MuseTalk are MIT, and the add-on describes them as fine for commercial use. The other models it downloads keep their own licenses, which it doesn't list; check them before commercial work.
Versions
- 0.2.0