claudeandClaude Opus 5
fc4b761f52
Converts SRT subtitles into a Polish lektor-style voiceover using a local
ComfyUI instance running Higgs Audio V3, then muxes the result back into the
source video as a second audio track. Video and original audio are always
stream-copied; only the new voiceover track is encoded.
Contents:
lektor_ai_comfy.py main pipeline (standard library only, needs ffmpeg)
lektor_ai_comfy_v2.py batched-context experiment (timing unresolved)
lektor_ai.py earlier version using the hosted Boson AI API
install_comfyui.sh reproduces the ComfyUI environment, versions pinned
docs/ requirements, installation, usage, architecture,
troubleshooting
gudowski/ voice references (.wav) with their transcripts (.txt)
Source media and generated video are gitignored.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
lektor_ai_comfy — Documentation
Converts SRT subtitles into a Polish AI voiceover in the lektor style (a single narrator read over the quietened original soundtrack) using a local ComfyUI instance running the Higgs Audio V3 TTS model, then muxes the result back into the video as a second audio track.
The source video is never re-encoded. Video and the original audio are stream-copied; only the new voiceover track is encoded.
Documentation map
| Document | What it covers |
|---|---|
| requirements.md | Hardware, software, and the exact dependency surface |
| installation.md | Installing ComfyUI + the TTS engine, and verifying it |
| usage.md | Every CLI flag, the working modes, and practical recipes |
| architecture.md | How the pipeline works internally, end to end |
| troubleshooting.md | Diagnosed failure modes and their fixes |
Quick start
Assuming ComfyUI is already installed (if not, see installation.md):
# 1. Start the TTS backend and leave it running
~/comfy-tts-app/start_comfyui.sh
# 2. In another terminal, generate the voiceover
cd ~/Programy/lektor_ai
python3 lektor_ai_comfy.py "input/film.mkv" --embedded-subs
The result is output/film_lektor.mkv with two audio tracks — Original and
Polish — plus every subtitle track from the source.
The two halves of this project
The pieces are deliberately decoupled and it helps to keep them apart when diagnosing anything:
ComfyUI + Higgs Audio V3 — a local HTTP service on port 8189 that turns text into speech. Heavy: ~22 GB on disk, needs an NVIDIA GPU, lives in its own conda environment. It knows nothing about subtitles or video.
lektor_ai_comfy.py — a single-file script that parses subtitles, drives
ComfyUI over HTTP, and does all timing and audio/video assembly with ffmpeg. It
is pure Python standard library, runs on the system Python, and needs no conda
environment or pip packages of its own.
So a TTS problem (bad pronunciation, hallucination, mumbling) belongs to the first half, and a timing, mixing, or muxing problem belongs to the second.
Conventions in these docs
Paths are written as they exist on the machine this was developed on:
~/Programy/lektor_ai the project
~/comfy-tts-app/ComfyUI the TTS backend
~/miniforge3 the conda distribution
If yours differ, update the constants at the top of lektor_ai_comfy.py — see
installation.md.