claudeandClaude Opus 5
fc4b761f52
Converts SRT subtitles into a Polish lektor-style voiceover using a local
ComfyUI instance running Higgs Audio V3, then muxes the result back into the
source video as a second audio track. Video and original audio are always
stream-copied; only the new voiceover track is encoded.
Contents:
lektor_ai_comfy.py main pipeline (standard library only, needs ffmpeg)
lektor_ai_comfy_v2.py batched-context experiment (timing unresolved)
lektor_ai.py earlier version using the hosted Boson AI API
install_comfyui.sh reproduces the ComfyUI environment, versions pinned
docs/ requirements, installation, usage, architecture,
troubleshooting
gudowski/ voice references (.wav) with their transcripts (.txt)
Source media and generated video are gitignored.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
67 lines
2.5 KiB
Markdown
67 lines
2.5 KiB
Markdown
# lektor_ai_comfy — Documentation
|
|
|
|
Converts SRT subtitles into a Polish AI voiceover in the *lektor* style (a single
|
|
narrator read over the quietened original soundtrack) using a local ComfyUI
|
|
instance running the Higgs Audio V3 TTS model, then muxes the result back into
|
|
the video as a second audio track.
|
|
|
|
The source video is never re-encoded. Video and the original audio are
|
|
stream-copied; only the new voiceover track is encoded.
|
|
|
|
## Documentation map
|
|
|
|
| Document | What it covers |
|
|
|---|---|
|
|
| [requirements.md](requirements.md) | Hardware, software, and the exact dependency surface |
|
|
| [installation.md](installation.md) | Installing ComfyUI + the TTS engine, and verifying it |
|
|
| [usage.md](usage.md) | Every CLI flag, the working modes, and practical recipes |
|
|
| [architecture.md](architecture.md) | How the pipeline works internally, end to end |
|
|
| [troubleshooting.md](troubleshooting.md) | Diagnosed failure modes and their fixes |
|
|
|
|
## Quick start
|
|
|
|
Assuming ComfyUI is already installed (if not, see
|
|
[installation.md](installation.md)):
|
|
|
|
```bash
|
|
# 1. Start the TTS backend and leave it running
|
|
~/comfy-tts-app/start_comfyui.sh
|
|
|
|
# 2. In another terminal, generate the voiceover
|
|
cd ~/Programy/lektor_ai
|
|
python3 lektor_ai_comfy.py "input/film.mkv" --embedded-subs
|
|
```
|
|
|
|
The result is `output/film_lektor.mkv` with two audio tracks — **Original** and
|
|
**Polish** — plus every subtitle track from the source.
|
|
|
|
## The two halves of this project
|
|
|
|
The pieces are deliberately decoupled and it helps to keep them apart when
|
|
diagnosing anything:
|
|
|
|
**ComfyUI + Higgs Audio V3** — a local HTTP service on port 8189 that turns text
|
|
into speech. Heavy: ~22 GB on disk, needs an NVIDIA GPU, lives in its own conda
|
|
environment. It knows nothing about subtitles or video.
|
|
|
|
**`lektor_ai_comfy.py`** — a single-file script that parses subtitles, drives
|
|
ComfyUI over HTTP, and does all timing and audio/video assembly with ffmpeg. It
|
|
is pure Python standard library, runs on the system Python, and needs no conda
|
|
environment or pip packages of its own.
|
|
|
|
So a TTS problem (bad pronunciation, hallucination, mumbling) belongs to the
|
|
first half, and a timing, mixing, or muxing problem belongs to the second.
|
|
|
|
## Conventions in these docs
|
|
|
|
Paths are written as they exist on the machine this was developed on:
|
|
|
|
```
|
|
~/Programy/lektor_ai the project
|
|
~/comfy-tts-app/ComfyUI the TTS backend
|
|
~/miniforge3 the conda distribution
|
|
```
|
|
|
|
If yours differ, update the constants at the top of `lektor_ai_comfy.py` — see
|
|
[installation.md](installation.md#configuring-the-script).
|