lukas/lektor-ai
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

Initial commit: Polish AI dubbing pipeline (ComfyUI + Higgs Audio V3)

Browse Source 
Converts SRT subtitles into a Polish lektor-style voiceover using a local
ComfyUI instance running Higgs Audio V3, then muxes the result back into the
source video as a second audio track. Video and original audio are always
stream-copied; only the new voiceover track is encoded.

Contents:
  lektor_ai_comfy.py    main pipeline (standard library only, needs ffmpeg)
  lektor_ai_comfy_v2.py batched-context experiment (timing unresolved)
  lektor_ai.py          earlier version using the hosted Boson AI API
  install_comfyui.sh    reproduces the ComfyUI environment, versions pinned
  docs/                 requirements, installation, usage, architecture,
                        troubleshooting
  gudowski/             voice references (.wav) with their transcripts (.txt)

Source media and generated video are gitignored.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
This commit is contained in:
claude
2026-08-09 14:29:53 +02:00
co-authored by Claude Opus 5
commit fc4b761f52
23 changed files with 4452 additions and 0 deletions
Download Patch File Download Diff File Expand all files Collapse all files
docs/README.md
+66
View File
@@ -0,0 +1,66 @@
# lektor_ai_comfy — Documentation
Converts SRT subtitles into a Polish AI voiceover in the *lektor* style (a single
narrator read over the quietened original soundtrack) using a local ComfyUI
instance running the Higgs Audio V3 TTS model, then muxes the result back into
the video as a second audio track.
The source video is never re-encoded. Video and the original audio are
stream-copied; only the new voiceover track is encoded.
## Documentation map
| Document | What it covers |
|---|---|
| [requirements.md](requirements.md) | Hardware, software, and the exact dependency surface |
| [installation.md](installation.md) | Installing ComfyUI + the TTS engine, and verifying it |
| [usage.md](usage.md) | Every CLI flag, the working modes, and practical recipes |
| [architecture.md](architecture.md) | How the pipeline works internally, end to end |
| [troubleshooting.md](troubleshooting.md) | Diagnosed failure modes and their fixes |
## Quick start
Assuming ComfyUI is already installed (if not, see
[installation.md](installation.md)):
```bash
# 1. Start the TTS backend and leave it running
~/comfy-tts-app/start_comfyui.sh
# 2. In another terminal, generate the voiceover
cd ~/Programy/lektor_ai
python3 lektor_ai_comfy.py "input/film.mkv" --embedded-subs
```
The result is `output/film_lektor.mkv` with two audio tracks — **Original** and
**Polish** — plus every subtitle track from the source.
## The two halves of this project
The pieces are deliberately decoupled and it helps to keep them apart when
diagnosing anything:
**ComfyUI + Higgs Audio V3** — a local HTTP service on port 8189 that turns text
into speech. Heavy: ~22 GB on disk, needs an NVIDIA GPU, lives in its own conda
environment. It knows nothing about subtitles or video.
**`lektor_ai_comfy.py`** — a single-file script that parses subtitles, drives
ComfyUI over HTTP, and does all timing and audio/video assembly with ffmpeg. It
is pure Python standard library, runs on the system Python, and needs no conda
environment or pip packages of its own.
So a TTS problem (bad pronunciation, hallucination, mumbling) belongs to the
first half, and a timing, mixing, or muxing problem belongs to the second.
## Conventions in these docs
Paths are written as they exist on the machine this was developed on:
```
~/Programy/lektor_ai the project
~/comfy-tts-app/ComfyUI the TTS backend
~/miniforge3 the conda distribution
```
If yours differ, update the constants at the top of `lektor_ai_comfy.py` — see
[installation.md](installation.md#configuring-the-script).