Files

T

claudeandClaude Opus 5 fc4b761f52 Initial commit: Polish AI dubbing pipeline (ComfyUI + Higgs Audio V3)

Converts SRT subtitles into a Polish lektor-style voiceover using a local
ComfyUI instance running Higgs Audio V3, then muxes the result back into the
source video as a second audio track. Video and original audio are always
stream-copied; only the new voiceover track is encoded.

Contents:
  lektor_ai_comfy.py    main pipeline (standard library only, needs ffmpeg)
  lektor_ai_comfy_v2.py batched-context experiment (timing unresolved)
  lektor_ai.py          earlier version using the hosted Boson AI API
  install_comfyui.sh    reproduces the ComfyUI environment, versions pinned
  docs/                 requirements, installation, usage, architecture,
                        troubleshooting
  gudowski/             voice references (.wav) with their transcripts (.txt)

Source media and generated video are gitignored.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>

2026-08-09 14:29:53 +02:00

2.5 KiB

Raw Blame History

lektor_ai_comfy — Documentation

Converts SRT subtitles into a Polish AI voiceover in the lektor style (a single narrator read over the quietened original soundtrack) using a local ComfyUI instance running the Higgs Audio V3 TTS model, then muxes the result back into the video as a second audio track.

The source video is never re-encoded. Video and the original audio are stream-copied; only the new voiceover track is encoded.

Documentation map

Document	What it covers
requirements.md	Hardware, software, and the exact dependency surface
installation.md	Installing ComfyUI + the TTS engine, and verifying it
usage.md	Every CLI flag, the working modes, and practical recipes
architecture.md	How the pipeline works internally, end to end
troubleshooting.md	Diagnosed failure modes and their fixes

Quick start

Assuming ComfyUI is already installed (if not, see installation.md):

# 1. Start the TTS backend and leave it running
~/comfy-tts-app/start_comfyui.sh

# 2. In another terminal, generate the voiceover
cd ~/Programy/lektor_ai
python3 lektor_ai_comfy.py "input/film.mkv" --embedded-subs

The result is output/film_lektor.mkv with two audio tracks — Original and Polish — plus every subtitle track from the source.

The two halves of this project

The pieces are deliberately decoupled and it helps to keep them apart when diagnosing anything:

ComfyUI + Higgs Audio V3 — a local HTTP service on port 8189 that turns text into speech. Heavy: ~22 GB on disk, needs an NVIDIA GPU, lives in its own conda environment. It knows nothing about subtitles or video.

lektor_ai_comfy.py — a single-file script that parses subtitles, drives ComfyUI over HTTP, and does all timing and audio/video assembly with ffmpeg. It is pure Python standard library, runs on the system Python, and needs no conda environment or pip packages of its own.

So a TTS problem (bad pronunciation, hallucination, mumbling) belongs to the first half, and a timing, mixing, or muxing problem belongs to the second.

Conventions in these docs

Paths are written as they exist on the machine this was developed on:

~/Programy/lektor_ai      the project
~/comfy-tts-app/ComfyUI   the TTS backend
~/miniforge3              the conda distribution

If yours differ, update the constants at the top of lektor_ai_comfy.py — see installation.md.

2.5 KiB Raw Blame History