claudeandClaude Opus 5
97866e6623
install_comfyui.sh becomes install_backend.sh and now builds everything the app needs, not just the speech half: - Step 7 runs normalizer/setup.sh --model, which fetches both models: the open inflection model first, so the install gets as far as it can before anything can ask for credentials, then the gated fine-tune. - A missing Hugging Face account is no longer fatal. Accepting a licence is the one thing an installer cannot do, so the rest installs, the verification reports "fallback only (gated model missing)", and the summary prints the three commands that finish it. - --skip-normalizer, disk requirement 35 -> 55 GB, removal instructions and the step list updated to match. Queue logging — queue.log in the first queued video's job folder, which for a single-video run is its own: - Records what went wrong rather than what happened: warnings, tracebacks, ffmpeg errors, guard rejections, clips cut off mid-word or that crashed the backend, and the --regen line carrying their numbers. - ComfyUI failures *and* restarts, so the log shows the recovery rather than reading as though the run simply stopped. GUI-initiated backend actions go in too; they never pass through a job's pipe. - A failure also dumps the job's last 15 lines. Keyword matching alone is not enough — ffmpeg says "Invalid data found when processing input" and nothing that looks like an error. UI: - The output pane follows new output only while already at the bottom. Scroll up and it stays there; scroll back down and it resumes. Reading an earlier warning during a run was impossible while every ffmpeg progress frame yanked the view to the end. - The ComfyUI button's status dot is drawn as its own canvas item, so it can be green (answering) or amber (holds the port, does not answer). A "." inside the label could only ever be the same grey as the words. comfyui.log is capped at 120,000 lines, trimmed on backend start and every five minutes. Rewritten in place, never replaced: tee holds it open in append mode, so swapping the inode would leave the backend writing to an unlinked file. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
lektor_ai_comfy — Documentation
Converts SRT subtitles into a Polish AI voiceover in the lektor style (a single narrator read over the quietened original soundtrack) using a local ComfyUI instance running the Higgs Audio V3 TTS model, then muxes the result back into the video as a second audio track.
The source video is never re-encoded. Video and the original audio are stream-copied; only the new voiceover track is encoded.
Documentation map
| Document | What it covers |
|---|---|
| requirements.md | Hardware, software, and the exact dependency surface |
| installation.md | Installing ComfyUI + the TTS engine, and verifying it |
| usage.md | Every CLI flag, the working modes, and practical recipes |
| architecture.md | How the pipeline works internally, end to end |
| troubleshooting.md | Diagnosed failure modes and their fixes |
| ../normalizer/README.md | The Polish number normalizer, its own component |
Quick start
Assuming ComfyUI is already installed (if not, see installation.md):
# 1. Start the TTS backend and leave it running
~/comfy-tts-app/start_comfyui.sh
# 2. In another terminal, generate the voiceover
cd ~/Programy/lektor_ai
python3 lektor_ai_comfy.py "input/film.mkv" --embedded-subs
The result is output/film_lektor.mkv with two audio tracks — Original and
Polish — plus every subtitle track from the source.
The three parts of this project
The pieces are deliberately decoupled — each in its own environment, talking over a process boundary — and it helps to keep them apart when diagnosing anything:
ComfyUI + Higgs Audio V3 — a local HTTP service on port 8189 that turns text into speech. Heavy: ~22 GB on disk, needs an NVIDIA GPU, lives in its own conda environment. It knows nothing about subtitles or video.
lektor_ai_comfy.py — a single-file script that parses subtitles, drives
ComfyUI over HTTP, and does all timing and audio/video assembly with ffmpeg. It
is pure Python standard library, runs on the system Python, and needs no conda
environment or pip packages of its own.
normalizer/ — an optional component that spells digits out as Polish words
before they reach the TTS. Its own venv (~6.9 GB, mostly CUDA libraries) and
its own 1.1 GB model, because it needs transformers >= 5 and the ComfyUI
environment must not be disturbed. Driven as a subprocess for one stage of each
run, then closed.
So a TTS problem (bad pronunciation, hallucination, mumbling) belongs to the first part, a timing, mixing or muxing problem to the second, and a number read in the wrong case to the third.
Conventions in these docs
Paths are written as they exist on the machine this was developed on:
~/Programy/lektor_ai the project
~/comfy-tts-app/ComfyUI the TTS backend
~/miniforge3 the conda distribution
If yours differ, update the constants at the top of lektor_ai_comfy.py — see
installation.md.