claudeandClaude Opus 5
e201aadc3e
Text processing before TTS:
- dictionary.txt rewrites words and phrases (km/h, np., units). Rules compile
into one alternation applied in a single pass, so replacements never chain
and longest-literal wins. Case-sensitive, whole-word.
- A per-queue second dictionary, written by the GUI into the first video's job
folder and merged with the shared one (--dictionary-extra).
- ALL-CAPS blocks become Title Case rather than sentence case: lowercasing
destroyed proper nouns inside captions ("NASA - MARGO MADISON") and invited
the normalizer to "correct" the name. Paired clips in output/clips/all-caps
showed the TTS reads both casings the same.
normalizer/ — digits to spoken Polish, in its own venv because it needs
transformers>=5, which must not touch ComfyUI's pinned versions:
- Two backends. Plain integers go to num2words for the value and a general
Polish LLM for the case; times, dates, decimals and identifiers go to a 0.6B
fine-tune. Measured on 40 real subtitle lines, each is clearly better than
the other on its half. 74% of numbered blocks take the first leg.
- Guards on every answer: runaway length, repetition loops, invalid bytes,
leftover digits, a lost minus sign, a changed number, a lost word. A failure
falls back to num2words, which is nominative but never wrong about the value.
- ComfyUI is asked to unload before normalizing; a normalizer that does not fit
in VRAM is spilled to RAM silently at 5x the cost.
Backend lifecycle:
- ComfyUI is found by port through /proc, so a backend started by hand is
managed too. SIGTERM escalates to SIGKILL, and a suspended process is killed
outright since it never runs its handlers.
- A clip that kills the backend is retried three times, each with a different
seed (the aborts are seed-dependent, not text-dependent), then skipped and
reported so an overnight queue keeps moving.
- start_comfyui.sh execs python, so the script's PID is ComfyUI's.
- GUI gains ComfyUI and Queue dictionary buttons; quitting stops the backend.
--resume-from now reuses the job folder's tts_input.srt instead of reparsing
the subtitles, so block N still pairs with clip_NNNN.flac.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
76 lines
3.1 KiB
Markdown
76 lines
3.1 KiB
Markdown
# lektor_ai_comfy — Documentation
|
|
|
|
Converts SRT subtitles into a Polish AI voiceover in the *lektor* style (a single
|
|
narrator read over the quietened original soundtrack) using a local ComfyUI
|
|
instance running the Higgs Audio V3 TTS model, then muxes the result back into
|
|
the video as a second audio track.
|
|
|
|
The source video is never re-encoded. Video and the original audio are
|
|
stream-copied; only the new voiceover track is encoded.
|
|
|
|
## Documentation map
|
|
|
|
| Document | What it covers |
|
|
|---|---|
|
|
| [requirements.md](requirements.md) | Hardware, software, and the exact dependency surface |
|
|
| [installation.md](installation.md) | Installing ComfyUI + the TTS engine, and verifying it |
|
|
| [usage.md](usage.md) | Every CLI flag, the working modes, and practical recipes |
|
|
| [architecture.md](architecture.md) | How the pipeline works internally, end to end |
|
|
| [troubleshooting.md](troubleshooting.md) | Diagnosed failure modes and their fixes |
|
|
| [../normalizer/README.md](../normalizer/README.md) | The Polish number normalizer, its own component |
|
|
|
|
## Quick start
|
|
|
|
Assuming ComfyUI is already installed (if not, see
|
|
[installation.md](installation.md)):
|
|
|
|
```bash
|
|
# 1. Start the TTS backend and leave it running
|
|
~/comfy-tts-app/start_comfyui.sh
|
|
|
|
# 2. In another terminal, generate the voiceover
|
|
cd ~/Programy/lektor_ai
|
|
python3 lektor_ai_comfy.py "input/film.mkv" --embedded-subs
|
|
```
|
|
|
|
The result is `output/film_lektor.mkv` with two audio tracks — **Original** and
|
|
**Polish** — plus every subtitle track from the source.
|
|
|
|
## The three parts of this project
|
|
|
|
The pieces are deliberately decoupled — each in its own environment, talking
|
|
over a process boundary — and it helps to keep them apart when diagnosing
|
|
anything:
|
|
|
|
**ComfyUI + Higgs Audio V3** — a local HTTP service on port 8189 that turns text
|
|
into speech. Heavy: ~22 GB on disk, needs an NVIDIA GPU, lives in its own conda
|
|
environment. It knows nothing about subtitles or video.
|
|
|
|
**`lektor_ai_comfy.py`** — a single-file script that parses subtitles, drives
|
|
ComfyUI over HTTP, and does all timing and audio/video assembly with ffmpeg. It
|
|
is pure Python standard library, runs on the system Python, and needs no conda
|
|
environment or pip packages of its own.
|
|
|
|
**`normalizer/`** — an optional component that spells digits out as Polish words
|
|
before they reach the TTS. Its own venv (~6.9 GB, mostly CUDA libraries) and
|
|
its own 1.1 GB model, because it needs `transformers >= 5` and the ComfyUI
|
|
environment must not be disturbed. Driven as a subprocess for one stage of each
|
|
run, then closed.
|
|
|
|
So a TTS problem (bad pronunciation, hallucination, mumbling) belongs to the
|
|
first part, a timing, mixing or muxing problem to the second, and a number read
|
|
in the wrong case to the third.
|
|
|
|
## Conventions in these docs
|
|
|
|
Paths are written as they exist on the machine this was developed on:
|
|
|
|
```
|
|
~/Programy/lektor_ai the project
|
|
~/comfy-tts-app/ComfyUI the TTS backend
|
|
~/miniforge3 the conda distribution
|
|
```
|
|
|
|
If yours differ, update the constants at the top of `lektor_ai_comfy.py` — see
|
|
[installation.md](installation.md#configuring-the-script).
|