lukas/lektor-ai
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

Initial commit: Polish AI dubbing pipeline (ComfyUI + Higgs Audio V3)

Browse Source 
Converts SRT subtitles into a Polish lektor-style voiceover using a local
ComfyUI instance running Higgs Audio V3, then muxes the result back into the
source video as a second audio track. Video and original audio are always
stream-copied; only the new voiceover track is encoded.

Contents:
  lektor_ai_comfy.py    main pipeline (standard library only, needs ffmpeg)
  lektor_ai_comfy_v2.py batched-context experiment (timing unresolved)
  lektor_ai.py          earlier version using the hosted Boson AI API
  install_comfyui.sh    reproduces the ComfyUI environment, versions pinned
  docs/                 requirements, installation, usage, architecture,
                        troubleshooting
  gudowski/             voice references (.wav) with their transcripts (.txt)

Source media and generated video are gitignored.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
This commit is contained in:
claude
2026-08-09 14:29:53 +02:00
co-authored by Claude Opus 5
commit fc4b761f52
23 changed files with 4452 additions and 0 deletions
Download Patch File Download Diff File Expand all files Collapse all files
docs/README.md
+66
View File
@@ -0,0 +1,66 @@
# lektor_ai_comfy — Documentation
Converts SRT subtitles into a Polish AI voiceover in the *lektor* style (a single
narrator read over the quietened original soundtrack) using a local ComfyUI
instance running the Higgs Audio V3 TTS model, then muxes the result back into
the video as a second audio track.
The source video is never re-encoded. Video and the original audio are
stream-copied; only the new voiceover track is encoded.
## Documentation map
| Document | What it covers |
|---|---|
| [requirements.md](requirements.md) | Hardware, software, and the exact dependency surface |
| [installation.md](installation.md) | Installing ComfyUI + the TTS engine, and verifying it |
| [usage.md](usage.md) | Every CLI flag, the working modes, and practical recipes |
| [architecture.md](architecture.md) | How the pipeline works internally, end to end |
| [troubleshooting.md](troubleshooting.md) | Diagnosed failure modes and their fixes |
## Quick start
Assuming ComfyUI is already installed (if not, see
[installation.md](installation.md)):
```bash
# 1. Start the TTS backend and leave it running
~/comfy-tts-app/start_comfyui.sh
# 2. In another terminal, generate the voiceover
cd ~/Programy/lektor_ai
python3 lektor_ai_comfy.py "input/film.mkv" --embedded-subs
```
The result is `output/film_lektor.mkv` with two audio tracks — **Original** and
**Polish** — plus every subtitle track from the source.
## The two halves of this project
The pieces are deliberately decoupled and it helps to keep them apart when
diagnosing anything:
**ComfyUI + Higgs Audio V3** — a local HTTP service on port 8189 that turns text
into speech. Heavy: ~22 GB on disk, needs an NVIDIA GPU, lives in its own conda
environment. It knows nothing about subtitles or video.
**`lektor_ai_comfy.py`** — a single-file script that parses subtitles, drives
ComfyUI over HTTP, and does all timing and audio/video assembly with ffmpeg. It
is pure Python standard library, runs on the system Python, and needs no conda
environment or pip packages of its own.
So a TTS problem (bad pronunciation, hallucination, mumbling) belongs to the
first half, and a timing, mixing, or muxing problem belongs to the second.
## Conventions in these docs
Paths are written as they exist on the machine this was developed on:
```
~/Programy/lektor_ai the project
~/comfy-tts-app/ComfyUI the TTS backend
~/miniforge3 the conda distribution
```
If yours differ, update the constants at the top of `lektor_ai_comfy.py` — see
[installation.md](installation.md#configuring-the-script).
docs/architecture.md
+215
View File
@@ -0,0 +1,215 @@
# Architecture
How `lektor_ai_comfy.py` turns subtitles into a finished video, and why each
stage works the way it does. Most of the non-obvious choices here exist because
the straightforward version was tried first and broke.
```
SRT / embedded track
│ parse, clean, split dialogue
▼
subtitle blocks ──────────────► output/tts_input.srt
│ one HTTP job per block
▼
ComfyUI + Higgs Audio V3
│ per-clip atempo + fade
▼
clips/clip_NNNN.flac
│ adelay + amix, batched
▼
tts_assembled.flac
│ pass 1: mix with original audio
│ pass 2: stream-copy mux
▼
{stem}_lektor.mkv
```
## 1. Subtitle parsing
`parse_subtitle_blocks()` reads the SRT with `utf-8-sig` (tolerating a BOM),
splits on blank lines, and keeps blocks whose start falls inside the requested
range. Times become **relative to `--start`**, which is why a later `--rebuild`
must repeat the same `--start`/`--end`.
`_clean_text()` then normalises each block, in this order:
1. **Strip formatting tags.** `{...}` ASS overrides (`{\an8}`, `{\pos(x,y)}`)
and `<...>` HTML tags (`<i>`, `<font …>`).
2. **Detect dialogue.** Lines beginning `- ` mean two speakers and are kept
separate; ordinary wrapped lines are joined with a space.
3. **Sentence-case ALL-CAPS blocks**, which the model would otherwise shout.
4. **Capitalise a lowercase first letter**, since a fragment continuing from the
previous subtitle confuses the model.
5. **Add terminal punctuation** so the model produces a natural falling tail.
Step 1 must come first. `<i>ALL CAPS</i>` looks mixed-case while the tag is
still present — the `i` in `<i>` is a lowercase letter — so the caps detection
in step 3 would silently miss it.
`split_dialogue_blocks()` then divides a two-speaker block into one clip per
line, apportioning the block's duration by character count.
The result is written to `output/tts_input.srt`. That file is the source of
truth for block numbering: block *N* there is `clips/clip_NNNN.flac` and the
argument to `--regen N`.
## 2. Speech generation
For each block the script POSTs a workflow graph to `/prompt`, polls
`/history/{id}` every 3 s, and reads the resulting file directly from ComfyUI's
output directory.
The graph has five nodes:
| Node | Class | Role |
|---|---|---|
| 1 | `HiggsAudioV3EngineNode` | Model, temperature, top_p 0.95, top_k 50 |
| 2 | `LoadAudio` | Voice reference WAV from ComfyUI's input dir |
| 3 | `CharacterVoicesNode` | Pairs that audio with its transcript |
| 4 | `UnifiedTTSTextNode` | The text, the seed, `enable_audio_cache: False` |
| 5 | `SaveAudio` | Writes FLAC |
Three details matter:
- **`enable_audio_cache: False`** is essential. With caching on, ComfyUI returns
a previous result for repeated text and every `--regen` is a no-op.
- **The seed is `base_seed + block_index`**, so a fixed `--seed` is fully
reproducible while each clip still differs.
- **No language tag is sent.** Higgs Audio V3 detects Polish from the text
itself; its tokenizer has no `<|lang:pl|>` token, and a `[pl:]` prefix is
stripped by the character parser before it reaches the model. A `lang: en`
line in the ComfyUI log is cosmetic.
Each finished clip is optionally head-trimmed (`--strip-leading-ms`), then
sped up with `atempo` and given a short fade-out — **per clip, before
assembly**. Applying `atempo` to the assembled track instead would speed up the
silences between clips and shift every subsequent timestamp.
## 3. Placement
Clips are positioned with a push-forward cursor:
```python
actual_start = max(t_start, cursor)
cursor = actual_start + duration
```
A clip starts at its subtitle time, or when the previous clip finished if that
is later. Clips are **never trimmed** to their subtitle window — cutting speech
mid-sentence sounds far worse than running a little long. Overrun cascades
forward and then resolves by itself at the next natural gap in the subtitles,
because a gap lets the cursor fall back below the next `t_start`.
The console prints `[+1.20s behind SRT]` when drift occurs. Steady growth means
`--speed` is too low for the material.
## 4. Assembly
Each clip becomes an `adelay` filter input and everything is mixed with `amix`:
```
[0:a]adelay=1340:all=1[a0]; [1:a]adelay=4820:all=1[a1]; …
[a0][a1]…amix=inputs=N:duration=longest:normalize=0[out]
```
`normalize=0` is important — `amix` otherwise divides by the input count, which
would make every clip inaudibly quiet across hundreds of inputs.
Above 500 clips the work is **batched**, because one ffmpeg process cannot hold
thousands of open inputs. Each batch is assembled to its own temporary FLAC and
the batches are then mixed together.
Batches use **relative** delays: the first clip's start time is subtracted from
every clip in the batch, and re-applied when the batches are combined. Without
that subtraction, batch 7 would begin with 40 minutes of silence, wasting time
and making the progress bar useless.
## 5. Muxing
`merge_voiceover()` runs **two separate ffmpeg passes**. This is deliberate and
should not be collapsed back into one.
### Pass 1 — build the lektor mix
```
[0:a:0]<norm>,aformat=channel_layouts=stereo,volume=<orig>[orig_mix];
[1:a]aformat=channel_layouts=stereo,volume=<tts>[tts];
[orig_mix][tts]amix=inputs=2:duration=first:normalize=0[lektor]
```
Written to a temporary `*_mix.m4a`, AAC 192k, and deleted in a `finally` block.
**Stereo, not 5.1.** ffmpeg's AAC encoder does not signal a 5.1 channel layout —
the track comes back tagged `unknown` in every container tested, and players
that cannot route the channels play silence. Stereo is tagged correctly, plays
everywhere, and spends the 192 kbit/s on two channels instead of six. Track 0
keeps the untouched 5.1, so no surround information is lost.
**PTS normalisation.** MKV rips often start their audio at a non-zero PTS (one
source here begins at 4.043 s). `amix` cannot emit output before both inputs
have frames, so the voiceover would be delayed by that offset. When ffprobe
reports an offset the filter becomes:
```
asetpts=PTS-STARTPTS,adelay=<offset_ms>:all=1,aresample=async=1
```
`asetpts` rebases the stream to zero and `adelay` reinserts the offset as real
silence, so the background audio stays where it belongs while `amix` receives
frames from both inputs starting at zero.
`duration=first` with `orig_mix` listed first means the mix runs as long as the
film, not as long as the voiceover.
### Pass 2 — stream-copy mux
Video, original audio, the mix, and every subtitle track are copied into the
final MKV with `-c copy`. Nothing is re-encoded, and it runs at several hundred
times real time.
Two flags carry the weight here:
**`-max_interleave_delta 0`**, placed **after the maps and before the output
path**. It is an output-muxer option; before `-i` it silently configures the
demuxer and does nothing. Without it, sparse subtitle tracks make the muxer
abandon interleaving after its default 10-second window, and the lektor audio
ends up written hundreds of megabytes away from the video it belongs with —
measurably fine to ffmpeg, but silent in any player that reads sequentially.
**MKV output**, not MP4. MKV copies subtitles of any codec without conversion,
carries DTS and TrueHD in track 0, and avoids the `mov_text` conversion that
made large multi-subtitle files abort mid-mux.
### Why two passes
Doing the filtering and the subtitle muxing in one command fails on sources with
many subtitle tracks. The sparse subtitle streams stall the muxer while the
audio filter graph keeps producing, and ffmpeg aborts partway through with:
```
Error while filtering: Invalid argument
Failed to inject frame into filter network: Invalid argument
```
The message points at audio, but audio is not the cause. Splitting the work
keeps the filter graph free of subtitle streams and makes the final mux a pure
copy. It is also about ten times faster.
## Progress reporting
`_run_progress()` adds `-progress pipe:1 -nostats` before the output path, reads
`out_time_us=` lines from stdout, and rewrites a single line with `\r`. A daemon
thread drains stderr concurrently, so ffmpeg cannot deadlock on a full stderr
pipe while the main thread reads stdout.
## Design rules
Worth preserving, since each was learned by breaking it:
1. **Per-clip processing before assembly**, never on the assembled track.
2. **Never trim clips** to the subtitle window.
3. **Keep every clip file** — `--regen` depends on them.
4. **FLAC throughout**, encoding only the final voiceover track.
5. **Two audio tracks**, with the original stream-copied and untouched.
6. **Verify muxed output by byte position**, not only by volume. A track can
measure perfectly and still be unplayable.
docs/installation.md
+207
View File
@@ -0,0 +1,207 @@
# Installation
Two things need to exist: the **ComfyUI backend** (large, isolated, GPU-bound)
and the **script** (which needs nothing installed at all).
## Automated install
`install_comfyui.sh` in the project root reproduces the exact working
environment. It is idempotent — safe to re-run, and it skips any step already
satisfied.
```bash
cd ~/Programy/lektor_ai
./install_comfyui.sh
```
Roughly 30–60 minutes, mostly downloading. Expect ~22 GB.
### Options
| Flag | Effect |
|---|---|
| *(none)* | Pinned versions — ComfyUI `7cb784e0`, suite `v5.3.0`. Recommended. |
| `--latest` | Newest ComfyUI and suite instead. You are on your own for breakage. |
| `--skip-model` | Skip the 8.7 GB download; it happens on first generation anyway. |
| `--lock FILE` | Restore an exact package set from a `pip freeze` lockfile. |
| `--help` | Usage summary. |
### Changing where things go
```bash
CONDA_ROOT=/opt/miniforge3 COMFY_ROOT=/data/comfy ./install_comfyui.sh
```
| Variable | Default |
|---|---|
| `CONDA_ROOT` | `~/miniforge3` |
| `COMFY_ROOT` | `~/comfy-tts-app` |
| `ENV_NAME` | `comfy-tts` |
| `COMFY_PORT` | `8189` |
| `PYTHON_VERSION` | `3.12` |
If you change `COMFY_ROOT` or `COMFY_PORT`, update the script constants too —
see [Configuring the script](#configuring-the-script).
### Exact reproduction
`comfyui-requirements.lock.txt` pins all 232 packages. To rebuild a
byte-identical environment:
```bash
./install_comfyui.sh --lock comfyui-requirements.lock.txt
```
Regenerate it after intentional changes:
```bash
source ~/miniforge3/etc/profile.d/conda.sh && conda activate comfy-tts
pip freeze > comfyui-requirements.lock.txt
```
## What the installer does
Useful to know when something fails midway, since each step can be repeated on
its own.
1. **Preflight** — checks Linux, git, a downloader, GPU, disk space, ffmpeg.
Only missing git or a downloader is fatal; the rest warn.
2. **Miniforge** → `~/miniforge3`, batch mode. `conda init` is deliberately not
run, so your `.bashrc` is never touched.
3. **Conda env** `comfy-tts` with Python 3.12, then a guard that refuses to
continue if the wrong Python somehow ended up active.
4. **PyTorch** from the cu128 index, pinned to 2.11.0.
5. **ComfyUI** cloned and checked out at the pinned commit (detached HEAD — this
is expected, not an error), then its `requirements.txt` installed.
6. **TTS-Audio-Suite** cloned, checked out at `v5.3.0`, then **`python
install.py`** — its own resolver, which must not be replaced with a plain
pip install (see [requirements.md](requirements.md#dependency-surface)).
7. **Model** — `bosonai/higgs-audio-v3-tts-4b` into
`ComfyUI/models/TTS/higgs_audio_v3/`.
8. **Launcher** written to `$COMFY_ROOT/start_comfyui.sh`.
9. **Verification** — imports torch, torchaudio, transformers, safetensors,
soundfile, librosa and reports CUDA availability.
## Manual install
The equivalent by hand, if you would rather not use the script:
```bash
# Miniforge, self-contained, no shell integration
wget https://github.com/conda-forge/miniforge/releases/latest/download/Miniforge3-Linux-x86_64.sh
bash Miniforge3-Linux-x86_64.sh -b -p ~/miniforge3
source ~/miniforge3/etc/profile.d/conda.sh
conda create -n comfy-tts python=3.12 -y
conda activate comfy-tts
# PyTorch with the CUDA 12.8 runtime bundled
pip install torch==2.11.0 torchvision==0.26.0 torchaudio==2.11.0 \
--index-url https://download.pytorch.org/whl/cu128
# ComfyUI, pinned
mkdir -p ~/comfy-tts-app && cd ~/comfy-tts-app
git clone https://github.com/comfyanonymous/ComfyUI.git
cd ComfyUI && git checkout 7cb784e0f48784bb6ed588912e186e5ee1e9ee68
pip install -r requirements.txt
# The TTS node, pinned — note install.py, NOT requirements.txt
cd custom_nodes
git clone https://github.com/diodiogod/TTS-Audio-Suite.git
cd TTS-Audio-Suite && git checkout tags/v5.3.0
python install.py
```
The model downloads itself on first generation. To fetch it ahead of time:
```python
from huggingface_hub import snapshot_download
snapshot_download(
repo_id="bosonai/higgs-audio-v3-tts-4b",
local_dir="~/comfy-tts-app/ComfyUI/models/TTS/higgs_audio_v3/higgs-audio-v3-tts-4b",
)
```
## Running the backend
```bash
~/comfy-tts-app/start_comfyui.sh
```
which is just:
```bash
#!/bin/bash
source ~/miniforge3/etc/profile.d/conda.sh
conda activate comfy-tts
cd ~/comfy-tts-app/ComfyUI && python main.py --port 8189
```
Leave it running in its own terminal for the whole session — model load takes
tens of seconds and would otherwise be repeated. The web UI at
<http://127.0.0.1:8189> is useful for watching progress and reading errors, but
is not needed; the script drives everything over HTTP.
## Configuring the script
`lektor_ai_comfy.py` has four constants at the top. If you installed anywhere
other than the defaults, edit them:
```python
COMFY_URL = "http://127.0.0.1:8189"
COMFY_OUTPUT_DIR = Path("/home/lukas/comfy-tts-app/ComfyUI/output")
COMFY_INPUT_DIR = Path("/home/lukas/comfy-tts-app/ComfyUI/input")
VOICE_PROMPTS_DIR = Path("/home/lukas/Programy/lektor_ai/gudowski")
```
`COMFY_URL` can also be overridden per-run with `--comfy-url`. The two directory
constants cannot — the script reads generated audio directly from ComfyUI's
output directory and copies voice references into its input directory, so both
must be real paths on the same filesystem.
The script itself needs no installation. Ensure ffmpeg is present:
```bash
ffmpeg -version && ffprobe -version
```
If missing, and you would rather not use sudo:
```bash
conda install -n comfy-tts -c conda-forge ffmpeg
```
Note that this puts ffmpeg inside the conda env, so the script must then be run
with that env active.
## Verifying
```bash
# 1. Backend responds
curl -s http://127.0.0.1:8189/system_stats | head -c 200
# 2. GPU visible to PyTorch
source ~/miniforge3/etc/profile.d/conda.sh && conda activate comfy-tts
python -c "import torch; print(torch.__version__, torch.cuda.is_available())"
# -> 2.11.0+cu128 True
# 3. Model present
ls -la ~/comfy-tts-app/ComfyUI/models/TTS/higgs_audio_v3/higgs-audio-v3-tts-4b/model.safetensors
# 4. End-to-end on 60 seconds of a real file
cd ~/Programy/lektor_ai
python3 lektor_ai_comfy.py "input/film.mkv" --embedded-subs --end 01:00
```
The short run is the real test — it exercises subtitle parsing, TTS, assembly
and muxing, and finishes in a couple of minutes.
## Uninstalling
Nothing was installed system-wide, so removal is just deleting directories:
```bash
conda env remove -n comfy-tts
rm -rf ~/comfy-tts-app
rm -rf ~/miniforge3 # only if nothing else uses it
```
docs/requirements.md
+132
View File
@@ -0,0 +1,132 @@
# Requirements
## Hardware
| | Minimum | Notes |
|---|---|---|
| GPU | NVIDIA, 12 GB VRAM | Developed on an RTX 4070 Ti SUPER (16 GB) |
| Driver | 525+ | Needed by the CUDA 12.8 wheels |
| Disk | ~35 GB free | ~22 GB installed, plus working room for output |
| RAM | 16 GB | The final mux briefly peaks around 2.6 GB |
The CUDA **toolkit** is not required. The PyTorch cu128 wheels bundle their own
CUDA runtime, so the system `nvcc` version is irrelevant — this machine has CUDA
11.5 installed system-wide while PyTorch runs cu128 perfectly well. Only the
driver version matters.
CPU-only inference technically works but is slow enough to be impractical for a
feature-length film.
### Disk footprint
```
~/miniforge3/envs/comfy-tts 9.7 GB conda env (PyTorch and ~233 packages)
~/comfy-tts-app/ComfyUI 3.2 GB application code and dependencies
└── models/TTS 8.8 GB Higgs Audio V3 weights
------
21.7 GB
```
Output is roughly the size of the source video, since video is stream-copied. A
60-minute 10 Mbit/s episode produces about 5 GB. Intermediate FLAC clips add a
few hundred MB per film and are kept deliberately, so individual clips can be
regenerated without redoing a whole run (see
[usage.md](usage.md#working-modes)).
## Software
| Component | Version here | Notes |
|---|---|---|
| OS | Linux (Ubuntu 22.04) | Only Linux is supported by the installer |
| ComfyUI | 0.26.0 (`7cb784e0`) | Pinned |
| TTS-Audio-Suite | v5.3.0 (`09df1681`) | The only custom node installed |
| Higgs Audio V3 | `bosonai/higgs-audio-v3-tts-4b` | 8.7 GB, auto-downloaded |
| Python (ComfyUI) | 3.12.13 | conda-forge, in the `comfy-tts` env |
| PyTorch | 2.11.0+cu128 | With torchvision 0.26.0, torchaudio 2.11.0 |
| Python (script) | 3.8+ | System Python; 3.10.12 here |
| ffmpeg / ffprobe | 4.4+ | System install; 4.4.2 here |
## Dependency surface
This is the point most worth internalising, because it makes debugging much
simpler: **the two halves have almost nothing in common.**
### `lektor_ai_comfy.py` — standard library only
```python
argparse json random re shutil subprocess sys threading time uuid
pathlib.Path urllib.request
```
No third-party packages. No conda environment. No virtualenv. It runs on the
system Python:
```bash
/usr/bin/python3 lektor_ai_comfy.py --help # works, nothing installed
```
Its only external requirements are **ffmpeg** and **ffprobe** on `PATH`, and a
reachable ComfyUI instance. It talks to ComfyUI over plain HTTP with
`urllib.request` — there is no ComfyUI client library involved.
> **`requirements.txt` in the project root does not apply here.** It belongs to
> the older `lektor_ai.py`, which used the hosted Boson AI API and needed
> `openai>=1.0.0`. `lektor_ai_comfy.py` requires nothing from it.
### ComfyUI — a large, isolated environment
233 pip packages in the `comfy-tts` conda env. The important ones:
```
torch 2.11.0+cu128 torchaudio 2.11.0+cu128 torchvision 0.26.0+cu128
transformers 5.12.1 tokenizers 0.22.2 accelerate 1.14.0
numpy 2.2.6 scipy 1.18.0 librosa 0.11.0
soundfile 0.14.0 safetensors 0.8.0 einops 0.8.2
```
The conda environment itself holds only the 27 base conda-forge packages that
make up the Python runtime. Everything else arrives via pip. In particular,
`portaudio`, `libsamplerate`, `ffmpeg` and `espeak-ng` are **not** installed in
the environment despite appearing in the original planning notes — the system
ffmpeg is used and espeak-ng is absent entirely. Everything works without them.
TTS-Audio-Suite installs its dependencies through its own `install.py`, not
through its `requirements.txt`. That file deliberately omits packages needing
special handling — `librosa`, `descript-audio-codec`, `onnxruntime`,
`opencv-python` and others are installed with `--no-deps` because pip would
otherwise drag numpy backwards and break the rest of the environment. Never
substitute `pip install -r requirements.txt` for `python install.py`.
## Input requirements
**Video** — anything ffmpeg can demux. MKV and MP4 are the usual cases. The
container must support what gets muxed back in; output is always MKV, which
accepts every codec involved.
**Subtitles** — either an external `.srt`, or a text-based track inside the
video (`subrip`, `ass`, `ssa`, `webvtt`, `mov_text`). Image-based tracks
(`hdmv_pgs_subtitle` from Blu-ray, `dvd_subtitle`) contain bitmaps rather than
text and cannot drive TTS without OCR; the `--embedded-subs` picker marks them
as unusable. They are still copied to the output untouched.
**Voice reference** — a `.wav` plus a `.txt` holding its exact transcript, both
in `gudowski/`. Three are bundled:
| Voice | Length | Format |
|---|---|---|
| `gudowski` | 2:10 | 44.1 kHz stereo |
| `gudowski_short` (default) | 0:52 | 44.1 kHz stereo |
| `gudowski_short_2` | 0:28 | 44.1 kHz stereo |
The transcript matters as much as the audio — voice cloning quality drops
noticeably without an accurate one. The script warns if the `.txt` is missing.
## Network
Needed only at install time, for the ComfyUI and TTS-Audio-Suite clones, the pip
downloads, and the 8.7 GB model. Generation afterwards is entirely local; no
API keys, no accounts, no per-use cost.
The model is published under a research / non-commercial licence and may require
accepting its terms on huggingface.co, then `hf auth login`, before it will
download.
docs/troubleshooting.md
+295
View File
@@ -0,0 +1,295 @@
# Troubleshooting
Failure modes that have actually occurred here, with the diagnosis that
identified each one. Where a symptom is misleading, that is called out — several
of these point somewhere other than their cause.
## Audio problems in the finished file
### The lektor track is silent in the player
The single most misleading failure in this project. It has **two distinct
causes**, and volume measurements cannot tell them apart — both produce a track
that measures perfectly loud.
Diagnose by checking the declared channel layout and the interleaving, in that
order.
**Cause 1 — unknown channel layout**
```bash
ffprobe -v error -select_streams a -show_entries stream=index,codec_name,channels,channel_layout -of csv=p=0 out.mkv
```
```
1,eac3,6,5.1(side) ← original, fine
2,aac,6,unknown ← broken: 6 channels, no routing information
```
ffmpeg's AAC encoder does not signal a 5.1 layout, in any container. Players
cannot map the channels and output silence. The fix is the stereo lektor track
now used by default (`aformat=channel_layouts=stereo` on both inputs). A correct
track reports `2,stereo`.
**Cause 2 — broken interleaving**
Compare byte positions for the same timestamp:
```bash
ffprobe -read_intervals 00:20:00%+1 -select_streams v -show_entries packet=pos -of csv=p=0 out.mkv | head -1
ffprobe -read_intervals 00:20:00%+1 -select_streams a:1 -show_entries packet=pos -of csv=p=0 out.mkv | head -1
```
They should be within a few MB. Observed on a broken file:
| time | drift from video |
|---|---|
| 02:00 | 477 MB |
| 10:00 | **2315 MB** |
| 50:00 | 740 MB |
Sparse subtitle tracks make the muxer give up interleaving after its default
10-second window. ffmpeg reads the file fine, so every level measurement looks
healthy — but players read MKV sequentially and never reach that audio.
The fix is `-max_interleave_delta 0` as an **output** option. An existing file
can be repaired without regenerating anything:
```bash
ffmpeg -i broken.mkv -map 0 -c copy -max_interleave_delta 0 fixed.mkv
```
14 seconds and ~2.6 GB of transient RAM for a 60-minute file.
> Placement is the trap. Before `-i` the flag configures the demuxer, changes
> nothing, and produces byte-identical output — which is easily mistaken for
> "this fix doesn't work".
### The voiceover is late by a constant offset
The source audio starts at a non-zero PTS. Confirm:
```bash
ffprobe -v error -select_streams a:0 -show_entries stream=start_time -of csv=p=0 film.mkv
# 4.043000 ← not 0
```
`amix` cannot emit output until both inputs have frames, so the voiceover is
delayed by that amount. Handled automatically now
(`asetpts=PTS-STARTPTS,adelay=<offset>`); the run prints:
```
Audio PTS offset: 4.043s — normalising orig_mix for lektor track
```
Note that this affects **only** the mixed track. Track 0 is stream-copied and
was never wrong, so "the original audio is offset" and "the original audio
inside the lektor mix is offset" are different problems with different fixes.
### Voiceover drifts progressively out of sync
Not a mixing fault — clips are overrunning their windows and pushing each other
forward. The console shows it:
```
→ placed at 00:12:03,400, duration 3.10s [+2.40s behind SRT]
```
Raise `--speed` (try 1.3–1.4) and rebuild. Small drift is expected and
self-corrects at the next gap in the subtitles; only sustained growth is a
problem.
### The whole voiceover sounds too quiet or too loud
Adjust and rebuild — no regeneration needed:
```bash
python3 lektor_ai_comfy.py film.mkv --rebuild --orig-volume 0.5 --tts-volume 2.0
```
Track 0 is unaffected by these, always.
## Muxing failures
### `Failed to inject frame into filter network: Invalid argument`
```
Error while filtering: Invalid argument
Failed to inject frame into filter network: Invalid argument
Error while processing the decoded data for stream #0:1
```
Aborts partway through, at a **different timestamp each run**. The message
blames audio; audio is not the cause. It appears when filtering and
subtitle muxing happen in one ffmpeg command on a source with many subtitle
tracks — the sparse streams stall the muxer while the filter graph keeps
producing.
The varying failure point is the tell. A genuinely corrupt frame fails at the
same place every time.
Fixed by the two-pass structure in `merge_voiceover()`. If you are editing that
function, do not merge the passes back together.
### The mux fails on DTS or TrueHD audio
Only an issue if output is changed to MP4, which cannot carry those codecs. The
default MKV output handles them. Keep it.
## TTS problems
### A clip recites the reference transcript
The model occasionally speaks the contents of `gudowski.txt` instead of the
subtitle. Most likely at `--temperature 1.0`; `0.9` is the default for this
reason.
```bash
python3 lektor_ai_comfy.py film.mkv --regen 88
python3 lektor_ai_comfy.py film.mkv --rebuild
```
`--seed 0` randomises each run, so a regeneration gets a different sample.
### A clip is cut off mid-word
The token ceiling was reached. Raise it and regenerate just that clip:
```bash
python3 lektor_ai_comfy.py film.mkv --regen 214 --max-tokens 8192
```
### Every clip returns identical audio
`enable_audio_cache` is being set to `True` somewhere in
`build_text_workflow()`. It must stay `False` or ComfyUI serves cached results
and `--regen` does nothing.
### Stiff pauses at every comma
Baked into the model weights. There is no SSML support and no parameter for it.
Rephrasing the subtitle text is the only lever.
### Audible breath at the start of clips
```bash
python3 lektor_ai_comfy.py film.mkv --regen 5 --strip-leading-ms 120
```
Values of 80–150 ms usually suffice. Too high clips the first consonant.
### Voice cloning quality is poor
Check that the reference `.txt` exists and matches the `.wav` exactly. The run
prints:
```
Reference text loaded (930 chars)
```
or warns if it is missing. An inaccurate transcript degrades cloning noticeably.
## Subtitle problems
### Formatting tags are being spoken
`<i>`, `</i>` and `{\an8}` are stripped by `_strip_sub_tags()` before anything
else. If new tags appear, extend that function — and keep the stripping first,
because tags interfere with the ALL-CAPS detection that follows.
### ALL-CAPS text is being shouted
Handled by `_clean_text()`, but only after tags are removed. `<i>HELLO</i>`
looked mixed-case while the tag was present, because `i` is a lowercase letter.
If shouting persists, check for an unhandled tag type.
### `--embedded-subs` shows no usable tracks
The video only carries image-based subtitles (`hdmv_pgs_subtitle`,
`dvd_subtitle`). These are bitmaps and need OCR. Supply an external `.srt`
instead — the image tracks are still copied to the output.
### Wrong language selected
The picker lists language tags but they are only as good as the source's
metadata. `output/embedded_sub.srt` holds exactly what was extracted; check it
before committing to a full run.
## Backend problems
### `Connection refused` on port 8189
ComfyUI is not running. Start it and wait for the model to load:
```bash
~/comfy-tts-app/start_comfyui.sh
curl -s http://127.0.0.1:8189/system_stats | head -c 200
```
### ComfyUI crashed and the GPU is stuck
```bash
sudo fuser -k /dev/nvidia*
sudo nvidia-smi --gpu-reset
```
Then restart and resume from the last completed block:
```bash
python3 lektor_ai_comfy.py film.mkv --resume-from 312
```
### Out of VRAM
Nothing else should be using the GPU during a run. Restart ComfyUI to clear
fragmentation. The 4B model needs roughly 10–12 GB.
### `torch.cuda.is_available()` is False
```bash
source ~/miniforge3/etc/profile.d/conda.sh && conda activate comfy-tts
python -c "import torch; print(torch.__version__, torch.cuda.is_available())"
```
Expect `2.11.0+cu128 True`. If False, check `nvidia-smi` works and the driver is
525 or newer. The CUDA **toolkit** version is irrelevant — the wheels bundle
their own runtime, and this machine runs cu128 wheels against a system CUDA
11.5 without issue.
## Script problems
### `Provide an SRT subtitle file, or use --embedded-subs`
A mode that needs subtitles was given only a video. Either pass the SRT first,
or add `--embedded-subs`.
### `No such file or directory` on the video
Almost always an unquoted path with spaces or parentheses, or a wrong extension
(`.mp4` where the file is `.mkv`). Quote it and check with `ls`.
### `Missing clip files — regenerate them first`
`--rebuild` needs a complete set. The listed clips must be regenerated with
`--regen`, or `--resume-from` used to fill the gap.
### `No tts_input.srt found`
`--rebuild` and `--regen` require a previous full run in the same
`--output-dir`.
## General diagnostic method
The lesson from the silent-track investigation, which took several wrong turns:
1. **Measure, don't assume.** Level, layout, and byte position each reveal a
different class of fault.
2. **Isolate by subtraction.** Removing subtitles from the failing command is
what proved they were the cause, not the audio the error named.
3. **A varying failure point means state, not data.** Corrupt input fails in the
same place every time; buffering and interleaving problems move around.
4. **"ffmpeg reads it fine" is not "a player can play it."** Verify position, not
just content.
5. **Check flag placement.** Several ffmpeg options mean different things before
`-i` versus before the output path, and the wrong position often fails
silently rather than erroring.
docs/usage.md
+266
View File
@@ -0,0 +1,266 @@
# Usage
ComfyUI must be running before any command that generates speech
(`~/comfy-tts-app/start_comfyui.sh`). `--rebuild` is the exception — it only
touches ffmpeg and works with the backend stopped.
## Invocation
```
python3 lektor_ai_comfy.py [SRT] VIDEO [options]
```
Both positionals are optional in the argparse sense, because several modes do
not need an SRT file. The rule:
```bash
# External subtitle file — pass both, SRT first
python3 lektor_ai_comfy.py input/film.pl.srt input/film.mkv
# Subtitles inside the video — pass the video alone
python3 lektor_ai_comfy.py input/film.mkv --embedded-subs
# --rebuild and --regen also need only the video
python3 lektor_ai_comfy.py input/film.mkv --rebuild
```
When a single positional is given in a mode that needs no SRT, the script moves
it into the video slot automatically. Always quote paths containing spaces or
parentheses:
```bash
python3 lektor_ai_comfy.py "input/Star City (2026)/S01_E01 The Eyes.mkv" --embedded-subs
```
## Working modes
### Full generation (default)
Parses subtitles, generates one TTS clip per block, assembles, muxes.
```bash
python3 lektor_ai_comfy.py input/film.mkv --embedded-subs
```
Roughly real-time-ish on a 4070 Ti SUPER: a 60-minute episode of ~700 subtitle
blocks takes on the order of an hour. Each clip is saved as it completes, so a
crash never loses more than the clip in flight.
### `--embedded-subs`
Lists the video's subtitle tracks and prompts for one:
```
Available subtitle tracks in S01_E01 The Eyes.mkv:
[1] Stream #2: subrip [eng] — Forced
[2] Stream #3: subrip [eng]
[3] Stream #4: subrip [eng] — SDH
...
[30] Stream #31: subrip [pol]
Select track number: 30
```
The chosen track is extracted to `output/embedded_sub.srt` and then processed
exactly like an external file. Image-based tracks are listed but marked
unusable — they hold bitmaps, not text.
### `--rebuild`
Reassembles audio from existing clips and rebuilds the video, with no TTS at
all. This is the fast iteration loop — seconds to minutes rather than an hour.
```bash
python3 lektor_ai_comfy.py input/film.mkv --rebuild
python3 lektor_ai_comfy.py input/film.mkv --rebuild --orig-volume 0.5 --tts-volume 2.0
```
Use it after `--regen`, or to adjust the mix. It requires `output/tts_input.srt`
and a complete set of `output/clips/clip_NNNN.flac`, and reports which are
missing if any are.
If the original run used `--end`, pass it again so the same video fragment is
used.
### `--regen`
Regenerates specific clips by block number, leaving everything else alone.
```bash
python3 lektor_ai_comfy.py input/film.mkv --regen 12 22 41
python3 lektor_ai_comfy.py input/film.mkv --rebuild # then apply
```
Block numbers match `output/tts_input.srt` and the `clip_NNNN.flac` filenames.
Only the selected FLAC files are replaced — `--rebuild` afterwards is what
actually updates the video.
This is the intended fix for a bad clip. There is no automatic retry; when the
model hallucinates, mumbles, or truncates, note the number and regenerate it. A
different seed usually resolves it, and `--seed 0` (the default) randomises
every run.
### `--resume-from`
Continues a crashed run, loading clips `1..N-1` from disk and generating from
`N` onward.
```bash
python3 lektor_ai_comfy.py input/film.mkv --resume-from 47
```
The console prints `[47/693]` style progress, so the number to resume from is
the last one that completed.
### `--skip-tts`
Uses a pre-assembled audio file and goes straight to the mux.
```bash
python3 lektor_ai_comfy.py input/film.mkv --skip-tts output/tts_assembled.flac
```
## Options
### Range
| Flag | Default | Meaning |
|---|---|---|
| `--start MM:SS` | `00:00` | Start of the range to process |
| `--end MM:SS` | *(end)* | End of the range |
With `--end`, a matching fragment is extracted to `output/video_fragment.mkv`
and all TTS timestamps become relative to `--start`. Pass the same values to any
later `--rebuild`, or the audio will not line up.
`--end 01:00` is the quickest way to sanity-check a new voice or speed setting.
### Voice and model
| Flag | Default | Meaning |
|---|---|---|
| `--voice NAME\|PATH` | `gudowski_short` | Name in `gudowski/`, or a path to a WAV |
| `--seed N` | `0` | Base seed; `0` randomises each run |
| `--temperature T` | `0.9` | Sampling temperature |
| `--max-tokens N` | `4096` | Token ceiling per clip |
`--temperature` is worth understanding. At `1.0` the model sometimes recites the
reference transcript instead of the subtitle. `0.9` largely prevents that;
lower is more deterministic but flatter. A per-clip seed of `base_seed + index`
means a fixed `--seed` reproduces a run exactly.
Raise `--max-tokens` only if long clips are cut off mid-word.
### Timing and cleanup
| Flag | Default | Meaning |
|---|---|---|
| `--speed FACTOR` | `1.2` | Per-clip speed-up via `atempo` |
| `--strip-leading-ms MS` | `0` | Trim the start of each clip |
| `--trailing-fade-ms MS` | `80` | Fade the end of each clip |
`--speed` compresses speech so a clip is more likely to fit its subtitle window.
It is applied **per clip, before assembly** — never to the assembled track,
which would shift every timestamp by speeding up the silences too.
`--strip-leading-ms` removes an inhale artifact at clip starts; try 80–150 if
you hear one. `--trailing-fade-ms` suppresses tail pops; `0` disables it.
### Mixing
| Flag | Default | Meaning |
|---|---|---|
| `--orig-volume 0-1` | `0.60` | Original audio level inside the lektor mix |
| `--tts-volume 0-2` | `1.8` | Voiceover level inside the lektor mix |
These affect **Track 1 only**. Track 0 is always the untouched original, so
nothing is lost by mixing aggressively. Changing them needs only `--rebuild`.
Typical starting point for dialogue-heavy material: `--orig-volume 0.5
--tts-volume 2.0`. For quiet sources, `--orig-volume 1.0 --tts-volume 1.4`.
### Paths
| Flag | Default | Meaning |
|---|---|---|
| `--output-dir DIR` | `output` | Where everything is written |
| `--comfy-url URL` | `http://127.0.0.1:8189` | Backend address |
## Output
```
output/
tts_input.srt processed text actually sent to TTS — inspect this
embedded_sub.srt extracted embedded subtitle (--embedded-subs only)
clips/clip_NNNN.flac one lossless clip per block, kept for --regen
tts_assembled.flac all clips at their final positions, lossless
video_fragment.mkv extracted segment (--end only)
{stem}_lektor.mkv the finished video
```
The finished file contains:
| Track | Content |
|---|---|
| Video | Stream-copied, untouched |
| Audio 0 | **Original** — stream-copied, full 5.1 if the source had it |
| Audio 1 | **Polish** — lektor mix, AAC 192k stereo, `language=pol` |
| Subtitles | Every track from the source, stream-copied |
The lektor track is stereo deliberately; see
[troubleshooting.md](troubleshooting.md#the-lektor-track-is-silent-in-the-player).
FLAC is used for every intermediate so repeated processing stays lossless. Only
the final voiceover track is ever encoded.
## Recipes
**Check a voice before committing to a full run**
```bash
python3 lektor_ai_comfy.py input/film.mkv --embedded-subs --end 02:00 --voice gudowski
```
**Tune the mix without regenerating speech**
```bash
python3 lektor_ai_comfy.py input/film.mkv --rebuild --orig-volume 0.45 --tts-volume 2.1
```
**Fix a handful of bad clips**
```bash
# read output/tts_input.srt, note the offending block numbers
python3 lektor_ai_comfy.py input/film.mkv --regen 88 91 104
python3 lektor_ai_comfy.py input/film.mkv --rebuild
```
**Reproduce a run exactly**
```bash
python3 lektor_ai_comfy.py input/film.mkv --embedded-subs --seed 123456
```
**Speech running long and overlapping the next line**
```bash
python3 lektor_ai_comfy.py input/film.mkv --embedded-subs --speed 1.35
```
## Reading the console
```
[1/4] Parsing subtitles 00:00–end…
693 entries after dialogue splitting.
[2/4] Generating TTS (693 clips, speed 1.2×)…
[12/693] 00:01:34,120 'Nie wiem, o czym mówisz.'
Waiting for a3f8b201… done.
→ placed at 00:01:34,120, duration 1.84s
```
`→ placed at` reveals timing drift. A clip that overruns its window pushes the
next one forward, shown as `[+1.20s behind SRT]`. Small values are normal and
self-correct at the next natural gap in the subtitles; values that keep growing
mean `--speed` is too low for the material.
Assembly then shows a progress bar per batch (clips are assembled 500 at a time
to stay under the open-file limit), followed by the two-pass mux.