Files

T

claudeandClaude Opus 5 e201aadc3e Add subtitle dictionaries, Polish number normalizer, ComfyUI lifecycle control

Text processing before TTS:

- dictionary.txt rewrites words and phrases (km/h, np., units). Rules compile
  into one alternation applied in a single pass, so replacements never chain
  and longest-literal wins. Case-sensitive, whole-word.
- A per-queue second dictionary, written by the GUI into the first video's job
  folder and merged with the shared one (--dictionary-extra).
- ALL-CAPS blocks become Title Case rather than sentence case: lowercasing
  destroyed proper nouns inside captions ("NASA - MARGO MADISON") and invited
  the normalizer to "correct" the name. Paired clips in output/clips/all-caps
  showed the TTS reads both casings the same.

normalizer/ — digits to spoken Polish, in its own venv because it needs
transformers>=5, which must not touch ComfyUI's pinned versions:

- Two backends. Plain integers go to num2words for the value and a general
  Polish LLM for the case; times, dates, decimals and identifiers go to a 0.6B
  fine-tune. Measured on 40 real subtitle lines, each is clearly better than
  the other on its half. 74% of numbered blocks take the first leg.
- Guards on every answer: runaway length, repetition loops, invalid bytes,
  leftover digits, a lost minus sign, a changed number, a lost word. A failure
  falls back to num2words, which is nominative but never wrong about the value.
- ComfyUI is asked to unload before normalizing; a normalizer that does not fit
  in VRAM is spilled to RAM silently at 5x the cost.

Backend lifecycle:

- ComfyUI is found by port through /proc, so a backend started by hand is
  managed too. SIGTERM escalates to SIGKILL, and a suspended process is killed
  outright since it never runs its handlers.
- A clip that kills the backend is retried three times, each with a different
  seed (the aborts are seed-dependent, not text-dependent), then skipped and
  reported so an overnight queue keeps moving.
- start_comfyui.sh execs python, so the script's PID is ComfyUI's.
- GUI gains ComfyUI and Queue dictionary buttons; quitting stops the backend.

--resume-from now reuses the job folder's tts_input.srt instead of reparsing
the subtitles, so block N still pairs with clip_NNNN.flac.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>

2026-08-12 21:55:31 +02:00

6.7 KiB

Raw Blame History

Requirements

Hardware

	Minimum	Notes
GPU	NVIDIA, 12 GB VRAM	Developed on an RTX 4070 Ti SUPER (16 GB)
Driver	525+	Needed by the CUDA 12.8 wheels
Disk	~35 GB free	~22 GB installed (+8 GB with the number normalizer)
RAM	16 GB	The final mux briefly peaks around 2.6 GB

The CUDA toolkit is not required. The PyTorch cu128 wheels bundle their own CUDA runtime, so the system nvcc version is irrelevant — this machine has CUDA 11.5 installed system-wide while PyTorch runs cu128 perfectly well. Only the driver version matters.

CPU-only inference technically works but is slow enough to be impractical for a feature-length film.

Disk footprint

~/miniforge3/envs/comfy-tts    9.7 GB   conda env (PyTorch and ~233 packages)
~/comfy-tts-app/ComfyUI        3.2 GB   application code and dependencies
  └── models/TTS               8.8 GB   Higgs Audio V3 weights
                              ------
                              21.7 GB

lektor_ai/normalizer/venv      6.9 GB   optional: number normalizer env
lektor_ai/normalizer/model     1.1 GB   optional: its weights

Output is roughly the size of the source video, since video is stream-copied. A 60-minute 10 Mbit/s episode produces about 5 GB. Intermediate FLAC clips add a few hundred MB per film and are kept deliberately, so individual clips can be regenerated without redoing a whole run (see usage.md).

Software

Component	Version here	Notes
OS	Linux (Ubuntu 22.04)	Only Linux is supported by the installer
ComfyUI	0.26.0 (`7cb784e0`)	Pinned
TTS-Audio-Suite	v5.3.0 (`09df1681`)	The only custom node installed
Higgs Audio V3	`bosonai/higgs-audio-v3-tts-4b`	8.7 GB, auto-downloaded
Python (ComfyUI)	3.12.13	conda-forge, in the `comfy-tts` env
PyTorch	2.11.0+cu128	With torchvision 0.26.0, torchaudio 2.11.0
Python (script)	3.8+	System Python; 3.10.12 here
ffmpeg / ffprobe	4.4+	System install; 4.4.2 here

Dependency surface

This is the point most worth internalising, because it makes debugging much simpler: the three parts have almost nothing in common.

`lektor_ai_comfy.py` — standard library only

argparse  configparser  json  os  queue  random  re  shlex  shutil  subprocess
sys  threading  time  uuid    pathlib.Path    urllib.request

No third-party packages. No conda environment. No virtualenv. It runs on the system Python:

/usr/bin/python3 lektor_ai_comfy.py --help    # works, nothing installed

Its only external requirements are ffmpeg and ffprobe on PATH, and a reachable ComfyUI instance. It talks to ComfyUI over plain HTTP with urllib.request — there is no ComfyUI client library involved.

requirements.txt in the project root does not apply here. It belongs to the older lektor_ai.py, which used the hosted Boson AI API and needed openai>=1.0.0. lektor_ai_comfy.py requires nothing from it. normalizer/requirements.txt is a different file and does apply — to the normalizer's own venv, never to the system Python.

`normalizer/` — optional third environment

torch 2.11.0+cu128      transformers 5.15.0     accelerate 1.14.0
num2words 0.5.x         (fallback path only)

Its own venv at normalizer/venv (~6.9 GB — the cu128 wheels bundle their own CUDA runtime, again) plus normalizer/model (1.1 GB). Separate from ComfyUI's environment because it hard-requires transformers >= 5 — 4.x tokenizes the Qwen3 chat template differently and silently produces wrong output. lektor_ai_comfy.py never imports it; it starts it as a subprocess and talks JSON lines over a pipe, so the version conflict never has to be resolved.

Entirely optional: without it the pipeline uses a num2words fallback, and with numbers = no it is not started at all.

ComfyUI — a large, isolated environment

233 pip packages in the comfy-tts conda env. The important ones:

torch 2.11.0+cu128      torchaudio 2.11.0+cu128     torchvision 0.26.0+cu128
transformers 5.12.1     tokenizers 0.22.2           accelerate 1.14.0
numpy 2.2.6             scipy 1.18.0                librosa 0.11.0
soundfile 0.14.0        safetensors 0.8.0           einops 0.8.2

The conda environment itself holds only the 27 base conda-forge packages that make up the Python runtime. Everything else arrives via pip. In particular, portaudio, libsamplerate, ffmpeg and espeak-ng are not installed in the environment despite appearing in the original planning notes — the system ffmpeg is used and espeak-ng is absent entirely. Everything works without them.

TTS-Audio-Suite installs its dependencies through its own install.py, not through its requirements.txt. That file deliberately omits packages needing special handling — librosa, descript-audio-codec, onnxruntime, opencv-python and others are installed with --no-deps because pip would otherwise drag numpy backwards and break the rest of the environment. Never substitute pip install -r requirements.txt for python install.py.

Input requirements

Video — anything ffmpeg can demux. MKV and MP4 are the usual cases. The container must support what gets muxed back in; output is always MKV, which accepts every codec involved.

Subtitles — either an external .srt, or a text-based track inside the video (subrip, ass, ssa, webvtt, mov_text). Image-based tracks (hdmv_pgs_subtitle from Blu-ray, dvd_subtitle) contain bitmaps rather than text and cannot drive TTS without OCR; the --embedded-subs picker marks them as unusable. They are still copied to the output untouched.

Voice reference — a .wav plus a .txt holding its exact transcript, both in voices/. Three are bundled:

Voice	Length	Format
`gudowski`	2:10	44.1 kHz stereo
`gudowski_short` (default)	0:52	44.1 kHz stereo
`gudowski_short_2`	0:28	44.1 kHz stereo

The transcript matters as much as the audio — voice cloning quality drops noticeably without an accurate one. The script warns if the .txt is missing.

Network

Needed only at install time, for the ComfyUI and TTS-Audio-Suite clones, the pip downloads, and the 8.7 GB model. Generation afterwards is entirely local; no API keys, no accounts, no per-use cost.

The model is published under a research / non-commercial licence and may require accepting its terms on huggingface.co, then hf auth login, before it will download.

The number normalizer's model is gated in the same way and needs an HF_TOKEN for its one-time download; see installation.md. Runs set HF_HUB_OFFLINE=1 and TRANSFORMERS_OFFLINE=1, so it never reaches the network afterwards.

6.7 KiB Raw Blame History