claudeandClaude Opus 5
e201aadc3e
Text processing before TTS:
- dictionary.txt rewrites words and phrases (km/h, np., units). Rules compile
into one alternation applied in a single pass, so replacements never chain
and longest-literal wins. Case-sensitive, whole-word.
- A per-queue second dictionary, written by the GUI into the first video's job
folder and merged with the shared one (--dictionary-extra).
- ALL-CAPS blocks become Title Case rather than sentence case: lowercasing
destroyed proper nouns inside captions ("NASA - MARGO MADISON") and invited
the normalizer to "correct" the name. Paired clips in output/clips/all-caps
showed the TTS reads both casings the same.
normalizer/ — digits to spoken Polish, in its own venv because it needs
transformers>=5, which must not touch ComfyUI's pinned versions:
- Two backends. Plain integers go to num2words for the value and a general
Polish LLM for the case; times, dates, decimals and identifiers go to a 0.6B
fine-tune. Measured on 40 real subtitle lines, each is clearly better than
the other on its half. 74% of numbered blocks take the first leg.
- Guards on every answer: runaway length, repetition loops, invalid bytes,
leftover digits, a lost minus sign, a changed number, a lost word. A failure
falls back to num2words, which is nominative but never wrong about the value.
- ComfyUI is asked to unload before normalizing; a normalizer that does not fit
in VRAM is spilled to RAM silently at 5x the cost.
Backend lifecycle:
- ComfyUI is found by port through /proc, so a backend started by hand is
managed too. SIGTERM escalates to SIGKILL, and a suspended process is killed
outright since it never runs its handlers.
- A clip that kills the backend is retried three times, each with a different
seed (the aborts are seed-dependent, not text-dependent), then skipped and
reported so an overnight queue keeps moving.
- start_comfyui.sh execs python, so the script's PID is ComfyUI's.
- GUI gains ComfyUI and Queue dictionary buttons; quitting stops the backend.
--resume-from now reuses the job folder's tts_input.srt instead of reparsing
the subtitles, so block N still pairs with clip_NNNN.flac.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
161 lines
6.7 KiB
Markdown
161 lines
6.7 KiB
Markdown
# Requirements
|
|
|
|
## Hardware
|
|
|
|
| | Minimum | Notes |
|
|
|---|---|---|
|
|
| GPU | NVIDIA, 12 GB VRAM | Developed on an RTX 4070 Ti SUPER (16 GB) |
|
|
| Driver | 525+ | Needed by the CUDA 12.8 wheels |
|
|
| Disk | ~35 GB free | ~22 GB installed (+8 GB with the number normalizer) |
|
|
| RAM | 16 GB | The final mux briefly peaks around 2.6 GB |
|
|
|
|
The CUDA **toolkit** is not required. The PyTorch cu128 wheels bundle their own
|
|
CUDA runtime, so the system `nvcc` version is irrelevant — this machine has CUDA
|
|
11.5 installed system-wide while PyTorch runs cu128 perfectly well. Only the
|
|
driver version matters.
|
|
|
|
CPU-only inference technically works but is slow enough to be impractical for a
|
|
feature-length film.
|
|
|
|
### Disk footprint
|
|
|
|
```
|
|
~/miniforge3/envs/comfy-tts 9.7 GB conda env (PyTorch and ~233 packages)
|
|
~/comfy-tts-app/ComfyUI 3.2 GB application code and dependencies
|
|
└── models/TTS 8.8 GB Higgs Audio V3 weights
|
|
------
|
|
21.7 GB
|
|
|
|
lektor_ai/normalizer/venv 6.9 GB optional: number normalizer env
|
|
lektor_ai/normalizer/model 1.1 GB optional: its weights
|
|
```
|
|
|
|
Output is roughly the size of the source video, since video is stream-copied. A
|
|
60-minute 10 Mbit/s episode produces about 5 GB. Intermediate FLAC clips add a
|
|
few hundred MB per film and are kept deliberately, so individual clips can be
|
|
regenerated without redoing a whole run (see
|
|
[usage.md](usage.md#working-modes)).
|
|
|
|
## Software
|
|
|
|
| Component | Version here | Notes |
|
|
|---|---|---|
|
|
| OS | Linux (Ubuntu 22.04) | Only Linux is supported by the installer |
|
|
| ComfyUI | 0.26.0 (`7cb784e0`) | Pinned |
|
|
| TTS-Audio-Suite | v5.3.0 (`09df1681`) | The only custom node installed |
|
|
| Higgs Audio V3 | `bosonai/higgs-audio-v3-tts-4b` | 8.7 GB, auto-downloaded |
|
|
| Python (ComfyUI) | 3.12.13 | conda-forge, in the `comfy-tts` env |
|
|
| PyTorch | 2.11.0+cu128 | With torchvision 0.26.0, torchaudio 2.11.0 |
|
|
| Python (script) | 3.8+ | System Python; 3.10.12 here |
|
|
| ffmpeg / ffprobe | 4.4+ | System install; 4.4.2 here |
|
|
|
|
## Dependency surface
|
|
|
|
This is the point most worth internalising, because it makes debugging much
|
|
simpler: **the three parts have almost nothing in common.**
|
|
|
|
### `lektor_ai_comfy.py` — standard library only
|
|
|
|
```python
|
|
argparse configparser json os queue random re shlex shutil subprocess
|
|
sys threading time uuid pathlib.Path urllib.request
|
|
```
|
|
|
|
No third-party packages. No conda environment. No virtualenv. It runs on the
|
|
system Python:
|
|
|
|
```bash
|
|
/usr/bin/python3 lektor_ai_comfy.py --help # works, nothing installed
|
|
```
|
|
|
|
Its only external requirements are **ffmpeg** and **ffprobe** on `PATH`, and a
|
|
reachable ComfyUI instance. It talks to ComfyUI over plain HTTP with
|
|
`urllib.request` — there is no ComfyUI client library involved.
|
|
|
|
> **`requirements.txt` in the project root does not apply here.** It belongs to
|
|
> the older `lektor_ai.py`, which used the hosted Boson AI API and needed
|
|
> `openai>=1.0.0`. `lektor_ai_comfy.py` requires nothing from it.
|
|
> `normalizer/requirements.txt` is a different file and does apply — to the
|
|
> normalizer's own venv, never to the system Python.
|
|
|
|
### `normalizer/` — optional third environment
|
|
|
|
```
|
|
torch 2.11.0+cu128 transformers 5.15.0 accelerate 1.14.0
|
|
num2words 0.5.x (fallback path only)
|
|
```
|
|
|
|
Its own venv at `normalizer/venv` (~6.9 GB — the cu128 wheels bundle their
|
|
own CUDA runtime, again) plus `normalizer/model` (1.1 GB).
|
|
Separate from ComfyUI's environment because it hard-requires `transformers >= 5`
|
|
— 4.x tokenizes the Qwen3 chat template differently and silently produces wrong
|
|
output. `lektor_ai_comfy.py` never imports it; it starts it as a subprocess and
|
|
talks JSON lines over a pipe, so the version conflict never has to be resolved.
|
|
|
|
Entirely optional: without it the pipeline uses a `num2words` fallback, and with
|
|
`numbers = no` it is not started at all.
|
|
|
|
### ComfyUI — a large, isolated environment
|
|
|
|
233 pip packages in the `comfy-tts` conda env. The important ones:
|
|
|
|
```
|
|
torch 2.11.0+cu128 torchaudio 2.11.0+cu128 torchvision 0.26.0+cu128
|
|
transformers 5.12.1 tokenizers 0.22.2 accelerate 1.14.0
|
|
numpy 2.2.6 scipy 1.18.0 librosa 0.11.0
|
|
soundfile 0.14.0 safetensors 0.8.0 einops 0.8.2
|
|
```
|
|
|
|
The conda environment itself holds only the 27 base conda-forge packages that
|
|
make up the Python runtime. Everything else arrives via pip. In particular,
|
|
`portaudio`, `libsamplerate`, `ffmpeg` and `espeak-ng` are **not** installed in
|
|
the environment despite appearing in the original planning notes — the system
|
|
ffmpeg is used and espeak-ng is absent entirely. Everything works without them.
|
|
|
|
TTS-Audio-Suite installs its dependencies through its own `install.py`, not
|
|
through its `requirements.txt`. That file deliberately omits packages needing
|
|
special handling — `librosa`, `descript-audio-codec`, `onnxruntime`,
|
|
`opencv-python` and others are installed with `--no-deps` because pip would
|
|
otherwise drag numpy backwards and break the rest of the environment. Never
|
|
substitute `pip install -r requirements.txt` for `python install.py`.
|
|
|
|
## Input requirements
|
|
|
|
**Video** — anything ffmpeg can demux. MKV and MP4 are the usual cases. The
|
|
container must support what gets muxed back in; output is always MKV, which
|
|
accepts every codec involved.
|
|
|
|
**Subtitles** — either an external `.srt`, or a text-based track inside the
|
|
video (`subrip`, `ass`, `ssa`, `webvtt`, `mov_text`). Image-based tracks
|
|
(`hdmv_pgs_subtitle` from Blu-ray, `dvd_subtitle`) contain bitmaps rather than
|
|
text and cannot drive TTS without OCR; the `--embedded-subs` picker marks them
|
|
as unusable. They are still copied to the output untouched.
|
|
|
|
**Voice reference** — a `.wav` plus a `.txt` holding its exact transcript, both
|
|
in `voices/`. Three are bundled:
|
|
|
|
| Voice | Length | Format |
|
|
|---|---|---|
|
|
| `gudowski` | 2:10 | 44.1 kHz stereo |
|
|
| `gudowski_short` (default) | 0:52 | 44.1 kHz stereo |
|
|
| `gudowski_short_2` | 0:28 | 44.1 kHz stereo |
|
|
|
|
The transcript matters as much as the audio — voice cloning quality drops
|
|
noticeably without an accurate one. The script warns if the `.txt` is missing.
|
|
|
|
## Network
|
|
|
|
Needed only at install time, for the ComfyUI and TTS-Audio-Suite clones, the pip
|
|
downloads, and the 8.7 GB model. Generation afterwards is entirely local; no
|
|
API keys, no accounts, no per-use cost.
|
|
|
|
The model is published under a research / non-commercial licence and may require
|
|
accepting its terms on huggingface.co, then `hf auth login`, before it will
|
|
download.
|
|
|
|
The number normalizer's model is **gated** in the same way and needs an
|
|
`HF_TOKEN` for its one-time download; see
|
|
[installation.md](installation.md#number-normalizer-optional-recommended). Runs
|
|
set `HF_HUB_OFFLINE=1` and `TRANSFORMERS_OFFLINE=1`, so it never reaches the
|
|
network afterwards.
|