claudeandClaude Opus 5
fc4b761f52
Converts SRT subtitles into a Polish lektor-style voiceover using a local
ComfyUI instance running Higgs Audio V3, then muxes the result back into the
source video as a second audio track. Video and original audio are always
stream-copied; only the new voiceover track is encoded.
Contents:
lektor_ai_comfy.py main pipeline (standard library only, needs ffmpeg)
lektor_ai_comfy_v2.py batched-context experiment (timing unresolved)
lektor_ai.py earlier version using the hosted Boson AI API
install_comfyui.sh reproduces the ComfyUI environment, versions pinned
docs/ requirements, installation, usage, architecture,
troubleshooting
gudowski/ voice references (.wav) with their transcripts (.txt)
Source media and generated video are gitignored.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
133 lines
5.4 KiB
Markdown
133 lines
5.4 KiB
Markdown
# Requirements
|
|
|
|
## Hardware
|
|
|
|
| | Minimum | Notes |
|
|
|---|---|---|
|
|
| GPU | NVIDIA, 12 GB VRAM | Developed on an RTX 4070 Ti SUPER (16 GB) |
|
|
| Driver | 525+ | Needed by the CUDA 12.8 wheels |
|
|
| Disk | ~35 GB free | ~22 GB installed, plus working room for output |
|
|
| RAM | 16 GB | The final mux briefly peaks around 2.6 GB |
|
|
|
|
The CUDA **toolkit** is not required. The PyTorch cu128 wheels bundle their own
|
|
CUDA runtime, so the system `nvcc` version is irrelevant — this machine has CUDA
|
|
11.5 installed system-wide while PyTorch runs cu128 perfectly well. Only the
|
|
driver version matters.
|
|
|
|
CPU-only inference technically works but is slow enough to be impractical for a
|
|
feature-length film.
|
|
|
|
### Disk footprint
|
|
|
|
```
|
|
~/miniforge3/envs/comfy-tts 9.7 GB conda env (PyTorch and ~233 packages)
|
|
~/comfy-tts-app/ComfyUI 3.2 GB application code and dependencies
|
|
└── models/TTS 8.8 GB Higgs Audio V3 weights
|
|
------
|
|
21.7 GB
|
|
```
|
|
|
|
Output is roughly the size of the source video, since video is stream-copied. A
|
|
60-minute 10 Mbit/s episode produces about 5 GB. Intermediate FLAC clips add a
|
|
few hundred MB per film and are kept deliberately, so individual clips can be
|
|
regenerated without redoing a whole run (see
|
|
[usage.md](usage.md#working-modes)).
|
|
|
|
## Software
|
|
|
|
| Component | Version here | Notes |
|
|
|---|---|---|
|
|
| OS | Linux (Ubuntu 22.04) | Only Linux is supported by the installer |
|
|
| ComfyUI | 0.26.0 (`7cb784e0`) | Pinned |
|
|
| TTS-Audio-Suite | v5.3.0 (`09df1681`) | The only custom node installed |
|
|
| Higgs Audio V3 | `bosonai/higgs-audio-v3-tts-4b` | 8.7 GB, auto-downloaded |
|
|
| Python (ComfyUI) | 3.12.13 | conda-forge, in the `comfy-tts` env |
|
|
| PyTorch | 2.11.0+cu128 | With torchvision 0.26.0, torchaudio 2.11.0 |
|
|
| Python (script) | 3.8+ | System Python; 3.10.12 here |
|
|
| ffmpeg / ffprobe | 4.4+ | System install; 4.4.2 here |
|
|
|
|
## Dependency surface
|
|
|
|
This is the point most worth internalising, because it makes debugging much
|
|
simpler: **the two halves have almost nothing in common.**
|
|
|
|
### `lektor_ai_comfy.py` — standard library only
|
|
|
|
```python
|
|
argparse json random re shutil subprocess sys threading time uuid
|
|
pathlib.Path urllib.request
|
|
```
|
|
|
|
No third-party packages. No conda environment. No virtualenv. It runs on the
|
|
system Python:
|
|
|
|
```bash
|
|
/usr/bin/python3 lektor_ai_comfy.py --help # works, nothing installed
|
|
```
|
|
|
|
Its only external requirements are **ffmpeg** and **ffprobe** on `PATH`, and a
|
|
reachable ComfyUI instance. It talks to ComfyUI over plain HTTP with
|
|
`urllib.request` — there is no ComfyUI client library involved.
|
|
|
|
> **`requirements.txt` in the project root does not apply here.** It belongs to
|
|
> the older `lektor_ai.py`, which used the hosted Boson AI API and needed
|
|
> `openai>=1.0.0`. `lektor_ai_comfy.py` requires nothing from it.
|
|
|
|
### ComfyUI — a large, isolated environment
|
|
|
|
233 pip packages in the `comfy-tts` conda env. The important ones:
|
|
|
|
```
|
|
torch 2.11.0+cu128 torchaudio 2.11.0+cu128 torchvision 0.26.0+cu128
|
|
transformers 5.12.1 tokenizers 0.22.2 accelerate 1.14.0
|
|
numpy 2.2.6 scipy 1.18.0 librosa 0.11.0
|
|
soundfile 0.14.0 safetensors 0.8.0 einops 0.8.2
|
|
```
|
|
|
|
The conda environment itself holds only the 27 base conda-forge packages that
|
|
make up the Python runtime. Everything else arrives via pip. In particular,
|
|
`portaudio`, `libsamplerate`, `ffmpeg` and `espeak-ng` are **not** installed in
|
|
the environment despite appearing in the original planning notes — the system
|
|
ffmpeg is used and espeak-ng is absent entirely. Everything works without them.
|
|
|
|
TTS-Audio-Suite installs its dependencies through its own `install.py`, not
|
|
through its `requirements.txt`. That file deliberately omits packages needing
|
|
special handling — `librosa`, `descript-audio-codec`, `onnxruntime`,
|
|
`opencv-python` and others are installed with `--no-deps` because pip would
|
|
otherwise drag numpy backwards and break the rest of the environment. Never
|
|
substitute `pip install -r requirements.txt` for `python install.py`.
|
|
|
|
## Input requirements
|
|
|
|
**Video** — anything ffmpeg can demux. MKV and MP4 are the usual cases. The
|
|
container must support what gets muxed back in; output is always MKV, which
|
|
accepts every codec involved.
|
|
|
|
**Subtitles** — either an external `.srt`, or a text-based track inside the
|
|
video (`subrip`, `ass`, `ssa`, `webvtt`, `mov_text`). Image-based tracks
|
|
(`hdmv_pgs_subtitle` from Blu-ray, `dvd_subtitle`) contain bitmaps rather than
|
|
text and cannot drive TTS without OCR; the `--embedded-subs` picker marks them
|
|
as unusable. They are still copied to the output untouched.
|
|
|
|
**Voice reference** — a `.wav` plus a `.txt` holding its exact transcript, both
|
|
in `gudowski/`. Three are bundled:
|
|
|
|
| Voice | Length | Format |
|
|
|---|---|---|
|
|
| `gudowski` | 2:10 | 44.1 kHz stereo |
|
|
| `gudowski_short` (default) | 0:52 | 44.1 kHz stereo |
|
|
| `gudowski_short_2` | 0:28 | 44.1 kHz stereo |
|
|
|
|
The transcript matters as much as the audio — voice cloning quality drops
|
|
noticeably without an accurate one. The script warns if the `.txt` is missing.
|
|
|
|
## Network
|
|
|
|
Needed only at install time, for the ComfyUI and TTS-Audio-Suite clones, the pip
|
|
downloads, and the 8.7 GB model. Generation afterwards is entirely local; no
|
|
API keys, no accounts, no per-use cost.
|
|
|
|
The model is published under a research / non-commercial licence and may require
|
|
accepting its terms on huggingface.co, then `hf auth login`, before it will
|
|
download.
|