Initial commit: Polish AI dubbing pipeline (ComfyUI + Higgs Audio V3)
Converts SRT subtitles into a Polish lektor-style voiceover using a local
ComfyUI instance running Higgs Audio V3, then muxes the result back into the
source video as a second audio track. Video and original audio are always
stream-copied; only the new voiceover track is encoded.
Contents:
lektor_ai_comfy.py main pipeline (standard library only, needs ffmpeg)
lektor_ai_comfy_v2.py batched-context experiment (timing unresolved)
lektor_ai.py earlier version using the hosted Boson AI API
install_comfyui.sh reproduces the ComfyUI environment, versions pinned
docs/ requirements, installation, usage, architecture,
troubleshooting
gudowski/ voice references (.wav) with their transcripts (.txt)
Source media and generated video are gitignored.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
This commit is contained in:
Claude Opus 5
@@ -0,0 +1,132 @@
|
||||
# Requirements
|
||||
|
||||
## Hardware
|
||||
|
||||
| | Minimum | Notes |
|
||||
|---|---|---|
|
||||
| GPU | NVIDIA, 12 GB VRAM | Developed on an RTX 4070 Ti SUPER (16 GB) |
|
||||
| Driver | 525+ | Needed by the CUDA 12.8 wheels |
|
||||
| Disk | ~35 GB free | ~22 GB installed, plus working room for output |
|
||||
| RAM | 16 GB | The final mux briefly peaks around 2.6 GB |
|
||||
|
||||
The CUDA **toolkit** is not required. The PyTorch cu128 wheels bundle their own
|
||||
CUDA runtime, so the system `nvcc` version is irrelevant — this machine has CUDA
|
||||
11.5 installed system-wide while PyTorch runs cu128 perfectly well. Only the
|
||||
driver version matters.
|
||||
|
||||
CPU-only inference technically works but is slow enough to be impractical for a
|
||||
feature-length film.
|
||||
|
||||
### Disk footprint
|
||||
|
||||
```
|
||||
~/miniforge3/envs/comfy-tts 9.7 GB conda env (PyTorch and ~233 packages)
|
||||
~/comfy-tts-app/ComfyUI 3.2 GB application code and dependencies
|
||||
└── models/TTS 8.8 GB Higgs Audio V3 weights
|
||||
------
|
||||
21.7 GB
|
||||
```
|
||||
|
||||
Output is roughly the size of the source video, since video is stream-copied. A
|
||||
60-minute 10 Mbit/s episode produces about 5 GB. Intermediate FLAC clips add a
|
||||
few hundred MB per film and are kept deliberately, so individual clips can be
|
||||
regenerated without redoing a whole run (see
|
||||
[usage.md](usage.md#working-modes)).
|
||||
|
||||
## Software
|
||||
|
||||
| Component | Version here | Notes |
|
||||
|---|---|---|
|
||||
| OS | Linux (Ubuntu 22.04) | Only Linux is supported by the installer |
|
||||
| ComfyUI | 0.26.0 (`7cb784e0`) | Pinned |
|
||||
| TTS-Audio-Suite | v5.3.0 (`09df1681`) | The only custom node installed |
|
||||
| Higgs Audio V3 | `bosonai/higgs-audio-v3-tts-4b` | 8.7 GB, auto-downloaded |
|
||||
| Python (ComfyUI) | 3.12.13 | conda-forge, in the `comfy-tts` env |
|
||||
| PyTorch | 2.11.0+cu128 | With torchvision 0.26.0, torchaudio 2.11.0 |
|
||||
| Python (script) | 3.8+ | System Python; 3.10.12 here |
|
||||
| ffmpeg / ffprobe | 4.4+ | System install; 4.4.2 here |
|
||||
|
||||
## Dependency surface
|
||||
|
||||
This is the point most worth internalising, because it makes debugging much
|
||||
simpler: **the two halves have almost nothing in common.**
|
||||
|
||||
### `lektor_ai_comfy.py` — standard library only
|
||||
|
||||
```python
|
||||
argparse json random re shutil subprocess sys threading time uuid
|
||||
pathlib.Path urllib.request
|
||||
```
|
||||
|
||||
No third-party packages. No conda environment. No virtualenv. It runs on the
|
||||
system Python:
|
||||
|
||||
```bash
|
||||
/usr/bin/python3 lektor_ai_comfy.py --help # works, nothing installed
|
||||
```
|
||||
|
||||
Its only external requirements are **ffmpeg** and **ffprobe** on `PATH`, and a
|
||||
reachable ComfyUI instance. It talks to ComfyUI over plain HTTP with
|
||||
`urllib.request` — there is no ComfyUI client library involved.
|
||||
|
||||
> **`requirements.txt` in the project root does not apply here.** It belongs to
|
||||
> the older `lektor_ai.py`, which used the hosted Boson AI API and needed
|
||||
> `openai>=1.0.0`. `lektor_ai_comfy.py` requires nothing from it.
|
||||
|
||||
### ComfyUI — a large, isolated environment
|
||||
|
||||
233 pip packages in the `comfy-tts` conda env. The important ones:
|
||||
|
||||
```
|
||||
torch 2.11.0+cu128 torchaudio 2.11.0+cu128 torchvision 0.26.0+cu128
|
||||
transformers 5.12.1 tokenizers 0.22.2 accelerate 1.14.0
|
||||
numpy 2.2.6 scipy 1.18.0 librosa 0.11.0
|
||||
soundfile 0.14.0 safetensors 0.8.0 einops 0.8.2
|
||||
```
|
||||
|
||||
The conda environment itself holds only the 27 base conda-forge packages that
|
||||
make up the Python runtime. Everything else arrives via pip. In particular,
|
||||
`portaudio`, `libsamplerate`, `ffmpeg` and `espeak-ng` are **not** installed in
|
||||
the environment despite appearing in the original planning notes — the system
|
||||
ffmpeg is used and espeak-ng is absent entirely. Everything works without them.
|
||||
|
||||
TTS-Audio-Suite installs its dependencies through its own `install.py`, not
|
||||
through its `requirements.txt`. That file deliberately omits packages needing
|
||||
special handling — `librosa`, `descript-audio-codec`, `onnxruntime`,
|
||||
`opencv-python` and others are installed with `--no-deps` because pip would
|
||||
otherwise drag numpy backwards and break the rest of the environment. Never
|
||||
substitute `pip install -r requirements.txt` for `python install.py`.
|
||||
|
||||
## Input requirements
|
||||
|
||||
**Video** — anything ffmpeg can demux. MKV and MP4 are the usual cases. The
|
||||
container must support what gets muxed back in; output is always MKV, which
|
||||
accepts every codec involved.
|
||||
|
||||
**Subtitles** — either an external `.srt`, or a text-based track inside the
|
||||
video (`subrip`, `ass`, `ssa`, `webvtt`, `mov_text`). Image-based tracks
|
||||
(`hdmv_pgs_subtitle` from Blu-ray, `dvd_subtitle`) contain bitmaps rather than
|
||||
text and cannot drive TTS without OCR; the `--embedded-subs` picker marks them
|
||||
as unusable. They are still copied to the output untouched.
|
||||
|
||||
**Voice reference** — a `.wav` plus a `.txt` holding its exact transcript, both
|
||||
in `gudowski/`. Three are bundled:
|
||||
|
||||
| Voice | Length | Format |
|
||||
|---|---|---|
|
||||
| `gudowski` | 2:10 | 44.1 kHz stereo |
|
||||
| `gudowski_short` (default) | 0:52 | 44.1 kHz stereo |
|
||||
| `gudowski_short_2` | 0:28 | 44.1 kHz stereo |
|
||||
|
||||
The transcript matters as much as the audio — voice cloning quality drops
|
||||
noticeably without an accurate one. The script warns if the `.txt` is missing.
|
||||
|
||||
## Network
|
||||
|
||||
Needed only at install time, for the ComfyUI and TTS-Audio-Suite clones, the pip
|
||||
downloads, and the 8.7 GB model. Generation afterwards is entirely local; no
|
||||
API keys, no accounts, no per-use cost.
|
||||
|
||||
The model is published under a research / non-commercial licence and may require
|
||||
accepting its terms on huggingface.co, then `hf auth login`, before it will
|
||||
download.
|
||||
Reference in New Issue
Block a user