claudeandClaude Opus 5
fc4b761f52
Converts SRT subtitles into a Polish lektor-style voiceover using a local
ComfyUI instance running Higgs Audio V3, then muxes the result back into the
source video as a second audio track. Video and original audio are always
stream-copied; only the new voiceover track is encoded.
Contents:
lektor_ai_comfy.py main pipeline (standard library only, needs ffmpeg)
lektor_ai_comfy_v2.py batched-context experiment (timing unresolved)
lektor_ai.py earlier version using the hosted Boson AI API
install_comfyui.sh reproduces the ComfyUI environment, versions pinned
docs/ requirements, installation, usage, architecture,
troubleshooting
gudowski/ voice references (.wav) with their transcripts (.txt)
Source media and generated video are gitignored.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
208 lines
6.5 KiB
Markdown
208 lines
6.5 KiB
Markdown
# Installation
|
||
|
||
Two things need to exist: the **ComfyUI backend** (large, isolated, GPU-bound)
|
||
and the **script** (which needs nothing installed at all).
|
||
|
||
## Automated install
|
||
|
||
`install_comfyui.sh` in the project root reproduces the exact working
|
||
environment. It is idempotent — safe to re-run, and it skips any step already
|
||
satisfied.
|
||
|
||
```bash
|
||
cd ~/Programy/lektor_ai
|
||
./install_comfyui.sh
|
||
```
|
||
|
||
Roughly 30–60 minutes, mostly downloading. Expect ~22 GB.
|
||
|
||
### Options
|
||
|
||
| Flag | Effect |
|
||
|---|---|
|
||
| *(none)* | Pinned versions — ComfyUI `7cb784e0`, suite `v5.3.0`. Recommended. |
|
||
| `--latest` | Newest ComfyUI and suite instead. You are on your own for breakage. |
|
||
| `--skip-model` | Skip the 8.7 GB download; it happens on first generation anyway. |
|
||
| `--lock FILE` | Restore an exact package set from a `pip freeze` lockfile. |
|
||
| `--help` | Usage summary. |
|
||
|
||
### Changing where things go
|
||
|
||
```bash
|
||
CONDA_ROOT=/opt/miniforge3 COMFY_ROOT=/data/comfy ./install_comfyui.sh
|
||
```
|
||
|
||
| Variable | Default |
|
||
|---|---|
|
||
| `CONDA_ROOT` | `~/miniforge3` |
|
||
| `COMFY_ROOT` | `~/comfy-tts-app` |
|
||
| `ENV_NAME` | `comfy-tts` |
|
||
| `COMFY_PORT` | `8189` |
|
||
| `PYTHON_VERSION` | `3.12` |
|
||
|
||
If you change `COMFY_ROOT` or `COMFY_PORT`, update the script constants too —
|
||
see [Configuring the script](#configuring-the-script).
|
||
|
||
### Exact reproduction
|
||
|
||
`comfyui-requirements.lock.txt` pins all 232 packages. To rebuild a
|
||
byte-identical environment:
|
||
|
||
```bash
|
||
./install_comfyui.sh --lock comfyui-requirements.lock.txt
|
||
```
|
||
|
||
Regenerate it after intentional changes:
|
||
|
||
```bash
|
||
source ~/miniforge3/etc/profile.d/conda.sh && conda activate comfy-tts
|
||
pip freeze > comfyui-requirements.lock.txt
|
||
```
|
||
|
||
## What the installer does
|
||
|
||
Useful to know when something fails midway, since each step can be repeated on
|
||
its own.
|
||
|
||
1. **Preflight** — checks Linux, git, a downloader, GPU, disk space, ffmpeg.
|
||
Only missing git or a downloader is fatal; the rest warn.
|
||
2. **Miniforge** → `~/miniforge3`, batch mode. `conda init` is deliberately not
|
||
run, so your `.bashrc` is never touched.
|
||
3. **Conda env** `comfy-tts` with Python 3.12, then a guard that refuses to
|
||
continue if the wrong Python somehow ended up active.
|
||
4. **PyTorch** from the cu128 index, pinned to 2.11.0.
|
||
5. **ComfyUI** cloned and checked out at the pinned commit (detached HEAD — this
|
||
is expected, not an error), then its `requirements.txt` installed.
|
||
6. **TTS-Audio-Suite** cloned, checked out at `v5.3.0`, then **`python
|
||
install.py`** — its own resolver, which must not be replaced with a plain
|
||
pip install (see [requirements.md](requirements.md#dependency-surface)).
|
||
7. **Model** — `bosonai/higgs-audio-v3-tts-4b` into
|
||
`ComfyUI/models/TTS/higgs_audio_v3/`.
|
||
8. **Launcher** written to `$COMFY_ROOT/start_comfyui.sh`.
|
||
9. **Verification** — imports torch, torchaudio, transformers, safetensors,
|
||
soundfile, librosa and reports CUDA availability.
|
||
|
||
## Manual install
|
||
|
||
The equivalent by hand, if you would rather not use the script:
|
||
|
||
```bash
|
||
# Miniforge, self-contained, no shell integration
|
||
wget https://github.com/conda-forge/miniforge/releases/latest/download/Miniforge3-Linux-x86_64.sh
|
||
bash Miniforge3-Linux-x86_64.sh -b -p ~/miniforge3
|
||
source ~/miniforge3/etc/profile.d/conda.sh
|
||
|
||
conda create -n comfy-tts python=3.12 -y
|
||
conda activate comfy-tts
|
||
|
||
# PyTorch with the CUDA 12.8 runtime bundled
|
||
pip install torch==2.11.0 torchvision==0.26.0 torchaudio==2.11.0 \
|
||
--index-url https://download.pytorch.org/whl/cu128
|
||
|
||
# ComfyUI, pinned
|
||
mkdir -p ~/comfy-tts-app && cd ~/comfy-tts-app
|
||
git clone https://github.com/comfyanonymous/ComfyUI.git
|
||
cd ComfyUI && git checkout 7cb784e0f48784bb6ed588912e186e5ee1e9ee68
|
||
pip install -r requirements.txt
|
||
|
||
# The TTS node, pinned — note install.py, NOT requirements.txt
|
||
cd custom_nodes
|
||
git clone https://github.com/diodiogod/TTS-Audio-Suite.git
|
||
cd TTS-Audio-Suite && git checkout tags/v5.3.0
|
||
python install.py
|
||
```
|
||
|
||
The model downloads itself on first generation. To fetch it ahead of time:
|
||
|
||
```python
|
||
from huggingface_hub import snapshot_download
|
||
snapshot_download(
|
||
repo_id="bosonai/higgs-audio-v3-tts-4b",
|
||
local_dir="~/comfy-tts-app/ComfyUI/models/TTS/higgs_audio_v3/higgs-audio-v3-tts-4b",
|
||
)
|
||
```
|
||
|
||
## Running the backend
|
||
|
||
```bash
|
||
~/comfy-tts-app/start_comfyui.sh
|
||
```
|
||
|
||
which is just:
|
||
|
||
```bash
|
||
#!/bin/bash
|
||
source ~/miniforge3/etc/profile.d/conda.sh
|
||
conda activate comfy-tts
|
||
cd ~/comfy-tts-app/ComfyUI && python main.py --port 8189
|
||
```
|
||
|
||
Leave it running in its own terminal for the whole session — model load takes
|
||
tens of seconds and would otherwise be repeated. The web UI at
|
||
<http://127.0.0.1:8189> is useful for watching progress and reading errors, but
|
||
is not needed; the script drives everything over HTTP.
|
||
|
||
## Configuring the script
|
||
|
||
`lektor_ai_comfy.py` has four constants at the top. If you installed anywhere
|
||
other than the defaults, edit them:
|
||
|
||
```python
|
||
COMFY_URL = "http://127.0.0.1:8189"
|
||
COMFY_OUTPUT_DIR = Path("/home/lukas/comfy-tts-app/ComfyUI/output")
|
||
COMFY_INPUT_DIR = Path("/home/lukas/comfy-tts-app/ComfyUI/input")
|
||
VOICE_PROMPTS_DIR = Path("/home/lukas/Programy/lektor_ai/gudowski")
|
||
```
|
||
|
||
`COMFY_URL` can also be overridden per-run with `--comfy-url`. The two directory
|
||
constants cannot — the script reads generated audio directly from ComfyUI's
|
||
output directory and copies voice references into its input directory, so both
|
||
must be real paths on the same filesystem.
|
||
|
||
The script itself needs no installation. Ensure ffmpeg is present:
|
||
|
||
```bash
|
||
ffmpeg -version && ffprobe -version
|
||
```
|
||
|
||
If missing, and you would rather not use sudo:
|
||
|
||
```bash
|
||
conda install -n comfy-tts -c conda-forge ffmpeg
|
||
```
|
||
|
||
Note that this puts ffmpeg inside the conda env, so the script must then be run
|
||
with that env active.
|
||
|
||
## Verifying
|
||
|
||
```bash
|
||
# 1. Backend responds
|
||
curl -s http://127.0.0.1:8189/system_stats | head -c 200
|
||
|
||
# 2. GPU visible to PyTorch
|
||
source ~/miniforge3/etc/profile.d/conda.sh && conda activate comfy-tts
|
||
python -c "import torch; print(torch.__version__, torch.cuda.is_available())"
|
||
# -> 2.11.0+cu128 True
|
||
|
||
# 3. Model present
|
||
ls -la ~/comfy-tts-app/ComfyUI/models/TTS/higgs_audio_v3/higgs-audio-v3-tts-4b/model.safetensors
|
||
|
||
# 4. End-to-end on 60 seconds of a real file
|
||
cd ~/Programy/lektor_ai
|
||
python3 lektor_ai_comfy.py "input/film.mkv" --embedded-subs --end 01:00
|
||
```
|
||
|
||
The short run is the real test — it exercises subtitle parsing, TTS, assembly
|
||
and muxing, and finishes in a couple of minutes.
|
||
|
||
## Uninstalling
|
||
|
||
Nothing was installed system-wide, so removal is just deleting directories:
|
||
|
||
```bash
|
||
conda env remove -n comfy-tts
|
||
rm -rf ~/comfy-tts-app
|
||
rm -rf ~/miniforge3 # only if nothing else uses it
|
||
```
|