Initial commit: Polish AI dubbing pipeline (ComfyUI + Higgs Audio V3)
Converts SRT subtitles into a Polish lektor-style voiceover using a local
ComfyUI instance running Higgs Audio V3, then muxes the result back into the
source video as a second audio track. Video and original audio are always
stream-copied; only the new voiceover track is encoded.
Contents:
lektor_ai_comfy.py main pipeline (standard library only, needs ffmpeg)
lektor_ai_comfy_v2.py batched-context experiment (timing unresolved)
lektor_ai.py earlier version using the hosted Boson AI API
install_comfyui.sh reproduces the ComfyUI environment, versions pinned
docs/ requirements, installation, usage, architecture,
troubleshooting
gudowski/ voice references (.wav) with their transcripts (.txt)
Source media and generated video are gitignored.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
This commit is contained in:
Claude Opus 5
+24
@@ -0,0 +1,24 @@
|
|||||||
|
# Media — source films and generated video. Tens of GB, and not ours to publish.
|
||||||
|
input/
|
||||||
|
output/
|
||||||
|
|
||||||
|
# Python
|
||||||
|
__pycache__/
|
||||||
|
*.py[cod]
|
||||||
|
*.egg-info/
|
||||||
|
.venv/
|
||||||
|
venv/
|
||||||
|
|
||||||
|
# Machine-local Claude Code permissions (intentionally not shared)
|
||||||
|
.claude/settings.local.json
|
||||||
|
|
||||||
|
# Editor / OS noise
|
||||||
|
.vscode/
|
||||||
|
.idea/
|
||||||
|
*.swp
|
||||||
|
.DS_Store
|
||||||
|
|
||||||
|
# Scratch
|
||||||
|
*.log
|
||||||
|
*.tmp
|
||||||
|
_batch_tmp/
|
||||||
@@ -0,0 +1,57 @@
|
|||||||
|
=============================================================================
|
||||||
|
NOTE: These are the original planning notes. Use install_comfyui.sh instead —
|
||||||
|
it reproduces what is ACTUALLY installed, with versions pinned.
|
||||||
|
|
||||||
|
Two things below do not match the real installation:
|
||||||
|
|
||||||
|
* "cu128 for your 3090" — the machine has an RTX 4070 Ti SUPER (16 GB).
|
||||||
|
The cu128 wheels are still correct; only the comment is wrong.
|
||||||
|
|
||||||
|
* The `conda install -c conda-forge portaudio libsamplerate ffmpeg espeak-ng
|
||||||
|
git` line was never actually applied. The environment contains only the 27
|
||||||
|
base Python packages from conda-forge. ffmpeg is used from the system
|
||||||
|
(/usr/bin/ffmpeg) and espeak-ng is not installed at all. Everything works
|
||||||
|
without them, so install_comfyui.sh does not install them either — it only
|
||||||
|
warns if ffmpeg is missing, since lektor_ai_comfy.py needs it (ComfyUI
|
||||||
|
itself does not).
|
||||||
|
=============================================================================
|
||||||
|
|
||||||
|
Option A — Conda/Mamba (cleanest match for "a virtual environment, no system changes")
|
||||||
|
|
||||||
|
Miniforge gives you its own Python and the native libs (portaudio, ffmpeg, espeak-ng…) inside the environment prefix, so there's no deadsnakes, no sudo apt, nothing in /usr.
|
||||||
|
|
||||||
|
bash
|
||||||
|
# Install Miniforge into its own dir — fully self-contained, removable
|
||||||
|
wget https://github.com/conda-forge/miniforge/releases/latest/download/Miniforge3-Linux-x86_64.sh
|
||||||
|
bash Miniforge3-Linux-x86_64.sh -b -p ~/miniforge3
|
||||||
|
|
||||||
|
# Activate WITHOUT running `conda init` so it doesn't even touch your .bashrc:
|
||||||
|
source ~/miniforge3/etc/profile.d/conda.sh
|
||||||
|
|
||||||
|
conda create -n comfy-tts python=3.12 -y
|
||||||
|
conda activate comfy-tts
|
||||||
|
|
||||||
|
# The libs that were apt before — now inside the env, no sudo:
|
||||||
|
conda install -c conda-forge portaudio libsamplerate ffmpeg espeak-ng git -y
|
||||||
|
|
||||||
|
# PyTorch via pip wheels (more current than conda's channel), cu128 for your 3090:
|
||||||
|
pip install torch torchvision torchaudio --index-url https://download.pytorch.org/whl/cu128
|
||||||
|
|
||||||
|
Then ComfyUI + the suite, all inside the env:
|
||||||
|
|
||||||
|
bash
|
||||||
|
mkdir -p ~/comfy-tts-app && cd ~/comfy-tts-app
|
||||||
|
git clone https://github.com/comfyanonymous/ComfyUI.git
|
||||||
|
cd ComfyUI && pip install -r requirements.txt
|
||||||
|
cd custom_nodes
|
||||||
|
git clone https://github.com/diodiogod/TTS-Audio-Suite.git
|
||||||
|
cd TTS-Audio-Suite && python install.py
|
||||||
|
|
||||||
|
Launch:
|
||||||
|
|
||||||
|
bash
|
||||||
|
source ~/miniforge3/etc/profile.d/conda.sh
|
||||||
|
conda activate comfy-tts
|
||||||
|
cd ~/comfy-tts-app/ComfyUI && python main.py --port 8189
|
||||||
|
|
||||||
|
To remove all of it: conda env remove -n comfy-tts, delete ~/comfy-tts-app, and if you want it truly gone, rm -rf ~/miniforge3. Your system Python and apt are never touched.
|
||||||
@@ -0,0 +1,232 @@
|
|||||||
|
absl-py==2.4.0
|
||||||
|
accelerate==1.14.0
|
||||||
|
aiohappyeyeballs==2.6.2
|
||||||
|
aiohttp==3.14.1
|
||||||
|
aiosignal==1.4.0
|
||||||
|
alembic==1.18.5
|
||||||
|
aliyun-python-sdk-core==2.16.0
|
||||||
|
aliyun-python-sdk-kms==2.16.5
|
||||||
|
annotated-doc==0.0.4
|
||||||
|
annotated-types==0.7.0
|
||||||
|
antlr4-python3-runtime==4.9.3
|
||||||
|
anyio==4.14.1
|
||||||
|
attrs==26.1.0
|
||||||
|
audioread==3.1.0
|
||||||
|
av==17.1.0
|
||||||
|
babel==2.18.0
|
||||||
|
bitsandbytes==0.49.2
|
||||||
|
blake3==1.0.9
|
||||||
|
braceexpand==0.1.7
|
||||||
|
certifi==2026.6.17
|
||||||
|
cffi==2.0.0
|
||||||
|
charset-normalizer==3.4.7
|
||||||
|
click==8.4.2
|
||||||
|
cn2an==0.5.24
|
||||||
|
comfy-aimdo==0.4.10
|
||||||
|
comfy-kitchen==0.2.12
|
||||||
|
comfyui-embedded-docs==0.5.5
|
||||||
|
comfyui-workflow-templates-core==0.3.263
|
||||||
|
comfyui-workflow-templates-media-api==0.3.81
|
||||||
|
comfyui-workflow-templates-media-image==0.3.157
|
||||||
|
comfyui-workflow-templates-media-other==0.3.226
|
||||||
|
comfyui-workflow-templates-media-video==0.3.98
|
||||||
|
comfyui_frontend_package==1.45.19
|
||||||
|
comfyui_workflow_templates==0.10.7
|
||||||
|
conformer==0.3.2
|
||||||
|
crcmod==1.7
|
||||||
|
cryptography==49.0.0
|
||||||
|
csvw==4.0.0
|
||||||
|
cuda-bindings==12.9.4
|
||||||
|
cuda-pathfinder==1.2.2
|
||||||
|
cuda-toolkit==12.8.1
|
||||||
|
Cython==3.2.6
|
||||||
|
dacite==1.9.2
|
||||||
|
datasets==5.0.0
|
||||||
|
decorator==5.3.1
|
||||||
|
dill==0.4.1
|
||||||
|
Distance==0.1.3
|
||||||
|
dlinfo==2.0.0
|
||||||
|
dyNET38==2.2
|
||||||
|
echo-tts==0.1.4
|
||||||
|
editdistance==0.8.1
|
||||||
|
einops==0.8.2
|
||||||
|
einx==0.4.3
|
||||||
|
ema-pytorch==0.7.9
|
||||||
|
encodec==0.1.1
|
||||||
|
faiss-cpu==1.14.3
|
||||||
|
filelock==3.29.0
|
||||||
|
flatbuffers==25.12.19
|
||||||
|
frozendict==2.4.7
|
||||||
|
frozenlist==1.8.0
|
||||||
|
fsspec==2026.4.0
|
||||||
|
funasr==1.3.14
|
||||||
|
g2p-en==2.1.0
|
||||||
|
gitdb==4.0.12
|
||||||
|
GitPython==3.1.50
|
||||||
|
glfw==2.10.0
|
||||||
|
greenlet==3.5.2
|
||||||
|
h11==0.16.0
|
||||||
|
h5py==3.16.0
|
||||||
|
hf-xet==1.5.1
|
||||||
|
httpcore==1.0.9
|
||||||
|
httpx==0.28.1
|
||||||
|
huggingface_hub==1.21.0
|
||||||
|
hydra-core==1.3.3
|
||||||
|
hyper-connections==0.4.11
|
||||||
|
HyperPyYAML==1.2.3
|
||||||
|
idna==3.18
|
||||||
|
inflect==7.5.0
|
||||||
|
isodate==0.7.2
|
||||||
|
jaconv==0.5.0
|
||||||
|
jamo==0.4.1
|
||||||
|
jieba==0.42.1
|
||||||
|
Jinja2==3.1.6
|
||||||
|
jmespath==0.10.0
|
||||||
|
joblib==1.5.3
|
||||||
|
json5==0.15.0
|
||||||
|
jsonschema==4.26.0
|
||||||
|
jsonschema-specifications==2025.9.1
|
||||||
|
kaldiio==2.18.1
|
||||||
|
keras==3.15.0
|
||||||
|
kornia==0.8.3
|
||||||
|
kornia_rs==0.1.14
|
||||||
|
langcodes==3.5.1
|
||||||
|
language-tags==1.3.1
|
||||||
|
lazy-loader==0.5
|
||||||
|
librosa==0.11.0
|
||||||
|
lingua-language-detector==2.2.0
|
||||||
|
llvmlite==0.47.0
|
||||||
|
local-attention==1.11.2
|
||||||
|
loguru==0.7.3
|
||||||
|
Mako==1.3.12
|
||||||
|
markdown-it-py==4.2.0
|
||||||
|
MarkupSafe==3.0.3
|
||||||
|
mdurl==0.1.2
|
||||||
|
ml_dtypes==0.5.4
|
||||||
|
modelscope==1.37.1
|
||||||
|
monotonic-alignment-search==0.2.1
|
||||||
|
more-itertools==11.1.0
|
||||||
|
mpmath==1.3.0
|
||||||
|
msgpack==1.2.1
|
||||||
|
multidict==6.7.1
|
||||||
|
multiprocess==0.70.19
|
||||||
|
munch==4.0.0
|
||||||
|
nagisa==0.2.12
|
||||||
|
namex==0.1.0
|
||||||
|
narwhals==2.22.1
|
||||||
|
networkx==3.6.1
|
||||||
|
ninja==1.13.0
|
||||||
|
nltk==3.9.4
|
||||||
|
numba==0.65.1
|
||||||
|
numpy==2.2.6
|
||||||
|
nvidia-cublas-cu12==12.8.4.1
|
||||||
|
nvidia-cuda-cupti-cu12==12.8.90
|
||||||
|
nvidia-cuda-nvrtc-cu12==12.8.93
|
||||||
|
nvidia-cuda-runtime-cu12==12.8.90
|
||||||
|
nvidia-cudnn-cu12==9.19.0.56
|
||||||
|
nvidia-cufft-cu12==11.3.3.83
|
||||||
|
nvidia-cufile-cu12==1.13.1.3
|
||||||
|
nvidia-curand-cu12==10.3.9.90
|
||||||
|
nvidia-cusolver-cu12==11.7.3.90
|
||||||
|
nvidia-cusparse-cu12==12.5.8.93
|
||||||
|
nvidia-cusparselt-cu12==0.7.1
|
||||||
|
nvidia-nccl-cu12==2.28.9
|
||||||
|
nvidia-nvjitlink-cu12==12.8.93
|
||||||
|
nvidia-nvshmem-cu12==3.4.5
|
||||||
|
nvidia-nvtx-cu12==12.8.90
|
||||||
|
omegaconf==2.3.1
|
||||||
|
onnxruntime==1.27.0
|
||||||
|
openai-whisper==20250625
|
||||||
|
optree==0.19.1
|
||||||
|
orjson==3.11.9
|
||||||
|
oss2==2.19.1
|
||||||
|
packaging @ file:///home/conda/feedstock_root/build_artifacts/bld/rattler-build_packaging_1777103621/work
|
||||||
|
pandas==3.0.3
|
||||||
|
peft==0.19.1
|
||||||
|
phonemizer==3.3.0
|
||||||
|
pillow==12.2.0
|
||||||
|
platformdirs==4.10.0
|
||||||
|
pooch==1.9.0
|
||||||
|
praat-parselmouth==0.4.7
|
||||||
|
proces==0.1.7
|
||||||
|
propcache==0.5.2
|
||||||
|
protobuf==7.35.1
|
||||||
|
psutil==7.2.2
|
||||||
|
punctuators==0.0.7
|
||||||
|
pyarrow==24.0.0
|
||||||
|
pycparser==3.0
|
||||||
|
pycryptodome==3.23.0
|
||||||
|
pydantic==2.13.4
|
||||||
|
pydantic-settings==2.14.2
|
||||||
|
pydantic_core==2.46.4
|
||||||
|
pydub==0.25.1
|
||||||
|
Pygments==2.20.0
|
||||||
|
pynndescent==0.6.0
|
||||||
|
PyOpenGL==3.1.10
|
||||||
|
pyparsing==3.3.2
|
||||||
|
pyphen==0.17.2
|
||||||
|
pypinyin==0.55.0
|
||||||
|
python-dateutil==2.9.0.post0
|
||||||
|
python-dotenv==1.2.2
|
||||||
|
pyworld==0.3.5
|
||||||
|
PyYAML==6.0.3
|
||||||
|
rdflib==7.6.0
|
||||||
|
referencing==0.37.0
|
||||||
|
regex==2026.5.9
|
||||||
|
requests==2.34.2
|
||||||
|
rfc3986==1.5.0
|
||||||
|
rich==15.0.0
|
||||||
|
rpds-py==2026.5.1
|
||||||
|
ruamel.yaml==0.18.17
|
||||||
|
ruamel.yaml.clib==0.2.15
|
||||||
|
safetensors==0.8.0
|
||||||
|
scikit-learn==1.9.0
|
||||||
|
scipy==1.18.0
|
||||||
|
segments==2.4.0
|
||||||
|
sentencepiece==0.2.1
|
||||||
|
sentry-sdk==2.63.0
|
||||||
|
setuptools==70.2.0
|
||||||
|
shellingham==1.5.4
|
||||||
|
simpleeval==1.0.7
|
||||||
|
six==1.17.0
|
||||||
|
smmap==5.0.3
|
||||||
|
sounddevice==0.5.5
|
||||||
|
soundfile==0.14.0
|
||||||
|
soxr==1.1.0
|
||||||
|
spandrel==0.4.2
|
||||||
|
SQLAlchemy==2.0.51
|
||||||
|
sympy==1.14.0
|
||||||
|
tensorboardX==2.6.5
|
||||||
|
termcolor==3.3.0
|
||||||
|
textstat==0.7.13
|
||||||
|
threadpoolctl==3.6.0
|
||||||
|
tiktoken==0.13.0
|
||||||
|
tokenizers==0.22.2
|
||||||
|
torch==2.11.0+cu128
|
||||||
|
torch-complex==0.4.4
|
||||||
|
torch-einops-utils==0.1.4
|
||||||
|
torchaudio==2.11.0+cu128
|
||||||
|
torchcodec==0.14.0
|
||||||
|
torchdiffeq==0.2.5
|
||||||
|
torchfcpe==0.0.4
|
||||||
|
torchsde==0.2.6
|
||||||
|
torchvision==0.26.0+cu128
|
||||||
|
tqdm==4.68.3
|
||||||
|
trampoline==0.1.2
|
||||||
|
transformers==5.12.1
|
||||||
|
triton==3.6.0
|
||||||
|
typeguard==4.5.2
|
||||||
|
typer==0.25.1
|
||||||
|
typing-inspection==0.4.2
|
||||||
|
typing_extensions==4.15.0
|
||||||
|
umap-learn==0.5.12
|
||||||
|
Unidecode==1.4.0
|
||||||
|
uritemplate==4.2.0
|
||||||
|
urllib3==2.7.0
|
||||||
|
vocos==0.1.0
|
||||||
|
wandb==0.28.0
|
||||||
|
webdataset==1.0.2
|
||||||
|
wheel==0.47.0
|
||||||
|
x-transformers==2.23.1
|
||||||
|
xxhash==3.7.1
|
||||||
|
yarl==1.24.2
|
||||||
@@ -0,0 +1,66 @@
|
|||||||
|
# lektor_ai_comfy — Documentation
|
||||||
|
|
||||||
|
Converts SRT subtitles into a Polish AI voiceover in the *lektor* style (a single
|
||||||
|
narrator read over the quietened original soundtrack) using a local ComfyUI
|
||||||
|
instance running the Higgs Audio V3 TTS model, then muxes the result back into
|
||||||
|
the video as a second audio track.
|
||||||
|
|
||||||
|
The source video is never re-encoded. Video and the original audio are
|
||||||
|
stream-copied; only the new voiceover track is encoded.
|
||||||
|
|
||||||
|
## Documentation map
|
||||||
|
|
||||||
|
| Document | What it covers |
|
||||||
|
|---|---|
|
||||||
|
| [requirements.md](requirements.md) | Hardware, software, and the exact dependency surface |
|
||||||
|
| [installation.md](installation.md) | Installing ComfyUI + the TTS engine, and verifying it |
|
||||||
|
| [usage.md](usage.md) | Every CLI flag, the working modes, and practical recipes |
|
||||||
|
| [architecture.md](architecture.md) | How the pipeline works internally, end to end |
|
||||||
|
| [troubleshooting.md](troubleshooting.md) | Diagnosed failure modes and their fixes |
|
||||||
|
|
||||||
|
## Quick start
|
||||||
|
|
||||||
|
Assuming ComfyUI is already installed (if not, see
|
||||||
|
[installation.md](installation.md)):
|
||||||
|
|
||||||
|
```bash
|
||||||
|
# 1. Start the TTS backend and leave it running
|
||||||
|
~/comfy-tts-app/start_comfyui.sh
|
||||||
|
|
||||||
|
# 2. In another terminal, generate the voiceover
|
||||||
|
cd ~/Programy/lektor_ai
|
||||||
|
python3 lektor_ai_comfy.py "input/film.mkv" --embedded-subs
|
||||||
|
```
|
||||||
|
|
||||||
|
The result is `output/film_lektor.mkv` with two audio tracks — **Original** and
|
||||||
|
**Polish** — plus every subtitle track from the source.
|
||||||
|
|
||||||
|
## The two halves of this project
|
||||||
|
|
||||||
|
The pieces are deliberately decoupled and it helps to keep them apart when
|
||||||
|
diagnosing anything:
|
||||||
|
|
||||||
|
**ComfyUI + Higgs Audio V3** — a local HTTP service on port 8189 that turns text
|
||||||
|
into speech. Heavy: ~22 GB on disk, needs an NVIDIA GPU, lives in its own conda
|
||||||
|
environment. It knows nothing about subtitles or video.
|
||||||
|
|
||||||
|
**`lektor_ai_comfy.py`** — a single-file script that parses subtitles, drives
|
||||||
|
ComfyUI over HTTP, and does all timing and audio/video assembly with ffmpeg. It
|
||||||
|
is pure Python standard library, runs on the system Python, and needs no conda
|
||||||
|
environment or pip packages of its own.
|
||||||
|
|
||||||
|
So a TTS problem (bad pronunciation, hallucination, mumbling) belongs to the
|
||||||
|
first half, and a timing, mixing, or muxing problem belongs to the second.
|
||||||
|
|
||||||
|
## Conventions in these docs
|
||||||
|
|
||||||
|
Paths are written as they exist on the machine this was developed on:
|
||||||
|
|
||||||
|
```
|
||||||
|
~/Programy/lektor_ai the project
|
||||||
|
~/comfy-tts-app/ComfyUI the TTS backend
|
||||||
|
~/miniforge3 the conda distribution
|
||||||
|
```
|
||||||
|
|
||||||
|
If yours differ, update the constants at the top of `lektor_ai_comfy.py` — see
|
||||||
|
[installation.md](installation.md#configuring-the-script).
|
||||||
@@ -0,0 +1,215 @@
|
|||||||
|
# Architecture
|
||||||
|
|
||||||
|
How `lektor_ai_comfy.py` turns subtitles into a finished video, and why each
|
||||||
|
stage works the way it does. Most of the non-obvious choices here exist because
|
||||||
|
the straightforward version was tried first and broke.
|
||||||
|
|
||||||
|
```
|
||||||
|
SRT / embedded track
|
||||||
|
│ parse, clean, split dialogue
|
||||||
|
▼
|
||||||
|
subtitle blocks ──────────────► output/tts_input.srt
|
||||||
|
│ one HTTP job per block
|
||||||
|
▼
|
||||||
|
ComfyUI + Higgs Audio V3
|
||||||
|
│ per-clip atempo + fade
|
||||||
|
▼
|
||||||
|
clips/clip_NNNN.flac
|
||||||
|
│ adelay + amix, batched
|
||||||
|
▼
|
||||||
|
tts_assembled.flac
|
||||||
|
│ pass 1: mix with original audio
|
||||||
|
│ pass 2: stream-copy mux
|
||||||
|
▼
|
||||||
|
{stem}_lektor.mkv
|
||||||
|
```
|
||||||
|
|
||||||
|
## 1. Subtitle parsing
|
||||||
|
|
||||||
|
`parse_subtitle_blocks()` reads the SRT with `utf-8-sig` (tolerating a BOM),
|
||||||
|
splits on blank lines, and keeps blocks whose start falls inside the requested
|
||||||
|
range. Times become **relative to `--start`**, which is why a later `--rebuild`
|
||||||
|
must repeat the same `--start`/`--end`.
|
||||||
|
|
||||||
|
`_clean_text()` then normalises each block, in this order:
|
||||||
|
|
||||||
|
1. **Strip formatting tags.** `{...}` ASS overrides (`{\an8}`, `{\pos(x,y)}`)
|
||||||
|
and `<...>` HTML tags (`<i>`, `<font …>`).
|
||||||
|
2. **Detect dialogue.** Lines beginning `- ` mean two speakers and are kept
|
||||||
|
separate; ordinary wrapped lines are joined with a space.
|
||||||
|
3. **Sentence-case ALL-CAPS blocks**, which the model would otherwise shout.
|
||||||
|
4. **Capitalise a lowercase first letter**, since a fragment continuing from the
|
||||||
|
previous subtitle confuses the model.
|
||||||
|
5. **Add terminal punctuation** so the model produces a natural falling tail.
|
||||||
|
|
||||||
|
Step 1 must come first. `<i>ALL CAPS</i>` looks mixed-case while the tag is
|
||||||
|
still present — the `i` in `<i>` is a lowercase letter — so the caps detection
|
||||||
|
in step 3 would silently miss it.
|
||||||
|
|
||||||
|
`split_dialogue_blocks()` then divides a two-speaker block into one clip per
|
||||||
|
line, apportioning the block's duration by character count.
|
||||||
|
|
||||||
|
The result is written to `output/tts_input.srt`. That file is the source of
|
||||||
|
truth for block numbering: block *N* there is `clips/clip_NNNN.flac` and the
|
||||||
|
argument to `--regen N`.
|
||||||
|
|
||||||
|
## 2. Speech generation
|
||||||
|
|
||||||
|
For each block the script POSTs a workflow graph to `/prompt`, polls
|
||||||
|
`/history/{id}` every 3 s, and reads the resulting file directly from ComfyUI's
|
||||||
|
output directory.
|
||||||
|
|
||||||
|
The graph has five nodes:
|
||||||
|
|
||||||
|
| Node | Class | Role |
|
||||||
|
|---|---|---|
|
||||||
|
| 1 | `HiggsAudioV3EngineNode` | Model, temperature, top_p 0.95, top_k 50 |
|
||||||
|
| 2 | `LoadAudio` | Voice reference WAV from ComfyUI's input dir |
|
||||||
|
| 3 | `CharacterVoicesNode` | Pairs that audio with its transcript |
|
||||||
|
| 4 | `UnifiedTTSTextNode` | The text, the seed, `enable_audio_cache: False` |
|
||||||
|
| 5 | `SaveAudio` | Writes FLAC |
|
||||||
|
|
||||||
|
Three details matter:
|
||||||
|
|
||||||
|
- **`enable_audio_cache: False`** is essential. With caching on, ComfyUI returns
|
||||||
|
a previous result for repeated text and every `--regen` is a no-op.
|
||||||
|
- **The seed is `base_seed + block_index`**, so a fixed `--seed` is fully
|
||||||
|
reproducible while each clip still differs.
|
||||||
|
- **No language tag is sent.** Higgs Audio V3 detects Polish from the text
|
||||||
|
itself; its tokenizer has no `<|lang:pl|>` token, and a `[pl:]` prefix is
|
||||||
|
stripped by the character parser before it reaches the model. A `lang: en`
|
||||||
|
line in the ComfyUI log is cosmetic.
|
||||||
|
|
||||||
|
Each finished clip is optionally head-trimmed (`--strip-leading-ms`), then
|
||||||
|
sped up with `atempo` and given a short fade-out — **per clip, before
|
||||||
|
assembly**. Applying `atempo` to the assembled track instead would speed up the
|
||||||
|
silences between clips and shift every subsequent timestamp.
|
||||||
|
|
||||||
|
## 3. Placement
|
||||||
|
|
||||||
|
Clips are positioned with a push-forward cursor:
|
||||||
|
|
||||||
|
```python
|
||||||
|
actual_start = max(t_start, cursor)
|
||||||
|
cursor = actual_start + duration
|
||||||
|
```
|
||||||
|
|
||||||
|
A clip starts at its subtitle time, or when the previous clip finished if that
|
||||||
|
is later. Clips are **never trimmed** to their subtitle window — cutting speech
|
||||||
|
mid-sentence sounds far worse than running a little long. Overrun cascades
|
||||||
|
forward and then resolves by itself at the next natural gap in the subtitles,
|
||||||
|
because a gap lets the cursor fall back below the next `t_start`.
|
||||||
|
|
||||||
|
The console prints `[+1.20s behind SRT]` when drift occurs. Steady growth means
|
||||||
|
`--speed` is too low for the material.
|
||||||
|
|
||||||
|
## 4. Assembly
|
||||||
|
|
||||||
|
Each clip becomes an `adelay` filter input and everything is mixed with `amix`:
|
||||||
|
|
||||||
|
```
|
||||||
|
[0:a]adelay=1340:all=1[a0]; [1:a]adelay=4820:all=1[a1]; …
|
||||||
|
[a0][a1]…amix=inputs=N:duration=longest:normalize=0[out]
|
||||||
|
```
|
||||||
|
|
||||||
|
`normalize=0` is important — `amix` otherwise divides by the input count, which
|
||||||
|
would make every clip inaudibly quiet across hundreds of inputs.
|
||||||
|
|
||||||
|
Above 500 clips the work is **batched**, because one ffmpeg process cannot hold
|
||||||
|
thousands of open inputs. Each batch is assembled to its own temporary FLAC and
|
||||||
|
the batches are then mixed together.
|
||||||
|
|
||||||
|
Batches use **relative** delays: the first clip's start time is subtracted from
|
||||||
|
every clip in the batch, and re-applied when the batches are combined. Without
|
||||||
|
that subtraction, batch 7 would begin with 40 minutes of silence, wasting time
|
||||||
|
and making the progress bar useless.
|
||||||
|
|
||||||
|
## 5. Muxing
|
||||||
|
|
||||||
|
`merge_voiceover()` runs **two separate ffmpeg passes**. This is deliberate and
|
||||||
|
should not be collapsed back into one.
|
||||||
|
|
||||||
|
### Pass 1 — build the lektor mix
|
||||||
|
|
||||||
|
```
|
||||||
|
[0:a:0]<norm>,aformat=channel_layouts=stereo,volume=<orig>[orig_mix];
|
||||||
|
[1:a]aformat=channel_layouts=stereo,volume=<tts>[tts];
|
||||||
|
[orig_mix][tts]amix=inputs=2:duration=first:normalize=0[lektor]
|
||||||
|
```
|
||||||
|
|
||||||
|
Written to a temporary `*_mix.m4a`, AAC 192k, and deleted in a `finally` block.
|
||||||
|
|
||||||
|
**Stereo, not 5.1.** ffmpeg's AAC encoder does not signal a 5.1 channel layout —
|
||||||
|
the track comes back tagged `unknown` in every container tested, and players
|
||||||
|
that cannot route the channels play silence. Stereo is tagged correctly, plays
|
||||||
|
everywhere, and spends the 192 kbit/s on two channels instead of six. Track 0
|
||||||
|
keeps the untouched 5.1, so no surround information is lost.
|
||||||
|
|
||||||
|
**PTS normalisation.** MKV rips often start their audio at a non-zero PTS (one
|
||||||
|
source here begins at 4.043 s). `amix` cannot emit output before both inputs
|
||||||
|
have frames, so the voiceover would be delayed by that offset. When ffprobe
|
||||||
|
reports an offset the filter becomes:
|
||||||
|
|
||||||
|
```
|
||||||
|
asetpts=PTS-STARTPTS,adelay=<offset_ms>:all=1,aresample=async=1
|
||||||
|
```
|
||||||
|
|
||||||
|
`asetpts` rebases the stream to zero and `adelay` reinserts the offset as real
|
||||||
|
silence, so the background audio stays where it belongs while `amix` receives
|
||||||
|
frames from both inputs starting at zero.
|
||||||
|
|
||||||
|
`duration=first` with `orig_mix` listed first means the mix runs as long as the
|
||||||
|
film, not as long as the voiceover.
|
||||||
|
|
||||||
|
### Pass 2 — stream-copy mux
|
||||||
|
|
||||||
|
Video, original audio, the mix, and every subtitle track are copied into the
|
||||||
|
final MKV with `-c copy`. Nothing is re-encoded, and it runs at several hundred
|
||||||
|
times real time.
|
||||||
|
|
||||||
|
Two flags carry the weight here:
|
||||||
|
|
||||||
|
**`-max_interleave_delta 0`**, placed **after the maps and before the output
|
||||||
|
path**. It is an output-muxer option; before `-i` it silently configures the
|
||||||
|
demuxer and does nothing. Without it, sparse subtitle tracks make the muxer
|
||||||
|
abandon interleaving after its default 10-second window, and the lektor audio
|
||||||
|
ends up written hundreds of megabytes away from the video it belongs with —
|
||||||
|
measurably fine to ffmpeg, but silent in any player that reads sequentially.
|
||||||
|
|
||||||
|
**MKV output**, not MP4. MKV copies subtitles of any codec without conversion,
|
||||||
|
carries DTS and TrueHD in track 0, and avoids the `mov_text` conversion that
|
||||||
|
made large multi-subtitle files abort mid-mux.
|
||||||
|
|
||||||
|
### Why two passes
|
||||||
|
|
||||||
|
Doing the filtering and the subtitle muxing in one command fails on sources with
|
||||||
|
many subtitle tracks. The sparse subtitle streams stall the muxer while the
|
||||||
|
audio filter graph keeps producing, and ffmpeg aborts partway through with:
|
||||||
|
|
||||||
|
```
|
||||||
|
Error while filtering: Invalid argument
|
||||||
|
Failed to inject frame into filter network: Invalid argument
|
||||||
|
```
|
||||||
|
|
||||||
|
The message points at audio, but audio is not the cause. Splitting the work
|
||||||
|
keeps the filter graph free of subtitle streams and makes the final mux a pure
|
||||||
|
copy. It is also about ten times faster.
|
||||||
|
|
||||||
|
## Progress reporting
|
||||||
|
|
||||||
|
`_run_progress()` adds `-progress pipe:1 -nostats` before the output path, reads
|
||||||
|
`out_time_us=` lines from stdout, and rewrites a single line with `\r`. A daemon
|
||||||
|
thread drains stderr concurrently, so ffmpeg cannot deadlock on a full stderr
|
||||||
|
pipe while the main thread reads stdout.
|
||||||
|
|
||||||
|
## Design rules
|
||||||
|
|
||||||
|
Worth preserving, since each was learned by breaking it:
|
||||||
|
|
||||||
|
1. **Per-clip processing before assembly**, never on the assembled track.
|
||||||
|
2. **Never trim clips** to the subtitle window.
|
||||||
|
3. **Keep every clip file** — `--regen` depends on them.
|
||||||
|
4. **FLAC throughout**, encoding only the final voiceover track.
|
||||||
|
5. **Two audio tracks**, with the original stream-copied and untouched.
|
||||||
|
6. **Verify muxed output by byte position**, not only by volume. A track can
|
||||||
|
measure perfectly and still be unplayable.
|
||||||
@@ -0,0 +1,207 @@
|
|||||||
|
# Installation
|
||||||
|
|
||||||
|
Two things need to exist: the **ComfyUI backend** (large, isolated, GPU-bound)
|
||||||
|
and the **script** (which needs nothing installed at all).
|
||||||
|
|
||||||
|
## Automated install
|
||||||
|
|
||||||
|
`install_comfyui.sh` in the project root reproduces the exact working
|
||||||
|
environment. It is idempotent — safe to re-run, and it skips any step already
|
||||||
|
satisfied.
|
||||||
|
|
||||||
|
```bash
|
||||||
|
cd ~/Programy/lektor_ai
|
||||||
|
./install_comfyui.sh
|
||||||
|
```
|
||||||
|
|
||||||
|
Roughly 30–60 minutes, mostly downloading. Expect ~22 GB.
|
||||||
|
|
||||||
|
### Options
|
||||||
|
|
||||||
|
| Flag | Effect |
|
||||||
|
|---|---|
|
||||||
|
| *(none)* | Pinned versions — ComfyUI `7cb784e0`, suite `v5.3.0`. Recommended. |
|
||||||
|
| `--latest` | Newest ComfyUI and suite instead. You are on your own for breakage. |
|
||||||
|
| `--skip-model` | Skip the 8.7 GB download; it happens on first generation anyway. |
|
||||||
|
| `--lock FILE` | Restore an exact package set from a `pip freeze` lockfile. |
|
||||||
|
| `--help` | Usage summary. |
|
||||||
|
|
||||||
|
### Changing where things go
|
||||||
|
|
||||||
|
```bash
|
||||||
|
CONDA_ROOT=/opt/miniforge3 COMFY_ROOT=/data/comfy ./install_comfyui.sh
|
||||||
|
```
|
||||||
|
|
||||||
|
| Variable | Default |
|
||||||
|
|---|---|
|
||||||
|
| `CONDA_ROOT` | `~/miniforge3` |
|
||||||
|
| `COMFY_ROOT` | `~/comfy-tts-app` |
|
||||||
|
| `ENV_NAME` | `comfy-tts` |
|
||||||
|
| `COMFY_PORT` | `8189` |
|
||||||
|
| `PYTHON_VERSION` | `3.12` |
|
||||||
|
|
||||||
|
If you change `COMFY_ROOT` or `COMFY_PORT`, update the script constants too —
|
||||||
|
see [Configuring the script](#configuring-the-script).
|
||||||
|
|
||||||
|
### Exact reproduction
|
||||||
|
|
||||||
|
`comfyui-requirements.lock.txt` pins all 232 packages. To rebuild a
|
||||||
|
byte-identical environment:
|
||||||
|
|
||||||
|
```bash
|
||||||
|
./install_comfyui.sh --lock comfyui-requirements.lock.txt
|
||||||
|
```
|
||||||
|
|
||||||
|
Regenerate it after intentional changes:
|
||||||
|
|
||||||
|
```bash
|
||||||
|
source ~/miniforge3/etc/profile.d/conda.sh && conda activate comfy-tts
|
||||||
|
pip freeze > comfyui-requirements.lock.txt
|
||||||
|
```
|
||||||
|
|
||||||
|
## What the installer does
|
||||||
|
|
||||||
|
Useful to know when something fails midway, since each step can be repeated on
|
||||||
|
its own.
|
||||||
|
|
||||||
|
1. **Preflight** — checks Linux, git, a downloader, GPU, disk space, ffmpeg.
|
||||||
|
Only missing git or a downloader is fatal; the rest warn.
|
||||||
|
2. **Miniforge** → `~/miniforge3`, batch mode. `conda init` is deliberately not
|
||||||
|
run, so your `.bashrc` is never touched.
|
||||||
|
3. **Conda env** `comfy-tts` with Python 3.12, then a guard that refuses to
|
||||||
|
continue if the wrong Python somehow ended up active.
|
||||||
|
4. **PyTorch** from the cu128 index, pinned to 2.11.0.
|
||||||
|
5. **ComfyUI** cloned and checked out at the pinned commit (detached HEAD — this
|
||||||
|
is expected, not an error), then its `requirements.txt` installed.
|
||||||
|
6. **TTS-Audio-Suite** cloned, checked out at `v5.3.0`, then **`python
|
||||||
|
install.py`** — its own resolver, which must not be replaced with a plain
|
||||||
|
pip install (see [requirements.md](requirements.md#dependency-surface)).
|
||||||
|
7. **Model** — `bosonai/higgs-audio-v3-tts-4b` into
|
||||||
|
`ComfyUI/models/TTS/higgs_audio_v3/`.
|
||||||
|
8. **Launcher** written to `$COMFY_ROOT/start_comfyui.sh`.
|
||||||
|
9. **Verification** — imports torch, torchaudio, transformers, safetensors,
|
||||||
|
soundfile, librosa and reports CUDA availability.
|
||||||
|
|
||||||
|
## Manual install
|
||||||
|
|
||||||
|
The equivalent by hand, if you would rather not use the script:
|
||||||
|
|
||||||
|
```bash
|
||||||
|
# Miniforge, self-contained, no shell integration
|
||||||
|
wget https://github.com/conda-forge/miniforge/releases/latest/download/Miniforge3-Linux-x86_64.sh
|
||||||
|
bash Miniforge3-Linux-x86_64.sh -b -p ~/miniforge3
|
||||||
|
source ~/miniforge3/etc/profile.d/conda.sh
|
||||||
|
|
||||||
|
conda create -n comfy-tts python=3.12 -y
|
||||||
|
conda activate comfy-tts
|
||||||
|
|
||||||
|
# PyTorch with the CUDA 12.8 runtime bundled
|
||||||
|
pip install torch==2.11.0 torchvision==0.26.0 torchaudio==2.11.0 \
|
||||||
|
--index-url https://download.pytorch.org/whl/cu128
|
||||||
|
|
||||||
|
# ComfyUI, pinned
|
||||||
|
mkdir -p ~/comfy-tts-app && cd ~/comfy-tts-app
|
||||||
|
git clone https://github.com/comfyanonymous/ComfyUI.git
|
||||||
|
cd ComfyUI && git checkout 7cb784e0f48784bb6ed588912e186e5ee1e9ee68
|
||||||
|
pip install -r requirements.txt
|
||||||
|
|
||||||
|
# The TTS node, pinned — note install.py, NOT requirements.txt
|
||||||
|
cd custom_nodes
|
||||||
|
git clone https://github.com/diodiogod/TTS-Audio-Suite.git
|
||||||
|
cd TTS-Audio-Suite && git checkout tags/v5.3.0
|
||||||
|
python install.py
|
||||||
|
```
|
||||||
|
|
||||||
|
The model downloads itself on first generation. To fetch it ahead of time:
|
||||||
|
|
||||||
|
```python
|
||||||
|
from huggingface_hub import snapshot_download
|
||||||
|
snapshot_download(
|
||||||
|
repo_id="bosonai/higgs-audio-v3-tts-4b",
|
||||||
|
local_dir="~/comfy-tts-app/ComfyUI/models/TTS/higgs_audio_v3/higgs-audio-v3-tts-4b",
|
||||||
|
)
|
||||||
|
```
|
||||||
|
|
||||||
|
## Running the backend
|
||||||
|
|
||||||
|
```bash
|
||||||
|
~/comfy-tts-app/start_comfyui.sh
|
||||||
|
```
|
||||||
|
|
||||||
|
which is just:
|
||||||
|
|
||||||
|
```bash
|
||||||
|
#!/bin/bash
|
||||||
|
source ~/miniforge3/etc/profile.d/conda.sh
|
||||||
|
conda activate comfy-tts
|
||||||
|
cd ~/comfy-tts-app/ComfyUI && python main.py --port 8189
|
||||||
|
```
|
||||||
|
|
||||||
|
Leave it running in its own terminal for the whole session — model load takes
|
||||||
|
tens of seconds and would otherwise be repeated. The web UI at
|
||||||
|
<http://127.0.0.1:8189> is useful for watching progress and reading errors, but
|
||||||
|
is not needed; the script drives everything over HTTP.
|
||||||
|
|
||||||
|
## Configuring the script
|
||||||
|
|
||||||
|
`lektor_ai_comfy.py` has four constants at the top. If you installed anywhere
|
||||||
|
other than the defaults, edit them:
|
||||||
|
|
||||||
|
```python
|
||||||
|
COMFY_URL = "http://127.0.0.1:8189"
|
||||||
|
COMFY_OUTPUT_DIR = Path("/home/lukas/comfy-tts-app/ComfyUI/output")
|
||||||
|
COMFY_INPUT_DIR = Path("/home/lukas/comfy-tts-app/ComfyUI/input")
|
||||||
|
VOICE_PROMPTS_DIR = Path("/home/lukas/Programy/lektor_ai/gudowski")
|
||||||
|
```
|
||||||
|
|
||||||
|
`COMFY_URL` can also be overridden per-run with `--comfy-url`. The two directory
|
||||||
|
constants cannot — the script reads generated audio directly from ComfyUI's
|
||||||
|
output directory and copies voice references into its input directory, so both
|
||||||
|
must be real paths on the same filesystem.
|
||||||
|
|
||||||
|
The script itself needs no installation. Ensure ffmpeg is present:
|
||||||
|
|
||||||
|
```bash
|
||||||
|
ffmpeg -version && ffprobe -version
|
||||||
|
```
|
||||||
|
|
||||||
|
If missing, and you would rather not use sudo:
|
||||||
|
|
||||||
|
```bash
|
||||||
|
conda install -n comfy-tts -c conda-forge ffmpeg
|
||||||
|
```
|
||||||
|
|
||||||
|
Note that this puts ffmpeg inside the conda env, so the script must then be run
|
||||||
|
with that env active.
|
||||||
|
|
||||||
|
## Verifying
|
||||||
|
|
||||||
|
```bash
|
||||||
|
# 1. Backend responds
|
||||||
|
curl -s http://127.0.0.1:8189/system_stats | head -c 200
|
||||||
|
|
||||||
|
# 2. GPU visible to PyTorch
|
||||||
|
source ~/miniforge3/etc/profile.d/conda.sh && conda activate comfy-tts
|
||||||
|
python -c "import torch; print(torch.__version__, torch.cuda.is_available())"
|
||||||
|
# -> 2.11.0+cu128 True
|
||||||
|
|
||||||
|
# 3. Model present
|
||||||
|
ls -la ~/comfy-tts-app/ComfyUI/models/TTS/higgs_audio_v3/higgs-audio-v3-tts-4b/model.safetensors
|
||||||
|
|
||||||
|
# 4. End-to-end on 60 seconds of a real file
|
||||||
|
cd ~/Programy/lektor_ai
|
||||||
|
python3 lektor_ai_comfy.py "input/film.mkv" --embedded-subs --end 01:00
|
||||||
|
```
|
||||||
|
|
||||||
|
The short run is the real test — it exercises subtitle parsing, TTS, assembly
|
||||||
|
and muxing, and finishes in a couple of minutes.
|
||||||
|
|
||||||
|
## Uninstalling
|
||||||
|
|
||||||
|
Nothing was installed system-wide, so removal is just deleting directories:
|
||||||
|
|
||||||
|
```bash
|
||||||
|
conda env remove -n comfy-tts
|
||||||
|
rm -rf ~/comfy-tts-app
|
||||||
|
rm -rf ~/miniforge3 # only if nothing else uses it
|
||||||
|
```
|
||||||
@@ -0,0 +1,132 @@
|
|||||||
|
# Requirements
|
||||||
|
|
||||||
|
## Hardware
|
||||||
|
|
||||||
|
| | Minimum | Notes |
|
||||||
|
|---|---|---|
|
||||||
|
| GPU | NVIDIA, 12 GB VRAM | Developed on an RTX 4070 Ti SUPER (16 GB) |
|
||||||
|
| Driver | 525+ | Needed by the CUDA 12.8 wheels |
|
||||||
|
| Disk | ~35 GB free | ~22 GB installed, plus working room for output |
|
||||||
|
| RAM | 16 GB | The final mux briefly peaks around 2.6 GB |
|
||||||
|
|
||||||
|
The CUDA **toolkit** is not required. The PyTorch cu128 wheels bundle their own
|
||||||
|
CUDA runtime, so the system `nvcc` version is irrelevant — this machine has CUDA
|
||||||
|
11.5 installed system-wide while PyTorch runs cu128 perfectly well. Only the
|
||||||
|
driver version matters.
|
||||||
|
|
||||||
|
CPU-only inference technically works but is slow enough to be impractical for a
|
||||||
|
feature-length film.
|
||||||
|
|
||||||
|
### Disk footprint
|
||||||
|
|
||||||
|
```
|
||||||
|
~/miniforge3/envs/comfy-tts 9.7 GB conda env (PyTorch and ~233 packages)
|
||||||
|
~/comfy-tts-app/ComfyUI 3.2 GB application code and dependencies
|
||||||
|
└── models/TTS 8.8 GB Higgs Audio V3 weights
|
||||||
|
------
|
||||||
|
21.7 GB
|
||||||
|
```
|
||||||
|
|
||||||
|
Output is roughly the size of the source video, since video is stream-copied. A
|
||||||
|
60-minute 10 Mbit/s episode produces about 5 GB. Intermediate FLAC clips add a
|
||||||
|
few hundred MB per film and are kept deliberately, so individual clips can be
|
||||||
|
regenerated without redoing a whole run (see
|
||||||
|
[usage.md](usage.md#working-modes)).
|
||||||
|
|
||||||
|
## Software
|
||||||
|
|
||||||
|
| Component | Version here | Notes |
|
||||||
|
|---|---|---|
|
||||||
|
| OS | Linux (Ubuntu 22.04) | Only Linux is supported by the installer |
|
||||||
|
| ComfyUI | 0.26.0 (`7cb784e0`) | Pinned |
|
||||||
|
| TTS-Audio-Suite | v5.3.0 (`09df1681`) | The only custom node installed |
|
||||||
|
| Higgs Audio V3 | `bosonai/higgs-audio-v3-tts-4b` | 8.7 GB, auto-downloaded |
|
||||||
|
| Python (ComfyUI) | 3.12.13 | conda-forge, in the `comfy-tts` env |
|
||||||
|
| PyTorch | 2.11.0+cu128 | With torchvision 0.26.0, torchaudio 2.11.0 |
|
||||||
|
| Python (script) | 3.8+ | System Python; 3.10.12 here |
|
||||||
|
| ffmpeg / ffprobe | 4.4+ | System install; 4.4.2 here |
|
||||||
|
|
||||||
|
## Dependency surface
|
||||||
|
|
||||||
|
This is the point most worth internalising, because it makes debugging much
|
||||||
|
simpler: **the two halves have almost nothing in common.**
|
||||||
|
|
||||||
|
### `lektor_ai_comfy.py` — standard library only
|
||||||
|
|
||||||
|
```python
|
||||||
|
argparse json random re shutil subprocess sys threading time uuid
|
||||||
|
pathlib.Path urllib.request
|
||||||
|
```
|
||||||
|
|
||||||
|
No third-party packages. No conda environment. No virtualenv. It runs on the
|
||||||
|
system Python:
|
||||||
|
|
||||||
|
```bash
|
||||||
|
/usr/bin/python3 lektor_ai_comfy.py --help # works, nothing installed
|
||||||
|
```
|
||||||
|
|
||||||
|
Its only external requirements are **ffmpeg** and **ffprobe** on `PATH`, and a
|
||||||
|
reachable ComfyUI instance. It talks to ComfyUI over plain HTTP with
|
||||||
|
`urllib.request` — there is no ComfyUI client library involved.
|
||||||
|
|
||||||
|
> **`requirements.txt` in the project root does not apply here.** It belongs to
|
||||||
|
> the older `lektor_ai.py`, which used the hosted Boson AI API and needed
|
||||||
|
> `openai>=1.0.0`. `lektor_ai_comfy.py` requires nothing from it.
|
||||||
|
|
||||||
|
### ComfyUI — a large, isolated environment
|
||||||
|
|
||||||
|
233 pip packages in the `comfy-tts` conda env. The important ones:
|
||||||
|
|
||||||
|
```
|
||||||
|
torch 2.11.0+cu128 torchaudio 2.11.0+cu128 torchvision 0.26.0+cu128
|
||||||
|
transformers 5.12.1 tokenizers 0.22.2 accelerate 1.14.0
|
||||||
|
numpy 2.2.6 scipy 1.18.0 librosa 0.11.0
|
||||||
|
soundfile 0.14.0 safetensors 0.8.0 einops 0.8.2
|
||||||
|
```
|
||||||
|
|
||||||
|
The conda environment itself holds only the 27 base conda-forge packages that
|
||||||
|
make up the Python runtime. Everything else arrives via pip. In particular,
|
||||||
|
`portaudio`, `libsamplerate`, `ffmpeg` and `espeak-ng` are **not** installed in
|
||||||
|
the environment despite appearing in the original planning notes — the system
|
||||||
|
ffmpeg is used and espeak-ng is absent entirely. Everything works without them.
|
||||||
|
|
||||||
|
TTS-Audio-Suite installs its dependencies through its own `install.py`, not
|
||||||
|
through its `requirements.txt`. That file deliberately omits packages needing
|
||||||
|
special handling — `librosa`, `descript-audio-codec`, `onnxruntime`,
|
||||||
|
`opencv-python` and others are installed with `--no-deps` because pip would
|
||||||
|
otherwise drag numpy backwards and break the rest of the environment. Never
|
||||||
|
substitute `pip install -r requirements.txt` for `python install.py`.
|
||||||
|
|
||||||
|
## Input requirements
|
||||||
|
|
||||||
|
**Video** — anything ffmpeg can demux. MKV and MP4 are the usual cases. The
|
||||||
|
container must support what gets muxed back in; output is always MKV, which
|
||||||
|
accepts every codec involved.
|
||||||
|
|
||||||
|
**Subtitles** — either an external `.srt`, or a text-based track inside the
|
||||||
|
video (`subrip`, `ass`, `ssa`, `webvtt`, `mov_text`). Image-based tracks
|
||||||
|
(`hdmv_pgs_subtitle` from Blu-ray, `dvd_subtitle`) contain bitmaps rather than
|
||||||
|
text and cannot drive TTS without OCR; the `--embedded-subs` picker marks them
|
||||||
|
as unusable. They are still copied to the output untouched.
|
||||||
|
|
||||||
|
**Voice reference** — a `.wav` plus a `.txt` holding its exact transcript, both
|
||||||
|
in `gudowski/`. Three are bundled:
|
||||||
|
|
||||||
|
| Voice | Length | Format |
|
||||||
|
|---|---|---|
|
||||||
|
| `gudowski` | 2:10 | 44.1 kHz stereo |
|
||||||
|
| `gudowski_short` (default) | 0:52 | 44.1 kHz stereo |
|
||||||
|
| `gudowski_short_2` | 0:28 | 44.1 kHz stereo |
|
||||||
|
|
||||||
|
The transcript matters as much as the audio — voice cloning quality drops
|
||||||
|
noticeably without an accurate one. The script warns if the `.txt` is missing.
|
||||||
|
|
||||||
|
## Network
|
||||||
|
|
||||||
|
Needed only at install time, for the ComfyUI and TTS-Audio-Suite clones, the pip
|
||||||
|
downloads, and the 8.7 GB model. Generation afterwards is entirely local; no
|
||||||
|
API keys, no accounts, no per-use cost.
|
||||||
|
|
||||||
|
The model is published under a research / non-commercial licence and may require
|
||||||
|
accepting its terms on huggingface.co, then `hf auth login`, before it will
|
||||||
|
download.
|
||||||
@@ -0,0 +1,295 @@
|
|||||||
|
# Troubleshooting
|
||||||
|
|
||||||
|
Failure modes that have actually occurred here, with the diagnosis that
|
||||||
|
identified each one. Where a symptom is misleading, that is called out — several
|
||||||
|
of these point somewhere other than their cause.
|
||||||
|
|
||||||
|
## Audio problems in the finished file
|
||||||
|
|
||||||
|
### The lektor track is silent in the player
|
||||||
|
|
||||||
|
The single most misleading failure in this project. It has **two distinct
|
||||||
|
causes**, and volume measurements cannot tell them apart — both produce a track
|
||||||
|
that measures perfectly loud.
|
||||||
|
|
||||||
|
Diagnose by checking the declared channel layout and the interleaving, in that
|
||||||
|
order.
|
||||||
|
|
||||||
|
**Cause 1 — unknown channel layout**
|
||||||
|
|
||||||
|
```bash
|
||||||
|
ffprobe -v error -select_streams a -show_entries stream=index,codec_name,channels,channel_layout -of csv=p=0 out.mkv
|
||||||
|
```
|
||||||
|
|
||||||
|
```
|
||||||
|
1,eac3,6,5.1(side) ← original, fine
|
||||||
|
2,aac,6,unknown ← broken: 6 channels, no routing information
|
||||||
|
```
|
||||||
|
|
||||||
|
ffmpeg's AAC encoder does not signal a 5.1 layout, in any container. Players
|
||||||
|
cannot map the channels and output silence. The fix is the stereo lektor track
|
||||||
|
now used by default (`aformat=channel_layouts=stereo` on both inputs). A correct
|
||||||
|
track reports `2,stereo`.
|
||||||
|
|
||||||
|
**Cause 2 — broken interleaving**
|
||||||
|
|
||||||
|
Compare byte positions for the same timestamp:
|
||||||
|
|
||||||
|
```bash
|
||||||
|
ffprobe -read_intervals 00:20:00%+1 -select_streams v -show_entries packet=pos -of csv=p=0 out.mkv | head -1
|
||||||
|
ffprobe -read_intervals 00:20:00%+1 -select_streams a:1 -show_entries packet=pos -of csv=p=0 out.mkv | head -1
|
||||||
|
```
|
||||||
|
|
||||||
|
They should be within a few MB. Observed on a broken file:
|
||||||
|
|
||||||
|
| time | drift from video |
|
||||||
|
|---|---|
|
||||||
|
| 02:00 | 477 MB |
|
||||||
|
| 10:00 | **2315 MB** |
|
||||||
|
| 50:00 | 740 MB |
|
||||||
|
|
||||||
|
Sparse subtitle tracks make the muxer give up interleaving after its default
|
||||||
|
10-second window. ffmpeg reads the file fine, so every level measurement looks
|
||||||
|
healthy — but players read MKV sequentially and never reach that audio.
|
||||||
|
|
||||||
|
The fix is `-max_interleave_delta 0` as an **output** option. An existing file
|
||||||
|
can be repaired without regenerating anything:
|
||||||
|
|
||||||
|
```bash
|
||||||
|
ffmpeg -i broken.mkv -map 0 -c copy -max_interleave_delta 0 fixed.mkv
|
||||||
|
```
|
||||||
|
|
||||||
|
14 seconds and ~2.6 GB of transient RAM for a 60-minute file.
|
||||||
|
|
||||||
|
> Placement is the trap. Before `-i` the flag configures the demuxer, changes
|
||||||
|
> nothing, and produces byte-identical output — which is easily mistaken for
|
||||||
|
> "this fix doesn't work".
|
||||||
|
|
||||||
|
### The voiceover is late by a constant offset
|
||||||
|
|
||||||
|
The source audio starts at a non-zero PTS. Confirm:
|
||||||
|
|
||||||
|
```bash
|
||||||
|
ffprobe -v error -select_streams a:0 -show_entries stream=start_time -of csv=p=0 film.mkv
|
||||||
|
# 4.043000 ← not 0
|
||||||
|
```
|
||||||
|
|
||||||
|
`amix` cannot emit output until both inputs have frames, so the voiceover is
|
||||||
|
delayed by that amount. Handled automatically now
|
||||||
|
(`asetpts=PTS-STARTPTS,adelay=<offset>`); the run prints:
|
||||||
|
|
||||||
|
```
|
||||||
|
Audio PTS offset: 4.043s — normalising orig_mix for lektor track
|
||||||
|
```
|
||||||
|
|
||||||
|
Note that this affects **only** the mixed track. Track 0 is stream-copied and
|
||||||
|
was never wrong, so "the original audio is offset" and "the original audio
|
||||||
|
inside the lektor mix is offset" are different problems with different fixes.
|
||||||
|
|
||||||
|
### Voiceover drifts progressively out of sync
|
||||||
|
|
||||||
|
Not a mixing fault — clips are overrunning their windows and pushing each other
|
||||||
|
forward. The console shows it:
|
||||||
|
|
||||||
|
```
|
||||||
|
→ placed at 00:12:03,400, duration 3.10s [+2.40s behind SRT]
|
||||||
|
```
|
||||||
|
|
||||||
|
Raise `--speed` (try 1.3–1.4) and rebuild. Small drift is expected and
|
||||||
|
self-corrects at the next gap in the subtitles; only sustained growth is a
|
||||||
|
problem.
|
||||||
|
|
||||||
|
### The whole voiceover sounds too quiet or too loud
|
||||||
|
|
||||||
|
Adjust and rebuild — no regeneration needed:
|
||||||
|
|
||||||
|
```bash
|
||||||
|
python3 lektor_ai_comfy.py film.mkv --rebuild --orig-volume 0.5 --tts-volume 2.0
|
||||||
|
```
|
||||||
|
|
||||||
|
Track 0 is unaffected by these, always.
|
||||||
|
|
||||||
|
## Muxing failures
|
||||||
|
|
||||||
|
### `Failed to inject frame into filter network: Invalid argument`
|
||||||
|
|
||||||
|
```
|
||||||
|
Error while filtering: Invalid argument
|
||||||
|
Failed to inject frame into filter network: Invalid argument
|
||||||
|
Error while processing the decoded data for stream #0:1
|
||||||
|
```
|
||||||
|
|
||||||
|
Aborts partway through, at a **different timestamp each run**. The message
|
||||||
|
blames audio; audio is not the cause. It appears when filtering and
|
||||||
|
subtitle muxing happen in one ffmpeg command on a source with many subtitle
|
||||||
|
tracks — the sparse streams stall the muxer while the filter graph keeps
|
||||||
|
producing.
|
||||||
|
|
||||||
|
The varying failure point is the tell. A genuinely corrupt frame fails at the
|
||||||
|
same place every time.
|
||||||
|
|
||||||
|
Fixed by the two-pass structure in `merge_voiceover()`. If you are editing that
|
||||||
|
function, do not merge the passes back together.
|
||||||
|
|
||||||
|
### The mux fails on DTS or TrueHD audio
|
||||||
|
|
||||||
|
Only an issue if output is changed to MP4, which cannot carry those codecs. The
|
||||||
|
default MKV output handles them. Keep it.
|
||||||
|
|
||||||
|
## TTS problems
|
||||||
|
|
||||||
|
### A clip recites the reference transcript
|
||||||
|
|
||||||
|
The model occasionally speaks the contents of `gudowski.txt` instead of the
|
||||||
|
subtitle. Most likely at `--temperature 1.0`; `0.9` is the default for this
|
||||||
|
reason.
|
||||||
|
|
||||||
|
```bash
|
||||||
|
python3 lektor_ai_comfy.py film.mkv --regen 88
|
||||||
|
python3 lektor_ai_comfy.py film.mkv --rebuild
|
||||||
|
```
|
||||||
|
|
||||||
|
`--seed 0` randomises each run, so a regeneration gets a different sample.
|
||||||
|
|
||||||
|
### A clip is cut off mid-word
|
||||||
|
|
||||||
|
The token ceiling was reached. Raise it and regenerate just that clip:
|
||||||
|
|
||||||
|
```bash
|
||||||
|
python3 lektor_ai_comfy.py film.mkv --regen 214 --max-tokens 8192
|
||||||
|
```
|
||||||
|
|
||||||
|
### Every clip returns identical audio
|
||||||
|
|
||||||
|
`enable_audio_cache` is being set to `True` somewhere in
|
||||||
|
`build_text_workflow()`. It must stay `False` or ComfyUI serves cached results
|
||||||
|
and `--regen` does nothing.
|
||||||
|
|
||||||
|
### Stiff pauses at every comma
|
||||||
|
|
||||||
|
Baked into the model weights. There is no SSML support and no parameter for it.
|
||||||
|
Rephrasing the subtitle text is the only lever.
|
||||||
|
|
||||||
|
### Audible breath at the start of clips
|
||||||
|
|
||||||
|
```bash
|
||||||
|
python3 lektor_ai_comfy.py film.mkv --regen 5 --strip-leading-ms 120
|
||||||
|
```
|
||||||
|
|
||||||
|
Values of 80–150 ms usually suffice. Too high clips the first consonant.
|
||||||
|
|
||||||
|
### Voice cloning quality is poor
|
||||||
|
|
||||||
|
Check that the reference `.txt` exists and matches the `.wav` exactly. The run
|
||||||
|
prints:
|
||||||
|
|
||||||
|
```
|
||||||
|
Reference text loaded (930 chars)
|
||||||
|
```
|
||||||
|
|
||||||
|
or warns if it is missing. An inaccurate transcript degrades cloning noticeably.
|
||||||
|
|
||||||
|
## Subtitle problems
|
||||||
|
|
||||||
|
### Formatting tags are being spoken
|
||||||
|
|
||||||
|
`<i>`, `</i>` and `{\an8}` are stripped by `_strip_sub_tags()` before anything
|
||||||
|
else. If new tags appear, extend that function — and keep the stripping first,
|
||||||
|
because tags interfere with the ALL-CAPS detection that follows.
|
||||||
|
|
||||||
|
### ALL-CAPS text is being shouted
|
||||||
|
|
||||||
|
Handled by `_clean_text()`, but only after tags are removed. `<i>HELLO</i>`
|
||||||
|
looked mixed-case while the tag was present, because `i` is a lowercase letter.
|
||||||
|
If shouting persists, check for an unhandled tag type.
|
||||||
|
|
||||||
|
### `--embedded-subs` shows no usable tracks
|
||||||
|
|
||||||
|
The video only carries image-based subtitles (`hdmv_pgs_subtitle`,
|
||||||
|
`dvd_subtitle`). These are bitmaps and need OCR. Supply an external `.srt`
|
||||||
|
instead — the image tracks are still copied to the output.
|
||||||
|
|
||||||
|
### Wrong language selected
|
||||||
|
|
||||||
|
The picker lists language tags but they are only as good as the source's
|
||||||
|
metadata. `output/embedded_sub.srt` holds exactly what was extracted; check it
|
||||||
|
before committing to a full run.
|
||||||
|
|
||||||
|
## Backend problems
|
||||||
|
|
||||||
|
### `Connection refused` on port 8189
|
||||||
|
|
||||||
|
ComfyUI is not running. Start it and wait for the model to load:
|
||||||
|
|
||||||
|
```bash
|
||||||
|
~/comfy-tts-app/start_comfyui.sh
|
||||||
|
curl -s http://127.0.0.1:8189/system_stats | head -c 200
|
||||||
|
```
|
||||||
|
|
||||||
|
### ComfyUI crashed and the GPU is stuck
|
||||||
|
|
||||||
|
```bash
|
||||||
|
sudo fuser -k /dev/nvidia*
|
||||||
|
sudo nvidia-smi --gpu-reset
|
||||||
|
```
|
||||||
|
|
||||||
|
Then restart and resume from the last completed block:
|
||||||
|
|
||||||
|
```bash
|
||||||
|
python3 lektor_ai_comfy.py film.mkv --resume-from 312
|
||||||
|
```
|
||||||
|
|
||||||
|
### Out of VRAM
|
||||||
|
|
||||||
|
Nothing else should be using the GPU during a run. Restart ComfyUI to clear
|
||||||
|
fragmentation. The 4B model needs roughly 10–12 GB.
|
||||||
|
|
||||||
|
### `torch.cuda.is_available()` is False
|
||||||
|
|
||||||
|
```bash
|
||||||
|
source ~/miniforge3/etc/profile.d/conda.sh && conda activate comfy-tts
|
||||||
|
python -c "import torch; print(torch.__version__, torch.cuda.is_available())"
|
||||||
|
```
|
||||||
|
|
||||||
|
Expect `2.11.0+cu128 True`. If False, check `nvidia-smi` works and the driver is
|
||||||
|
525 or newer. The CUDA **toolkit** version is irrelevant — the wheels bundle
|
||||||
|
their own runtime, and this machine runs cu128 wheels against a system CUDA
|
||||||
|
11.5 without issue.
|
||||||
|
|
||||||
|
## Script problems
|
||||||
|
|
||||||
|
### `Provide an SRT subtitle file, or use --embedded-subs`
|
||||||
|
|
||||||
|
A mode that needs subtitles was given only a video. Either pass the SRT first,
|
||||||
|
or add `--embedded-subs`.
|
||||||
|
|
||||||
|
### `No such file or directory` on the video
|
||||||
|
|
||||||
|
Almost always an unquoted path with spaces or parentheses, or a wrong extension
|
||||||
|
(`.mp4` where the file is `.mkv`). Quote it and check with `ls`.
|
||||||
|
|
||||||
|
### `Missing clip files — regenerate them first`
|
||||||
|
|
||||||
|
`--rebuild` needs a complete set. The listed clips must be regenerated with
|
||||||
|
`--regen`, or `--resume-from` used to fill the gap.
|
||||||
|
|
||||||
|
### `No tts_input.srt found`
|
||||||
|
|
||||||
|
`--rebuild` and `--regen` require a previous full run in the same
|
||||||
|
`--output-dir`.
|
||||||
|
|
||||||
|
## General diagnostic method
|
||||||
|
|
||||||
|
The lesson from the silent-track investigation, which took several wrong turns:
|
||||||
|
|
||||||
|
1. **Measure, don't assume.** Level, layout, and byte position each reveal a
|
||||||
|
different class of fault.
|
||||||
|
2. **Isolate by subtraction.** Removing subtitles from the failing command is
|
||||||
|
what proved they were the cause, not the audio the error named.
|
||||||
|
3. **A varying failure point means state, not data.** Corrupt input fails in the
|
||||||
|
same place every time; buffering and interleaving problems move around.
|
||||||
|
4. **"ffmpeg reads it fine" is not "a player can play it."** Verify position, not
|
||||||
|
just content.
|
||||||
|
5. **Check flag placement.** Several ffmpeg options mean different things before
|
||||||
|
`-i` versus before the output path, and the wrong position often fails
|
||||||
|
silently rather than erroring.
|
||||||
+266
@@ -0,0 +1,266 @@
|
|||||||
|
# Usage
|
||||||
|
|
||||||
|
ComfyUI must be running before any command that generates speech
|
||||||
|
(`~/comfy-tts-app/start_comfyui.sh`). `--rebuild` is the exception — it only
|
||||||
|
touches ffmpeg and works with the backend stopped.
|
||||||
|
|
||||||
|
## Invocation
|
||||||
|
|
||||||
|
```
|
||||||
|
python3 lektor_ai_comfy.py [SRT] VIDEO [options]
|
||||||
|
```
|
||||||
|
|
||||||
|
Both positionals are optional in the argparse sense, because several modes do
|
||||||
|
not need an SRT file. The rule:
|
||||||
|
|
||||||
|
```bash
|
||||||
|
# External subtitle file — pass both, SRT first
|
||||||
|
python3 lektor_ai_comfy.py input/film.pl.srt input/film.mkv
|
||||||
|
|
||||||
|
# Subtitles inside the video — pass the video alone
|
||||||
|
python3 lektor_ai_comfy.py input/film.mkv --embedded-subs
|
||||||
|
|
||||||
|
# --rebuild and --regen also need only the video
|
||||||
|
python3 lektor_ai_comfy.py input/film.mkv --rebuild
|
||||||
|
```
|
||||||
|
|
||||||
|
When a single positional is given in a mode that needs no SRT, the script moves
|
||||||
|
it into the video slot automatically. Always quote paths containing spaces or
|
||||||
|
parentheses:
|
||||||
|
|
||||||
|
```bash
|
||||||
|
python3 lektor_ai_comfy.py "input/Star City (2026)/S01_E01 The Eyes.mkv" --embedded-subs
|
||||||
|
```
|
||||||
|
|
||||||
|
## Working modes
|
||||||
|
|
||||||
|
### Full generation (default)
|
||||||
|
|
||||||
|
Parses subtitles, generates one TTS clip per block, assembles, muxes.
|
||||||
|
|
||||||
|
```bash
|
||||||
|
python3 lektor_ai_comfy.py input/film.mkv --embedded-subs
|
||||||
|
```
|
||||||
|
|
||||||
|
Roughly real-time-ish on a 4070 Ti SUPER: a 60-minute episode of ~700 subtitle
|
||||||
|
blocks takes on the order of an hour. Each clip is saved as it completes, so a
|
||||||
|
crash never loses more than the clip in flight.
|
||||||
|
|
||||||
|
### `--embedded-subs`
|
||||||
|
|
||||||
|
Lists the video's subtitle tracks and prompts for one:
|
||||||
|
|
||||||
|
```
|
||||||
|
Available subtitle tracks in S01_E01 The Eyes.mkv:
|
||||||
|
[1] Stream #2: subrip [eng] — Forced
|
||||||
|
[2] Stream #3: subrip [eng]
|
||||||
|
[3] Stream #4: subrip [eng] — SDH
|
||||||
|
...
|
||||||
|
[30] Stream #31: subrip [pol]
|
||||||
|
Select track number: 30
|
||||||
|
```
|
||||||
|
|
||||||
|
The chosen track is extracted to `output/embedded_sub.srt` and then processed
|
||||||
|
exactly like an external file. Image-based tracks are listed but marked
|
||||||
|
unusable — they hold bitmaps, not text.
|
||||||
|
|
||||||
|
### `--rebuild`
|
||||||
|
|
||||||
|
Reassembles audio from existing clips and rebuilds the video, with no TTS at
|
||||||
|
all. This is the fast iteration loop — seconds to minutes rather than an hour.
|
||||||
|
|
||||||
|
```bash
|
||||||
|
python3 lektor_ai_comfy.py input/film.mkv --rebuild
|
||||||
|
python3 lektor_ai_comfy.py input/film.mkv --rebuild --orig-volume 0.5 --tts-volume 2.0
|
||||||
|
```
|
||||||
|
|
||||||
|
Use it after `--regen`, or to adjust the mix. It requires `output/tts_input.srt`
|
||||||
|
and a complete set of `output/clips/clip_NNNN.flac`, and reports which are
|
||||||
|
missing if any are.
|
||||||
|
|
||||||
|
If the original run used `--end`, pass it again so the same video fragment is
|
||||||
|
used.
|
||||||
|
|
||||||
|
### `--regen`
|
||||||
|
|
||||||
|
Regenerates specific clips by block number, leaving everything else alone.
|
||||||
|
|
||||||
|
```bash
|
||||||
|
python3 lektor_ai_comfy.py input/film.mkv --regen 12 22 41
|
||||||
|
python3 lektor_ai_comfy.py input/film.mkv --rebuild # then apply
|
||||||
|
```
|
||||||
|
|
||||||
|
Block numbers match `output/tts_input.srt` and the `clip_NNNN.flac` filenames.
|
||||||
|
Only the selected FLAC files are replaced — `--rebuild` afterwards is what
|
||||||
|
actually updates the video.
|
||||||
|
|
||||||
|
This is the intended fix for a bad clip. There is no automatic retry; when the
|
||||||
|
model hallucinates, mumbles, or truncates, note the number and regenerate it. A
|
||||||
|
different seed usually resolves it, and `--seed 0` (the default) randomises
|
||||||
|
every run.
|
||||||
|
|
||||||
|
### `--resume-from`
|
||||||
|
|
||||||
|
Continues a crashed run, loading clips `1..N-1` from disk and generating from
|
||||||
|
`N` onward.
|
||||||
|
|
||||||
|
```bash
|
||||||
|
python3 lektor_ai_comfy.py input/film.mkv --resume-from 47
|
||||||
|
```
|
||||||
|
|
||||||
|
The console prints `[47/693]` style progress, so the number to resume from is
|
||||||
|
the last one that completed.
|
||||||
|
|
||||||
|
### `--skip-tts`
|
||||||
|
|
||||||
|
Uses a pre-assembled audio file and goes straight to the mux.
|
||||||
|
|
||||||
|
```bash
|
||||||
|
python3 lektor_ai_comfy.py input/film.mkv --skip-tts output/tts_assembled.flac
|
||||||
|
```
|
||||||
|
|
||||||
|
## Options
|
||||||
|
|
||||||
|
### Range
|
||||||
|
|
||||||
|
| Flag | Default | Meaning |
|
||||||
|
|---|---|---|
|
||||||
|
| `--start MM:SS` | `00:00` | Start of the range to process |
|
||||||
|
| `--end MM:SS` | *(end)* | End of the range |
|
||||||
|
|
||||||
|
With `--end`, a matching fragment is extracted to `output/video_fragment.mkv`
|
||||||
|
and all TTS timestamps become relative to `--start`. Pass the same values to any
|
||||||
|
later `--rebuild`, or the audio will not line up.
|
||||||
|
|
||||||
|
`--end 01:00` is the quickest way to sanity-check a new voice or speed setting.
|
||||||
|
|
||||||
|
### Voice and model
|
||||||
|
|
||||||
|
| Flag | Default | Meaning |
|
||||||
|
|---|---|---|
|
||||||
|
| `--voice NAME\|PATH` | `gudowski_short` | Name in `gudowski/`, or a path to a WAV |
|
||||||
|
| `--seed N` | `0` | Base seed; `0` randomises each run |
|
||||||
|
| `--temperature T` | `0.9` | Sampling temperature |
|
||||||
|
| `--max-tokens N` | `4096` | Token ceiling per clip |
|
||||||
|
|
||||||
|
`--temperature` is worth understanding. At `1.0` the model sometimes recites the
|
||||||
|
reference transcript instead of the subtitle. `0.9` largely prevents that;
|
||||||
|
lower is more deterministic but flatter. A per-clip seed of `base_seed + index`
|
||||||
|
means a fixed `--seed` reproduces a run exactly.
|
||||||
|
|
||||||
|
Raise `--max-tokens` only if long clips are cut off mid-word.
|
||||||
|
|
||||||
|
### Timing and cleanup
|
||||||
|
|
||||||
|
| Flag | Default | Meaning |
|
||||||
|
|---|---|---|
|
||||||
|
| `--speed FACTOR` | `1.2` | Per-clip speed-up via `atempo` |
|
||||||
|
| `--strip-leading-ms MS` | `0` | Trim the start of each clip |
|
||||||
|
| `--trailing-fade-ms MS` | `80` | Fade the end of each clip |
|
||||||
|
|
||||||
|
`--speed` compresses speech so a clip is more likely to fit its subtitle window.
|
||||||
|
It is applied **per clip, before assembly** — never to the assembled track,
|
||||||
|
which would shift every timestamp by speeding up the silences too.
|
||||||
|
|
||||||
|
`--strip-leading-ms` removes an inhale artifact at clip starts; try 80–150 if
|
||||||
|
you hear one. `--trailing-fade-ms` suppresses tail pops; `0` disables it.
|
||||||
|
|
||||||
|
### Mixing
|
||||||
|
|
||||||
|
| Flag | Default | Meaning |
|
||||||
|
|---|---|---|
|
||||||
|
| `--orig-volume 0-1` | `0.60` | Original audio level inside the lektor mix |
|
||||||
|
| `--tts-volume 0-2` | `1.8` | Voiceover level inside the lektor mix |
|
||||||
|
|
||||||
|
These affect **Track 1 only**. Track 0 is always the untouched original, so
|
||||||
|
nothing is lost by mixing aggressively. Changing them needs only `--rebuild`.
|
||||||
|
|
||||||
|
Typical starting point for dialogue-heavy material: `--orig-volume 0.5
|
||||||
|
--tts-volume 2.0`. For quiet sources, `--orig-volume 1.0 --tts-volume 1.4`.
|
||||||
|
|
||||||
|
### Paths
|
||||||
|
|
||||||
|
| Flag | Default | Meaning |
|
||||||
|
|---|---|---|
|
||||||
|
| `--output-dir DIR` | `output` | Where everything is written |
|
||||||
|
| `--comfy-url URL` | `http://127.0.0.1:8189` | Backend address |
|
||||||
|
|
||||||
|
## Output
|
||||||
|
|
||||||
|
```
|
||||||
|
output/
|
||||||
|
tts_input.srt processed text actually sent to TTS — inspect this
|
||||||
|
embedded_sub.srt extracted embedded subtitle (--embedded-subs only)
|
||||||
|
clips/clip_NNNN.flac one lossless clip per block, kept for --regen
|
||||||
|
tts_assembled.flac all clips at their final positions, lossless
|
||||||
|
video_fragment.mkv extracted segment (--end only)
|
||||||
|
{stem}_lektor.mkv the finished video
|
||||||
|
```
|
||||||
|
|
||||||
|
The finished file contains:
|
||||||
|
|
||||||
|
| Track | Content |
|
||||||
|
|---|---|
|
||||||
|
| Video | Stream-copied, untouched |
|
||||||
|
| Audio 0 | **Original** — stream-copied, full 5.1 if the source had it |
|
||||||
|
| Audio 1 | **Polish** — lektor mix, AAC 192k stereo, `language=pol` |
|
||||||
|
| Subtitles | Every track from the source, stream-copied |
|
||||||
|
|
||||||
|
The lektor track is stereo deliberately; see
|
||||||
|
[troubleshooting.md](troubleshooting.md#the-lektor-track-is-silent-in-the-player).
|
||||||
|
|
||||||
|
FLAC is used for every intermediate so repeated processing stays lossless. Only
|
||||||
|
the final voiceover track is ever encoded.
|
||||||
|
|
||||||
|
## Recipes
|
||||||
|
|
||||||
|
**Check a voice before committing to a full run**
|
||||||
|
|
||||||
|
```bash
|
||||||
|
python3 lektor_ai_comfy.py input/film.mkv --embedded-subs --end 02:00 --voice gudowski
|
||||||
|
```
|
||||||
|
|
||||||
|
**Tune the mix without regenerating speech**
|
||||||
|
|
||||||
|
```bash
|
||||||
|
python3 lektor_ai_comfy.py input/film.mkv --rebuild --orig-volume 0.45 --tts-volume 2.1
|
||||||
|
```
|
||||||
|
|
||||||
|
**Fix a handful of bad clips**
|
||||||
|
|
||||||
|
```bash
|
||||||
|
# read output/tts_input.srt, note the offending block numbers
|
||||||
|
python3 lektor_ai_comfy.py input/film.mkv --regen 88 91 104
|
||||||
|
python3 lektor_ai_comfy.py input/film.mkv --rebuild
|
||||||
|
```
|
||||||
|
|
||||||
|
**Reproduce a run exactly**
|
||||||
|
|
||||||
|
```bash
|
||||||
|
python3 lektor_ai_comfy.py input/film.mkv --embedded-subs --seed 123456
|
||||||
|
```
|
||||||
|
|
||||||
|
**Speech running long and overlapping the next line**
|
||||||
|
|
||||||
|
```bash
|
||||||
|
python3 lektor_ai_comfy.py input/film.mkv --embedded-subs --speed 1.35
|
||||||
|
```
|
||||||
|
|
||||||
|
## Reading the console
|
||||||
|
|
||||||
|
```
|
||||||
|
[1/4] Parsing subtitles 00:00–end…
|
||||||
|
693 entries after dialogue splitting.
|
||||||
|
[2/4] Generating TTS (693 clips, speed 1.2×)…
|
||||||
|
[12/693] 00:01:34,120 'Nie wiem, o czym mówisz.'
|
||||||
|
Waiting for a3f8b201… done.
|
||||||
|
→ placed at 00:01:34,120, duration 1.84s
|
||||||
|
```
|
||||||
|
|
||||||
|
`→ placed at` reveals timing drift. A clip that overruns its window pushes the
|
||||||
|
next one forward, shown as `[+1.20s behind SRT]`. Small values are normal and
|
||||||
|
self-correct at the next natural gap in the subtitles; values that keep growing
|
||||||
|
mean `--speed` is too low for the material.
|
||||||
|
|
||||||
|
Assembly then shows a progress bar per batch (clips are assembled 500 at a time
|
||||||
|
to stay under the open-file limit), followed by the two-pass mux.
|
||||||
@@ -0,0 +1,2 @@
|
|||||||
|
14 listopada 1917 roku było już wiadomo, że hrabia Adam Tarnowski nie zostanie premierem Rady Regencyjnej. Niemcy nie wyraziły zgody. Żadnego powodu nie podano. Nie, bo nie. Według polskich polityków Tarnowski był idealnym kandydatem na premiera. Wszechstronnie wykształcony, władał biegle kilkoma językami, znał świat. Od 1897 roku pracował w służbie dyplomatycznej Austro-Węgier w Waszyngtonie, Paryżu, Dreźnie, Brukseli, Madrycie i Londynie. W roku 1911 został mianowany ambasadorem Austro-Węgier w Sofii. Sprawował dyplomatyczną misję w trudnym okresie, w czasie wojen bałkańskich i na początku Wielkiej Wojny. W roku 1916 prosto z Sofii udał się do Stanów Zjednoczonych, by objąć funkcję ambasadora Austro-Węgier w tym kraju. Nie otrzymał jednak od Aliantów dyplomatycznego listu żelaznego. Podróżował zatem do Ameryki okrężną drogą. Zjawił się w Waszyngtonie dopiero pod koniec stycznia 1917 roku, kiedy Niemcy zaczynały swoją nieograniczoną wojnę podwodną na Atlantyku. Tarnowski tłumaczył amerykańskiemu sekretarzowi stanu Robertowi Lansingowi, że Austro-Węgry nie mają nic wspólnego z atakowaniem i zatapianiem przez okręty podwodne statków pasażerskich i handlowych. Kiedy okazało się, że prawda jest inna, Tarnowski wyjaśniał amerykańskim politykom, że to Niemcy zmusiły swego sojusznika do uprawiania tego procederu. Nic to Tarnowskiemu nie pomogło. Nie został ambasadorem Stanach Zjednoczonych i wrócił do Wiednia. Wtedy próbował robić karierę w Królestwie Polskim, ale nie został ani członkiem Rady Regencyjnej, ani premierem. Nie życzył sobie tego sam cesarz Wilhelm II. Dla Niemców Tarnowski był nie tylko Polakiem, polskim patriotą, ale przede wszystkim zaufanym człowiekiem Wiednia. A Niemcy i Austriacy, chociaż ciągle sojusznicy, coraz częściej chadzali różnymi drogami. Ofiarą tych sojuszniczych nieporozumień padł właśnie hrabia Adam Tarnowski, który w wolnej Polsce też nie zrobił dyplomatycznej kariery. Może był człowiekiem zupełnie innej epoki.
|
||||||
|
|
||||||
Binary file not shown.
@@ -0,0 +1,2 @@
|
|||||||
|
Sprawował dyplomatyczną misję w trudnym okresie, w czasie wojen bałkańskich i na początku Wielkiej Wojny. W roku tysiąc dziewięćset szesnastym prosto z Sofii udał się do Stanów Zjednoczonych, by objąć funkcję ambasadora Austro-Węgier w tym kraju. Nie otrzymał jednak od aliantów dyplomatycznego listu żelaznego. Podróżował zatem do Ameryki okrężną drogą. Zjawił się w Waszyngtonie dopiero pod koniec stycznia tysiąc dziewięćset siedemnastego roku, kiedy Niemcy zaczynały swoją nieograniczoną wojnę podwodną na Atlantyku. Tarnowski tłumaczył amerykańskiemu sekretarzowi stanu Robertowi Lansingowi, że Austro-Węgry nie mają nic wspólnego z atakowaniem i zatapianiem przez okręty podwodne statków pasażerskich i handlowych. Kiedy okazało się, że prawda jest inna, Tarnowski wyjaśniał amerykańskim politykom, że to Niemcy zmusiły swojego sojusznika do uprawiania tego procederu.
|
||||||
|
|
||||||
Binary file not shown.
@@ -0,0 +1,2 @@
|
|||||||
|
Nie życzył sobie tego sam cesarz Wilhelm II. Dla Niemców Tarnowski był nie tylko Polakiem, polskim patriotą, ale przede wszystkim zaufanym człowiekiem Wiednia. A Niemcy i Austriacy, chociaż ciągle sojusznicy, coraz częściej chadzali różnymi drogami. Ofiarą tych sojuszniczych nieporozumień padł właśnie hrabia Adam Tarnowski, który w wolnej Polsce też nie zrobił dyplomatycznej kariery. Może był człowiekiem zupełnie innej epoki.
|
||||||
|
|
||||||
Binary file not shown.
@@ -0,0 +1,57 @@
|
|||||||
|
#!/usr/bin/env python3
|
||||||
|
"""Inspect voice picker and Text mode on AI Studio TTS page."""
|
||||||
|
import json, time
|
||||||
|
from pathlib import Path
|
||||||
|
from playwright.sync_api import sync_playwright
|
||||||
|
|
||||||
|
profile_dir = Path.home() / ".config" / "lektor_ai" / "browser_profile"
|
||||||
|
|
||||||
|
def dump_elements(page, label):
|
||||||
|
elements = page.evaluate("""() => {
|
||||||
|
const sel = 'input, textarea, select, [role="combobox"], [role="listbox"], [role="textbox"], [role="option"], button, [contenteditable]';
|
||||||
|
return [...document.querySelectorAll(sel)].map(el => ({
|
||||||
|
tag: el.tagName,
|
||||||
|
role: el.getAttribute('role') || '',
|
||||||
|
ariaLabel: el.getAttribute('aria-label') || '',
|
||||||
|
placeholder: el.getAttribute('placeholder') || '',
|
||||||
|
class: el.className?.toString().slice(0, 100) || '',
|
||||||
|
text: (el.innerText || el.value || '').slice(0, 80),
|
||||||
|
}));
|
||||||
|
}""")
|
||||||
|
path = f"output/elements_{label}.json"
|
||||||
|
Path(path).write_text(json.dumps(elements, indent=2, ensure_ascii=False))
|
||||||
|
print(f"[{label}] {len(elements)} elements → {path}")
|
||||||
|
|
||||||
|
with sync_playwright() as pw:
|
||||||
|
ctx = pw.chromium.launch_persistent_context(
|
||||||
|
str(profile_dir),
|
||||||
|
channel="chrome",
|
||||||
|
headless=False,
|
||||||
|
viewport={"width": 1400, "height": 900},
|
||||||
|
ignore_default_args=["--enable-automation"],
|
||||||
|
args=["--disable-blink-features=AutomationControlled", "--start-maximized"],
|
||||||
|
)
|
||||||
|
page = ctx.new_page()
|
||||||
|
page.goto("https://aistudio.google.com/generate-speech", wait_until="networkidle", timeout=60_000)
|
||||||
|
time.sleep(3)
|
||||||
|
|
||||||
|
# Open editor
|
||||||
|
page.locator('.text-input-container').first.click()
|
||||||
|
time.sleep(2)
|
||||||
|
|
||||||
|
# Step 1: Click Text mode (the other toggle)
|
||||||
|
page.locator('button[role="radio"]').filter(has_text="Text").click()
|
||||||
|
time.sleep(2)
|
||||||
|
page.screenshot(path="output/shot_3_text_mode.png", full_page=True)
|
||||||
|
dump_elements(page, "3_text_mode")
|
||||||
|
|
||||||
|
# Step 2: Back to Composer, click voice chip
|
||||||
|
page.locator('button[role="radio"]').filter(has_text="Composer").click()
|
||||||
|
time.sleep(2)
|
||||||
|
page.locator('button.voice-chip').first.click()
|
||||||
|
time.sleep(2)
|
||||||
|
page.screenshot(path="output/shot_4_voice_picker.png", full_page=True)
|
||||||
|
dump_elements(page, "4_voice_picker")
|
||||||
|
|
||||||
|
input("Press Enter to close...")
|
||||||
|
ctx.close()
|
||||||
Executable
+328
@@ -0,0 +1,328 @@
|
|||||||
|
#!/usr/bin/env bash
|
||||||
|
#
|
||||||
|
# install_comfyui.sh — reproduce the ComfyUI + Higgs Audio V3 TTS environment
|
||||||
|
# used by lektor_ai_comfy.py, on this or any other Linux machine.
|
||||||
|
#
|
||||||
|
# Everything lands in two removable directories and nothing touches the system:
|
||||||
|
# ~/miniforge3 — conda distribution (its own Python)
|
||||||
|
# ~/comfy-tts-app — ComfyUI, the TTS-Audio-Suite node, and the models
|
||||||
|
#
|
||||||
|
# No sudo, no apt, no changes to system Python. `conda init` is never run, so
|
||||||
|
# your .bashrc is left alone.
|
||||||
|
#
|
||||||
|
# Usage:
|
||||||
|
# ./install_comfyui.sh # pinned versions (recommended)
|
||||||
|
# ./install_comfyui.sh --latest # newest ComfyUI + suite instead
|
||||||
|
# ./install_comfyui.sh --skip-model # don't pre-download the 8.7 GB model
|
||||||
|
# ./install_comfyui.sh --lock FILE # restore exact package set from a lockfile
|
||||||
|
# ./install_comfyui.sh --help
|
||||||
|
#
|
||||||
|
# Override locations with environment variables:
|
||||||
|
# CONDA_ROOT=/opt/miniforge3 COMFY_ROOT=/data/comfy ./install_comfyui.sh
|
||||||
|
#
|
||||||
|
set -euo pipefail
|
||||||
|
|
||||||
|
# ── Configuration ────────────────────────────────────────────────────────────
|
||||||
|
CONDA_ROOT="${CONDA_ROOT:-$HOME/miniforge3}"
|
||||||
|
COMFY_ROOT="${COMFY_ROOT:-$HOME/comfy-tts-app}"
|
||||||
|
ENV_NAME="${ENV_NAME:-comfy-tts}"
|
||||||
|
COMFY_PORT="${COMFY_PORT:-8189}"
|
||||||
|
PYTHON_VERSION="${PYTHON_VERSION:-3.12}"
|
||||||
|
|
||||||
|
# Exact versions of the working installation this script reproduces.
|
||||||
|
COMFYUI_REPO="https://github.com/comfyanonymous/ComfyUI.git"
|
||||||
|
COMFYUI_COMMIT="7cb784e0f48784bb6ed588912e186e5ee1e9ee68" # v0.26.0-13-g7cb784e0
|
||||||
|
TTS_SUITE_REPO="https://github.com/diodiogod/TTS-Audio-Suite.git"
|
||||||
|
TTS_SUITE_TAG="v5.3.0" # 09df1681
|
||||||
|
MINIFORGE_URL="https://github.com/conda-forge/miniforge/releases/latest/download/Miniforge3-Linux-x86_64.sh"
|
||||||
|
|
||||||
|
# PyTorch: CUDA 12.8 wheels. These carry their own CUDA runtime, so the
|
||||||
|
# system CUDA toolkit version is irrelevant — only the NVIDIA driver matters.
|
||||||
|
TORCH_INDEX="https://download.pytorch.org/whl/cu128"
|
||||||
|
TORCH_PINS=(torch==2.11.0 torchvision==0.26.0 torchaudio==2.11.0)
|
||||||
|
|
||||||
|
# The TTS model. The suite downloads this automatically on first use; doing it
|
||||||
|
# here just means the first generation isn't a surprise 8.7 GB wait.
|
||||||
|
HF_MODEL_REPO="bosonai/higgs-audio-v3-tts-4b"
|
||||||
|
MODEL_SUBDIR="models/TTS/higgs_audio_v3/higgs-audio-v3-tts-4b"
|
||||||
|
|
||||||
|
REQUIRED_DISK_GB=35 # ~10 env + ~3 ComfyUI + ~9 model + headroom
|
||||||
|
|
||||||
|
# ── Options ──────────────────────────────────────────────────────────────────
|
||||||
|
USE_LATEST=0
|
||||||
|
SKIP_MODEL=0
|
||||||
|
LOCKFILE=""
|
||||||
|
|
||||||
|
while [[ $# -gt 0 ]]; do
|
||||||
|
case "$1" in
|
||||||
|
--latest) USE_LATEST=1; shift ;;
|
||||||
|
--skip-model) SKIP_MODEL=1; shift ;;
|
||||||
|
--lock) LOCKFILE="${2:-}"; shift 2 ;;
|
||||||
|
-h|--help) awk 'NR>1 {if (/^#/) {sub(/^# ?/,""); print} else exit}' "$0"; exit 0 ;;
|
||||||
|
*) echo "Unknown option: $1 (try --help)" >&2; exit 1 ;;
|
||||||
|
esac
|
||||||
|
done
|
||||||
|
|
||||||
|
# ── Output helpers ───────────────────────────────────────────────────────────
|
||||||
|
if [[ -t 1 ]]; then
|
||||||
|
B=$'\033[1m'; G=$'\033[32m'; Y=$'\033[33m'; R=$'\033[31m'; N=$'\033[0m'
|
||||||
|
else
|
||||||
|
B=""; G=""; Y=""; R=""; N=""
|
||||||
|
fi
|
||||||
|
step() { echo; echo "${B}==> $*${N}"; }
|
||||||
|
ok() { echo " ${G}[+]${N} $*"; }
|
||||||
|
warn() { echo " ${Y}[!]${N} $*"; }
|
||||||
|
die() { echo " ${R}[X]${N} $*" >&2; exit 1; }
|
||||||
|
have() { command -v "$1" >/dev/null 2>&1; }
|
||||||
|
|
||||||
|
# Activate the conda env. conda's shell functions dislike `set -u`, so it is
|
||||||
|
# lifted for the duration of the call only.
|
||||||
|
activate_env() {
|
||||||
|
# shellcheck disable=SC1091
|
||||||
|
source "$CONDA_ROOT/etc/profile.d/conda.sh"
|
||||||
|
set +u
|
||||||
|
conda activate "$ENV_NAME"
|
||||||
|
set -u
|
||||||
|
}
|
||||||
|
|
||||||
|
ENV_PY="$CONDA_ROOT/envs/$ENV_NAME/bin/python"
|
||||||
|
pip_install() { "$ENV_PY" -m pip install --no-input "$@"; }
|
||||||
|
|
||||||
|
# ── 0. Preflight ─────────────────────────────────────────────────────────────
|
||||||
|
step "Preflight checks"
|
||||||
|
|
||||||
|
[[ "$(uname -s)" == "Linux" ]] || die "This script targets Linux (found $(uname -s))."
|
||||||
|
|
||||||
|
have git || die "git is required. Install it first (e.g. sudo apt install git)."
|
||||||
|
if have curl; then DL=(curl -fL -o); elif have wget; then DL=(wget -O); else
|
||||||
|
die "Need curl or wget to download Miniforge."
|
||||||
|
fi
|
||||||
|
ok "git and a downloader are present"
|
||||||
|
|
||||||
|
if have nvidia-smi && nvidia-smi >/dev/null 2>&1; then
|
||||||
|
ok "GPU: $(nvidia-smi --query-gpu=name,driver_version,memory.total --format=csv,noheader | head -1)"
|
||||||
|
else
|
||||||
|
warn "No usable nvidia-smi. Install will proceed but TTS will run on CPU (very slow)."
|
||||||
|
warn "The cu128 wheels need an NVIDIA driver >= 525; the CUDA toolkit is NOT required."
|
||||||
|
fi
|
||||||
|
|
||||||
|
mkdir -p "$(dirname "$COMFY_ROOT")"
|
||||||
|
avail_gb=$(df -BG --output=avail "$(dirname "$COMFY_ROOT")" | tail -1 | tr -dc '0-9')
|
||||||
|
if (( avail_gb < REQUIRED_DISK_GB )); then
|
||||||
|
warn "Only ${avail_gb} GB free where $COMFY_ROOT will live; ~${REQUIRED_DISK_GB} GB recommended."
|
||||||
|
warn "Use --skip-model, or set COMFY_ROOT to a roomier disk."
|
||||||
|
else
|
||||||
|
ok "Disk space: ${avail_gb} GB available"
|
||||||
|
fi
|
||||||
|
|
||||||
|
if have ffmpeg; then
|
||||||
|
ok "ffmpeg found at $(command -v ffmpeg) (used by lektor_ai_comfy.py, not by ComfyUI)"
|
||||||
|
else
|
||||||
|
warn "ffmpeg not found. ComfyUI does not need it, but lektor_ai_comfy.py does."
|
||||||
|
warn "Install it with: conda install -n $ENV_NAME -c conda-forge ffmpeg (no sudo)"
|
||||||
|
fi
|
||||||
|
|
||||||
|
if (( USE_LATEST )); then
|
||||||
|
warn "--latest: installing newest ComfyUI and TTS-Audio-Suite rather than the pinned"
|
||||||
|
warn "versions this project was validated against. Expect to debug breakage yourself."
|
||||||
|
else
|
||||||
|
ok "Pinned: ComfyUI ${COMFYUI_COMMIT:0:8} (v0.26.0), TTS-Audio-Suite $TTS_SUITE_TAG"
|
||||||
|
fi
|
||||||
|
|
||||||
|
# ── 1. Miniforge ─────────────────────────────────────────────────────────────
|
||||||
|
step "Miniforge (self-contained conda in $CONDA_ROOT)"
|
||||||
|
|
||||||
|
if [[ -x "$CONDA_ROOT/bin/conda" ]]; then
|
||||||
|
ok "Already installed: $("$CONDA_ROOT/bin/conda" --version)"
|
||||||
|
else
|
||||||
|
tmp_installer="$(mktemp -t miniforge-XXXXXX.sh)"
|
||||||
|
trap 'rm -f "$tmp_installer"' EXIT
|
||||||
|
echo " Downloading Miniforge…"
|
||||||
|
"${DL[@]}" "$tmp_installer" "$MINIFORGE_URL"
|
||||||
|
echo " Installing to $CONDA_ROOT (batch mode, no shell integration)…"
|
||||||
|
bash "$tmp_installer" -b -p "$CONDA_ROOT"
|
||||||
|
rm -f "$tmp_installer"; trap - EXIT
|
||||||
|
ok "Installed $("$CONDA_ROOT/bin/conda" --version)"
|
||||||
|
fi
|
||||||
|
|
||||||
|
# ── 2. Conda environment ─────────────────────────────────────────────────────
|
||||||
|
step "Conda environment '$ENV_NAME' (Python $PYTHON_VERSION)"
|
||||||
|
|
||||||
|
# shellcheck disable=SC1091
|
||||||
|
source "$CONDA_ROOT/etc/profile.d/conda.sh"
|
||||||
|
if conda env list | awk '{print $1}' | grep -qx "$ENV_NAME"; then
|
||||||
|
ok "Environment already exists — reusing it"
|
||||||
|
else
|
||||||
|
conda create -n "$ENV_NAME" "python=$PYTHON_VERSION" -y
|
||||||
|
ok "Created"
|
||||||
|
fi
|
||||||
|
|
||||||
|
activate_env
|
||||||
|
ok "Active: $(python -V) at $(command -v python)"
|
||||||
|
|
||||||
|
[[ "$(command -v python)" == "$CONDA_ROOT/envs/$ENV_NAME/"* ]] \
|
||||||
|
|| die "Wrong Python active — refusing to install into the wrong environment."
|
||||||
|
|
||||||
|
pip_install --upgrade pip setuptools wheel >/dev/null
|
||||||
|
ok "pip toolchain up to date"
|
||||||
|
|
||||||
|
# ── Exact restore path ───────────────────────────────────────────────────────
|
||||||
|
if [[ -n "$LOCKFILE" ]]; then
|
||||||
|
step "Restoring exact package set from $LOCKFILE"
|
||||||
|
[[ -f "$LOCKFILE" ]] || die "Lockfile not found: $LOCKFILE"
|
||||||
|
pip_install --extra-index-url "$TORCH_INDEX" -r "$LOCKFILE"
|
||||||
|
ok "Lockfile applied — skipping individual package steps"
|
||||||
|
fi
|
||||||
|
|
||||||
|
# ── 3. PyTorch ───────────────────────────────────────────────────────────────
|
||||||
|
step "PyTorch (CUDA 12.8 wheels)"
|
||||||
|
|
||||||
|
if [[ -z "$LOCKFILE" ]]; then
|
||||||
|
if (( USE_LATEST )); then
|
||||||
|
pip_install --index-url "$TORCH_INDEX" torch torchvision torchaudio
|
||||||
|
else
|
||||||
|
pip_install --index-url "$TORCH_INDEX" "${TORCH_PINS[@]}"
|
||||||
|
fi
|
||||||
|
fi
|
||||||
|
python - <<'PY'
|
||||||
|
import torch
|
||||||
|
print(f" torch {torch.__version__} cuda_available={torch.cuda.is_available()}")
|
||||||
|
if torch.cuda.is_available():
|
||||||
|
print(f" device: {torch.cuda.get_device_name(0)}")
|
||||||
|
PY
|
||||||
|
ok "PyTorch ready"
|
||||||
|
|
||||||
|
# ── 4. ComfyUI ───────────────────────────────────────────────────────────────
|
||||||
|
step "ComfyUI"
|
||||||
|
|
||||||
|
mkdir -p "$COMFY_ROOT"
|
||||||
|
COMFY_DIR="$COMFY_ROOT/ComfyUI"
|
||||||
|
|
||||||
|
if [[ -d "$COMFY_DIR/.git" ]]; then
|
||||||
|
ok "Repository already present at $COMFY_DIR"
|
||||||
|
else
|
||||||
|
git clone "$COMFYUI_REPO" "$COMFY_DIR"
|
||||||
|
fi
|
||||||
|
|
||||||
|
if (( ! USE_LATEST )); then
|
||||||
|
git -C "$COMFY_DIR" fetch --quiet origin
|
||||||
|
git -C "$COMFY_DIR" checkout --quiet "$COMFYUI_COMMIT"
|
||||||
|
ok "Checked out $COMFYUI_COMMIT (detached HEAD — expected)"
|
||||||
|
else
|
||||||
|
git -C "$COMFY_DIR" pull --quiet --ff-only origin master || warn "Could not fast-forward; leaving as-is"
|
||||||
|
fi
|
||||||
|
|
||||||
|
[[ -z "$LOCKFILE" ]] && pip_install -r "$COMFY_DIR/requirements.txt"
|
||||||
|
ok "ComfyUI dependencies installed"
|
||||||
|
|
||||||
|
# ── 5. TTS-Audio-Suite ───────────────────────────────────────────────────────
|
||||||
|
step "TTS-Audio-Suite custom node"
|
||||||
|
|
||||||
|
SUITE_DIR="$COMFY_DIR/custom_nodes/TTS-Audio-Suite"
|
||||||
|
mkdir -p "$COMFY_DIR/custom_nodes"
|
||||||
|
|
||||||
|
if [[ -d "$SUITE_DIR/.git" ]]; then
|
||||||
|
ok "Repository already present"
|
||||||
|
else
|
||||||
|
git clone "$TTS_SUITE_REPO" "$SUITE_DIR"
|
||||||
|
fi
|
||||||
|
|
||||||
|
if (( ! USE_LATEST )); then
|
||||||
|
git -C "$SUITE_DIR" fetch --quiet --tags origin
|
||||||
|
git -C "$SUITE_DIR" checkout --quiet "tags/$TTS_SUITE_TAG"
|
||||||
|
ok "Checked out $TTS_SUITE_TAG"
|
||||||
|
else
|
||||||
|
git -C "$SUITE_DIR" pull --quiet --ff-only || warn "Could not fast-forward; leaving as-is"
|
||||||
|
fi
|
||||||
|
|
||||||
|
if [[ -z "$LOCKFILE" ]]; then
|
||||||
|
# install.py resolves this suite's many conflicting dependencies itself
|
||||||
|
# (selective --no-deps installs, numpy pinning). Do not replace it with a
|
||||||
|
# plain `pip install -r requirements.txt` — that file deliberately omits
|
||||||
|
# the packages that need special handling.
|
||||||
|
echo " Running the suite's own installer (this pulls a lot and takes a while)…"
|
||||||
|
( cd "$SUITE_DIR" && python install.py )
|
||||||
|
fi
|
||||||
|
ok "TTS-Audio-Suite installed"
|
||||||
|
|
||||||
|
# ── 6. Model ─────────────────────────────────────────────────────────────────
|
||||||
|
step "Higgs Audio V3 model ($HF_MODEL_REPO)"
|
||||||
|
|
||||||
|
MODEL_DIR="$COMFY_DIR/$MODEL_SUBDIR"
|
||||||
|
if (( SKIP_MODEL )); then
|
||||||
|
warn "Skipped (--skip-model). It downloads automatically on the first generation."
|
||||||
|
elif [[ -f "$MODEL_DIR/model.safetensors" ]]; then
|
||||||
|
ok "Already present ($(du -h "$MODEL_DIR/model.safetensors" | cut -f1))"
|
||||||
|
else
|
||||||
|
echo " Downloading ~8.7 GB into $MODEL_DIR …"
|
||||||
|
mkdir -p "$MODEL_DIR"
|
||||||
|
if ! HF_MODEL_REPO="$HF_MODEL_REPO" MODEL_DIR="$MODEL_DIR" python - <<'PY'
|
||||||
|
import os, sys
|
||||||
|
from huggingface_hub import snapshot_download
|
||||||
|
try:
|
||||||
|
snapshot_download(repo_id=os.environ["HF_MODEL_REPO"],
|
||||||
|
local_dir=os.environ["MODEL_DIR"])
|
||||||
|
except Exception as exc:
|
||||||
|
print(f"download failed: {exc}", file=sys.stderr)
|
||||||
|
sys.exit(1)
|
||||||
|
PY
|
||||||
|
then
|
||||||
|
warn "Model download failed."
|
||||||
|
warn "This model is licensed for research / non-commercial use and may require"
|
||||||
|
warn "accepting its terms on huggingface.co first, then: hf auth login"
|
||||||
|
warn "The suite will retry automatically on the first generation."
|
||||||
|
else
|
||||||
|
ok "Model downloaded"
|
||||||
|
fi
|
||||||
|
fi
|
||||||
|
|
||||||
|
# ── 7. Launcher ──────────────────────────────────────────────────────────────
|
||||||
|
step "Launcher script"
|
||||||
|
|
||||||
|
LAUNCHER="$COMFY_ROOT/start_comfyui.sh"
|
||||||
|
cat > "$LAUNCHER" <<EOF
|
||||||
|
#!/bin/bash
|
||||||
|
# Generated by install_comfyui.sh — starts ComfyUI on port $COMFY_PORT.
|
||||||
|
source "$CONDA_ROOT/etc/profile.d/conda.sh"
|
||||||
|
conda activate $ENV_NAME
|
||||||
|
cd "$COMFY_DIR" && python main.py --port $COMFY_PORT
|
||||||
|
EOF
|
||||||
|
chmod +x "$LAUNCHER"
|
||||||
|
ok "Wrote $LAUNCHER"
|
||||||
|
|
||||||
|
# ── 8. Verification ──────────────────────────────────────────────────────────
|
||||||
|
step "Verifying installation"
|
||||||
|
|
||||||
|
python - <<'PY'
|
||||||
|
import importlib, sys
|
||||||
|
failed = []
|
||||||
|
for mod in ("torch", "torchaudio", "transformers", "safetensors", "soundfile", "librosa"):
|
||||||
|
try:
|
||||||
|
m = importlib.import_module(mod)
|
||||||
|
print(f" [+] {mod:<13} {getattr(m, '__version__', '?')}")
|
||||||
|
except Exception as exc:
|
||||||
|
print(f" [X] {mod:<13} {exc}")
|
||||||
|
failed.append(mod)
|
||||||
|
sys.exit(1 if failed else 0)
|
||||||
|
PY
|
||||||
|
ok "Core imports succeed"
|
||||||
|
|
||||||
|
echo
|
||||||
|
echo "${B}${G}Installation complete.${N}"
|
||||||
|
echo
|
||||||
|
echo " Start ComfyUI: $LAUNCHER"
|
||||||
|
echo " Web UI: http://127.0.0.1:$COMFY_PORT"
|
||||||
|
echo " Then run: python3 lektor_ai_comfy.py <video.mkv> --embedded-subs"
|
||||||
|
echo
|
||||||
|
echo " lektor_ai_comfy.py expects these paths — edit its constants if you changed them:"
|
||||||
|
echo " COMFY_URL = \"http://127.0.0.1:$COMFY_PORT\""
|
||||||
|
echo " COMFY_OUTPUT_DIR = \"$COMFY_DIR/output\""
|
||||||
|
echo " COMFY_INPUT_DIR = \"$COMFY_DIR/input\""
|
||||||
|
echo
|
||||||
|
echo " Record this environment for later:"
|
||||||
|
echo " source $CONDA_ROOT/etc/profile.d/conda.sh && conda activate $ENV_NAME"
|
||||||
|
echo " pip freeze > comfyui-requirements.lock.txt"
|
||||||
|
echo
|
||||||
|
echo " Remove everything:"
|
||||||
|
echo " conda env remove -n $ENV_NAME && rm -rf $COMFY_ROOT"
|
||||||
|
echo " rm -rf $CONDA_ROOT # only if nothing else uses it"
|
||||||
+389
@@ -0,0 +1,389 @@
|
|||||||
|
#!/usr/bin/env python3
|
||||||
|
"""
|
||||||
|
lektor_ai — Convert SRT subtitles to AI voiceover using Boson AI TTS (higgs-audio-v3),
|
||||||
|
then mix the result with a video file's original audio.
|
||||||
|
|
||||||
|
Usage:
|
||||||
|
python3 lektor_ai.py input/film.srt input/film.mkv --start 04:07 --end 05:45
|
||||||
|
|
||||||
|
Requires a free Boson AI API key from https://boson.ai/workspace
|
||||||
|
Set it via: export BOSON_API_KEY=bai-xxxx
|
||||||
|
"""
|
||||||
|
|
||||||
|
import argparse
|
||||||
|
import base64
|
||||||
|
import os
|
||||||
|
import re
|
||||||
|
import subprocess
|
||||||
|
import sys
|
||||||
|
from pathlib import Path
|
||||||
|
|
||||||
|
# ---------------------------------------------------------------------------
|
||||||
|
# SRT helpers
|
||||||
|
# ---------------------------------------------------------------------------
|
||||||
|
|
||||||
|
_SRT_TIME_RE = re.compile(r'(\d+):(\d+):(\d+)[,.](\d+)')
|
||||||
|
|
||||||
|
|
||||||
|
def _parse_srt_time(s: str) -> float:
|
||||||
|
m = _SRT_TIME_RE.match(s.strip())
|
||||||
|
if not m:
|
||||||
|
raise ValueError(f"Unrecognised SRT timestamp: {s!r}")
|
||||||
|
h, mn, sec, ms = int(m[1]), int(m[2]), int(m[3]), int(m[4])
|
||||||
|
return h * 3600 + mn * 60 + sec + ms / 1000
|
||||||
|
|
||||||
|
|
||||||
|
def _fmt_srt_time(seconds: float) -> str:
|
||||||
|
h = int(seconds // 3600)
|
||||||
|
m = int((seconds % 3600) // 60)
|
||||||
|
s = int(seconds % 60)
|
||||||
|
ms = int(round((seconds % 1) * 1000))
|
||||||
|
if ms == 1000:
|
||||||
|
ms = 999
|
||||||
|
return f"{h:02d}:{m:02d}:{s:02d},{ms:03d}"
|
||||||
|
|
||||||
|
|
||||||
|
def _parse_mmss(value: str) -> float:
|
||||||
|
"""Accept HH:MM:SS or MM:SS."""
|
||||||
|
parts = value.strip().split(":")
|
||||||
|
if len(parts) == 2:
|
||||||
|
return int(parts[0]) * 60 + float(parts[1])
|
||||||
|
if len(parts) == 3:
|
||||||
|
return int(parts[0]) * 3600 + int(parts[1]) * 60 + float(parts[2])
|
||||||
|
raise ValueError(f"Cannot parse time: {value!r}")
|
||||||
|
|
||||||
|
|
||||||
|
def _clean_text(lines: list) -> str:
|
||||||
|
"""Strip dialogue dashes and return plain text."""
|
||||||
|
out = []
|
||||||
|
for line in lines:
|
||||||
|
line = line.strip()
|
||||||
|
line = re.sub(r"^-\s*", "", line)
|
||||||
|
if line:
|
||||||
|
out.append(line)
|
||||||
|
return "\n".join(out)
|
||||||
|
|
||||||
|
|
||||||
|
def parse_subtitle_blocks(srt_path: str, start_sec: float, end_sec: float) -> list:
|
||||||
|
"""
|
||||||
|
Return list of (start_relative, end_relative, plain_text) for each subtitle
|
||||||
|
block whose start falls within [start_sec, end_sec].
|
||||||
|
Times are relative to start_sec.
|
||||||
|
"""
|
||||||
|
text = Path(srt_path).read_text(encoding="utf-8-sig")
|
||||||
|
blocks = re.split(r"\n{2,}", text.strip())
|
||||||
|
|
||||||
|
result = []
|
||||||
|
for block in blocks:
|
||||||
|
lines = block.strip().splitlines()
|
||||||
|
ti = next((i for i, l in enumerate(lines) if "-->" in l), None)
|
||||||
|
if ti is None:
|
||||||
|
continue
|
||||||
|
parts = lines[ti].split("-->")
|
||||||
|
if len(parts) != 2:
|
||||||
|
continue
|
||||||
|
t_start = _parse_srt_time(parts[0])
|
||||||
|
t_end = _parse_srt_time(parts[1])
|
||||||
|
if t_start < start_sec or t_start > end_sec:
|
||||||
|
continue
|
||||||
|
plain = _clean_text(lines[ti + 1:])
|
||||||
|
if plain:
|
||||||
|
result.append((t_start - start_sec, t_end - start_sec, plain))
|
||||||
|
|
||||||
|
return result
|
||||||
|
|
||||||
|
|
||||||
|
# ---------------------------------------------------------------------------
|
||||||
|
# FFmpeg helpers
|
||||||
|
# ---------------------------------------------------------------------------
|
||||||
|
|
||||||
|
def _ffmpeg_ts(seconds: float) -> str:
|
||||||
|
h = int(seconds // 3600)
|
||||||
|
m = int((seconds % 3600) // 60)
|
||||||
|
s = seconds % 60
|
||||||
|
return f"{h:02d}:{m:02d}:{s:06.3f}"
|
||||||
|
|
||||||
|
|
||||||
|
def _run(cmd: list, label: str):
|
||||||
|
print(f" {label}...")
|
||||||
|
result = subprocess.run(cmd, capture_output=True, text=True)
|
||||||
|
if result.returncode != 0:
|
||||||
|
print(f"ffmpeg error ({label}):\n{result.stderr[-2000:]}", file=sys.stderr)
|
||||||
|
sys.exit(1)
|
||||||
|
|
||||||
|
|
||||||
|
def extract_video_fragment(video_path: str, start_sec: float, end_sec: float,
|
||||||
|
output_path: str):
|
||||||
|
duration = end_sec - start_sec
|
||||||
|
_run([
|
||||||
|
"ffmpeg", "-y",
|
||||||
|
"-ss", _ffmpeg_ts(start_sec),
|
||||||
|
"-i", video_path,
|
||||||
|
"-t", str(duration),
|
||||||
|
"-c", "copy",
|
||||||
|
output_path,
|
||||||
|
], f"Extracting video {_ffmpeg_ts(start_sec)}–{_ffmpeg_ts(end_sec)}")
|
||||||
|
|
||||||
|
|
||||||
|
def _assemble_batch(clips: list, output_path: str, label: str):
|
||||||
|
"""Mix a batch of (start_sec, path) clips with absolute adelay into output_path."""
|
||||||
|
if len(clips) == 1:
|
||||||
|
delay_ms = int(clips[0][0] * 1000)
|
||||||
|
_run([
|
||||||
|
"ffmpeg", "-y",
|
||||||
|
"-i", str(clips[0][1]),
|
||||||
|
"-af", f"adelay={delay_ms}:all=1",
|
||||||
|
output_path,
|
||||||
|
], label)
|
||||||
|
return
|
||||||
|
inputs, filter_parts = [], []
|
||||||
|
for i, (start_sec, path) in enumerate(clips):
|
||||||
|
inputs += ["-i", str(path)]
|
||||||
|
delay_ms = int(start_sec * 1000)
|
||||||
|
filter_parts.append(f"[{i}:a]adelay={delay_ms}:all=1[a{i}]")
|
||||||
|
mix_inputs = "".join(f"[a{i}]" for i in range(len(clips)))
|
||||||
|
filter_parts.append(
|
||||||
|
f"{mix_inputs}amix=inputs={len(clips)}:duration=longest:normalize=0[out]"
|
||||||
|
)
|
||||||
|
_run(
|
||||||
|
["ffmpeg", "-y"] + inputs +
|
||||||
|
["-filter_complex", ";".join(filter_parts), "-map", "[out]", output_path],
|
||||||
|
label,
|
||||||
|
)
|
||||||
|
|
||||||
|
|
||||||
|
def assemble_timed_audio(clips: list, output_path: str, batch_size: int = 500):
|
||||||
|
"""
|
||||||
|
Place each audio clip at its correct timestamp using adelay, then mix.
|
||||||
|
clips: list of (start_sec, audio_path).
|
||||||
|
Processes in batches to stay under OS file-descriptor limits.
|
||||||
|
"""
|
||||||
|
if len(clips) <= batch_size:
|
||||||
|
_assemble_batch(clips, output_path, f"Assembling {len(clips)} timed clips")
|
||||||
|
return
|
||||||
|
|
||||||
|
tmp_dir = Path(output_path).parent / "_batch_tmp"
|
||||||
|
tmp_dir.mkdir(exist_ok=True)
|
||||||
|
batch_files = []
|
||||||
|
try:
|
||||||
|
for b, start in enumerate(range(0, len(clips), batch_size)):
|
||||||
|
batch = clips[start:start + batch_size]
|
||||||
|
batch_path = str(tmp_dir / f"batch_{b:04d}.flac")
|
||||||
|
_assemble_batch(
|
||||||
|
batch, batch_path,
|
||||||
|
f"Assembling batch {b + 1}/{(len(clips) - 1) // batch_size + 1} ({len(batch)} clips)",
|
||||||
|
)
|
||||||
|
batch_files.append(batch_path)
|
||||||
|
# merge batch files — they already carry absolute timing, so delay=0
|
||||||
|
batch_clips = [(0.0, Path(p)) for p in batch_files]
|
||||||
|
_assemble_batch(batch_clips, output_path, f"Merging {len(batch_files)} batches")
|
||||||
|
finally:
|
||||||
|
for p in batch_files:
|
||||||
|
Path(p).unlink(missing_ok=True)
|
||||||
|
tmp_dir.rmdir() if tmp_dir.exists() and not any(tmp_dir.iterdir()) else None
|
||||||
|
|
||||||
|
|
||||||
|
def merge_voiceover(video_path: str, tts_path: str, output_path: str,
|
||||||
|
orig_volume: float = 0.75):
|
||||||
|
_run([
|
||||||
|
"ffmpeg", "-y",
|
||||||
|
"-i", video_path,
|
||||||
|
"-i", tts_path,
|
||||||
|
"-filter_complex",
|
||||||
|
(
|
||||||
|
f"[0:a]volume={orig_volume}[orig];"
|
||||||
|
"[1:a]volume=1.0[tts];"
|
||||||
|
"[orig][tts]amix=inputs=2:duration=first:normalize=0[out]"
|
||||||
|
),
|
||||||
|
"-map", "0:v",
|
||||||
|
"-map", "[out]",
|
||||||
|
"-c:v", "copy",
|
||||||
|
"-c:a", "aac",
|
||||||
|
"-b:a", "192k",
|
||||||
|
output_path,
|
||||||
|
], "Merging voiceover with video")
|
||||||
|
|
||||||
|
|
||||||
|
# ---------------------------------------------------------------------------
|
||||||
|
# Boson AI TTS (higgs-audio-v3, OpenAI-compatible API)
|
||||||
|
# ---------------------------------------------------------------------------
|
||||||
|
|
||||||
|
_LOCAL_API_BASE = "http://localhost:8000/v1"
|
||||||
|
_BOSON_API_BASE = "https://api.boson.ai/v1"
|
||||||
|
_BOSON_MODEL = "higgs-audio-v3-tts"
|
||||||
|
_VLLM_MODEL = "bosonai/higgs-audio-v3-tts-4b"
|
||||||
|
_VOICE_PROMPTS_DIR = "/home/lukas/higgs-audio/examples/voice_prompts"
|
||||||
|
|
||||||
|
|
||||||
|
def _normalize_tts_text(text: str) -> str:
|
||||||
|
text = text.replace("\n", " ").strip()
|
||||||
|
text = re.sub(r"\s+", " ", text)
|
||||||
|
text = text.replace("(", " ").replace(")", " ")
|
||||||
|
if text and text[-1] not in ".!?,;\"'":
|
||||||
|
text += "."
|
||||||
|
return text.strip()
|
||||||
|
|
||||||
|
|
||||||
|
def _load_voice_prompt(voice: str) -> tuple:
|
||||||
|
"""Return (ref_audio_b64, ref_text) for the given voice name or wav path."""
|
||||||
|
wav_path = Path(voice)
|
||||||
|
if not wav_path.exists():
|
||||||
|
wav_path = Path(_VOICE_PROMPTS_DIR) / f"{voice}.wav"
|
||||||
|
txt_path = wav_path.with_suffix(".txt")
|
||||||
|
|
||||||
|
if not wav_path.exists():
|
||||||
|
raise FileNotFoundError(f"Voice prompt WAV not found: {wav_path}")
|
||||||
|
|
||||||
|
ref_audio_b64 = base64.b64encode(wav_path.read_bytes()).decode()
|
||||||
|
ref_text = txt_path.read_text(encoding="utf-8").strip() if txt_path.exists() else ""
|
||||||
|
return ref_audio_b64, ref_text
|
||||||
|
|
||||||
|
|
||||||
|
def generate_tts_per_block(
|
||||||
|
blocks: list,
|
||||||
|
download_dir: str,
|
||||||
|
voice: str = "gudowski",
|
||||||
|
api_key: str = "",
|
||||||
|
local: bool = True,
|
||||||
|
) -> list:
|
||||||
|
"""
|
||||||
|
Generate one audio clip per subtitle block.
|
||||||
|
local=True → calls local vllm-omni server at localhost:8095 (no API key needed)
|
||||||
|
local=False → calls Boson AI cloud API (needs BOSON_API_KEY)
|
||||||
|
blocks: list of (start_sec, end_sec, plain_text)
|
||||||
|
Returns list of (start_sec, audio_Path).
|
||||||
|
"""
|
||||||
|
try:
|
||||||
|
from openai import OpenAI
|
||||||
|
except ImportError:
|
||||||
|
sys.exit("openai package not found. Run: pip install openai")
|
||||||
|
|
||||||
|
if local:
|
||||||
|
client = OpenAI(api_key="local", base_url=_LOCAL_API_BASE)
|
||||||
|
model = _VLLM_MODEL
|
||||||
|
print(f" Using local vllm-omni server at {_LOCAL_API_BASE}")
|
||||||
|
else:
|
||||||
|
if not api_key:
|
||||||
|
api_key = os.environ.get("BOSON_API_KEY", "")
|
||||||
|
if not api_key:
|
||||||
|
sys.exit(
|
||||||
|
"Boson AI API key required.\n"
|
||||||
|
"Get a free key at https://boson.ai/workspace then:\n"
|
||||||
|
" export BOSON_API_KEY=bai-xxxx"
|
||||||
|
)
|
||||||
|
client = OpenAI(api_key=api_key, base_url=_BOSON_API_BASE)
|
||||||
|
model = _BOSON_MODEL
|
||||||
|
print(f" Using Boson AI cloud API")
|
||||||
|
|
||||||
|
ref_audio_b64, ref_text = _load_voice_prompt(voice)
|
||||||
|
print(f" Voice: {voice} (ref text: {ref_text[:60]!r}…)")
|
||||||
|
|
||||||
|
dl_dir = Path(download_dir).resolve()
|
||||||
|
dl_dir.mkdir(parents=True, exist_ok=True)
|
||||||
|
|
||||||
|
results = []
|
||||||
|
total = len(blocks)
|
||||||
|
|
||||||
|
for idx, (start_sec, end_sec, text) in enumerate(blocks):
|
||||||
|
text = _normalize_tts_text(text)
|
||||||
|
print(f" [{idx+1}/{total}] {_fmt_srt_time(start_sec)} {text[:70]!r}")
|
||||||
|
|
||||||
|
response = client.audio.speech.create(
|
||||||
|
model=model,
|
||||||
|
input=text,
|
||||||
|
voice="alloy",
|
||||||
|
response_format="mp3",
|
||||||
|
extra_body={
|
||||||
|
"ref_audio": f"data:audio/wav;base64,{ref_audio_b64}",
|
||||||
|
"ref_text": ref_text,
|
||||||
|
},
|
||||||
|
)
|
||||||
|
|
||||||
|
out_path = dl_dir / f"clip_{idx+1:04d}.mp3"
|
||||||
|
out_path.write_bytes(response.content)
|
||||||
|
results.append((start_sec, out_path))
|
||||||
|
|
||||||
|
return results
|
||||||
|
|
||||||
|
|
||||||
|
# ---------------------------------------------------------------------------
|
||||||
|
# CLI
|
||||||
|
# ---------------------------------------------------------------------------
|
||||||
|
|
||||||
|
def main():
|
||||||
|
parser = argparse.ArgumentParser(
|
||||||
|
description="Generate per-subtitle AI voiceover using Boson AI TTS (higgs-audio-v3)."
|
||||||
|
)
|
||||||
|
parser.add_argument("srt", help="Input SRT subtitle file")
|
||||||
|
parser.add_argument("video", help="Input video file (MKV, MP4, …)")
|
||||||
|
parser.add_argument("--start", default="00:00", metavar="MM:SS",
|
||||||
|
help="Fragment start time (default: 00:00)")
|
||||||
|
parser.add_argument("--end", default=None, metavar="MM:SS",
|
||||||
|
help="Fragment end time")
|
||||||
|
parser.add_argument("--voice", default="gudowski",
|
||||||
|
help="Voice prompt name or WAV path (default: gudowski)")
|
||||||
|
parser.add_argument("--cloud", action="store_true",
|
||||||
|
help="Use Boson AI cloud API instead of local vllm-omni server")
|
||||||
|
parser.add_argument("--api-key", default="",
|
||||||
|
help="Boson AI API key for --cloud mode (or set BOSON_API_KEY)")
|
||||||
|
parser.add_argument("--orig-volume", type=float, default=0.75, metavar="0-1",
|
||||||
|
help="Original audio volume in mix (default: 0.75)")
|
||||||
|
parser.add_argument("--output-dir", default="output",
|
||||||
|
help="Output directory (default: output/)")
|
||||||
|
parser.add_argument("--skip-tts", metavar="AUDIO_FILE",
|
||||||
|
help="Skip TTS and use this pre-assembled audio file")
|
||||||
|
args = parser.parse_args()
|
||||||
|
|
||||||
|
out_dir = Path(args.output_dir)
|
||||||
|
out_dir.mkdir(parents=True, exist_ok=True)
|
||||||
|
|
||||||
|
start_sec = _parse_mmss(args.start)
|
||||||
|
end_sec = _parse_mmss(args.end) if args.end else None
|
||||||
|
|
||||||
|
# ── Step 1: Parse subtitle blocks ─────────────────────────────────────────
|
||||||
|
print(f"\n[1/4] Parsing subtitles {args.start}–{args.end or 'end'}…")
|
||||||
|
if end_sec is None:
|
||||||
|
end_sec = 99 * 3600
|
||||||
|
blocks = parse_subtitle_blocks(args.srt, start_sec, end_sec)
|
||||||
|
if not blocks:
|
||||||
|
sys.exit("No subtitle blocks found in the specified range.")
|
||||||
|
print(f" {len(blocks)} subtitle block(s).")
|
||||||
|
|
||||||
|
# ── Step 2: Generate TTS per block ────────────────────────────────────────
|
||||||
|
if args.skip_tts:
|
||||||
|
assembled_tts = Path(args.skip_tts)
|
||||||
|
print(f"\n[2/4] Using existing TTS audio: {assembled_tts}")
|
||||||
|
else:
|
||||||
|
print(f"\n[2/4] Generating TTS ({len(blocks)} clips via Boson AI)…")
|
||||||
|
clips = generate_tts_per_block(
|
||||||
|
blocks,
|
||||||
|
str(out_dir),
|
||||||
|
voice=args.voice,
|
||||||
|
api_key=args.api_key,
|
||||||
|
local=not args.cloud,
|
||||||
|
)
|
||||||
|
assembled_tts = out_dir / "tts_assembled.wav"
|
||||||
|
print(f"\n Assembling timed audio…")
|
||||||
|
assemble_timed_audio(clips, str(assembled_tts))
|
||||||
|
|
||||||
|
# ── Step 3: Extract video fragment ───────────────────────────────────────
|
||||||
|
print(f"\n[3/4] Extracting video fragment…")
|
||||||
|
real_end = _parse_mmss(args.end) if args.end else None
|
||||||
|
if real_end is not None:
|
||||||
|
video_fragment = out_dir / "video_fragment.mkv"
|
||||||
|
extract_video_fragment(args.video, start_sec, real_end, str(video_fragment))
|
||||||
|
else:
|
||||||
|
video_fragment = Path(args.video)
|
||||||
|
|
||||||
|
# ── Step 4: Mix voiceover with video ─────────────────────────────────────
|
||||||
|
stem = Path(args.video).stem
|
||||||
|
final_output = out_dir / f"{stem}_lektor.mp4"
|
||||||
|
print(f"\n[4/4] Mixing (TTS 100%, original at {int(args.orig_volume * 100)}%)…")
|
||||||
|
merge_voiceover(str(video_fragment), str(assembled_tts), str(final_output),
|
||||||
|
orig_volume=args.orig_volume)
|
||||||
|
|
||||||
|
print(f"\nDone! Output: {final_output}")
|
||||||
|
|
||||||
|
|
||||||
|
if __name__ == "__main__":
|
||||||
|
main()
|
||||||
+1064
File diff suppressed because it is too large
Load Diff
@@ -0,0 +1,884 @@
|
|||||||
|
#!/usr/bin/env python3
|
||||||
|
"""
|
||||||
|
lektor_ai_comfy_v2 — SRT-to-voiceover via ComfyUI + Higgs Audio V3.
|
||||||
|
Sends subtitles in batches (default: 5) so the TTS model sees multi-sentence
|
||||||
|
context, which can improve prosody and naturalness compared to per-clip generation.
|
||||||
|
|
||||||
|
Usage:
|
||||||
|
python3 lektor_ai_comfy_v2.py input/film.srt input/film.mkv --start 00:00 --end 01:40
|
||||||
|
|
||||||
|
Requires ComfyUI with TTS-Audio-Suite running at http://127.0.0.1:8189
|
||||||
|
"""
|
||||||
|
|
||||||
|
import argparse
|
||||||
|
import json
|
||||||
|
import random
|
||||||
|
import re
|
||||||
|
import shutil
|
||||||
|
import subprocess
|
||||||
|
import sys
|
||||||
|
import time
|
||||||
|
import uuid
|
||||||
|
from pathlib import Path
|
||||||
|
from urllib.request import urlopen, Request
|
||||||
|
|
||||||
|
# ---------------------------------------------------------------------------
|
||||||
|
# Config
|
||||||
|
# ---------------------------------------------------------------------------
|
||||||
|
|
||||||
|
COMFY_URL = "http://127.0.0.1:8189"
|
||||||
|
COMFY_OUTPUT_DIR = Path("/home/lukas/comfy-tts-app/ComfyUI/output")
|
||||||
|
COMFY_INPUT_DIR = Path("/home/lukas/comfy-tts-app/ComfyUI/input")
|
||||||
|
VOICE_PROMPTS_DIR = Path("/home/lukas/Programy/lektor_ai/gudowski")
|
||||||
|
|
||||||
|
# ---------------------------------------------------------------------------
|
||||||
|
# SRT helpers
|
||||||
|
# ---------------------------------------------------------------------------
|
||||||
|
|
||||||
|
_SRT_TIME_RE = re.compile(r'(\d+):(\d+):(\d+)[,.](\d+)')
|
||||||
|
|
||||||
|
|
||||||
|
def _parse_srt_time(s: str) -> float:
|
||||||
|
m = _SRT_TIME_RE.match(s.strip())
|
||||||
|
if not m:
|
||||||
|
raise ValueError(f"Bad SRT timestamp: {s!r}")
|
||||||
|
h, mn, sec, ms = int(m[1]), int(m[2]), int(m[3]), int(m[4])
|
||||||
|
return h * 3600 + mn * 60 + sec + ms / 1000
|
||||||
|
|
||||||
|
|
||||||
|
def _fmt_srt_time(seconds: float) -> str:
|
||||||
|
h = int(seconds // 3600)
|
||||||
|
m = int((seconds % 3600) // 60)
|
||||||
|
s = int(seconds % 60)
|
||||||
|
ms = int(round((seconds % 1) * 1000))
|
||||||
|
if ms == 1000:
|
||||||
|
ms = 999
|
||||||
|
return f"{h:02d}:{m:02d}:{s:02d},{ms:03d}"
|
||||||
|
|
||||||
|
|
||||||
|
def _parse_mmss(value: str) -> float:
|
||||||
|
parts = value.strip().split(":")
|
||||||
|
if len(parts) == 2:
|
||||||
|
return int(parts[0]) * 60 + float(parts[1])
|
||||||
|
if len(parts) == 3:
|
||||||
|
return int(parts[0]) * 3600 + int(parts[1]) * 60 + float(parts[2])
|
||||||
|
raise ValueError(f"Cannot parse time: {value!r}")
|
||||||
|
|
||||||
|
|
||||||
|
def _ensure_terminal_punct(text: str, single_dot: bool = False) -> str:
|
||||||
|
if not text or text[-1] in ('.', ',', '!', '?', ';', ':'):
|
||||||
|
return text
|
||||||
|
return text + ('.' if single_dot else '...')
|
||||||
|
|
||||||
|
|
||||||
|
def _clean_text(lines: list) -> str:
|
||||||
|
has_dialogue = any(re.match(r"^-\s*\S", l.strip()) for l in lines if l.strip())
|
||||||
|
out = []
|
||||||
|
for line in lines:
|
||||||
|
line = line.strip()
|
||||||
|
line = re.sub(r"^-\s*", "", line)
|
||||||
|
if line:
|
||||||
|
out.append(line)
|
||||||
|
result = "\n".join(out) if has_dialogue else " ".join(out)
|
||||||
|
letters = [c for c in result if c.isalpha()]
|
||||||
|
was_allcaps = bool(letters) and all(c == c.upper() for c in letters)
|
||||||
|
if was_allcaps:
|
||||||
|
result = result.lower()
|
||||||
|
if result and result[0].islower():
|
||||||
|
result = result[0].upper() + result[1:]
|
||||||
|
if has_dialogue:
|
||||||
|
result = "\n".join(_ensure_terminal_punct(p, was_allcaps) for p in result.split("\n"))
|
||||||
|
else:
|
||||||
|
result = _ensure_terminal_punct(result, was_allcaps)
|
||||||
|
return result
|
||||||
|
|
||||||
|
|
||||||
|
def load_tts_input_srt(path: Path) -> list:
|
||||||
|
"""Read a pre-processed tts_input.srt into (t_start, t_end, text) tuples."""
|
||||||
|
text = path.read_text(encoding="utf-8")
|
||||||
|
blocks = re.split(r"\n{2,}", text.strip())
|
||||||
|
result = []
|
||||||
|
for block in blocks:
|
||||||
|
lines = block.strip().splitlines()
|
||||||
|
ti = next((i for i, l in enumerate(lines) if "-->" in l), None)
|
||||||
|
if ti is None:
|
||||||
|
continue
|
||||||
|
parts = lines[ti].split("-->")
|
||||||
|
if len(parts) != 2:
|
||||||
|
continue
|
||||||
|
t_start = _parse_srt_time(parts[0])
|
||||||
|
t_end = _parse_srt_time(parts[1])
|
||||||
|
payload = " ".join(l.strip() for l in lines[ti + 1:] if l.strip())
|
||||||
|
if payload:
|
||||||
|
result.append((t_start, t_end, payload))
|
||||||
|
return result
|
||||||
|
|
||||||
|
|
||||||
|
def parse_subtitle_blocks(srt_path: str, start_sec: float, end_sec: float) -> list:
|
||||||
|
"""Return list of (start_relative, end_relative, plain_text) blocks."""
|
||||||
|
text = Path(srt_path).read_text(encoding="utf-8-sig")
|
||||||
|
blocks = re.split(r"\n{2,}", text.strip())
|
||||||
|
result = []
|
||||||
|
for block in blocks:
|
||||||
|
lines = block.strip().splitlines()
|
||||||
|
ti = next((i for i, l in enumerate(lines) if "-->" in l), None)
|
||||||
|
if ti is None:
|
||||||
|
continue
|
||||||
|
parts = lines[ti].split("-->")
|
||||||
|
if len(parts) != 2:
|
||||||
|
continue
|
||||||
|
t_start = _parse_srt_time(parts[0])
|
||||||
|
t_end = _parse_srt_time(parts[1])
|
||||||
|
if t_start < start_sec or t_start > end_sec:
|
||||||
|
continue
|
||||||
|
plain = _clean_text(lines[ti + 1:])
|
||||||
|
if plain:
|
||||||
|
result.append((t_start - start_sec, t_end - start_sec, plain))
|
||||||
|
return result
|
||||||
|
|
||||||
|
|
||||||
|
def split_dialogue_blocks(blocks: list) -> list:
|
||||||
|
"""Split multi-line blocks into one entry per line with proportional timing."""
|
||||||
|
result = []
|
||||||
|
for t_start, t_end, text in blocks:
|
||||||
|
lines = [l for l in text.split("\n") if l.strip()]
|
||||||
|
if len(lines) <= 1:
|
||||||
|
result.append((t_start, t_end, text))
|
||||||
|
continue
|
||||||
|
duration = t_end - t_start
|
||||||
|
total_chars = sum(len(l) for l in lines)
|
||||||
|
current = t_start
|
||||||
|
for i, line in enumerate(lines):
|
||||||
|
share = len(line) / total_chars if total_chars else 1 / len(lines)
|
||||||
|
end = t_end if i == len(lines) - 1 else current + duration * share
|
||||||
|
result.append((current, end, line))
|
||||||
|
current = end
|
||||||
|
return result
|
||||||
|
|
||||||
|
|
||||||
|
def blocks_to_srt(blocks: list) -> str:
|
||||||
|
"""Convert (start, end, text) tuples back to SRT string for inspection."""
|
||||||
|
lines = []
|
||||||
|
for i, (t_start, t_end, text) in enumerate(blocks, 1):
|
||||||
|
lines.append(str(i))
|
||||||
|
lines.append(f"{_fmt_srt_time(t_start)} --> {_fmt_srt_time(t_end)}")
|
||||||
|
lines.append(text)
|
||||||
|
lines.append("")
|
||||||
|
return "\n".join(lines)
|
||||||
|
|
||||||
|
|
||||||
|
# ---------------------------------------------------------------------------
|
||||||
|
# FFmpeg helpers
|
||||||
|
# ---------------------------------------------------------------------------
|
||||||
|
|
||||||
|
def _ffmpeg_ts(seconds: float) -> str:
|
||||||
|
h = int(seconds // 3600)
|
||||||
|
m = int((seconds % 3600) // 60)
|
||||||
|
s = seconds % 60
|
||||||
|
return f"{h:02d}:{m:02d}:{s:06.3f}"
|
||||||
|
|
||||||
|
|
||||||
|
def _run(cmd: list, label: str):
|
||||||
|
print(f" {label}...")
|
||||||
|
result = subprocess.run(cmd, capture_output=True, text=True)
|
||||||
|
if result.returncode != 0:
|
||||||
|
print(f"ffmpeg error ({label}):\n{result.stderr[-2000:]}", file=sys.stderr)
|
||||||
|
sys.exit(1)
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
def get_audio_duration(path: Path) -> float:
|
||||||
|
result = subprocess.run(
|
||||||
|
["ffprobe", "-v", "quiet", "-print_format", "json", "-show_streams", str(path)],
|
||||||
|
capture_output=True, text=True,
|
||||||
|
)
|
||||||
|
info = json.loads(result.stdout)
|
||||||
|
for stream in info.get("streams", []):
|
||||||
|
if "duration" in stream:
|
||||||
|
return float(stream["duration"])
|
||||||
|
raise RuntimeError(f"Cannot determine duration of {path}")
|
||||||
|
|
||||||
|
|
||||||
|
def speed_up_audio(input_path: Path, output_path: Path, speed: float,
|
||||||
|
fade_out_ms: float = 0):
|
||||||
|
"""Apply atempo (and optional fade-out) to a single clip."""
|
||||||
|
filters = []
|
||||||
|
if speed != 1.0:
|
||||||
|
remaining = speed
|
||||||
|
while remaining > 2.0:
|
||||||
|
filters.append("atempo=2.0")
|
||||||
|
remaining /= 2.0
|
||||||
|
filters.append(f"atempo={remaining:.4f}")
|
||||||
|
if fade_out_ms > 0:
|
||||||
|
fade_sec = fade_out_ms / 1000
|
||||||
|
raw_dur = get_audio_duration(input_path)
|
||||||
|
post_dur = raw_dur / speed if speed != 1.0 else raw_dur
|
||||||
|
fade_start = max(0.0, post_dur - fade_sec)
|
||||||
|
filters.append(f"afade=t=out:st={fade_start:.3f}:d={fade_sec:.3f}")
|
||||||
|
if not filters:
|
||||||
|
shutil.copy2(input_path, output_path)
|
||||||
|
return
|
||||||
|
label = f"Speed {speed:.2f}×"
|
||||||
|
if fade_out_ms:
|
||||||
|
label += f" + fade-out {fade_out_ms:.0f}ms"
|
||||||
|
_run([
|
||||||
|
"ffmpeg", "-y",
|
||||||
|
"-i", str(input_path),
|
||||||
|
"-filter:a", ",".join(filters),
|
||||||
|
str(output_path),
|
||||||
|
], label)
|
||||||
|
|
||||||
|
|
||||||
|
def strip_leading_audio(input_path: Path, output_path: Path, strip_ms: float):
|
||||||
|
"""Remove the first strip_ms milliseconds from a clip."""
|
||||||
|
_run([
|
||||||
|
"ffmpeg", "-y",
|
||||||
|
"-i", str(input_path),
|
||||||
|
"-ss", f"{strip_ms / 1000:.3f}",
|
||||||
|
"-c", "copy",
|
||||||
|
str(output_path),
|
||||||
|
], f"Strip {strip_ms:.0f}ms leading audio")
|
||||||
|
|
||||||
|
|
||||||
|
def _assemble_batch(clips: list, output_path: str, label: str):
|
||||||
|
"""Mix a batch of (start_sec, path) clips with absolute adelay into output_path."""
|
||||||
|
if len(clips) == 1:
|
||||||
|
delay_ms = int(clips[0][0] * 1000)
|
||||||
|
_run([
|
||||||
|
"ffmpeg", "-y",
|
||||||
|
"-i", str(clips[0][1]),
|
||||||
|
"-af", f"adelay={delay_ms}:all=1",
|
||||||
|
output_path,
|
||||||
|
], label)
|
||||||
|
return
|
||||||
|
inputs, filter_parts = [], []
|
||||||
|
for i, (start_sec, path) in enumerate(clips):
|
||||||
|
inputs += ["-i", str(path)]
|
||||||
|
delay_ms = int(start_sec * 1000)
|
||||||
|
filter_parts.append(f"[{i}:a]adelay={delay_ms}:all=1[a{i}]")
|
||||||
|
mix_inputs = "".join(f"[a{i}]" for i in range(len(clips)))
|
||||||
|
filter_parts.append(
|
||||||
|
f"{mix_inputs}amix=inputs={len(clips)}:duration=longest:normalize=0[out]"
|
||||||
|
)
|
||||||
|
_run(
|
||||||
|
["ffmpeg", "-y"] + inputs +
|
||||||
|
["-filter_complex", ";".join(filter_parts), "-map", "[out]", output_path],
|
||||||
|
label,
|
||||||
|
)
|
||||||
|
|
||||||
|
|
||||||
|
def assemble_timed_audio(clips: list, output_path: str, batch_size: int = 500):
|
||||||
|
"""Place each clip at its start time via adelay then mix.
|
||||||
|
clips: list of (start_sec, Path).
|
||||||
|
Processes in batches to stay under OS file-descriptor limits.
|
||||||
|
"""
|
||||||
|
if len(clips) <= batch_size:
|
||||||
|
_assemble_batch(clips, output_path, f"Assembling {len(clips)} timed clips")
|
||||||
|
return
|
||||||
|
|
||||||
|
tmp_dir = Path(output_path).parent / "_batch_tmp"
|
||||||
|
tmp_dir.mkdir(exist_ok=True)
|
||||||
|
batch_files = []
|
||||||
|
try:
|
||||||
|
for b, start in enumerate(range(0, len(clips), batch_size)):
|
||||||
|
batch = clips[start:start + batch_size]
|
||||||
|
batch_path = str(tmp_dir / f"batch_{b:04d}.flac")
|
||||||
|
_assemble_batch(
|
||||||
|
batch, batch_path,
|
||||||
|
f"Assembling batch {b + 1}/{(len(clips) - 1) // batch_size + 1} ({len(batch)} clips)",
|
||||||
|
)
|
||||||
|
batch_files.append(batch_path)
|
||||||
|
batch_clips = [(0.0, Path(p)) for p in batch_files]
|
||||||
|
_assemble_batch(batch_clips, output_path, f"Merging {len(batch_files)} batches")
|
||||||
|
finally:
|
||||||
|
for p in batch_files:
|
||||||
|
Path(p).unlink(missing_ok=True)
|
||||||
|
tmp_dir.rmdir() if tmp_dir.exists() and not any(tmp_dir.iterdir()) else None
|
||||||
|
|
||||||
|
|
||||||
|
def extract_video_fragment(video_path: str, start_sec: float, end_sec: float,
|
||||||
|
output_path: str):
|
||||||
|
_run([
|
||||||
|
"ffmpeg", "-y",
|
||||||
|
"-ss", _ffmpeg_ts(start_sec),
|
||||||
|
"-i", video_path,
|
||||||
|
"-t", str(end_sec - start_sec),
|
||||||
|
"-c", "copy",
|
||||||
|
output_path,
|
||||||
|
], f"Extracting video {_ffmpeg_ts(start_sec)}–{_ffmpeg_ts(end_sec)}")
|
||||||
|
|
||||||
|
|
||||||
|
def merge_voiceover(video_path: str, tts_path: str, output_path: str,
|
||||||
|
orig_volume: float = 0.60, tts_volume: float = 1.8):
|
||||||
|
"""Produce a video with two audio tracks:
|
||||||
|
track 0 — Original (first audio stream from source, re-encoded to AAC)
|
||||||
|
track 1 — Polish (lektor mix: TTS at tts_volume + original at orig_volume)
|
||||||
|
"""
|
||||||
|
probe = subprocess.run(
|
||||||
|
["ffprobe", "-v", "quiet", "-print_format", "json", "-show_streams", video_path],
|
||||||
|
capture_output=True, text=True,
|
||||||
|
)
|
||||||
|
streams = json.loads(probe.stdout).get("streams", [])
|
||||||
|
audio_start = next(
|
||||||
|
(float(s.get("start_time", 0)) for s in streams if s.get("codec_type") == "audio"),
|
||||||
|
0.0,
|
||||||
|
)
|
||||||
|
pts_reset = ",asetpts=PTS-STARTPTS" if audio_start > 0.01 else ""
|
||||||
|
if audio_start > 0.01:
|
||||||
|
print(f" Video audio start offset: {audio_start:.3f}s — resetting PTS in lektor mix")
|
||||||
|
|
||||||
|
_run([
|
||||||
|
"ffmpeg", "-y",
|
||||||
|
"-i", video_path,
|
||||||
|
"-i", tts_path,
|
||||||
|
"-filter_complex",
|
||||||
|
(
|
||||||
|
f"[0:a:0]volume={orig_volume}{pts_reset}[orig_mix];"
|
||||||
|
f"[1:a]volume={tts_volume}[tts];"
|
||||||
|
"[orig_mix][tts]amix=inputs=2:duration=longest:normalize=0[lektor]"
|
||||||
|
),
|
||||||
|
"-map", "0:v",
|
||||||
|
"-map", "0:a:0",
|
||||||
|
"-map", "[lektor]",
|
||||||
|
"-c:v", "copy",
|
||||||
|
"-c:a:0", "copy",
|
||||||
|
"-c:a:1", "aac", "-b:a:1", "192k",
|
||||||
|
"-metadata:s:a:0", "title=Original",
|
||||||
|
"-metadata:s:a:1", "title=Polish",
|
||||||
|
"-metadata:s:a:1", "language=pol",
|
||||||
|
output_path,
|
||||||
|
], "Merging voiceover with video (2 audio tracks)")
|
||||||
|
|
||||||
|
|
||||||
|
# ---------------------------------------------------------------------------
|
||||||
|
# ComfyUI API helpers
|
||||||
|
# ---------------------------------------------------------------------------
|
||||||
|
|
||||||
|
def _comfy_post(path: str, data: dict) -> dict:
|
||||||
|
payload = json.dumps(data).encode()
|
||||||
|
req = Request(
|
||||||
|
f"{COMFY_URL}{path}",
|
||||||
|
data=payload,
|
||||||
|
headers={"Content-Type": "application/json"},
|
||||||
|
)
|
||||||
|
with urlopen(req, timeout=30) as r:
|
||||||
|
return json.loads(r.read())
|
||||||
|
|
||||||
|
|
||||||
|
def _comfy_get(path: str) -> dict:
|
||||||
|
with urlopen(f"{COMFY_URL}{path}", timeout=30) as r:
|
||||||
|
return json.loads(r.read())
|
||||||
|
|
||||||
|
|
||||||
|
def upload_voice_wav(wav_path: Path) -> tuple:
|
||||||
|
"""Copy voice WAV into ComfyUI's input dir; return (filename, reference_text)."""
|
||||||
|
dst = COMFY_INPUT_DIR / wav_path.name
|
||||||
|
if not dst.exists() or dst.stat().st_mtime < wav_path.stat().st_mtime:
|
||||||
|
shutil.copy2(wav_path, dst)
|
||||||
|
print(f" Copied {wav_path.name} → {dst}")
|
||||||
|
txt_path = wav_path.with_suffix(".txt")
|
||||||
|
ref_text = txt_path.read_text(encoding="utf-8").strip() if txt_path.exists() else ""
|
||||||
|
if ref_text:
|
||||||
|
print(f" Reference text loaded ({len(ref_text)} chars)")
|
||||||
|
else:
|
||||||
|
print(" WARNING: no reference .txt found — voice cloning quality will be lower")
|
||||||
|
return wav_path.name, ref_text
|
||||||
|
|
||||||
|
|
||||||
|
def build_text_workflow(text: str, voice_filename: str, voice_ref_text: str,
|
||||||
|
seed: int, temperature: float = 0.9,
|
||||||
|
max_new_tokens: int = 4096) -> dict:
|
||||||
|
"""ComfyUI workflow for a single text utterance with proper voice reference."""
|
||||||
|
return {
|
||||||
|
"1": {
|
||||||
|
"class_type": "HiggsAudioV3EngineNode",
|
||||||
|
"inputs": {
|
||||||
|
"model": "higgs-audio-v3-tts-4b",
|
||||||
|
"device": "auto",
|
||||||
|
"dtype": "auto",
|
||||||
|
"attention": "auto",
|
||||||
|
"temperature": temperature,
|
||||||
|
"top_p": 0.95,
|
||||||
|
"top_k": 50,
|
||||||
|
"max_new_tokens": max_new_tokens,
|
||||||
|
},
|
||||||
|
},
|
||||||
|
"2": {
|
||||||
|
"class_type": "LoadAudio",
|
||||||
|
"inputs": {"audio": voice_filename},
|
||||||
|
},
|
||||||
|
"3": {
|
||||||
|
"class_type": "CharacterVoicesNode",
|
||||||
|
"inputs": {
|
||||||
|
"voice_name": "none",
|
||||||
|
"reference_text": voice_ref_text,
|
||||||
|
"opt_audio_input": ["2", 0],
|
||||||
|
},
|
||||||
|
},
|
||||||
|
"4": {
|
||||||
|
"class_type": "UnifiedTTSTextNode",
|
||||||
|
"inputs": {
|
||||||
|
"TTS_engine": ["1", 0],
|
||||||
|
"text": text,
|
||||||
|
"narrator_voice": "none",
|
||||||
|
"seed": seed,
|
||||||
|
"opt_narrator": ["3", 0],
|
||||||
|
"enable_audio_cache": False,
|
||||||
|
},
|
||||||
|
},
|
||||||
|
"5": {
|
||||||
|
"class_type": "SaveAudio",
|
||||||
|
"inputs": {
|
||||||
|
"audio": ["4", 0],
|
||||||
|
"filename_prefix": "audio/lektor_clip",
|
||||||
|
},
|
||||||
|
},
|
||||||
|
}
|
||||||
|
|
||||||
|
|
||||||
|
def submit_workflow(workflow: dict) -> str:
|
||||||
|
client_id = str(uuid.uuid4())
|
||||||
|
resp = _comfy_post("/prompt", {"prompt": workflow, "client_id": client_id})
|
||||||
|
return resp["prompt_id"]
|
||||||
|
|
||||||
|
|
||||||
|
def wait_for_completion(prompt_id: str, poll_interval: int = 3) -> dict:
|
||||||
|
print(f" Waiting for {prompt_id[:8]}…", end="", flush=True)
|
||||||
|
while True:
|
||||||
|
try:
|
||||||
|
history = _comfy_get(f"/history/{prompt_id}")
|
||||||
|
except Exception as e:
|
||||||
|
print(f"\n Poll error: {e}", file=sys.stderr)
|
||||||
|
time.sleep(poll_interval)
|
||||||
|
continue
|
||||||
|
|
||||||
|
if prompt_id in history:
|
||||||
|
entry = history[prompt_id]
|
||||||
|
status = entry.get("status", {})
|
||||||
|
for msg_type, msg_data in status.get("messages", []):
|
||||||
|
if msg_type == "execution_error":
|
||||||
|
print(f"\nComfyUI error: {msg_data}", file=sys.stderr)
|
||||||
|
sys.exit(1)
|
||||||
|
if status.get("completed"):
|
||||||
|
print(" done.")
|
||||||
|
return entry
|
||||||
|
|
||||||
|
print(".", end="", flush=True)
|
||||||
|
time.sleep(poll_interval)
|
||||||
|
|
||||||
|
|
||||||
|
def extract_output_audio(history_entry: dict) -> Path:
|
||||||
|
outputs = history_entry.get("outputs", {})
|
||||||
|
for node_output in outputs.values():
|
||||||
|
for audio_info in node_output.get("audio", []):
|
||||||
|
subfolder = audio_info.get("subfolder", "")
|
||||||
|
path = COMFY_OUTPUT_DIR / subfolder / audio_info["filename"]
|
||||||
|
if path.exists():
|
||||||
|
return path
|
||||||
|
raise RuntimeError(f"Output audio not found in history: {outputs}")
|
||||||
|
|
||||||
|
|
||||||
|
def _generate_one(text: str, voice_filename: str, voice_ref_text: str,
|
||||||
|
seed: int, speed: float, out_dir: Path, label: str,
|
||||||
|
strip_leading_ms: float, temperature: float = 0.9,
|
||||||
|
trailing_fade_ms: float = 0, max_new_tokens: int = 4096) -> tuple:
|
||||||
|
"""Submit one TTS job and return (path, duration_after_speedup)."""
|
||||||
|
workflow = build_text_workflow(text, voice_filename, voice_ref_text, seed,
|
||||||
|
temperature, max_new_tokens)
|
||||||
|
prompt_id = submit_workflow(workflow)
|
||||||
|
history_entry = wait_for_completion(prompt_id)
|
||||||
|
raw_clip = extract_output_audio(history_entry)
|
||||||
|
|
||||||
|
if strip_leading_ms > 0:
|
||||||
|
stripped = out_dir / f"{label}_s.flac"
|
||||||
|
strip_leading_audio(raw_clip, stripped, strip_leading_ms)
|
||||||
|
raw_clip = stripped
|
||||||
|
|
||||||
|
fast_clip = out_dir / f"{label}.flac"
|
||||||
|
speed_up_audio(raw_clip, fast_clip, speed, fade_out_ms=trailing_fade_ms)
|
||||||
|
|
||||||
|
return fast_clip, get_audio_duration(fast_clip)
|
||||||
|
|
||||||
|
|
||||||
|
# ---------------------------------------------------------------------------
|
||||||
|
# Batch generation
|
||||||
|
# ---------------------------------------------------------------------------
|
||||||
|
|
||||||
|
def _group_into_batches(blocks: list, batch_size: int) -> list:
|
||||||
|
"""Group (t_start, t_end, text) blocks into batches of batch_size.
|
||||||
|
|
||||||
|
Returns list of dicts:
|
||||||
|
batch_num — 1-based batch index
|
||||||
|
t_start — SRT start of the first block in the batch
|
||||||
|
t_end — SRT end of the last block in the batch
|
||||||
|
block_indices — 1-based indices of the blocks that belong to this batch
|
||||||
|
texts — list of individual block texts (already have terminal punct)
|
||||||
|
"""
|
||||||
|
batches = []
|
||||||
|
for i in range(0, len(blocks), batch_size):
|
||||||
|
group = blocks[i:i + batch_size]
|
||||||
|
batches.append({
|
||||||
|
"batch_num": len(batches) + 1,
|
||||||
|
"t_start": group[0][0],
|
||||||
|
"t_end": group[-1][1],
|
||||||
|
"block_indices": list(range(i + 1, i + len(group) + 1)),
|
||||||
|
"texts": [text for _, _, text in group],
|
||||||
|
})
|
||||||
|
return batches
|
||||||
|
|
||||||
|
|
||||||
|
def _join_batch_texts(texts: list) -> str:
|
||||||
|
"""Join multiple subtitle texts into a single TTS prompt.
|
||||||
|
|
||||||
|
Each text already ends with punctuation from _clean_text. We join with a
|
||||||
|
space so the model reads them as one continuous multi-sentence utterance.
|
||||||
|
"""
|
||||||
|
return " ".join(texts)
|
||||||
|
|
||||||
|
|
||||||
|
def generate_batch_clips(
|
||||||
|
blocks: list, batch_size: int,
|
||||||
|
voice_filename: str, voice_ref_text: str,
|
||||||
|
base_seed: int, speed: float, clips_dir: Path,
|
||||||
|
strip_leading_ms: float = 0.0,
|
||||||
|
resume_from: int = 0,
|
||||||
|
temperature: float = 0.9,
|
||||||
|
trailing_fade_ms: float = 0,
|
||||||
|
max_new_tokens: int = 4096,
|
||||||
|
) -> tuple:
|
||||||
|
"""
|
||||||
|
Generate one audio clip per batch of `batch_size` consecutive subtitle blocks.
|
||||||
|
The texts of all blocks in a batch are joined into one TTS call, giving the
|
||||||
|
model sentence-level context to produce more natural prosody.
|
||||||
|
|
||||||
|
resume_from: skip batches 1..(resume_from-1) by loading existing clip files.
|
||||||
|
|
||||||
|
Returns (clips, batches) where:
|
||||||
|
clips — list of (actual_start_sec, clip_path) for assembly
|
||||||
|
batches — list of batch metadata dicts (saved to batches.json)
|
||||||
|
"""
|
||||||
|
batches = _group_into_batches(blocks, batch_size)
|
||||||
|
total = len(batches)
|
||||||
|
clips = []
|
||||||
|
cursor = 0.0
|
||||||
|
|
||||||
|
for batch in batches:
|
||||||
|
batch_num = batch["batch_num"]
|
||||||
|
t_start = batch["t_start"]
|
||||||
|
label = f"clip_{batch_num:04d}"
|
||||||
|
canonical = clips_dir / f"{label}.flac"
|
||||||
|
|
||||||
|
if resume_from > 0 and batch_num < resume_from:
|
||||||
|
if not canonical.exists():
|
||||||
|
sys.exit(
|
||||||
|
f"Resume from batch {resume_from}: {canonical} not found — "
|
||||||
|
f"lower --resume-from or regenerate from an earlier batch."
|
||||||
|
)
|
||||||
|
duration = get_audio_duration(canonical)
|
||||||
|
blocks_str = f"blocks {batch['block_indices'][0]}–{batch['block_indices'][-1]}"
|
||||||
|
print(f" [batch {batch_num}/{total}] skip (resuming) — "
|
||||||
|
f"{blocks_str} {canonical.name} {duration:.2f}s")
|
||||||
|
actual_start = max(t_start, cursor)
|
||||||
|
clips.append((actual_start, canonical))
|
||||||
|
cursor = actual_start + duration
|
||||||
|
continue
|
||||||
|
|
||||||
|
blocks_str = (
|
||||||
|
f"block {batch['block_indices'][0]}"
|
||||||
|
if len(batch["block_indices"]) == 1
|
||||||
|
else f"blocks {batch['block_indices'][0]}–{batch['block_indices'][-1]}"
|
||||||
|
)
|
||||||
|
print(f"\n [batch {batch_num}/{total}] {blocks_str} @ {_fmt_srt_time(t_start)}")
|
||||||
|
for i, text in enumerate(batch["texts"]):
|
||||||
|
print(f" [{i + 1}] {text[:80]!r}")
|
||||||
|
|
||||||
|
joined = _join_batch_texts(batch["texts"])
|
||||||
|
seed = base_seed + batch_num - 1
|
||||||
|
fast_clip, duration = _generate_one(
|
||||||
|
joined, voice_filename, voice_ref_text,
|
||||||
|
seed, speed, clips_dir, label, strip_leading_ms, temperature,
|
||||||
|
trailing_fade_ms=trailing_fade_ms, max_new_tokens=max_new_tokens,
|
||||||
|
)
|
||||||
|
|
||||||
|
actual_start = max(t_start, cursor)
|
||||||
|
clips.append((actual_start, fast_clip))
|
||||||
|
cursor = actual_start + duration
|
||||||
|
|
||||||
|
delay = actual_start - t_start
|
||||||
|
delay_str = f" [+{delay:.2f}s behind SRT]" if delay > 0.05 else ""
|
||||||
|
print(f" → placed at {_fmt_srt_time(actual_start)}, "
|
||||||
|
f"duration {duration:.2f}s{delay_str}")
|
||||||
|
|
||||||
|
return clips, batches
|
||||||
|
|
||||||
|
|
||||||
|
# ---------------------------------------------------------------------------
|
||||||
|
# Rebuild / Regen helpers
|
||||||
|
# ---------------------------------------------------------------------------
|
||||||
|
|
||||||
|
def _rebuild_video(args, out_dir: Path):
|
||||||
|
"""Re-assemble audio from existing batch clips and rebuild the final video."""
|
||||||
|
batches_json = out_dir / "batches.json"
|
||||||
|
if not batches_json.exists():
|
||||||
|
sys.exit(f"No batches.json found in {out_dir} — run a full v2 generation first.")
|
||||||
|
|
||||||
|
batches = json.loads(batches_json.read_text(encoding="utf-8"))
|
||||||
|
total = len(batches)
|
||||||
|
clips_dir = out_dir / "clips"
|
||||||
|
|
||||||
|
missing = [
|
||||||
|
clips_dir / f"clip_{b['batch_num']:04d}.flac"
|
||||||
|
for b in batches
|
||||||
|
if not (clips_dir / f"clip_{b['batch_num']:04d}.flac").exists()
|
||||||
|
]
|
||||||
|
if missing:
|
||||||
|
sys.exit("Missing batch clip files:\n "
|
||||||
|
+ "\n ".join(str(p) for p in missing[:10]))
|
||||||
|
|
||||||
|
clips = []
|
||||||
|
cursor = 0.0
|
||||||
|
for batch in batches:
|
||||||
|
clip_path = clips_dir / f"clip_{batch['batch_num']:04d}.flac"
|
||||||
|
duration = get_audio_duration(clip_path)
|
||||||
|
t_start = batch["t_start"]
|
||||||
|
actual_start = max(t_start, cursor)
|
||||||
|
clips.append((actual_start, clip_path))
|
||||||
|
cursor = actual_start + duration
|
||||||
|
|
||||||
|
assembled_tts = out_dir / "tts_assembled.flac"
|
||||||
|
print(f"\n Assembling {total} batch clips…")
|
||||||
|
assemble_timed_audio(clips, str(assembled_tts))
|
||||||
|
print(f" Assembled: {assembled_tts} ({assembled_tts.stat().st_size // 1024} KB)")
|
||||||
|
|
||||||
|
if args.end:
|
||||||
|
start_sec = _parse_mmss(args.start)
|
||||||
|
end_sec = _parse_mmss(args.end)
|
||||||
|
video_fragment = out_dir / "video_fragment.mkv"
|
||||||
|
if not video_fragment.exists():
|
||||||
|
extract_video_fragment(args.video, start_sec, end_sec, str(video_fragment))
|
||||||
|
video_src = str(video_fragment)
|
||||||
|
else:
|
||||||
|
video_src = args.video
|
||||||
|
|
||||||
|
stem = Path(args.video).stem
|
||||||
|
final_output = out_dir / f"{stem}_lektor.mp4"
|
||||||
|
print(f"\n Building 2-track video "
|
||||||
|
f"(TTS {int(args.tts_volume * 100)}% / orig {int(args.orig_volume * 100)}%)…")
|
||||||
|
merge_voiceover(video_src, str(assembled_tts), str(final_output),
|
||||||
|
orig_volume=args.orig_volume, tts_volume=args.tts_volume)
|
||||||
|
|
||||||
|
print(f"\nDone! Output: {final_output}")
|
||||||
|
|
||||||
|
|
||||||
|
def _regen_clips(args, out_dir: Path):
|
||||||
|
"""Regenerate specific batch clips by batch number.
|
||||||
|
Only replaces FLAC files — run --rebuild afterwards to reassemble audio and video.
|
||||||
|
"""
|
||||||
|
batches_json = out_dir / "batches.json"
|
||||||
|
if not batches_json.exists():
|
||||||
|
sys.exit(f"No batches.json found in {out_dir} — run a full v2 generation first.")
|
||||||
|
|
||||||
|
batches = json.loads(batches_json.read_text(encoding="utf-8"))
|
||||||
|
total = len(batches)
|
||||||
|
regen_set = set(args.regen)
|
||||||
|
clips_dir = out_dir / "clips"
|
||||||
|
clips_dir.mkdir(exist_ok=True)
|
||||||
|
|
||||||
|
bad = [n for n in regen_set if n < 1 or n > total]
|
||||||
|
if bad:
|
||||||
|
sys.exit(f"Batch number(s) out of range (1–{total}): {bad}")
|
||||||
|
|
||||||
|
wav_path = Path(args.voice)
|
||||||
|
if not wav_path.is_file():
|
||||||
|
wav_path = VOICE_PROMPTS_DIR / f"{args.voice}.wav"
|
||||||
|
if not wav_path.is_file():
|
||||||
|
sys.exit(f"Voice WAV not found: {wav_path}")
|
||||||
|
voice_filename, voice_ref_text = upload_voice_wav(wav_path)
|
||||||
|
|
||||||
|
base_seed = args.seed if args.seed != 0 else random.randint(1, 2**31 - 1)
|
||||||
|
print(f"\n Regenerating batches {sorted(regen_set)} with seed {base_seed}…")
|
||||||
|
|
||||||
|
for batch in batches:
|
||||||
|
batch_num = batch["batch_num"]
|
||||||
|
if batch_num not in regen_set:
|
||||||
|
continue
|
||||||
|
|
||||||
|
joined = _join_batch_texts(batch["texts"])
|
||||||
|
label = f"clip_{batch_num:04d}"
|
||||||
|
blocks_str = f"blocks {batch['block_indices'][0]}–{batch['block_indices'][-1]}"
|
||||||
|
print(f"\n REGEN [batch {batch_num}/{total}] {blocks_str}")
|
||||||
|
for i, text in enumerate(batch["texts"]):
|
||||||
|
print(f" [{i + 1}] {text[:80]!r}")
|
||||||
|
|
||||||
|
seed = base_seed + batch_num - 1
|
||||||
|
clip_path, duration = _generate_one(
|
||||||
|
joined, voice_filename, voice_ref_text,
|
||||||
|
seed, args.speed, clips_dir, label,
|
||||||
|
args.strip_leading_ms, args.temperature,
|
||||||
|
trailing_fade_ms=args.trailing_fade_ms,
|
||||||
|
max_new_tokens=args.max_tokens,
|
||||||
|
)
|
||||||
|
print(f" → saved as {clip_path.name} ({duration:.2f}s)")
|
||||||
|
|
||||||
|
print(f"\nDone. Run --rebuild to reassemble audio and rebuild the video.")
|
||||||
|
|
||||||
|
|
||||||
|
# ---------------------------------------------------------------------------
|
||||||
|
# CLI
|
||||||
|
# ---------------------------------------------------------------------------
|
||||||
|
|
||||||
|
def main():
|
||||||
|
global COMFY_URL
|
||||||
|
parser = argparse.ArgumentParser(
|
||||||
|
description=(
|
||||||
|
"Generate AI voiceover via ComfyUI + Higgs Audio V3, in batches of "
|
||||||
|
"N subtitle blocks per TTS call for contextual multi-sentence generation."
|
||||||
|
)
|
||||||
|
)
|
||||||
|
parser.add_argument("srt", help="Input SRT subtitle file")
|
||||||
|
parser.add_argument("video", help="Input video file (MKV, MP4, …)")
|
||||||
|
parser.add_argument("--start", default="00:00", metavar="MM:SS",
|
||||||
|
help="Fragment start time (default: 00:00)")
|
||||||
|
parser.add_argument("--end", default=None, metavar="MM:SS",
|
||||||
|
help="Fragment end time")
|
||||||
|
parser.add_argument("--voice", default="gudowski_short",
|
||||||
|
help="Voice prompt name or WAV path (default: gudowski_short)")
|
||||||
|
parser.add_argument("--batch-size", type=int, default=5, metavar="N",
|
||||||
|
help="Subtitle blocks per TTS call (default: 5). "
|
||||||
|
"Larger batches give the model more context but produce "
|
||||||
|
"longer clips that are harder to regen individually.")
|
||||||
|
parser.add_argument("--speed", type=float, default=1.2, metavar="FACTOR",
|
||||||
|
help="Per-clip playback speed via atempo (default: 1.2)")
|
||||||
|
parser.add_argument("--seed", type=int, default=0,
|
||||||
|
help="Base generation seed (default: 0 = random each run)")
|
||||||
|
parser.add_argument("--orig-volume", type=float, default=0.60, metavar="0-1",
|
||||||
|
help="Original audio volume in mix (default: 0.60)")
|
||||||
|
parser.add_argument("--tts-volume", type=float, default=1.8, metavar="0-2",
|
||||||
|
help="Voiceover volume in mix (default: 1.8)")
|
||||||
|
parser.add_argument("--output-dir", default="output",
|
||||||
|
help="Output directory (default: output/)")
|
||||||
|
parser.add_argument("--comfy-url", default=COMFY_URL,
|
||||||
|
help=f"ComfyUI URL (default: {COMFY_URL})")
|
||||||
|
parser.add_argument("--skip-tts", metavar="AUDIO_FILE",
|
||||||
|
help="Skip TTS and use this pre-assembled audio file")
|
||||||
|
parser.add_argument("--temperature", type=float, default=0.9, metavar="T",
|
||||||
|
help="Sampling temperature (default: 0.9)")
|
||||||
|
parser.add_argument("--strip-leading-ms", type=float, default=0.0, metavar="MS",
|
||||||
|
help="Strip N ms from the start of each clip (default: 0)")
|
||||||
|
parser.add_argument("--trailing-fade-ms", type=float, default=80.0, metavar="MS",
|
||||||
|
help="Fade out the last N ms of each clip (default: 80)")
|
||||||
|
parser.add_argument("--max-tokens", type=int, default=4096, metavar="N",
|
||||||
|
help="Max audio tokens per TTS call (default: 4096). "
|
||||||
|
"Raise if batch clips are cut off mid-sentence.")
|
||||||
|
parser.add_argument("--resume-from", type=int, default=0, metavar="N",
|
||||||
|
help="Resume a crashed run: skip batches 1..(N-1) by loading "
|
||||||
|
"their existing clip files, generate from batch N onwards.")
|
||||||
|
parser.add_argument("--rebuild", action="store_true",
|
||||||
|
help="Re-assemble audio from existing clips and rebuild the "
|
||||||
|
"final video without any TTS generation.")
|
||||||
|
parser.add_argument("--regen", nargs="+", type=int, metavar="N",
|
||||||
|
help="Regenerate specific batch clip(s) by batch number "
|
||||||
|
"(as shown during generation and in batches.json). "
|
||||||
|
"Run --rebuild afterwards to reassemble the video.")
|
||||||
|
args = parser.parse_args()
|
||||||
|
|
||||||
|
COMFY_URL = args.comfy_url
|
||||||
|
|
||||||
|
out_dir = Path(args.output_dir)
|
||||||
|
out_dir.mkdir(parents=True, exist_ok=True)
|
||||||
|
clips_dir = out_dir / "clips"
|
||||||
|
clips_dir.mkdir(exist_ok=True)
|
||||||
|
|
||||||
|
# ── Rebuild mode ──────────────────────────────────────────────────────────
|
||||||
|
if args.rebuild:
|
||||||
|
_rebuild_video(args, out_dir)
|
||||||
|
return
|
||||||
|
|
||||||
|
# ── Regen mode ────────────────────────────────────────────────────────────
|
||||||
|
if args.regen:
|
||||||
|
_regen_clips(args, out_dir)
|
||||||
|
return
|
||||||
|
|
||||||
|
start_sec = _parse_mmss(args.start)
|
||||||
|
end_sec = _parse_mmss(args.end) if args.end else None
|
||||||
|
|
||||||
|
# ── Step 1: Parse subtitles ───────────────────────────────────────────────
|
||||||
|
print(f"\n[1/4] Parsing subtitles {args.start}–{args.end or 'end'}…")
|
||||||
|
if end_sec is None:
|
||||||
|
end_sec = 99 * 3600
|
||||||
|
blocks = parse_subtitle_blocks(args.srt, start_sec, end_sec)
|
||||||
|
if not blocks:
|
||||||
|
sys.exit("No subtitle blocks found in the specified range.")
|
||||||
|
blocks = split_dialogue_blocks(blocks)
|
||||||
|
n_batches = (len(blocks) + args.batch_size - 1) // args.batch_size
|
||||||
|
print(f" {len(blocks)} subtitle blocks → {n_batches} batches "
|
||||||
|
f"of up to {args.batch_size}.")
|
||||||
|
|
||||||
|
srt_preview = out_dir / "tts_input.srt"
|
||||||
|
srt_preview.write_text(blocks_to_srt(blocks), encoding="utf-8")
|
||||||
|
print(f" Saved individual blocks to {srt_preview} for reference.")
|
||||||
|
|
||||||
|
# ── Step 2: Generate TTS per batch ────────────────────────────────────────
|
||||||
|
if args.skip_tts:
|
||||||
|
assembled_tts = Path(args.skip_tts)
|
||||||
|
print(f"\n[2/4] Using existing TTS audio: {assembled_tts}")
|
||||||
|
else:
|
||||||
|
print(f"\n[2/4] Generating TTS ({n_batches} batches × up to "
|
||||||
|
f"{args.batch_size} blocks, speed {args.speed}×)…")
|
||||||
|
|
||||||
|
wav_path = Path(args.voice)
|
||||||
|
if not wav_path.is_file():
|
||||||
|
wav_path = VOICE_PROMPTS_DIR / f"{args.voice}.wav"
|
||||||
|
if not wav_path.is_file():
|
||||||
|
sys.exit(f"Voice WAV not found: {wav_path}")
|
||||||
|
print(f" Voice: {wav_path.name}")
|
||||||
|
|
||||||
|
base_seed = args.seed if args.seed != 0 else random.randint(1, 2**31 - 1)
|
||||||
|
print(f" Base seed: {base_seed}")
|
||||||
|
voice_filename, voice_ref_text = upload_voice_wav(wav_path)
|
||||||
|
|
||||||
|
clips, batches = generate_batch_clips(
|
||||||
|
blocks, args.batch_size,
|
||||||
|
voice_filename, voice_ref_text,
|
||||||
|
base_seed, args.speed, clips_dir,
|
||||||
|
strip_leading_ms=args.strip_leading_ms,
|
||||||
|
resume_from=args.resume_from,
|
||||||
|
temperature=args.temperature,
|
||||||
|
trailing_fade_ms=args.trailing_fade_ms,
|
||||||
|
max_new_tokens=args.max_tokens,
|
||||||
|
)
|
||||||
|
|
||||||
|
batches_json = out_dir / "batches.json"
|
||||||
|
batches_json.write_text(
|
||||||
|
json.dumps(batches, ensure_ascii=False, indent=2),
|
||||||
|
encoding="utf-8",
|
||||||
|
)
|
||||||
|
print(f"\n Batch map saved to {batches_json}")
|
||||||
|
|
||||||
|
assembled_tts = out_dir / "tts_assembled.flac"
|
||||||
|
assemble_timed_audio(clips, str(assembled_tts))
|
||||||
|
print(f" Assembled: {assembled_tts} ({assembled_tts.stat().st_size // 1024} KB)")
|
||||||
|
print(f" Clips saved to {clips_dir}")
|
||||||
|
|
||||||
|
# ── Step 3: Extract video fragment ───────────────────────────────────────
|
||||||
|
print(f"\n[3/4] Extracting video fragment…")
|
||||||
|
real_end = _parse_mmss(args.end) if args.end else None
|
||||||
|
if real_end is not None:
|
||||||
|
video_fragment = out_dir / "video_fragment.mkv"
|
||||||
|
extract_video_fragment(args.video, start_sec, real_end, str(video_fragment))
|
||||||
|
else:
|
||||||
|
video_fragment = Path(args.video)
|
||||||
|
|
||||||
|
# ── Step 4: Mix voiceover with video ─────────────────────────────────────
|
||||||
|
stem = Path(args.video).stem
|
||||||
|
final_output = out_dir / f"{stem}_lektor.mp4"
|
||||||
|
print(f"\n[4/4] Building 2-track video (Original + Polish lektor "
|
||||||
|
f"TTS {int(args.tts_volume * 100)}% / orig {int(args.orig_volume * 100)}%)…")
|
||||||
|
merge_voiceover(str(video_fragment), str(assembled_tts), str(final_output),
|
||||||
|
orig_volume=args.orig_volume, tts_volume=args.tts_volume)
|
||||||
|
|
||||||
|
print(f"\nDone! Output: {final_output}")
|
||||||
|
|
||||||
|
|
||||||
|
if __name__ == "__main__":
|
||||||
|
main()
|
||||||
+216
@@ -0,0 +1,216 @@
|
|||||||
|
lektor_ai_comfy — Polish AI Dubbing Tool
|
||||||
|
=========================================
|
||||||
|
|
||||||
|
FULL DOCUMENTATION: see docs/
|
||||||
|
docs/README.md start here — overview and quick start
|
||||||
|
docs/requirements.md hardware, software, dependencies
|
||||||
|
docs/installation.md installing ComfyUI + the TTS engine
|
||||||
|
docs/usage.md every CLI flag, working modes, recipes
|
||||||
|
docs/architecture.md how the pipeline works internally
|
||||||
|
docs/troubleshooting.md diagnosed failure modes and their fixes
|
||||||
|
|
||||||
|
Installer: ./install_comfyui.sh (see docs/installation.md)
|
||||||
|
|
||||||
|
This file remains as a condensed quick reference.
|
||||||
|
|
||||||
|
|
||||||
|
Converts SRT subtitles to AI voiceover (lektor style) using ComfyUI + Higgs Audio V3 TTS,
|
||||||
|
then mixes the result with the original video audio into a two-track MKV.
|
||||||
|
All subtitle tracks in the source video are carried over to the output.
|
||||||
|
|
||||||
|
Requirements
|
||||||
|
------------
|
||||||
|
- ComfyUI running at http://127.0.0.1:8189 with TTS-Audio-Suite custom node
|
||||||
|
- Model: higgs-audio-v3-tts-4b (bosonai/higgs-audio-v3)
|
||||||
|
- ffmpeg and ffprobe available on PATH
|
||||||
|
- Voice reference: a WAV file + matching .txt transcript in the voice directory
|
||||||
|
(default: gudowski/gudowski.wav + gudowski/gudowski.txt)
|
||||||
|
|
||||||
|
|
||||||
|
Basic Usage
|
||||||
|
-----------
|
||||||
|
|
||||||
|
Full run with an external SRT file:
|
||||||
|
|
||||||
|
python3 lektor_ai_comfy.py input/film.srt input/film.mkv
|
||||||
|
|
||||||
|
Use subtitles embedded in the video (lists tracks, prompts for a choice):
|
||||||
|
|
||||||
|
python3 lektor_ai_comfy.py input/film.mkv --embedded-subs
|
||||||
|
|
||||||
|
Process only a time range (extracts a video fragment automatically):
|
||||||
|
|
||||||
|
python3 lektor_ai_comfy.py input/film.srt input/film.mkv --start 01:20 --end 05:40
|
||||||
|
|
||||||
|
|
||||||
|
Subtitle Handling
|
||||||
|
-----------------
|
||||||
|
Both external SRT files and embedded subtitle tracks go through the same cleaning:
|
||||||
|
|
||||||
|
- HTML tags stripped: <i>, </i>, <b>, <font color="…">, etc.
|
||||||
|
- ASS override codes stripped: {\an8}, {\pos(x,y)}, {\c&H…&}, etc.
|
||||||
|
- ALL-CAPS blocks lowercased and converted to sentence case
|
||||||
|
- Sentence fragments starting with a lowercase letter are capitalised
|
||||||
|
- Missing terminal punctuation added (… or . depending on context)
|
||||||
|
- Dialogue lines (prefixed with -) are split into separate clips with proportional timing
|
||||||
|
|
||||||
|
With --embedded-subs the extracted subtitles are saved to output/embedded_sub.srt
|
||||||
|
so you can inspect them before or after the run.
|
||||||
|
|
||||||
|
The final MKV preserves all subtitle tracks from the source video — text-based
|
||||||
|
(SRT, ASS, SSA, WebVTT) and image-based (PGS Blu-ray, VOBSUB) alike — using
|
||||||
|
stream copy with no conversion.
|
||||||
|
|
||||||
|
|
||||||
|
Output Files
|
||||||
|
------------
|
||||||
|
output/
|
||||||
|
tts_input.srt — processed subtitle text sent to TTS (inspect to verify)
|
||||||
|
embedded_sub.srt — extracted embedded subtitle (only when --embedded-subs)
|
||||||
|
clips/clip_NNNN.flac — individual TTS clips, one per subtitle block (lossless)
|
||||||
|
tts_assembled.flac — all clips assembled with correct timing (lossless)
|
||||||
|
video_fragment.mkv — extracted video segment (only when --end is used)
|
||||||
|
{stem}_lektor.mkv — final video with two audio tracks + all original subtitles:
|
||||||
|
Track 0 — Original (source audio, stream-copied as-is)
|
||||||
|
Track 1 — Polish lektor (TTS + original mixed,
|
||||||
|
AAC 192k stereo)
|
||||||
|
|
||||||
|
|
||||||
|
All Options
|
||||||
|
-----------
|
||||||
|
|
||||||
|
srt Input SRT subtitle file (omit when using --embedded-subs)
|
||||||
|
video Input video file (MKV, MP4, ...)
|
||||||
|
|
||||||
|
--embedded-subs Use a subtitle track embedded in the video instead of an
|
||||||
|
external SRT file. The script lists all subtitle tracks
|
||||||
|
found in the video and prompts you to choose one.
|
||||||
|
The video path can be given as the only positional argument:
|
||||||
|
python3 lektor_ai_comfy.py film.mkv --embedded-subs
|
||||||
|
|
||||||
|
--start MM:SS Start time of the range to process (default: 00:00)
|
||||||
|
--end MM:SS End time of the range to process (default: end of file)
|
||||||
|
|
||||||
|
--voice NAME|PATH Voice prompt: name of a subdirectory under the voice prompts
|
||||||
|
directory, or a direct path to a WAV file.
|
||||||
|
The directory must contain a .wav and matching .txt transcript.
|
||||||
|
(default: gudowski_short)
|
||||||
|
|
||||||
|
--speed FACTOR Per-clip playback speed via atempo (default: 1.2)
|
||||||
|
Values above 1.0 speed up speech; 1.0 = no change.
|
||||||
|
|
||||||
|
--seed N Base generation seed. 0 = random each run. (default: 0)
|
||||||
|
A fixed seed produces deterministic output.
|
||||||
|
|
||||||
|
--temperature T TTS sampling temperature (default: 0.9)
|
||||||
|
Lower = more deterministic; higher = more expressive but
|
||||||
|
risks hallucination (1.0 can cause the model to recite
|
||||||
|
the reference text instead of the subtitle).
|
||||||
|
|
||||||
|
--max-tokens N Max audio tokens the model may generate per clip (default: 4096)
|
||||||
|
Raise this if long clips are cut off mid-word.
|
||||||
|
|
||||||
|
--orig-volume 0-1 Volume of the original audio in the Polish lektor mix (default: 0.60)
|
||||||
|
--tts-volume 0-2 Volume of the TTS voiceover in the Polish lektor mix (default: 1.8)
|
||||||
|
Note: these affect Track 1 only. Track 0 is always the raw original.
|
||||||
|
|
||||||
|
--strip-leading-ms MS Strip N milliseconds from the start of each clip to remove
|
||||||
|
leading breath or noise artifacts (default: 0, try 80-150)
|
||||||
|
|
||||||
|
--trailing-fade-ms MS Fade out the last N milliseconds of each clip to suppress
|
||||||
|
tail pops or noise (default: 80, set 0 to disable)
|
||||||
|
|
||||||
|
--output-dir DIR Output directory (default: output/)
|
||||||
|
|
||||||
|
--comfy-url URL ComfyUI server URL (default: http://127.0.0.1:8189)
|
||||||
|
|
||||||
|
--skip-tts AUDIO Skip TTS generation entirely and use this pre-assembled
|
||||||
|
audio file instead. Goes straight to video merge.
|
||||||
|
|
||||||
|
|
||||||
|
Recovery and Fixing Bad Clips
|
||||||
|
------------------------------
|
||||||
|
|
||||||
|
Resume a crashed run from block N (reuses existing clips 1..N-1):
|
||||||
|
|
||||||
|
python3 lektor_ai_comfy.py input/film.srt input/film.mkv --resume-from 47
|
||||||
|
|
||||||
|
Regenerate specific clips by block number (same numbering as tts_input.srt):
|
||||||
|
|
||||||
|
python3 lektor_ai_comfy.py input/film.mkv --regen 12 22 41
|
||||||
|
|
||||||
|
Only replaces the selected clips/clip_NNNN.flac files — does not reassemble
|
||||||
|
or rebuild the video. Run --rebuild afterwards to apply the changes.
|
||||||
|
Useful for fixing hallucinations, mumbling, or cut-off clips.
|
||||||
|
|
||||||
|
Reassemble audio from all existing clips and rebuild the final video
|
||||||
|
(no TTS generation — use this after --regen, or to adjust volumes):
|
||||||
|
|
||||||
|
python3 lektor_ai_comfy.py input/film.mkv --rebuild
|
||||||
|
|
||||||
|
python3 lektor_ai_comfy.py input/film.mkv --rebuild --orig-volume 0.5 --tts-volume 2.0
|
||||||
|
|
||||||
|
For --rebuild and --regen the SRT file is not needed — just pass the video path.
|
||||||
|
When --end is given, rebuilds from the same video fragment as the original run
|
||||||
|
(re-extracts it if needed). Without --end, uses the full video file.
|
||||||
|
|
||||||
|
|
||||||
|
Tips
|
||||||
|
----
|
||||||
|
|
||||||
|
- Check output/tts_input.srt before or after a run to see the exact text sent to TTS.
|
||||||
|
Block numbers in that file match the clip_NNNN.flac filenames.
|
||||||
|
|
||||||
|
- If a clip sounds wrong (hallucination, wrong language, cut off), note its block
|
||||||
|
number from tts_input.srt and use --regen N to replace just that clip,
|
||||||
|
then run --rebuild to reassemble the video.
|
||||||
|
|
||||||
|
- If ComfyUI crashes mid-run, note the last completed block number and use
|
||||||
|
--resume-from N to continue without regenerating finished clips.
|
||||||
|
|
||||||
|
- The final MP4 has two audio tracks. Most players let you switch between them:
|
||||||
|
Track 0 — Original soundtrack
|
||||||
|
Track 1 — Polish lektor (AI voiceover mixed with quiet original)
|
||||||
|
|
||||||
|
- Track 0 (Original) is stream-copied without re-encoding, so source quality is preserved.
|
||||||
|
The MKV container supports all common audio codecs (AAC, AC3, EAC3, DTS, TrueHD),
|
||||||
|
so no workarounds are needed for the original audio track.
|
||||||
|
|
||||||
|
- Videos with audio starting at a non-zero PTS offset (common in MKV rips) are handled
|
||||||
|
automatically — the original audio background in the lektor mix is normalised so it
|
||||||
|
stays in sync with the video.
|
||||||
|
|
||||||
|
- The lektor track (Track 1) is stereo by design, while Track 0 keeps the original
|
||||||
|
5.1 untouched. ffmpeg's AAC encoder does not signal a 5.1 channel layout — it ends up
|
||||||
|
tagged "unknown" in every container, and players that cannot route the channels play
|
||||||
|
the track as silence. Stereo is tagged correctly and plays everywhere. It also spends
|
||||||
|
the 192k on two channels instead of spreading it thinly across six. The 5.1 original
|
||||||
|
is downmixed with ffmpeg's standard coefficients and the mono TTS is centred.
|
||||||
|
|
||||||
|
- The final video is built in two passes: first the lektor audio mix is rendered to a
|
||||||
|
temporary file, then everything is stream-copied into the MKV. This is deliberate.
|
||||||
|
Doing both at once fails on sources with many subtitle tracks (some releases carry
|
||||||
|
40+) — the sparse subtitle streams stall the muxer while the audio filter keeps
|
||||||
|
producing, and ffmpeg aborts with "Failed to inject frame into filter network".
|
||||||
|
The two-pass approach also makes the final mux about 10x faster.
|
||||||
|
|
||||||
|
- The final mux passes -max_interleave_delta 0 as an OUTPUT option (it must come after
|
||||||
|
the maps and before the output path; placed before -i it configures the demuxer and
|
||||||
|
silently does nothing). Without it, sparse subtitle tracks make the muxer abandon
|
||||||
|
interleaving after its default 10-second window, and the lektor audio ends up written
|
||||||
|
hundreds of megabytes away from the video it belongs with. The file measures perfectly
|
||||||
|
fine with ffmpeg, but players read MKV sequentially and never reach that audio, so the
|
||||||
|
Polish track plays as complete silence. Setting it to 0 buffers until every stream has
|
||||||
|
a packet, costing a couple of GB of transient RAM for a few seconds.
|
||||||
|
|
||||||
|
- If a lektor track ever plays silent again, check interleaving before suspecting the
|
||||||
|
audio. Compare byte positions at the same timestamp:
|
||||||
|
ffprobe -read_intervals 00:20:00%+1 -select_streams v -show_entries packet=pos file.mkv
|
||||||
|
ffprobe -read_intervals 00:20:00%+1 -select_streams a:1 -show_entries packet=pos file.mkv
|
||||||
|
They should be within a few MB of each other. An existing file can be repaired without
|
||||||
|
regenerating anything:
|
||||||
|
ffmpeg -i in.mkv -map 0 -c copy -max_interleave_delta 0 out.mkv
|
||||||
|
|
||||||
|
- ComfyUI crash recovery:
|
||||||
|
sudo fuser -k /dev/nvidia*
|
||||||
|
sudo nvidia-smi --gpu-reset
|
||||||
@@ -0,0 +1,10 @@
|
|||||||
|
# Dependencies for lektor_ai.py
|
||||||
|
openai>=1.0.0
|
||||||
|
|
||||||
|
# System dependency: ffmpeg must be on PATH.
|
||||||
|
# Voice prompts (gudowski.wav etc.) live in:
|
||||||
|
# /home/lukas/higgs-audio/examples/voice_prompts/
|
||||||
|
#
|
||||||
|
# Boson AI API key required (free tier):
|
||||||
|
# export BOSON_API_KEY=bai-xxxx
|
||||||
|
# Get one at https://boson.ai/workspace
|
||||||
Executable
+4
@@ -0,0 +1,4 @@
|
|||||||
|
#!/bin/bash
|
||||||
|
source ~/miniforge3/etc/profile.d/conda.sh
|
||||||
|
conda activate comfy-tts
|
||||||
|
cd ~/comfy-tts-app/ComfyUI && python main.py --port 8189
|
||||||
Reference in New Issue
Block a user