Initial commit: Polish AI dubbing pipeline (ComfyUI + Higgs Audio V3)
Converts SRT subtitles into a Polish lektor-style voiceover using a local
ComfyUI instance running Higgs Audio V3, then muxes the result back into the
source video as a second audio track. Video and original audio are always
stream-copied; only the new voiceover track is encoded.
Contents:
lektor_ai_comfy.py main pipeline (standard library only, needs ffmpeg)
lektor_ai_comfy_v2.py batched-context experiment (timing unresolved)
lektor_ai.py earlier version using the hosted Boson AI API
install_comfyui.sh reproduces the ComfyUI environment, versions pinned
docs/ requirements, installation, usage, architecture,
troubleshooting
gudowski/ voice references (.wav) with their transcripts (.txt)
Source media and generated video are gitignored.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
This commit is contained in:
Claude Opus 5
@@ -0,0 +1,295 @@
|
||||
# Troubleshooting
|
||||
|
||||
Failure modes that have actually occurred here, with the diagnosis that
|
||||
identified each one. Where a symptom is misleading, that is called out — several
|
||||
of these point somewhere other than their cause.
|
||||
|
||||
## Audio problems in the finished file
|
||||
|
||||
### The lektor track is silent in the player
|
||||
|
||||
The single most misleading failure in this project. It has **two distinct
|
||||
causes**, and volume measurements cannot tell them apart — both produce a track
|
||||
that measures perfectly loud.
|
||||
|
||||
Diagnose by checking the declared channel layout and the interleaving, in that
|
||||
order.
|
||||
|
||||
**Cause 1 — unknown channel layout**
|
||||
|
||||
```bash
|
||||
ffprobe -v error -select_streams a -show_entries stream=index,codec_name,channels,channel_layout -of csv=p=0 out.mkv
|
||||
```
|
||||
|
||||
```
|
||||
1,eac3,6,5.1(side) ← original, fine
|
||||
2,aac,6,unknown ← broken: 6 channels, no routing information
|
||||
```
|
||||
|
||||
ffmpeg's AAC encoder does not signal a 5.1 layout, in any container. Players
|
||||
cannot map the channels and output silence. The fix is the stereo lektor track
|
||||
now used by default (`aformat=channel_layouts=stereo` on both inputs). A correct
|
||||
track reports `2,stereo`.
|
||||
|
||||
**Cause 2 — broken interleaving**
|
||||
|
||||
Compare byte positions for the same timestamp:
|
||||
|
||||
```bash
|
||||
ffprobe -read_intervals 00:20:00%+1 -select_streams v -show_entries packet=pos -of csv=p=0 out.mkv | head -1
|
||||
ffprobe -read_intervals 00:20:00%+1 -select_streams a:1 -show_entries packet=pos -of csv=p=0 out.mkv | head -1
|
||||
```
|
||||
|
||||
They should be within a few MB. Observed on a broken file:
|
||||
|
||||
| time | drift from video |
|
||||
|---|---|
|
||||
| 02:00 | 477 MB |
|
||||
| 10:00 | **2315 MB** |
|
||||
| 50:00 | 740 MB |
|
||||
|
||||
Sparse subtitle tracks make the muxer give up interleaving after its default
|
||||
10-second window. ffmpeg reads the file fine, so every level measurement looks
|
||||
healthy — but players read MKV sequentially and never reach that audio.
|
||||
|
||||
The fix is `-max_interleave_delta 0` as an **output** option. An existing file
|
||||
can be repaired without regenerating anything:
|
||||
|
||||
```bash
|
||||
ffmpeg -i broken.mkv -map 0 -c copy -max_interleave_delta 0 fixed.mkv
|
||||
```
|
||||
|
||||
14 seconds and ~2.6 GB of transient RAM for a 60-minute file.
|
||||
|
||||
> Placement is the trap. Before `-i` the flag configures the demuxer, changes
|
||||
> nothing, and produces byte-identical output — which is easily mistaken for
|
||||
> "this fix doesn't work".
|
||||
|
||||
### The voiceover is late by a constant offset
|
||||
|
||||
The source audio starts at a non-zero PTS. Confirm:
|
||||
|
||||
```bash
|
||||
ffprobe -v error -select_streams a:0 -show_entries stream=start_time -of csv=p=0 film.mkv
|
||||
# 4.043000 ← not 0
|
||||
```
|
||||
|
||||
`amix` cannot emit output until both inputs have frames, so the voiceover is
|
||||
delayed by that amount. Handled automatically now
|
||||
(`asetpts=PTS-STARTPTS,adelay=<offset>`); the run prints:
|
||||
|
||||
```
|
||||
Audio PTS offset: 4.043s — normalising orig_mix for lektor track
|
||||
```
|
||||
|
||||
Note that this affects **only** the mixed track. Track 0 is stream-copied and
|
||||
was never wrong, so "the original audio is offset" and "the original audio
|
||||
inside the lektor mix is offset" are different problems with different fixes.
|
||||
|
||||
### Voiceover drifts progressively out of sync
|
||||
|
||||
Not a mixing fault — clips are overrunning their windows and pushing each other
|
||||
forward. The console shows it:
|
||||
|
||||
```
|
||||
→ placed at 00:12:03,400, duration 3.10s [+2.40s behind SRT]
|
||||
```
|
||||
|
||||
Raise `--speed` (try 1.3–1.4) and rebuild. Small drift is expected and
|
||||
self-corrects at the next gap in the subtitles; only sustained growth is a
|
||||
problem.
|
||||
|
||||
### The whole voiceover sounds too quiet or too loud
|
||||
|
||||
Adjust and rebuild — no regeneration needed:
|
||||
|
||||
```bash
|
||||
python3 lektor_ai_comfy.py film.mkv --rebuild --orig-volume 0.5 --tts-volume 2.0
|
||||
```
|
||||
|
||||
Track 0 is unaffected by these, always.
|
||||
|
||||
## Muxing failures
|
||||
|
||||
### `Failed to inject frame into filter network: Invalid argument`
|
||||
|
||||
```
|
||||
Error while filtering: Invalid argument
|
||||
Failed to inject frame into filter network: Invalid argument
|
||||
Error while processing the decoded data for stream #0:1
|
||||
```
|
||||
|
||||
Aborts partway through, at a **different timestamp each run**. The message
|
||||
blames audio; audio is not the cause. It appears when filtering and
|
||||
subtitle muxing happen in one ffmpeg command on a source with many subtitle
|
||||
tracks — the sparse streams stall the muxer while the filter graph keeps
|
||||
producing.
|
||||
|
||||
The varying failure point is the tell. A genuinely corrupt frame fails at the
|
||||
same place every time.
|
||||
|
||||
Fixed by the two-pass structure in `merge_voiceover()`. If you are editing that
|
||||
function, do not merge the passes back together.
|
||||
|
||||
### The mux fails on DTS or TrueHD audio
|
||||
|
||||
Only an issue if output is changed to MP4, which cannot carry those codecs. The
|
||||
default MKV output handles them. Keep it.
|
||||
|
||||
## TTS problems
|
||||
|
||||
### A clip recites the reference transcript
|
||||
|
||||
The model occasionally speaks the contents of `gudowski.txt` instead of the
|
||||
subtitle. Most likely at `--temperature 1.0`; `0.9` is the default for this
|
||||
reason.
|
||||
|
||||
```bash
|
||||
python3 lektor_ai_comfy.py film.mkv --regen 88
|
||||
python3 lektor_ai_comfy.py film.mkv --rebuild
|
||||
```
|
||||
|
||||
`--seed 0` randomises each run, so a regeneration gets a different sample.
|
||||
|
||||
### A clip is cut off mid-word
|
||||
|
||||
The token ceiling was reached. Raise it and regenerate just that clip:
|
||||
|
||||
```bash
|
||||
python3 lektor_ai_comfy.py film.mkv --regen 214 --max-tokens 8192
|
||||
```
|
||||
|
||||
### Every clip returns identical audio
|
||||
|
||||
`enable_audio_cache` is being set to `True` somewhere in
|
||||
`build_text_workflow()`. It must stay `False` or ComfyUI serves cached results
|
||||
and `--regen` does nothing.
|
||||
|
||||
### Stiff pauses at every comma
|
||||
|
||||
Baked into the model weights. There is no SSML support and no parameter for it.
|
||||
Rephrasing the subtitle text is the only lever.
|
||||
|
||||
### Audible breath at the start of clips
|
||||
|
||||
```bash
|
||||
python3 lektor_ai_comfy.py film.mkv --regen 5 --strip-leading-ms 120
|
||||
```
|
||||
|
||||
Values of 80–150 ms usually suffice. Too high clips the first consonant.
|
||||
|
||||
### Voice cloning quality is poor
|
||||
|
||||
Check that the reference `.txt` exists and matches the `.wav` exactly. The run
|
||||
prints:
|
||||
|
||||
```
|
||||
Reference text loaded (930 chars)
|
||||
```
|
||||
|
||||
or warns if it is missing. An inaccurate transcript degrades cloning noticeably.
|
||||
|
||||
## Subtitle problems
|
||||
|
||||
### Formatting tags are being spoken
|
||||
|
||||
`<i>`, `</i>` and `{\an8}` are stripped by `_strip_sub_tags()` before anything
|
||||
else. If new tags appear, extend that function — and keep the stripping first,
|
||||
because tags interfere with the ALL-CAPS detection that follows.
|
||||
|
||||
### ALL-CAPS text is being shouted
|
||||
|
||||
Handled by `_clean_text()`, but only after tags are removed. `<i>HELLO</i>`
|
||||
looked mixed-case while the tag was present, because `i` is a lowercase letter.
|
||||
If shouting persists, check for an unhandled tag type.
|
||||
|
||||
### `--embedded-subs` shows no usable tracks
|
||||
|
||||
The video only carries image-based subtitles (`hdmv_pgs_subtitle`,
|
||||
`dvd_subtitle`). These are bitmaps and need OCR. Supply an external `.srt`
|
||||
instead — the image tracks are still copied to the output.
|
||||
|
||||
### Wrong language selected
|
||||
|
||||
The picker lists language tags but they are only as good as the source's
|
||||
metadata. `output/embedded_sub.srt` holds exactly what was extracted; check it
|
||||
before committing to a full run.
|
||||
|
||||
## Backend problems
|
||||
|
||||
### `Connection refused` on port 8189
|
||||
|
||||
ComfyUI is not running. Start it and wait for the model to load:
|
||||
|
||||
```bash
|
||||
~/comfy-tts-app/start_comfyui.sh
|
||||
curl -s http://127.0.0.1:8189/system_stats | head -c 200
|
||||
```
|
||||
|
||||
### ComfyUI crashed and the GPU is stuck
|
||||
|
||||
```bash
|
||||
sudo fuser -k /dev/nvidia*
|
||||
sudo nvidia-smi --gpu-reset
|
||||
```
|
||||
|
||||
Then restart and resume from the last completed block:
|
||||
|
||||
```bash
|
||||
python3 lektor_ai_comfy.py film.mkv --resume-from 312
|
||||
```
|
||||
|
||||
### Out of VRAM
|
||||
|
||||
Nothing else should be using the GPU during a run. Restart ComfyUI to clear
|
||||
fragmentation. The 4B model needs roughly 10–12 GB.
|
||||
|
||||
### `torch.cuda.is_available()` is False
|
||||
|
||||
```bash
|
||||
source ~/miniforge3/etc/profile.d/conda.sh && conda activate comfy-tts
|
||||
python -c "import torch; print(torch.__version__, torch.cuda.is_available())"
|
||||
```
|
||||
|
||||
Expect `2.11.0+cu128 True`. If False, check `nvidia-smi` works and the driver is
|
||||
525 or newer. The CUDA **toolkit** version is irrelevant — the wheels bundle
|
||||
their own runtime, and this machine runs cu128 wheels against a system CUDA
|
||||
11.5 without issue.
|
||||
|
||||
## Script problems
|
||||
|
||||
### `Provide an SRT subtitle file, or use --embedded-subs`
|
||||
|
||||
A mode that needs subtitles was given only a video. Either pass the SRT first,
|
||||
or add `--embedded-subs`.
|
||||
|
||||
### `No such file or directory` on the video
|
||||
|
||||
Almost always an unquoted path with spaces or parentheses, or a wrong extension
|
||||
(`.mp4` where the file is `.mkv`). Quote it and check with `ls`.
|
||||
|
||||
### `Missing clip files — regenerate them first`
|
||||
|
||||
`--rebuild` needs a complete set. The listed clips must be regenerated with
|
||||
`--regen`, or `--resume-from` used to fill the gap.
|
||||
|
||||
### `No tts_input.srt found`
|
||||
|
||||
`--rebuild` and `--regen` require a previous full run in the same
|
||||
`--output-dir`.
|
||||
|
||||
## General diagnostic method
|
||||
|
||||
The lesson from the silent-track investigation, which took several wrong turns:
|
||||
|
||||
1. **Measure, don't assume.** Level, layout, and byte position each reveal a
|
||||
different class of fault.
|
||||
2. **Isolate by subtraction.** Removing subtitles from the failing command is
|
||||
what proved they were the cause, not the audio the error named.
|
||||
3. **A varying failure point means state, not data.** Corrupt input fails in the
|
||||
same place every time; buffering and interleaving problems move around.
|
||||
4. **"ffmpeg reads it fine" is not "a player can play it."** Verify position, not
|
||||
just content.
|
||||
5. **Check flag placement.** Several ffmpeg options mean different things before
|
||||
`-i` versus before the output path, and the wrong position often fails
|
||||
silently rather than erroring.
|
||||
Reference in New Issue
Block a user