lektor-ai/docs/troubleshooting.md

Troubleshooting

Failure modes that have actually occurred here, with the diagnosis that identified each one. Where a symptom is misleading, that is called out — several of these point somewhere other than their cause.

Audio problems in the finished file

The lektor track is silent in the player

The single most misleading failure in this project. It has two distinct causes, and volume measurements cannot tell them apart — both produce a track that measures perfectly loud.

Diagnose by checking the declared channel layout and the interleaving, in that order.

Cause 1 — unknown channel layout

ffprobe -v error -select_streams a -show_entries stream=index,codec_name,channels,channel_layout -of csv=p=0 out.mkv

1,eac3,6,5.1(side)     ← original, fine
2,aac,6,unknown        ← broken: 6 channels, no routing information

ffmpeg's AAC encoder does not signal a 5.1 layout, in any container. Players cannot map the channels and output silence. The fix is the stereo lektor track now used by default (aformat=channel_layouts=stereo on both inputs). A correct track reports 2,stereo.

Cause 2 — broken interleaving

Compare byte positions for the same timestamp:

ffprobe -read_intervals 00:20:00%+1 -select_streams v   -show_entries packet=pos -of csv=p=0 out.mkv | head -1
ffprobe -read_intervals 00:20:00%+1 -select_streams a:1 -show_entries packet=pos -of csv=p=0 out.mkv | head -1

They should be within a few MB. Observed on a broken file:

time	drift from video
02:00	477 MB
10:00	2315 MB
50:00	740 MB

Sparse subtitle tracks make the muxer give up interleaving after its default 10-second window. ffmpeg reads the file fine, so every level measurement looks healthy — but players read MKV sequentially and never reach that audio.

The fix is -max_interleave_delta 0 as an output option. An existing file can be repaired without regenerating anything:

ffmpeg -i broken.mkv -map 0 -c copy -max_interleave_delta 0 fixed.mkv

14 seconds and ~2.6 GB of transient RAM for a 60-minute file.

Placement is the trap. Before -i the flag configures the demuxer, changes nothing, and produces byte-identical output — which is easily mistaken for "this fix doesn't work".

The voiceover is late by a constant offset

The source audio starts at a non-zero PTS. Confirm:

ffprobe -v error -select_streams a:0 -show_entries stream=start_time -of csv=p=0 film.mkv
# 4.043000  ← not 0

amix cannot emit output until both inputs have frames, so the voiceover is delayed by that amount. Handled automatically now (asetpts=PTS-STARTPTS,adelay=<offset>); the run prints:

Audio PTS offset: 4.043s — normalising orig_mix for lektor track

Note that this affects only the mixed track. Track 0 is stream-copied and was never wrong, so "the original audio is offset" and "the original audio inside the lektor mix is offset" are different problems with different fixes.

Voiceover drifts progressively out of sync

Not a mixing fault — clips are overrunning their windows and pushing each other forward. The console shows it:

→ placed at 00:12:03,400, duration 3.10s [+2.40s behind SRT]

Raise --speed (try 1.3–1.4) and rebuild. Small drift is expected and self-corrects at the next gap in the subtitles; only sustained growth is a problem.

The whole voiceover sounds too quiet or too loud

Adjust and rebuild — no regeneration needed:

python3 lektor_ai_comfy.py film.mkv --rebuild --orig-volume 0.5 --tts-volume 2.0

Track 0 is unaffected by these, always.

Muxing failures

Failed to inject frame into filter network: Invalid argument

Error while filtering: Invalid argument
Failed to inject frame into filter network: Invalid argument
Error while processing the decoded data for stream #0:1

Aborts partway through, at a different timestamp each run. The message blames audio; audio is not the cause. It appears when filtering and subtitle muxing happen in one ffmpeg command on a source with many subtitle tracks — the sparse streams stall the muxer while the filter graph keeps producing.

The varying failure point is the tell. A genuinely corrupt frame fails at the same place every time.

Fixed by the two-pass structure in merge_voiceover(). If you are editing that function, do not merge the passes back together.

The mux fails on DTS or TrueHD audio

Only an issue if output is changed to MP4, which cannot carry those codecs. The default MKV output handles them. Keep it.

TTS problems

A clip recites the reference transcript

The model occasionally speaks the contents of gudowski.txt instead of the subtitle. Most likely at --temperature 1.0; 0.9 is the default for this reason.

python3 lektor_ai_comfy.py film.mkv --regen 88
python3 lektor_ai_comfy.py film.mkv --rebuild

--seed 0 randomises each run, so a regeneration gets a different sample.

A clip is cut off mid-word

The token ceiling was reached. Raise it and regenerate just that clip:

python3 lektor_ai_comfy.py film.mkv --regen 214 --max-tokens 8192

Every clip returns identical audio

enable_audio_cache is being set to True somewhere in build_text_workflow(). It must stay False or ComfyUI serves cached results and --regen does nothing.

Stiff pauses at every comma

Baked into the model weights. There is no SSML support and no parameter for it. Rephrasing the subtitle text is the only lever.

Audible breath at the start of clips

python3 lektor_ai_comfy.py film.mkv --regen 5 --strip-leading-ms 120

Values of 80–150 ms usually suffice. Too high clips the first consonant.

Voice cloning quality is poor

Check that the reference .txt exists and matches the .wav exactly. The run prints:

Reference text loaded (930 chars)

or warns if it is missing. An inaccurate transcript degrades cloning noticeably.

Subtitle problems

Formatting tags are being spoken

<i>, </i> and {\an8} are stripped by _strip_sub_tags() before anything else. If new tags appear, extend that function — and keep the stripping first, because tags interfere with the ALL-CAPS detection that follows.

ALL-CAPS text is being shouted

Handled by _clean_text(), but only after tags are removed. <i>HELLO</i> looked mixed-case while the tag was present, because i is a lowercase letter. If shouting persists, check for an unhandled tag type.

Such blocks are converted to Title Case (Nasa – Margo Madison), not to sentence case. Lowercasing everything but the first letter also destroys the proper nouns in a caption, and the number normalizer then tries to repair the lowercase name — it turned margo into marga. Paired clips in output/clips/all-caps/ show the TTS reads both casings identically, so the names are kept.

A clip of an ALL-CAPS caption is cut off mid-word

Check whether the block still contains digits. Higgs truncates on raw digits far more often than on words: the same caption generated eight times, four with 21 MAJA 1983 and four with the date spelled out, was cut in 3 of 4 digit versions and 0 of 4 word versions — and the digit clips ran barely half as long as the speech required. Casing made no difference in that test.

That is the number normalizer's job, so if you see it, the block reached the TTS without being normalized: check numbers = yes and the Normalizer ready line in step 1.

An abbreviation, unit or name is read out wrong

Put it in dictionary.txt: km/h = kilometrów na godzinę. The replacement happens before the text reaches the model, and the result is visible in the job folder's tts_input.srt. See usage.md.

If a rule appears to do nothing, check the run's Dictionary: line — a rule that failed to parse was reported there and skipped, and a rule cannot match across a line break because each dialogue line is treated separately.

--regen and --rebuild read tts_input.srt, which already has the replacements baked in, so adding a rule afterwards needs either a fresh run from the SRT or a manual edit of tts_input.srt before --regen.

Numbers are read as garbled non-words

That is what the number normalizer is for; check what the run printed in step 1.

• Normalizer ready (model on cuda:0) — it ran. Look at tts_input.srt: if the digits are still there, the block may hold only standalone single digits, which are left alone deliberately.
• Normalizer ready (fallback) — the model is missing or failed to load, and num2words is standing in: bare integers only, nominative case only, times and decimals untouched. Install the model, see usage.md.
• normalizer interpreter not found — run normalizer/setup.sh.
• Nothing at all — the stage is off (numbers = no, or --no-normalize-numbers).

For the underlying cause of a bad load, run the component directly; it prints the real error instead of a summary:

normalizer/venv/bin/python normalizer/normalizer.py --status
normalizer/venv/bin/python normalizer/normalizer.py "Wygrał 1247 złotych."

A number is expanded in the wrong grammatical case

Nominative everywhere (dwadzieścia pięć psach instead of dwudziestu pięciu psach) means the num2words fallback is running, not the model — see above. The neural model is the only path that declines correctly.

The normalizer is unexpectedly slow

Check which device the run reported. Normalizer ready (model on cpu) means it never got the GPU. If it says cuda:0 and is still slow, it may have only got part of it: with Higgs resident there is not enough VRAM, and accelerate spills the rest to RAM without raising anything — about 5× slower.

The script prevents this by asking ComfyUI to unload before normalizing. If you do not see Freed N GB in step 1, either free_comfy_vram = no, or ComfyUI had nothing loaded (normal on a freshly started backend), or the call failed and said so.

The normalizer times out on every block

timeout under [normalize] is per block, and a block takes well under a second on a GPU. Timing out means the model landed on the CPU or the worker is wedged. Check torch.cuda.is_available() in that venv:

normalizer/venv/bin/python -c "import torch; print(torch.cuda.is_available())"

The stage disables itself after the first timeout rather than repeating the wait on every remaining block, so a run costs one timeout at worst.

--embedded-subs shows no usable tracks

The video only carries image-based subtitles (hdmv_pgs_subtitle, dvd_subtitle). These are bitmaps and need OCR. Supply an external .srt instead — the image tracks are still copied to the output.

Wrong language selected

The picker lists language tags but they are only as good as the source's metadata. embedded_sub.srt in the job folder holds exactly what was extracted; check it before committing to a full run.

Backend problems

Connection refused on port 8189

ComfyUI is not running. Start it and wait for the model to load:

~/comfy-tts-app/start_comfyui.sh
curl -s http://127.0.0.1:8189/system_stats | head -c 200

ComfyUI dies partway through a long queue

The run recovers by itself now, and says so:

  [2035/2073] 02:22:59  'Joshua.'
  ComfyUI went away ([Errno 104] Connection reset by peer) [1/3 for this clip].
  Stopping ComfyUI (pid 530039, suspended)…
  Starting ComfyUI in a new gnome-terminal window…
    ComfyUI died on this clip (1/3) — retrying with a different seed.

The stale process is stopped before the restart, because a wedged or suspended backend still holds the port and the GPU and the new one then cannot start. A process in state T is killed outright — it never runs its signal handlers, so SIGTERM is accepted and ignored.

Each retry uses a different seed, which matters: these aborts are seed-dependent, not text-dependent. The block that killed ComfyUI three times in a 2073-clip run generated perfectly on its own with another seed.

After three deaths on the same clip the clip is skipped and the queue continues, with a report at the end:

  2 clip(s) crashed ComfyUI and were skipped — the voiceover is silent there.
  Edit their text in tts_input.srt, then:
    --regen 2035 4180

Skipping only happens when the backend comes back healthy. If it does not, the run waits for it instead — otherwise a backend that is simply not running would cause every remaining clip to be "skipped" and produce a silent film.

The clip that crashes ComfyUI

The abort looks like this in comfyui.log:

/pytorch/aten/src/ATen/native/cuda/Indexing.cu:1515: indexSelectSmallIndex:
    Assertion `srcIndex < srcSelectDimSize` failed.
Aborted (core dumped)

An out-of-range index in an embedding lookup inside the Higgs audio tokenizer. It is a CUDA device-side assert, so it cannot be caught in Python — the process dies outright, which is why it looks like ComfyUI "will not start" when a client is resubmitting the same clip on every restart.

--regen N with a different seed is the fix; rewording the block helps if it persists.

ComfyUI crashed and the GPU is stuck

sudo fuser -k /dev/nvidia*
sudo nvidia-smi --gpu-reset

Then restart and resume from the last completed block:

python3 lektor_ai_comfy.py film.mkv --resume-from 312

Out of VRAM

Nothing else should be using the GPU during a run. Restart ComfyUI to clear fragmentation. The 4B model needs roughly 10–12 GB.

The number normalizer adds ~1.5 GB, but only during step 1 — its process exits before TTS generation starts, so the two never hold VRAM at the same time. If it cannot fit anyway, it logs the CUDA error and finishes the run on the num2words fallback rather than failing.

torch.cuda.is_available() is False

source ~/miniforge3/etc/profile.d/conda.sh && conda activate comfy-tts
python -c "import torch; print(torch.__version__, torch.cuda.is_available())"

Expect 2.11.0+cu128 True. If False, check nvidia-smi works and the driver is 525 or newer. The CUDA toolkit version is irrelevant — the wheels bundle their own runtime, and this machine runs cu128 wheels against a system CUDA 11.5 without issue.

Script problems

Provide an SRT subtitle file, or use --embedded-subs

A mode that needs subtitles was given only a video. Either pass the SRT first, or add --embedded-subs.

No such file or directory on the video

Almost always an unquoted path with spaces or parentheses, or a wrong extension (.mp4 where the file is .mkv). Quote it and check with ls.

Missing clip files — regenerate them first

--rebuild needs a complete set. The listed clips must be regenerated with --regen, or --resume-from used to fill the gap.

No tts_input.srt found

--rebuild and --regen require a previous full run in the same --output-dir.

General diagnostic method

The lesson from the silent-track investigation, which took several wrong turns:

1. Measure, don't assume. Level, layout, and byte position each reveal a different class of fault.
2. Isolate by subtraction. Removing subtitles from the failing command is what proved they were the cause, not the audio the error named.
3. A varying failure point means state, not data. Corrupt input fails in the same place every time; buffering and interleaving problems move around.
4. "ffmpeg reads it fine" is not "a player can play it." Verify position, not just content.
5. Check flag placement. Several ffmpeg options mean different things before -i versus before the output path, and the wrong position often fails silently rather than erroring.