lektor-ai/docs/troubleshooting.md

# Troubleshooting

Failure modes that have actually occurred here, with the diagnosis that
identified each one. Where a symptom is misleading, that is called out — several
of these point somewhere other than their cause.

## Audio problems in the finished file

### The lektor track is silent in the player

The single most misleading failure in this project. It has **two distinct
causes**, and volume measurements cannot tell them apart — both produce a track
that measures perfectly loud.

Diagnose by checking the declared channel layout and the interleaving, in that
order.

**Cause 1 — unknown channel layout**

```bash
ffprobe -v error -select_streams a -show_entries stream=index,codec_name,channels,channel_layout -of csv=p=0 out.mkv
```

```
1,eac3,6,5.1(side)     ← original, fine
2,aac,6,unknown        ← broken: 6 channels, no routing information
```

ffmpeg's AAC encoder does not signal a 5.1 layout, in any container. Players
cannot map the channels and output silence. The fix is the stereo lektor track
now used by default (`aformat=channel_layouts=stereo` on both inputs). A correct
track reports `2,stereo`.

**Cause 2 — broken interleaving**

Compare byte positions for the same timestamp:

```bash
ffprobe -read_intervals 00:20:00%+1 -select_streams v   -show_entries packet=pos -of csv=p=0 out.mkv | head -1
ffprobe -read_intervals 00:20:00%+1 -select_streams a:1 -show_entries packet=pos -of csv=p=0 out.mkv | head -1
```

They should be within a few MB. Observed on a broken file:

| time | drift from video |
|---|---|
| 02:00 | 477 MB |
| 10:00 | **2315 MB** |
| 50:00 | 740 MB |

Sparse subtitle tracks make the muxer give up interleaving after its default
10-second window. ffmpeg reads the file fine, so every level measurement looks
healthy — but players read MKV sequentially and never reach that audio.

The fix is `-max_interleave_delta 0` as an **output** option. An existing file
can be repaired without regenerating anything:

```bash
ffmpeg -i broken.mkv -map 0 -c copy -max_interleave_delta 0 fixed.mkv
```

14 seconds and ~2.6 GB of transient RAM for a 60-minute file.

> Placement is the trap. Before `-i` the flag configures the demuxer, changes
> nothing, and produces byte-identical output — which is easily mistaken for
> "this fix doesn't work".

### The voiceover is late by a constant offset

The source audio starts at a non-zero PTS. Confirm:

```bash
ffprobe -v error -select_streams a:0 -show_entries stream=start_time -of csv=p=0 film.mkv
# 4.043000  ← not 0
```

`amix` cannot emit output until both inputs have frames, so the voiceover is
delayed by that amount. Handled automatically now
(`asetpts=PTS-STARTPTS,adelay=<offset>`); the run prints:

```
Audio PTS offset: 4.043s — normalising orig_mix for lektor track
```

Note that this affects **only** the mixed track. Track 0 is stream-copied and
was never wrong, so "the original audio is offset" and "the original audio
inside the lektor mix is offset" are different problems with different fixes.

### Voiceover drifts progressively out of sync

Not a mixing fault — clips are overrunning their windows and pushing each other
forward. The console shows it:

```
→ placed at 00:12:03,400, duration 3.10s [+2.40s behind SRT]
```

Raise `--speed` (try 1.3–1.4) and rebuild. Small drift is expected and
self-corrects at the next gap in the subtitles; only sustained growth is a
problem.

### The whole voiceover sounds too quiet or too loud

Adjust and rebuild — no regeneration needed:

```bash
python3 lektor_ai_comfy.py film.mkv --rebuild --orig-volume 0.5 --tts-volume 2.0
```

Track 0 is unaffected by these, always.

## Muxing failures

### `Failed to inject frame into filter network: Invalid argument`

```
Error while filtering: Invalid argument
Failed to inject frame into filter network: Invalid argument
Error while processing the decoded data for stream #0:1
```

Aborts partway through, at a **different timestamp each run**. The message
blames audio; audio is not the cause. It appears when filtering and
subtitle muxing happen in one ffmpeg command on a source with many subtitle
tracks — the sparse streams stall the muxer while the filter graph keeps
producing.

The varying failure point is the tell. A genuinely corrupt frame fails at the
same place every time.

Fixed by the two-pass structure in `merge_voiceover()`. If you are editing that
function, do not merge the passes back together.

### The mux fails on DTS or TrueHD audio

Only an issue if output is changed to MP4, which cannot carry those codecs. The
default MKV output handles them. Keep it.

## TTS problems

### A clip recites the reference transcript

The model occasionally speaks the contents of `gudowski.txt` instead of the
subtitle. Most likely at `--temperature 1.0`; `0.9` is the default for this
reason.

```bash
python3 lektor_ai_comfy.py film.mkv --regen 88
python3 lektor_ai_comfy.py film.mkv --rebuild
```

`--seed 0` randomises each run, so a regeneration gets a different sample.

### A clip is cut off mid-word

The token ceiling was reached. Raise it and regenerate just that clip:

```bash
python3 lektor_ai_comfy.py film.mkv --regen 214 --max-tokens 8192
```

### Every clip returns identical audio

`enable_audio_cache` is being set to `True` somewhere in
`build_text_workflow()`. It must stay `False` or ComfyUI serves cached results
and `--regen` does nothing.

### Stiff pauses at every comma

Baked into the model weights. There is no SSML support and no parameter for it.
Rephrasing the subtitle text is the only lever.

### Audible breath at the start of clips

```bash
python3 lektor_ai_comfy.py film.mkv --regen 5 --strip-leading-ms 120
```

Values of 80–150 ms usually suffice. Too high clips the first consonant.

### Voice cloning quality is poor

Check that the reference `.txt` exists and matches the `.wav` exactly. The run
prints:

```
Reference text loaded (930 chars)
```

or warns if it is missing. An inaccurate transcript degrades cloning noticeably.

## Subtitle problems

### Formatting tags are being spoken

`<i>`, `</i>` and `{\an8}` are stripped by `_strip_sub_tags()` before anything
else. If new tags appear, extend that function — and keep the stripping first,
because tags interfere with the ALL-CAPS detection that follows.

### ALL-CAPS text is being shouted

Handled by `_clean_text()`, but only after tags are removed. `<i>HELLO</i>`
looked mixed-case while the tag was present, because `i` is a lowercase letter.
If shouting persists, check for an unhandled tag type.

Such blocks are converted to Title Case (`Nasa – Margo Madison`), not to
sentence case. Lowercasing everything but the first letter also destroys the
proper nouns in a caption, and the number normalizer then tries to repair the
lowercase name — it turned `margo` into `marga`. Paired clips in
`output/clips/all-caps/` show the TTS reads both casings identically, so the
names are kept.

### A clip of an ALL-CAPS caption is cut off mid-word

Check whether the block still contains digits. Higgs truncates on raw digits far
more often than on words: the same caption generated eight times, four with
`21 MAJA 1983` and four with the date spelled out, was cut in 3 of 4 digit
versions and 0 of 4 word versions — and the digit clips ran barely half as long
as the speech required. Casing made no difference in that test.

That is the number normalizer's job, so if you see it, the block reached the TTS
without being normalized: check `numbers = yes` and the `Normalizer ready` line
in step 1.

### An abbreviation, unit or name is read out wrong

Put it in `dictionary.txt`: `km/h = kilometrów na godzinę`. The replacement
happens before the text reaches the model, and the result is visible in the job
folder's `tts_input.srt`. See [usage.md](usage.md#replacement-dictionary).

If a rule appears to do nothing, check the run's `Dictionary:` line — a rule
that failed to parse was reported there and skipped, and a rule cannot match
across a line break because each dialogue line is treated separately.

`--regen` and `--rebuild` read `tts_input.srt`, which already has the
replacements baked in, so adding a rule afterwards needs either a fresh run
from the SRT or a manual edit of `tts_input.srt` before `--regen`.

### Numbers are read as garbled non-words

That is what the number normalizer is for; check what the run printed in step 1.

- `Normalizer ready (model on cuda:0)` — it ran. Look at `tts_input.srt`: if the
  digits are still there, the block may hold only standalone single digits,
  which are left alone deliberately.
- `Normalizer ready (fallback)` — the model is missing or failed to load, and
  `num2words` is standing in: bare integers only, nominative case only, times
  and decimals untouched. Install the model, see
  [usage.md](usage.md#number-normalization).
- `normalizer interpreter not found` — run `normalizer/setup.sh`.
- Nothing at all — the stage is off (`numbers = no`, or
  `--no-normalize-numbers`).

For the underlying cause of a bad load, run the component directly; it prints
the real error instead of a summary:

```bash
normalizer/venv/bin/python normalizer/normalizer.py --status
normalizer/venv/bin/python normalizer/normalizer.py "Wygrał 1247 złotych."
```

### A number is expanded in the wrong grammatical case

Nominative everywhere (`dwadzieścia pięć psach` instead of `dwudziestu pięciu
psach`) means the `num2words` fallback is running, not the model — see above.
The neural model is the only path that declines correctly.

### The normalizer is unexpectedly slow

Check which device the run reported. `Normalizer ready (model on cpu)` means it
never got the GPU. If it says `cuda:0` and is still slow, it may have only got
part of it: with Higgs resident there is not enough VRAM, and `accelerate`
spills the rest to RAM without raising anything — about 5× slower.

The script prevents this by asking ComfyUI to unload before normalizing. If you
do not see `Freed N GB` in step 1, either `free_comfy_vram = no`, or ComfyUI had
nothing loaded (normal on a freshly started backend), or the call failed and
said so.

### The normalizer times out on every block

`timeout` under `[normalize]` is per block, and a block takes well under a second
on a GPU. Timing out means the model landed on the CPU or the worker is wedged.
Check `torch.cuda.is_available()` in that venv:

```bash
normalizer/venv/bin/python -c "import torch; print(torch.cuda.is_available())"
```

The stage disables itself after the first timeout rather than repeating the wait
on every remaining block, so a run costs one `timeout` at worst.

### `--embedded-subs` shows no usable tracks

The video only carries image-based subtitles (`hdmv_pgs_subtitle`,
`dvd_subtitle`). These are bitmaps and need OCR. Supply an external `.srt`
instead — the image tracks are still copied to the output.

### Wrong language selected

The picker lists language tags but they are only as good as the source's
metadata. `embedded_sub.srt` in the job folder holds exactly what was extracted; check it
before committing to a full run.

## Backend problems

### `Connection refused` on port 8189

ComfyUI is not running. Start it and wait for the model to load:

```bash
~/comfy-tts-app/start_comfyui.sh
curl -s http://127.0.0.1:8189/system_stats | head -c 200
```

### ComfyUI dies partway through a long queue

The run recovers by itself now, and says so:

```
  [2035/2073] 02:22:59  'Joshua.'
  ComfyUI went away ([Errno 104] Connection reset by peer) [1/3 for this clip].
  Stopping ComfyUI (pid 530039, suspended)…
  Starting ComfyUI in a new gnome-terminal window…
    ComfyUI died on this clip (1/3) — retrying with a different seed.
```

The stale process is stopped before the restart, because a wedged or suspended
backend still holds the port and the GPU and the new one then cannot start. A
process in state `T` is killed outright — it never runs its signal handlers, so
SIGTERM is accepted and ignored.

**Each retry uses a different seed**, which matters: these aborts are
seed-dependent, not text-dependent. The block that killed ComfyUI three times in
a 2073-clip run generated perfectly on its own with another seed.

After three deaths on the same clip the clip is skipped and the queue continues,
with a report at the end:

```
  2 clip(s) crashed ComfyUI and were skipped — the voiceover is silent there.
  Edit their text in tts_input.srt, then:
    --regen 2035 4180
```

Skipping only happens when the backend comes back healthy. If it does not, the
run waits for it instead — otherwise a backend that is simply not running would
cause every remaining clip to be "skipped" and produce a silent film.

### The clip that crashes ComfyUI

The abort looks like this in `comfyui.log`:

```
/pytorch/aten/src/ATen/native/cuda/Indexing.cu:1515: indexSelectSmallIndex:
    Assertion `srcIndex < srcSelectDimSize` failed.
Aborted (core dumped)
```

An out-of-range index in an embedding lookup inside the Higgs audio tokenizer.
It is a CUDA device-side assert, so it cannot be caught in Python — the process
dies outright, which is why it looks like ComfyUI "will not start" when a client
is resubmitting the same clip on every restart.

`--regen N` with a different seed is the fix; rewording the block helps if it
persists.

### ComfyUI crashed and the GPU is stuck

```bash
sudo fuser -k /dev/nvidia*
sudo nvidia-smi --gpu-reset
```

Then restart and resume from the last completed block:

```bash
python3 lektor_ai_comfy.py film.mkv --resume-from 312
```

### Out of VRAM

Nothing else should be using the GPU during a run. Restart ComfyUI to clear
fragmentation. The 4B model needs roughly 10–12 GB.

The number normalizer adds ~1.5 GB, but only during step 1 — its process exits
before TTS generation starts, so the two never hold VRAM at the same time. If it
cannot fit anyway, it logs the CUDA error and finishes the run on the `num2words`
fallback rather than failing.

### `torch.cuda.is_available()` is False

```bash
source ~/miniforge3/etc/profile.d/conda.sh && conda activate comfy-tts
python -c "import torch; print(torch.__version__, torch.cuda.is_available())"
```

Expect `2.11.0+cu128 True`. If False, check `nvidia-smi` works and the driver is
525 or newer. The CUDA **toolkit** version is irrelevant — the wheels bundle
their own runtime, and this machine runs cu128 wheels against a system CUDA
11.5 without issue.

## Script problems

### `Provide an SRT subtitle file, or use --embedded-subs`

A mode that needs subtitles was given only a video. Either pass the SRT first,
or add `--embedded-subs`.

### `No such file or directory` on the video

Almost always an unquoted path with spaces or parentheses, or a wrong extension
(`.mp4` where the file is `.mkv`). Quote it and check with `ls`.

### `Missing clip files — regenerate them first`

`--rebuild` needs a complete set. The listed clips must be regenerated with
`--regen`, or `--resume-from` used to fill the gap.

### `No tts_input.srt found`

`--rebuild` and `--regen` require a previous full run in the same
`--output-dir`.

## General diagnostic method

The lesson from the silent-track investigation, which took several wrong turns:

1. **Measure, don't assume.** Level, layout, and byte position each reveal a
   different class of fault.
2. **Isolate by subtraction.** Removing subtitles from the failing command is
   what proved they were the cause, not the audio the error named.
3. **A varying failure point means state, not data.** Corrupt input fails in the
   same place every time; buffering and interleaving problems move around.
4. **"ffmpeg reads it fine" is not "a player can play it."** Verify position, not
   just content.
5. **Check flag placement.** Several ffmpeg options mean different things before
   `-i` versus before the output path, and the wrong position often fails
   silently rather than erroring.