Add subtitle dictionaries, Polish number normalizer, ComfyUI lifecycle control

Text processing before TTS:

- dictionary.txt rewrites words and phrases (km/h, np., units). Rules compile
  into one alternation applied in a single pass, so replacements never chain
  and longest-literal wins. Case-sensitive, whole-word.
- A per-queue second dictionary, written by the GUI into the first video's job
  folder and merged with the shared one (--dictionary-extra).
- ALL-CAPS blocks become Title Case rather than sentence case: lowercasing
  destroyed proper nouns inside captions ("NASA - MARGO MADISON") and invited
  the normalizer to "correct" the name. Paired clips in output/clips/all-caps
  showed the TTS reads both casings the same.

normalizer/ — digits to spoken Polish, in its own venv because it needs
transformers>=5, which must not touch ComfyUI's pinned versions:

- Two backends. Plain integers go to num2words for the value and a general
  Polish LLM for the case; times, dates, decimals and identifiers go to a 0.6B
  fine-tune. Measured on 40 real subtitle lines, each is clearly better than
  the other on its half. 74% of numbered blocks take the first leg.
- Guards on every answer: runaway length, repetition loops, invalid bytes,
  leftover digits, a lost minus sign, a changed number, a lost word. A failure
  falls back to num2words, which is nominative but never wrong about the value.
- ComfyUI is asked to unload before normalizing; a normalizer that does not fit
  in VRAM is spilled to RAM silently at 5x the cost.

Backend lifecycle:

- ComfyUI is found by port through /proc, so a backend started by hand is
  managed too. SIGTERM escalates to SIGKILL, and a suspended process is killed
  outright since it never runs its handlers.
- A clip that kills the backend is retried three times, each with a different
  seed (the aborts are seed-dependent, not text-dependent), then skipped and
  reported so an overnight queue keeps moving.
- start_comfyui.sh execs python, so the script's PID is ComfyUI's.
- GUI gains ComfyUI and Queue dictionary buttons; quitting stops the backend.

--resume-from now reuses the job folder's tts_input.srt instead of reparsing
the subtitles, so block N still pairs with clip_NNNN.flac.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
This commit is contained in:
2026-08-12 21:55:31 +02:00
co-authored by Claude Opus 5
parent 7e2cee02e5
commit e201aadc3e
21 changed files with 3759 additions and 52 deletions
+144
View File
@@ -203,6 +203,93 @@ Handled by `_clean_text()`, but only after tags are removed. `<i>HELLO</i>`
looked mixed-case while the tag was present, because `i` is a lowercase letter.
If shouting persists, check for an unhandled tag type.
Such blocks are converted to Title Case (`Nasa – Margo Madison`), not to
sentence case. Lowercasing everything but the first letter also destroys the
proper nouns in a caption, and the number normalizer then tries to repair the
lowercase name — it turned `margo` into `marga`. Paired clips in
`output/clips/all-caps/` show the TTS reads both casings identically, so the
names are kept.
### A clip of an ALL-CAPS caption is cut off mid-word
Check whether the block still contains digits. Higgs truncates on raw digits far
more often than on words: the same caption generated eight times, four with
`21 MAJA 1983` and four with the date spelled out, was cut in 3 of 4 digit
versions and 0 of 4 word versions — and the digit clips ran barely half as long
as the speech required. Casing made no difference in that test.
That is the number normalizer's job, so if you see it, the block reached the TTS
without being normalized: check `numbers = yes` and the `Normalizer ready` line
in step 1.
### An abbreviation, unit or name is read out wrong
Put it in `dictionary.txt`: `km/h = kilometrów na godzinę`. The replacement
happens before the text reaches the model, and the result is visible in the job
folder's `tts_input.srt`. See [usage.md](usage.md#replacement-dictionary).
If a rule appears to do nothing, check the run's `Dictionary:` line — a rule
that failed to parse was reported there and skipped, and a rule cannot match
across a line break because each dialogue line is treated separately.
`--regen` and `--rebuild` read `tts_input.srt`, which already has the
replacements baked in, so adding a rule afterwards needs either a fresh run
from the SRT or a manual edit of `tts_input.srt` before `--regen`.
### Numbers are read as garbled non-words
That is what the number normalizer is for; check what the run printed in step 1.
- `Normalizer ready (model on cuda:0)` — it ran. Look at `tts_input.srt`: if the
digits are still there, the block may hold only standalone single digits,
which are left alone deliberately.
- `Normalizer ready (fallback)` — the model is missing or failed to load, and
`num2words` is standing in: bare integers only, nominative case only, times
and decimals untouched. Install the model, see
[usage.md](usage.md#number-normalization).
- `normalizer interpreter not found` — run `normalizer/setup.sh`.
- Nothing at all — the stage is off (`numbers = no`, or
`--no-normalize-numbers`).
For the underlying cause of a bad load, run the component directly; it prints
the real error instead of a summary:
```bash
normalizer/venv/bin/python normalizer/normalizer.py --status
normalizer/venv/bin/python normalizer/normalizer.py "Wygrał 1247 złotych."
```
### A number is expanded in the wrong grammatical case
Nominative everywhere (`dwadzieścia pięć psach` instead of `dwudziestu pięciu
psach`) means the `num2words` fallback is running, not the model — see above.
The neural model is the only path that declines correctly.
### The normalizer is unexpectedly slow
Check which device the run reported. `Normalizer ready (model on cpu)` means it
never got the GPU. If it says `cuda:0` and is still slow, it may have only got
part of it: with Higgs resident there is not enough VRAM, and `accelerate`
spills the rest to RAM without raising anything — about 5× slower.
The script prevents this by asking ComfyUI to unload before normalizing. If you
do not see `Freed N GB` in step 1, either `free_comfy_vram = no`, or ComfyUI had
nothing loaded (normal on a freshly started backend), or the call failed and
said so.
### The normalizer times out on every block
`timeout` under `[normalize]` is per block, and a block takes well under a second
on a GPU. Timing out means the model landed on the CPU or the worker is wedged.
Check `torch.cuda.is_available()` in that venv:
```bash
normalizer/venv/bin/python -c "import torch; print(torch.cuda.is_available())"
```
The stage disables itself after the first timeout rather than repeating the wait
on every remaining block, so a run costs one `timeout` at worst.
### `--embedded-subs` shows no usable tracks
The video only carries image-based subtitles (`hdmv_pgs_subtitle`,
@@ -226,6 +313,58 @@ ComfyUI is not running. Start it and wait for the model to load:
curl -s http://127.0.0.1:8189/system_stats | head -c 200
```
### ComfyUI dies partway through a long queue
The run recovers by itself now, and says so:
```
[2035/2073] 02:22:59 'Joshua.'
ComfyUI went away ([Errno 104] Connection reset by peer) [1/3 for this clip].
Stopping ComfyUI (pid 530039, suspended)…
Starting ComfyUI in a new gnome-terminal window…
ComfyUI died on this clip (1/3) — retrying with a different seed.
```
The stale process is stopped before the restart, because a wedged or suspended
backend still holds the port and the GPU and the new one then cannot start. A
process in state `T` is killed outright — it never runs its signal handlers, so
SIGTERM is accepted and ignored.
**Each retry uses a different seed**, which matters: these aborts are
seed-dependent, not text-dependent. The block that killed ComfyUI three times in
a 2073-clip run generated perfectly on its own with another seed.
After three deaths on the same clip the clip is skipped and the queue continues,
with a report at the end:
```
2 clip(s) crashed ComfyUI and were skipped — the voiceover is silent there.
Edit their text in tts_input.srt, then:
--regen 2035 4180
```
Skipping only happens when the backend comes back healthy. If it does not, the
run waits for it instead — otherwise a backend that is simply not running would
cause every remaining clip to be "skipped" and produce a silent film.
### The clip that crashes ComfyUI
The abort looks like this in `comfyui.log`:
```
/pytorch/aten/src/ATen/native/cuda/Indexing.cu:1515: indexSelectSmallIndex:
Assertion `srcIndex < srcSelectDimSize` failed.
Aborted (core dumped)
```
An out-of-range index in an embedding lookup inside the Higgs audio tokenizer.
It is a CUDA device-side assert, so it cannot be caught in Python — the process
dies outright, which is why it looks like ComfyUI "will not start" when a client
is resubmitting the same clip on every restart.
`--regen N` with a different seed is the fix; rewording the block helps if it
persists.
### ComfyUI crashed and the GPU is stuck
```bash
@@ -244,6 +383,11 @@ python3 lektor_ai_comfy.py film.mkv --resume-from 312
Nothing else should be using the GPU during a run. Restart ComfyUI to clear
fragmentation. The 4B model needs roughly 10–12 GB.
The number normalizer adds ~1.5 GB, but only during step 1 — its process exits
before TTS generation starts, so the two never hold VRAM at the same time. If it
cannot fit anyway, it logs the CUDA error and finishes the run on the `num2words`
fallback rather than failing.
### `torch.cuda.is_available()` is False
```bash