Add subtitle dictionaries, Polish number normalizer, ComfyUI lifecycle control
Text processing before TTS:
- dictionary.txt rewrites words and phrases (km/h, np., units). Rules compile
into one alternation applied in a single pass, so replacements never chain
and longest-literal wins. Case-sensitive, whole-word.
- A per-queue second dictionary, written by the GUI into the first video's job
folder and merged with the shared one (--dictionary-extra).
- ALL-CAPS blocks become Title Case rather than sentence case: lowercasing
destroyed proper nouns inside captions ("NASA - MARGO MADISON") and invited
the normalizer to "correct" the name. Paired clips in output/clips/all-caps
showed the TTS reads both casings the same.
normalizer/ — digits to spoken Polish, in its own venv because it needs
transformers>=5, which must not touch ComfyUI's pinned versions:
- Two backends. Plain integers go to num2words for the value and a general
Polish LLM for the case; times, dates, decimals and identifiers go to a 0.6B
fine-tune. Measured on 40 real subtitle lines, each is clearly better than
the other on its half. 74% of numbered blocks take the first leg.
- Guards on every answer: runaway length, repetition loops, invalid bytes,
leftover digits, a lost minus sign, a changed number, a lost word. A failure
falls back to num2words, which is nominative but never wrong about the value.
- ComfyUI is asked to unload before normalizing; a normalizer that does not fit
in VRAM is spilled to RAM silently at 5x the cost.
Backend lifecycle:
- ComfyUI is found by port through /proc, so a backend started by hand is
managed too. SIGTERM escalates to SIGKILL, and a suspended process is killed
outright since it never runs its handlers.
- A clip that kills the backend is retried three times, each with a different
seed (the aborts are seed-dependent, not text-dependent), then skipped and
reported so an overnight queue keeps moving.
- start_comfyui.sh execs python, so the script's PID is ComfyUI's.
- GUI gains ComfyUI and Queue dictionary buttons; quitting stops the backend.
--resume-from now reuses the job folder's tts_input.srt instead of reparsing
the subtitles, so block N still pairs with clip_NNNN.flac.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
This commit is contained in:
@@ -203,6 +203,93 @@ Handled by `_clean_text()`, but only after tags are removed. `<i>HELLO</i>`
|
||||
looked mixed-case while the tag was present, because `i` is a lowercase letter.
|
||||
If shouting persists, check for an unhandled tag type.
|
||||
|
||||
Such blocks are converted to Title Case (`Nasa – Margo Madison`), not to
|
||||
sentence case. Lowercasing everything but the first letter also destroys the
|
||||
proper nouns in a caption, and the number normalizer then tries to repair the
|
||||
lowercase name — it turned `margo` into `marga`. Paired clips in
|
||||
`output/clips/all-caps/` show the TTS reads both casings identically, so the
|
||||
names are kept.
|
||||
|
||||
### A clip of an ALL-CAPS caption is cut off mid-word
|
||||
|
||||
Check whether the block still contains digits. Higgs truncates on raw digits far
|
||||
more often than on words: the same caption generated eight times, four with
|
||||
`21 MAJA 1983` and four with the date spelled out, was cut in 3 of 4 digit
|
||||
versions and 0 of 4 word versions — and the digit clips ran barely half as long
|
||||
as the speech required. Casing made no difference in that test.
|
||||
|
||||
That is the number normalizer's job, so if you see it, the block reached the TTS
|
||||
without being normalized: check `numbers = yes` and the `Normalizer ready` line
|
||||
in step 1.
|
||||
|
||||
### An abbreviation, unit or name is read out wrong
|
||||
|
||||
Put it in `dictionary.txt`: `km/h = kilometrów na godzinę`. The replacement
|
||||
happens before the text reaches the model, and the result is visible in the job
|
||||
folder's `tts_input.srt`. See [usage.md](usage.md#replacement-dictionary).
|
||||
|
||||
If a rule appears to do nothing, check the run's `Dictionary:` line — a rule
|
||||
that failed to parse was reported there and skipped, and a rule cannot match
|
||||
across a line break because each dialogue line is treated separately.
|
||||
|
||||
`--regen` and `--rebuild` read `tts_input.srt`, which already has the
|
||||
replacements baked in, so adding a rule afterwards needs either a fresh run
|
||||
from the SRT or a manual edit of `tts_input.srt` before `--regen`.
|
||||
|
||||
### Numbers are read as garbled non-words
|
||||
|
||||
That is what the number normalizer is for; check what the run printed in step 1.
|
||||
|
||||
- `Normalizer ready (model on cuda:0)` — it ran. Look at `tts_input.srt`: if the
|
||||
digits are still there, the block may hold only standalone single digits,
|
||||
which are left alone deliberately.
|
||||
- `Normalizer ready (fallback)` — the model is missing or failed to load, and
|
||||
`num2words` is standing in: bare integers only, nominative case only, times
|
||||
and decimals untouched. Install the model, see
|
||||
[usage.md](usage.md#number-normalization).
|
||||
- `normalizer interpreter not found` — run `normalizer/setup.sh`.
|
||||
- Nothing at all — the stage is off (`numbers = no`, or
|
||||
`--no-normalize-numbers`).
|
||||
|
||||
For the underlying cause of a bad load, run the component directly; it prints
|
||||
the real error instead of a summary:
|
||||
|
||||
```bash
|
||||
normalizer/venv/bin/python normalizer/normalizer.py --status
|
||||
normalizer/venv/bin/python normalizer/normalizer.py "Wygrał 1247 złotych."
|
||||
```
|
||||
|
||||
### A number is expanded in the wrong grammatical case
|
||||
|
||||
Nominative everywhere (`dwadzieścia pięć psach` instead of `dwudziestu pięciu
|
||||
psach`) means the `num2words` fallback is running, not the model — see above.
|
||||
The neural model is the only path that declines correctly.
|
||||
|
||||
### The normalizer is unexpectedly slow
|
||||
|
||||
Check which device the run reported. `Normalizer ready (model on cpu)` means it
|
||||
never got the GPU. If it says `cuda:0` and is still slow, it may have only got
|
||||
part of it: with Higgs resident there is not enough VRAM, and `accelerate`
|
||||
spills the rest to RAM without raising anything — about 5× slower.
|
||||
|
||||
The script prevents this by asking ComfyUI to unload before normalizing. If you
|
||||
do not see `Freed N GB` in step 1, either `free_comfy_vram = no`, or ComfyUI had
|
||||
nothing loaded (normal on a freshly started backend), or the call failed and
|
||||
said so.
|
||||
|
||||
### The normalizer times out on every block
|
||||
|
||||
`timeout` under `[normalize]` is per block, and a block takes well under a second
|
||||
on a GPU. Timing out means the model landed on the CPU or the worker is wedged.
|
||||
Check `torch.cuda.is_available()` in that venv:
|
||||
|
||||
```bash
|
||||
normalizer/venv/bin/python -c "import torch; print(torch.cuda.is_available())"
|
||||
```
|
||||
|
||||
The stage disables itself after the first timeout rather than repeating the wait
|
||||
on every remaining block, so a run costs one `timeout` at worst.
|
||||
|
||||
### `--embedded-subs` shows no usable tracks
|
||||
|
||||
The video only carries image-based subtitles (`hdmv_pgs_subtitle`,
|
||||
@@ -226,6 +313,58 @@ ComfyUI is not running. Start it and wait for the model to load:
|
||||
curl -s http://127.0.0.1:8189/system_stats | head -c 200
|
||||
```
|
||||
|
||||
### ComfyUI dies partway through a long queue
|
||||
|
||||
The run recovers by itself now, and says so:
|
||||
|
||||
```
|
||||
[2035/2073] 02:22:59 'Joshua.'
|
||||
ComfyUI went away ([Errno 104] Connection reset by peer) [1/3 for this clip].
|
||||
Stopping ComfyUI (pid 530039, suspended)…
|
||||
Starting ComfyUI in a new gnome-terminal window…
|
||||
ComfyUI died on this clip (1/3) — retrying with a different seed.
|
||||
```
|
||||
|
||||
The stale process is stopped before the restart, because a wedged or suspended
|
||||
backend still holds the port and the GPU and the new one then cannot start. A
|
||||
process in state `T` is killed outright — it never runs its signal handlers, so
|
||||
SIGTERM is accepted and ignored.
|
||||
|
||||
**Each retry uses a different seed**, which matters: these aborts are
|
||||
seed-dependent, not text-dependent. The block that killed ComfyUI three times in
|
||||
a 2073-clip run generated perfectly on its own with another seed.
|
||||
|
||||
After three deaths on the same clip the clip is skipped and the queue continues,
|
||||
with a report at the end:
|
||||
|
||||
```
|
||||
2 clip(s) crashed ComfyUI and were skipped — the voiceover is silent there.
|
||||
Edit their text in tts_input.srt, then:
|
||||
--regen 2035 4180
|
||||
```
|
||||
|
||||
Skipping only happens when the backend comes back healthy. If it does not, the
|
||||
run waits for it instead — otherwise a backend that is simply not running would
|
||||
cause every remaining clip to be "skipped" and produce a silent film.
|
||||
|
||||
### The clip that crashes ComfyUI
|
||||
|
||||
The abort looks like this in `comfyui.log`:
|
||||
|
||||
```
|
||||
/pytorch/aten/src/ATen/native/cuda/Indexing.cu:1515: indexSelectSmallIndex:
|
||||
Assertion `srcIndex < srcSelectDimSize` failed.
|
||||
Aborted (core dumped)
|
||||
```
|
||||
|
||||
An out-of-range index in an embedding lookup inside the Higgs audio tokenizer.
|
||||
It is a CUDA device-side assert, so it cannot be caught in Python — the process
|
||||
dies outright, which is why it looks like ComfyUI "will not start" when a client
|
||||
is resubmitting the same clip on every restart.
|
||||
|
||||
`--regen N` with a different seed is the fix; rewording the block helps if it
|
||||
persists.
|
||||
|
||||
### ComfyUI crashed and the GPU is stuck
|
||||
|
||||
```bash
|
||||
@@ -244,6 +383,11 @@ python3 lektor_ai_comfy.py film.mkv --resume-from 312
|
||||
Nothing else should be using the GPU during a run. Restart ComfyUI to clear
|
||||
fragmentation. The 4B model needs roughly 10–12 GB.
|
||||
|
||||
The number normalizer adds ~1.5 GB, but only during step 1 — its process exits
|
||||
before TTS generation starts, so the two never hold VRAM at the same time. If it
|
||||
cannot fit anyway, it logs the CUDA error and finishes the run on the `num2words`
|
||||
fallback rather than failing.
|
||||
|
||||
### `torch.cuda.is_available()` is False
|
||||
|
||||
```bash
|
||||
|
||||
Reference in New Issue
Block a user