Add subtitle dictionaries, Polish number normalizer, ComfyUI lifecycle control
Text processing before TTS:
- dictionary.txt rewrites words and phrases (km/h, np., units). Rules compile
into one alternation applied in a single pass, so replacements never chain
and longest-literal wins. Case-sensitive, whole-word.
- A per-queue second dictionary, written by the GUI into the first video's job
folder and merged with the shared one (--dictionary-extra).
- ALL-CAPS blocks become Title Case rather than sentence case: lowercasing
destroyed proper nouns inside captions ("NASA - MARGO MADISON") and invited
the normalizer to "correct" the name. Paired clips in output/clips/all-caps
showed the TTS reads both casings the same.
normalizer/ — digits to spoken Polish, in its own venv because it needs
transformers>=5, which must not touch ComfyUI's pinned versions:
- Two backends. Plain integers go to num2words for the value and a general
Polish LLM for the case; times, dates, decimals and identifiers go to a 0.6B
fine-tune. Measured on 40 real subtitle lines, each is clearly better than
the other on its half. 74% of numbered blocks take the first leg.
- Guards on every answer: runaway length, repetition loops, invalid bytes,
leftover digits, a lost minus sign, a changed number, a lost word. A failure
falls back to num2words, which is nominative but never wrong about the value.
- ComfyUI is asked to unload before normalizing; a normalizer that does not fit
in VRAM is spilled to RAM silently at 5x the cost.
Backend lifecycle:
- ComfyUI is found by port through /proc, so a backend started by hand is
managed too. SIGTERM escalates to SIGKILL, and a suspended process is killed
outright since it never runs its handlers.
- A clip that kills the backend is retried three times, each with a different
seed (the aborts are seed-dependent, not text-dependent), then skipped and
reported so an overnight queue keeps moving.
- start_comfyui.sh execs python, so the script's PID is ComfyUI's.
- GUI gains ComfyUI and Queue dictionary buttons; quitting stops the backend.
--resume-from now reuses the job folder's tts_input.srt instead of reparsing
the subtitles, so block N still pairs with clip_NNNN.flac.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
This commit is contained in:
+96
-2
@@ -37,23 +37,117 @@ must repeat the same `--start`/`--end`.
|
||||
and `<...>` HTML tags (`<i>`, `<font …>`).
|
||||
2. **Detect dialogue.** Lines beginning `- ` mean two speakers and are kept
|
||||
separate; ordinary wrapped lines are joined with a space.
|
||||
3. **Sentence-case ALL-CAPS blocks**, which the model would otherwise shout.
|
||||
3. **Title-case ALL-CAPS blocks**, which the model would otherwise shout.
|
||||
Every word is capitalised rather than lowercasing all but the first letter,
|
||||
because the obvious version also lowercases the proper nouns inside a
|
||||
caption: `NASA – MARGO MADISON` became `Nasa – margo madison`, which is wrong
|
||||
to read aloud and invites the number normalizer to "correct" the name (it
|
||||
produced `marga`). A/B clips showed the TTS reads both casings the same, so
|
||||
keeping the names costs nothing.
|
||||
4. **Capitalise a lowercase first letter**, since a fragment continuing from the
|
||||
previous subtitle confuses the model.
|
||||
5. **Add terminal punctuation** so the model produces a natural falling tail.
|
||||
5. **Apply the replacement dictionary** — `km/h` → `kilometrów na godzinę` and
|
||||
anything else the model reads out wrong (see below).
|
||||
6. **Add terminal punctuation** so the model produces a natural falling tail.
|
||||
|
||||
Step 1 must come first. `<i>ALL CAPS</i>` looks mixed-case while the tag is
|
||||
still present — the `i` in `<i>` is a lowercase letter — so the caps detection
|
||||
in step 3 would silently miss it.
|
||||
|
||||
Step 5 must come after step 3 for the same reason in reverse: a lowercase
|
||||
replacement dropped into `JEDZIE 200 KM/H` would make the block look mixed-case
|
||||
and the shouting would survive. It comes before step 6 so a rule can supply
|
||||
punctuation of its own.
|
||||
|
||||
### The replacement dictionary
|
||||
|
||||
`Dictionary.load()` reads `dictionary.txt` (`[paths] dictionary`, or
|
||||
`--dictionary`) into `pattern = replacement` rules — literal and
|
||||
case-insensitive, or a regular expression when the pattern is wrapped in
|
||||
slashes. Every rule is compiled into **one alternation** applied in a single
|
||||
`re.sub` pass, which buys three properties worth keeping:
|
||||
|
||||
- Replacement text is never rescanned, so rules cannot chain or loop.
|
||||
- Literals are ordered longest-first inside the alternation, so `km/h` beats a
|
||||
plain `km` rule regardless of the order in the file.
|
||||
- Rules cost one pass over the text no matter how many there are.
|
||||
|
||||
Literal patterns get `(?<!\w)` / `(?!\w)` guards only on edges that are word
|
||||
characters, which is what lets `km/h` and `np.` match on their own while `znp.`
|
||||
does not — a guard on a `/` or `.` edge would stop them matching at all.
|
||||
|
||||
Matching is case-sensitive and the replacement is verbatim. The alternative —
|
||||
match loosely, then infer how to re-case the replacement — was tried and
|
||||
removed: it silently rewrote `NASA` as `Nasa`, and a rule that fires on a case
|
||||
you did not write is hard to reason about. The cost is that the first letter of
|
||||
a block is capitalised before the dictionary runs, so a lowercase rule cannot
|
||||
fire there; on the seven subtitle files here that changed nothing, but it is why
|
||||
a word that can open a subtitle needs both forms.
|
||||
|
||||
A missing file, a line without a separator and an invalid regex are all
|
||||
warnings, never fatal — the same rule as `settings.ini`, for the same reason: a
|
||||
typo should not cost a long run.
|
||||
|
||||
`split_dialogue_blocks()` then divides a two-speaker block into one clip per
|
||||
line, apportioning the block's duration by character count.
|
||||
|
||||
### Number normalization (last stage)
|
||||
|
||||
`normalize_block_numbers()` rewrites digits as Polish words — `1247` →
|
||||
`tysiąc dwieście czterdzieści siedem` — because Higgs Audio misreads long digit
|
||||
strings and sometimes emits garbled non-words. It is **last** for two reasons:
|
||||
the dictionary may have produced the words a numeral has to agree with (`16%` →
|
||||
`16 procent` → `szesnaście procent`), and dialogue splitting decides which clip
|
||||
a number ends up in.
|
||||
|
||||
The model that does it needs `transformers>=5`, which cannot be installed beside
|
||||
ComfyUI's pinned dependencies, so it lives in `normalizer/` with its own venv and
|
||||
runs as a **separate process**. `NumberNormalizer` is the client half:
|
||||
|
||||
- One worker per run, not per block — the 1.1 GB load dominates everything else.
|
||||
It is started only if some block actually contains a number worth expanding,
|
||||
and closed before step 2, so it is not holding VRAM while Higgs generates.
|
||||
- **ComfyUI is asked to unload first** (`free_comfy_vram`). Higgs stays resident
|
||||
between runs and leaves too little VRAM; a normalizer that does not fit is
|
||||
spilled to RAM by accelerate *silently*, at about 5× the time per block. A
|
||||
`POST /free` is cheaper than a restart — the server stays warm — and the wait
|
||||
is on `torch_vram_total` in `/system_stats` rather than a fixed sleep, because
|
||||
the flag is consumed by ComfyUI's worker loop rather than by the request.
|
||||
- JSON lines over stdin/stdout, strictly one request per response. stdout is the
|
||||
protocol; logs go to stderr and are kept in a 40-line ring for diagnostics.
|
||||
- Reads are on a queue fed by a reader thread, which is what makes a per-block
|
||||
timeout possible at all — a wedged worker cannot block the run.
|
||||
- **Nothing here is fatal.** Missing venv, dead worker, timeout, out-of-order
|
||||
reply: each prints a warning, disables the stage, and lets the digits through
|
||||
untouched. A number read badly is a bad clip; a crash here is a lost film.
|
||||
|
||||
- **Two models behind one interface.** Plain integers go to num2words for the
|
||||
value and a general Polish LLM for the case; anything with a time, date,
|
||||
decimal or identifier goes to the 0.6B fine-tune. Measured on 40 real subtitle
|
||||
lines, each is clearly better than the other on its half and clearly worse on
|
||||
the other's. The worker reports which leg answered, and the run prints the
|
||||
split.
|
||||
- **Guards, not trust.** Both models can emit something worse than nothing — a
|
||||
repetition loop to the token ceiling, a dropped minus that inverts a
|
||||
temperature, a vanished noun, a number changed from 8250 to 8205. Each answer
|
||||
is checked against the input and falls back to num2words when it fails.
|
||||
|
||||
`needs_number_normalization()` duplicates the worker's own selection rule so
|
||||
blocks that need nothing never cross the process boundary. The two must stay in
|
||||
step; the worker's copy is the authoritative one, and a test in the normalizer
|
||||
suite imports the caller and compares both on a shared sample.
|
||||
|
||||
The result is written to `tts_input.srt` in that video's job folder
|
||||
(`output/clips/<video name>/`). It is the source of truth for block
|
||||
numbering: block *N* there is `clip_NNNN.flac` beside it and the
|
||||
argument to `--regen N`.
|
||||
|
||||
Which is why `--regen`, `--rebuild` **and `--resume-from`** all read that file
|
||||
rather than the original SRT. Everything upstream of it — the dictionary, the
|
||||
normalizer, dialogue splitting — can produce different text than it did on the
|
||||
first run, and any difference breaks the pairing with the clips already on disk.
|
||||
Step 1 is skipped entirely in those modes.
|
||||
|
||||
## 2. Speech generation
|
||||
|
||||
For each block the script POSTs a workflow graph to `/prompt`, polls
|
||||
|
||||
Reference in New Issue
Block a user