Add subtitle dictionaries, Polish number normalizer, ComfyUI lifecycle control
Text processing before TTS:
- dictionary.txt rewrites words and phrases (km/h, np., units). Rules compile
into one alternation applied in a single pass, so replacements never chain
and longest-literal wins. Case-sensitive, whole-word.
- A per-queue second dictionary, written by the GUI into the first video's job
folder and merged with the shared one (--dictionary-extra).
- ALL-CAPS blocks become Title Case rather than sentence case: lowercasing
destroyed proper nouns inside captions ("NASA - MARGO MADISON") and invited
the normalizer to "correct" the name. Paired clips in output/clips/all-caps
showed the TTS reads both casings the same.
normalizer/ — digits to spoken Polish, in its own venv because it needs
transformers>=5, which must not touch ComfyUI's pinned versions:
- Two backends. Plain integers go to num2words for the value and a general
Polish LLM for the case; times, dates, decimals and identifiers go to a 0.6B
fine-tune. Measured on 40 real subtitle lines, each is clearly better than
the other on its half. 74% of numbered blocks take the first leg.
- Guards on every answer: runaway length, repetition loops, invalid bytes,
leftover digits, a lost minus sign, a changed number, a lost word. A failure
falls back to num2words, which is nominative but never wrong about the value.
- ComfyUI is asked to unload before normalizing; a normalizer that does not fit
in VRAM is spilled to RAM silently at 5x the cost.
Backend lifecycle:
- ComfyUI is found by port through /proc, so a backend started by hand is
managed too. SIGTERM escalates to SIGKILL, and a suspended process is killed
outright since it never runs its handlers.
- A clip that kills the backend is retried three times, each with a different
seed (the aborts are seed-dependent, not text-dependent), then skipped and
reported so an overnight queue keeps moving.
- start_comfyui.sh execs python, so the script's PID is ComfyUI's.
- GUI gains ComfyUI and Queue dictionary buttons; quitting stops the backend.
--resume-from now reuses the job folder's tts_input.srt instead of reparsing
the subtitles, so block N still pairs with clip_NNNN.flac.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
This commit is contained in:
Claude Opus 5
@@ -0,0 +1,253 @@
|
||||
# Build brief — Polish number normalizer (offline `transformers` component)
|
||||
|
||||
## Scope of this document
|
||||
|
||||
This brief covers **only the normalizer component**: installing it, running it
|
||||
offline with plain `transformers`, and what it does. **Integration is out of
|
||||
scope** — the calling app (`lektor_ai`) handles SRT parsing, decides which
|
||||
subtitle blocks to normalize, and invokes this component. The user will specify
|
||||
that wiring separately.
|
||||
|
||||
Assume the caller sends **one sentence / subtitle block at a time**, and only
|
||||
sends blocks that already contain at least one digit. This component's job is to
|
||||
return that text with numbers converted to spoken Polish words, per the rules
|
||||
below.
|
||||
|
||||
## Purpose
|
||||
|
||||
The downstream TTS (Higgs Audio v3) mispronounces raw digits — especially long
|
||||
numbers — sometimes emitting garbled non-words. This component removes that
|
||||
failure by converting digits to fully spelled-out Polish words **before** the
|
||||
text reaches the TTS, with correct grammatical case (the reason a neural
|
||||
normalizer is used rather than a lookup table: Polish numerals decline by case,
|
||||
gender, and the noun they govern).
|
||||
|
||||
Behaviour rules:
|
||||
|
||||
1. **Skip single-digit numbers.** A standalone one-digit integer (e.g. `3` in
|
||||
`Mam 3 koty`) is read correctly by the TTS and must be **left as a digit**.
|
||||
2. **Expand two-or-more-digit numbers** to words (`25`, `2020`, `1 350 000`).
|
||||
3. Complex numeric expressions — decimals (`3,5`), times (`14:30`), dates
|
||||
(`3 maja`), ratios/addresses (`12/45`), currency, units — are handled by the
|
||||
model. They are not "single-digit numbers" and are not skipped.
|
||||
4. A block that (after removing single digits) contains no multi-digit number is
|
||||
returned unchanged.
|
||||
|
||||
## The model
|
||||
|
||||
- **Repo:** `Folx/qwen3-0.6b-pl-text-normalization` (Hugging Face). ~1.1 GB,
|
||||
bfloat16, 600M params, `Qwen3ForCausalLM`. Fine-tuned specifically for Polish
|
||||
TTS text normalization (numbers, dates, times, units, abbreviations).
|
||||
- **GATED REPO.** The user must accept the terms on the model page and provide an
|
||||
`HF_TOKEN` for the **one-time** download. The build agent cannot accept the
|
||||
licence. Load the model from a **local path / local cache** at run time — never
|
||||
re-download.
|
||||
- **CRITICAL version requirement:** `transformers >= 5.0.0` and `torch >= 2.1.0`.
|
||||
Older transformers (e.g. 4.57) tokenize the Qwen3 chat template differently and
|
||||
silently produce wrong output even under greedy decode. Pin `transformers>=5.0`.
|
||||
- **System prompt (fixed, required):** `Zamieniasz tekst na formę mówioną.`
|
||||
- **Context:** max 512 tokens; quality degrades past ~300 characters. Input is one
|
||||
short block per call, so this is fine — warn (don't truncate) on longer inputs.
|
||||
- **Decoding:** greedy (`do_sample=False`), with **constrained decoding** that
|
||||
bans digit tokens from the output so no un-normalized number can leak through.
|
||||
|
||||
## Environment & installation
|
||||
|
||||
Install into a **dedicated, isolated venv/conda env**, separate from the ComfyUI
|
||||
/ Higgs TTS environment. Rationale: this model hard-requires `transformers>=5.0`,
|
||||
and installing that into the existing working TTS environment risks disturbing
|
||||
its pinned dependencies. Keep them apart; this component is invoked as its own
|
||||
process.
|
||||
|
||||
Pinned dependencies:
|
||||
|
||||
- `transformers>=5.0.0`
|
||||
- `torch>=2.1.0` — CUDA build matching the host (target: RTX 3090, 24 GB)
|
||||
- `accelerate` — for `device_map`
|
||||
- `num2words` — used only by the fallback path
|
||||
|
||||
One-time model fetch (network + token required):
|
||||
|
||||
```bash
|
||||
HF_TOKEN=... huggingface-cli download Folx/qwen3-0.6b-pl-text-normalization \
|
||||
--local-dir ./model
|
||||
```
|
||||
|
||||
Run time is **offline**. Set `HF_HUB_OFFLINE=1` and `TRANSFORMERS_OFFLINE=1`, and
|
||||
load the model from `./model` (or the local HF cache). Add a check that a full run
|
||||
succeeds with networking disabled.
|
||||
|
||||
Load the model, tokenizer, and banned-token list **once** (module level) and reuse
|
||||
across calls — model load dominates cost.
|
||||
|
||||
## Normalization logic
|
||||
|
||||
For a given input string:
|
||||
|
||||
1. **Protect single-digit numbers.** Replace each standalone one-digit integer
|
||||
with a **digit-free sentinel**, remembering the mapping. The boundary set
|
||||
excludes adjacent digits and numeric punctuation so decimals/times/dates are
|
||||
not caught:
|
||||
|
||||
```python
|
||||
KEEP_RE = re.compile(r'(?<![\d.,:/\-])\d(?![\d.,:/\-])')
|
||||
```
|
||||
|
||||
Sentinels must contain **no digits** (digit tokens are banned in output, and
|
||||
the check in step 2 looks for leftover digits). Use letter-indexed sentinels,
|
||||
e.g. `KEEPNUMA`, `KEEPNUMB`, … .
|
||||
|
||||
2. **Decide.** If the protected text still contains a digit, that digit belongs
|
||||
to a 2+ digit / complex number → call the model. If no digit remains, skip the
|
||||
model and return the text with sentinels restored (only single digits were
|
||||
present).
|
||||
|
||||
3. **Model call.** Build the chat prompt (fixed system prompt + protected text),
|
||||
greedy decode with digit tokens banned, decode the completion.
|
||||
|
||||
4. **Restore** sentinels to their original single digits.
|
||||
|
||||
5. On any generation error, use the fallback instead of raising.
|
||||
|
||||
### Reference: model wrapper (adapt, then verify against the gated model)
|
||||
|
||||
```python
|
||||
import re, torch
|
||||
from transformers import AutoModelForCausalLM, AutoTokenizer
|
||||
|
||||
MODEL = "./model" # local, offline
|
||||
SYS = "Zamieniasz tekst na formę mówioną."
|
||||
|
||||
tok = AutoTokenizer.from_pretrained(MODEL)
|
||||
model = AutoModelForCausalLM.from_pretrained(
|
||||
MODEL, torch_dtype=torch.bfloat16, device_map="auto"
|
||||
).eval()
|
||||
|
||||
def _banned_ids(tok):
|
||||
banned = set()
|
||||
for tid in range(tok.vocab_size):
|
||||
d = tok.decode([tid]).strip()
|
||||
if d and d.isdigit():
|
||||
banned.add(tid)
|
||||
for sym in ["$","%","€","£","¥","°","§","@","#","&","×","÷","±",
|
||||
"©","®","™","→","←","↑","↓","√","∞","≈","≠","≤","≥"]:
|
||||
ids = tok.encode(sym, add_special_tokens=False)
|
||||
if len(ids) == 1:
|
||||
banned.add(ids[0])
|
||||
return sorted(banned)
|
||||
|
||||
BANNED = _banned_ids(tok)
|
||||
|
||||
def _model_normalize(text: str) -> str:
|
||||
messages = [{"role": "system", "content": SYS},
|
||||
{"role": "user", "content": text}]
|
||||
prompt = tok.apply_chat_template(messages, tokenize=False,
|
||||
add_generation_prompt=True)
|
||||
inputs = tok(prompt, return_tensors="pt").to(model.device)
|
||||
n = inputs["input_ids"].shape[1]
|
||||
with torch.no_grad():
|
||||
out = model.generate(**inputs, max_new_tokens=400, do_sample=False,
|
||||
pad_token_id=tok.pad_token_id,
|
||||
suppress_tokens=BANNED)
|
||||
return tok.decode(out[0][n:], skip_special_tokens=True).strip()
|
||||
```
|
||||
|
||||
### Reference: skip / decide / restore + public function
|
||||
|
||||
```python
|
||||
KEEP_RE = re.compile(r'(?<![\d.,:/\-])\d(?![\d.,:/\-])')
|
||||
_LETTERS = "ABCDEFGHIJKLMNOPQRSTUVWXYZ"
|
||||
|
||||
def _sentinel(i: int) -> str:
|
||||
s = ""; i += 1
|
||||
while i:
|
||||
i, r = divmod(i - 1, 26); s = _LETTERS[r] + s
|
||||
return f"KEEPNUM{s}"
|
||||
|
||||
def normalize(text: str) -> str:
|
||||
"""Return `text` with 2+ digit numbers spelled out in Polish; single-digit
|
||||
numbers left as digits. Safe to call on any string."""
|
||||
mapping = {}
|
||||
def protect(m):
|
||||
key = _sentinel(len(mapping)); mapping[key] = m.group(0); return key
|
||||
protected = KEEP_RE.sub(protect, text)
|
||||
|
||||
if re.search(r'\d', protected):
|
||||
try:
|
||||
result = _model_normalize(protected)
|
||||
except Exception:
|
||||
result = _fallback(protected)
|
||||
else:
|
||||
result = protected
|
||||
|
||||
for key, digit in mapping.items():
|
||||
result = result.replace(key, digit)
|
||||
return result
|
||||
```
|
||||
|
||||
### Reference: fallback (model unavailable / generation error)
|
||||
|
||||
Guarantees the component never raises. Degraded: nominative case only, bare
|
||||
multi-digit integers only (leaves decimals/times/dates untouched rather than
|
||||
mangle them). Log a warning when used.
|
||||
|
||||
```python
|
||||
from num2words import num2words
|
||||
|
||||
_INT_RE = re.compile(r'(?<![\d.,:/\-])\d{2,}(?:[\u00a0 .]\d{3})*(?![\d.,:/\-])')
|
||||
|
||||
def _fallback(text: str) -> str:
|
||||
def repl(m):
|
||||
digits = re.sub(r'[\u00a0 .]', '', m.group(0))
|
||||
return num2words(int(digits), lang="pl")
|
||||
return _INT_RE.sub(repl, text)
|
||||
```
|
||||
|
||||
## Error handling
|
||||
|
||||
- Input longer than ~300 chars: log a warning (quality drops), still process,
|
||||
never truncate silently.
|
||||
- Model load failure: fall back to `num2words` and log clearly that the neural
|
||||
normalizer is unavailable — never crash.
|
||||
- Per-call generation error: use the fallback for that call only.
|
||||
|
||||
## Testing / acceptance
|
||||
|
||||
The model produces valid case variants, so assert on **properties**, not exact
|
||||
strings, except where noted.
|
||||
|
||||
- **Single digit preserved:** `Mam 3 koty.` → still contains `3`.
|
||||
- **Multi-digit expanded:** `Wygrał 1247 złotych.` → no `1247`; contains `tysiąc`.
|
||||
- **Large number:** `populacja 1 350 000 osób` → no digits from that number;
|
||||
contains `milion`.
|
||||
- **Mixed (key sentinel round-trip):** `Mam 3 koty i 25 psów.` → keeps `3`,
|
||||
expands `25` → `dwadzieścia pięć`.
|
||||
- **Only single digits:** model is not called; text returned unchanged (assert
|
||||
via a spy on `_model_normalize`).
|
||||
- **No digits leak:** expanded output contains no digit characters except
|
||||
restored single digits.
|
||||
- **Sentinel round-trip:** `KEEPNUM…` never appears in final output. **If the
|
||||
model is found to alter/inflect the sentinel, flag it** — the fix is a more
|
||||
inert sentinel, or dropping protection and documenting that single digits
|
||||
co-occurring with multi-digit numbers get expanded.
|
||||
- **Offline:** the suite passes with networking disabled, `HF_HUB_OFFLINE=1`,
|
||||
`TRANSFORMERS_OFFLINE=1`.
|
||||
|
||||
## Deliverables
|
||||
|
||||
- `normalizer.py` — model/tokenizer/banned-list loaded once at import; public
|
||||
`normalize(text: str) -> str`; `_fallback`; skip/decide/restore logic.
|
||||
- `requirements.txt` — pinned as above.
|
||||
- `README.md` — env creation, one-time gated download with `HF_TOKEN`, offline
|
||||
run variables. (No caller/integration details — those are provided separately.)
|
||||
- `tests/` — the checks above with sample inputs.
|
||||
|
||||
## Design decisions (defaults — confirm if you disagree)
|
||||
|
||||
1. **Single digit inside a date/time.** `3 maja` keeps its `3` (space-separated
|
||||
standalone single digit), while `9:30`, `3,5`, `12/45` go to the model and are
|
||||
expanded. If you want `3 maja` → `trzeciego maja`, that's a one-line change to
|
||||
the boundary set.
|
||||
2. **Sentinel robustness.** Small residual risk the model alters the placeholder
|
||||
instead of passing it through. Encoded as an explicit test with a documented
|
||||
fallback so it's caught, not shipped silently.
|
||||
Reference in New Issue
Block a user