From e201aadc3e68bf50da8255ce76f5d8bc35e624a9 Mon Sep 17 00:00:00 2001 From: claude Date: Wed, 12 Aug 2026 19:40:44 +0200 Subject: [PATCH] Add subtitle dictionaries, Polish number normalizer, ComfyUI lifecycle control MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Text processing before TTS: - dictionary.txt rewrites words and phrases (km/h, np., units). Rules compile into one alternation applied in a single pass, so replacements never chain and longest-literal wins. Case-sensitive, whole-word. - A per-queue second dictionary, written by the GUI into the first video's job folder and merged with the shared one (--dictionary-extra). - ALL-CAPS blocks become Title Case rather than sentence case: lowercasing destroyed proper nouns inside captions ("NASA - MARGO MADISON") and invited the normalizer to "correct" the name. Paired clips in output/clips/all-caps showed the TTS reads both casings the same. normalizer/ — digits to spoken Polish, in its own venv because it needs transformers>=5, which must not touch ComfyUI's pinned versions: - Two backends. Plain integers go to num2words for the value and a general Polish LLM for the case; times, dates, decimals and identifiers go to a 0.6B fine-tune. Measured on 40 real subtitle lines, each is clearly better than the other on its half. 74% of numbered blocks take the first leg. - Guards on every answer: runaway length, repetition loops, invalid bytes, leftover digits, a lost minus sign, a changed number, a lost word. A failure falls back to num2words, which is nominative but never wrong about the value. - ComfyUI is asked to unload before normalizing; a normalizer that does not fit in VRAM is spilled to RAM silently at 5x the cost. Backend lifecycle: - ComfyUI is found by port through /proc, so a backend started by hand is managed too. SIGTERM escalates to SIGKILL, and a suspended process is killed outright since it never runs its handlers. - A clip that kills the backend is retried three times, each with a different seed (the aborts are seed-dependent, not text-dependent), then skipped and reported so an overnight queue keeps moving. - start_comfyui.sh execs python, so the script's PID is ComfyUI's. - GUI gains ComfyUI and Queue dictionary buttons; quitting stops the backend. --resume-from now reuses the job folder's tts_input.srt instead of reparsing the subtitles, so block N still pairs with clip_NNNN.flac. Co-Authored-By: Claude Opus 5 --- .gitignore | 9 + BUILD_BRIEF_pl_number_normalizer.md | 253 ++++++++ dictionary.txt | 49 ++ docs/README.md | 17 +- docs/architecture.md | 98 ++- docs/installation.md | 29 +- docs/requirements.md | 36 +- docs/troubleshooting.md | 144 +++++ docs/usage.md | 256 ++++++++ lektor_ai_comfy.py | 939 +++++++++++++++++++++++++++- normalizer/README.md | 179 ++++++ normalizer/bench.py | 251 ++++++++ normalizer/normalizer.py | 645 +++++++++++++++++++ normalizer/requirements.txt | 13 + normalizer/setup.sh | 98 +++ normalizer/tests/test_normalizer.py | 375 +++++++++++ normalizer/worker.py | 74 +++ readme.txt | 73 ++- settings.ini | 43 +- start_comfyui.sh | 5 +- ui.py | 225 ++++++- 21 files changed, 3759 insertions(+), 52 deletions(-) create mode 100644 BUILD_BRIEF_pl_number_normalizer.md create mode 100644 dictionary.txt create mode 100644 normalizer/README.md create mode 100644 normalizer/bench.py create mode 100755 normalizer/normalizer.py create mode 100644 normalizer/requirements.txt create mode 100755 normalizer/setup.sh create mode 100644 normalizer/tests/test_normalizer.py create mode 100755 normalizer/worker.py diff --git a/.gitignore b/.gitignore index 0988c55..8a9b372 100644 --- a/.gitignore +++ b/.gitignore @@ -2,6 +2,12 @@ input/ output/ +# Number normalizer: its own venv, and 1.1 GB of gated model weights that must +# be fetched with the user's own HF token. Recreate with normalizer/setup.sh. +normalizer/venv/ +normalizer/model/ +normalizer/models/ + # Python __pycache__/ *.py[cod] @@ -9,6 +15,9 @@ __pycache__/ .venv/ venv/ +# Server credentials — never commit these +gitea-server-credentials.md + # Machine-local Claude Code permissions (intentionally not shared) .claude/settings.local.json diff --git a/BUILD_BRIEF_pl_number_normalizer.md b/BUILD_BRIEF_pl_number_normalizer.md new file mode 100644 index 0000000..093c190 --- /dev/null +++ b/BUILD_BRIEF_pl_number_normalizer.md @@ -0,0 +1,253 @@ +# Build brief — Polish number normalizer (offline `transformers` component) + +## Scope of this document + +This brief covers **only the normalizer component**: installing it, running it +offline with plain `transformers`, and what it does. **Integration is out of +scope** — the calling app (`lektor_ai`) handles SRT parsing, decides which +subtitle blocks to normalize, and invokes this component. The user will specify +that wiring separately. + +Assume the caller sends **one sentence / subtitle block at a time**, and only +sends blocks that already contain at least one digit. This component's job is to +return that text with numbers converted to spoken Polish words, per the rules +below. + +## Purpose + +The downstream TTS (Higgs Audio v3) mispronounces raw digits — especially long +numbers — sometimes emitting garbled non-words. This component removes that +failure by converting digits to fully spelled-out Polish words **before** the +text reaches the TTS, with correct grammatical case (the reason a neural +normalizer is used rather than a lookup table: Polish numerals decline by case, +gender, and the noun they govern). + +Behaviour rules: + +1. **Skip single-digit numbers.** A standalone one-digit integer (e.g. `3` in + `Mam 3 koty`) is read correctly by the TTS and must be **left as a digit**. +2. **Expand two-or-more-digit numbers** to words (`25`, `2020`, `1 350 000`). +3. Complex numeric expressions — decimals (`3,5`), times (`14:30`), dates + (`3 maja`), ratios/addresses (`12/45`), currency, units — are handled by the + model. They are not "single-digit numbers" and are not skipped. +4. A block that (after removing single digits) contains no multi-digit number is + returned unchanged. + +## The model + +- **Repo:** `Folx/qwen3-0.6b-pl-text-normalization` (Hugging Face). ~1.1 GB, + bfloat16, 600M params, `Qwen3ForCausalLM`. Fine-tuned specifically for Polish + TTS text normalization (numbers, dates, times, units, abbreviations). +- **GATED REPO.** The user must accept the terms on the model page and provide an + `HF_TOKEN` for the **one-time** download. The build agent cannot accept the + licence. Load the model from a **local path / local cache** at run time — never + re-download. +- **CRITICAL version requirement:** `transformers >= 5.0.0` and `torch >= 2.1.0`. + Older transformers (e.g. 4.57) tokenize the Qwen3 chat template differently and + silently produce wrong output even under greedy decode. Pin `transformers>=5.0`. +- **System prompt (fixed, required):** `Zamieniasz tekst na formę mówioną.` +- **Context:** max 512 tokens; quality degrades past ~300 characters. Input is one + short block per call, so this is fine — warn (don't truncate) on longer inputs. +- **Decoding:** greedy (`do_sample=False`), with **constrained decoding** that + bans digit tokens from the output so no un-normalized number can leak through. + +## Environment & installation + +Install into a **dedicated, isolated venv/conda env**, separate from the ComfyUI +/ Higgs TTS environment. Rationale: this model hard-requires `transformers>=5.0`, +and installing that into the existing working TTS environment risks disturbing +its pinned dependencies. Keep them apart; this component is invoked as its own +process. + +Pinned dependencies: + +- `transformers>=5.0.0` +- `torch>=2.1.0` — CUDA build matching the host (target: RTX 3090, 24 GB) +- `accelerate` — for `device_map` +- `num2words` — used only by the fallback path + +One-time model fetch (network + token required): + +```bash +HF_TOKEN=... huggingface-cli download Folx/qwen3-0.6b-pl-text-normalization \ + --local-dir ./model +``` + +Run time is **offline**. Set `HF_HUB_OFFLINE=1` and `TRANSFORMERS_OFFLINE=1`, and +load the model from `./model` (or the local HF cache). Add a check that a full run +succeeds with networking disabled. + +Load the model, tokenizer, and banned-token list **once** (module level) and reuse +across calls — model load dominates cost. + +## Normalization logic + +For a given input string: + +1. **Protect single-digit numbers.** Replace each standalone one-digit integer + with a **digit-free sentinel**, remembering the mapping. The boundary set + excludes adjacent digits and numeric punctuation so decimals/times/dates are + not caught: + + ```python + KEEP_RE = re.compile(r'(? str: + messages = [{"role": "system", "content": SYS}, + {"role": "user", "content": text}] + prompt = tok.apply_chat_template(messages, tokenize=False, + add_generation_prompt=True) + inputs = tok(prompt, return_tensors="pt").to(model.device) + n = inputs["input_ids"].shape[1] + with torch.no_grad(): + out = model.generate(**inputs, max_new_tokens=400, do_sample=False, + pad_token_id=tok.pad_token_id, + suppress_tokens=BANNED) + return tok.decode(out[0][n:], skip_special_tokens=True).strip() +``` + +### Reference: skip / decide / restore + public function + +```python +KEEP_RE = re.compile(r'(? str: + s = ""; i += 1 + while i: + i, r = divmod(i - 1, 26); s = _LETTERS[r] + s + return f"KEEPNUM{s}" + +def normalize(text: str) -> str: + """Return `text` with 2+ digit numbers spelled out in Polish; single-digit + numbers left as digits. Safe to call on any string.""" + mapping = {} + def protect(m): + key = _sentinel(len(mapping)); mapping[key] = m.group(0); return key + protected = KEEP_RE.sub(protect, text) + + if re.search(r'\d', protected): + try: + result = _model_normalize(protected) + except Exception: + result = _fallback(protected) + else: + result = protected + + for key, digit in mapping.items(): + result = result.replace(key, digit) + return result +``` + +### Reference: fallback (model unavailable / generation error) + +Guarantees the component never raises. Degraded: nominative case only, bare +multi-digit integers only (leaves decimals/times/dates untouched rather than +mangle them). Log a warning when used. + +```python +from num2words import num2words + +_INT_RE = re.compile(r'(? str: + def repl(m): + digits = re.sub(r'[\u00a0 .]', '', m.group(0)) + return num2words(int(digits), lang="pl") + return _INT_RE.sub(repl, text) +``` + +## Error handling + +- Input longer than ~300 chars: log a warning (quality drops), still process, + never truncate silently. +- Model load failure: fall back to `num2words` and log clearly that the neural + normalizer is unavailable — never crash. +- Per-call generation error: use the fallback for that call only. + +## Testing / acceptance + +The model produces valid case variants, so assert on **properties**, not exact +strings, except where noted. + +- **Single digit preserved:** `Mam 3 koty.` → still contains `3`. +- **Multi-digit expanded:** `Wygrał 1247 złotych.` → no `1247`; contains `tysiąc`. +- **Large number:** `populacja 1 350 000 osób` → no digits from that number; + contains `milion`. +- **Mixed (key sentinel round-trip):** `Mam 3 koty i 25 psów.` → keeps `3`, + expands `25` → `dwadzieścia pięć`. +- **Only single digits:** model is not called; text returned unchanged (assert + via a spy on `_model_normalize`). +- **No digits leak:** expanded output contains no digit characters except + restored single digits. +- **Sentinel round-trip:** `KEEPNUM…` never appears in final output. **If the + model is found to alter/inflect the sentinel, flag it** — the fix is a more + inert sentinel, or dropping protection and documenting that single digits + co-occurring with multi-digit numbers get expanded. +- **Offline:** the suite passes with networking disabled, `HF_HUB_OFFLINE=1`, + `TRANSFORMERS_OFFLINE=1`. + +## Deliverables + +- `normalizer.py` — model/tokenizer/banned-list loaded once at import; public + `normalize(text: str) -> str`; `_fallback`; skip/decide/restore logic. +- `requirements.txt` — pinned as above. +- `README.md` — env creation, one-time gated download with `HF_TOKEN`, offline + run variables. (No caller/integration details — those are provided separately.) +- `tests/` — the checks above with sample inputs. + +## Design decisions (defaults — confirm if you disagree) + +1. **Single digit inside a date/time.** `3 maja` keeps its `3` (space-separated + standalone single digit), while `9:30`, `3,5`, `12/45` go to the model and are + expanded. If you want `3 maja` → `trzeciego maja`, that's a one-line change to + the boundary set. +2. **Sentinel robustness.** Small residual risk the model alters the placeholder + instead of passing it through. Encoded as an explicit test with a documented + fallback so it's caught, not shipped silently. diff --git a/dictionary.txt b/dictionary.txt new file mode 100644 index 0000000..4e069ac --- /dev/null +++ b/dictionary.txt @@ -0,0 +1,49 @@ +# lektor_ai_comfy — replacement dictionary +# +# Words and phrases rewritten in the subtitles before they are sent to the TTS +# engine, for anything the model reads out wrong: abbreviations, units, symbols, +# names spelled in a way it mispronounces. +# +# One rule per line: +# +# pattern = replacement +# +# * Matching is CASE-SENSITIVE and matches whole words only: "np." leaves +# "znp." alone, and "Joshua" leaves "joshua" alone. +# * The replacement is inserted exactly as written — nothing is re-capitalised +# for you. Note that the first letter of every subtitle block is already +# capitalised before these rules run, so if a word can start a block, add a +# second rule for the capitalised form. +# * Longer patterns win: "km/h" below is used even though a plain "km" rule +# would also fit, whatever order the two are written in. +# * The replacement is never re-scanned, so rules cannot feed into each other. +# * A pattern wrapped in slashes is a regular expression, and \1 \2 … in the +# replacement refer to its groups. +# * A line starting with # or ; is a comment. An empty replacement deletes the +# phrase. +# * "->" and "=>" work as separators too, if you prefer them to "=". +# +# The changes are visible in clips/tts_input.srt after a run — check there if a +# rule is not doing what you expect. + + +# ── Units ────────────────────────────────────────────────────────────────── +km/h = kilometrów na godzinę +m/s = metrów na sekundę + +# ── Abbreviations ────────────────────────────────────────────────────────── +np. = na przykład +tzn. = to znaczy +# These two almost always close a sentence, so the dot is kept; drop it if you +# see an unwanted pause mid-sentence. +itp. = i tym podobne. +itd. = i tak dalej. +m.in. = między innymi +24/7 = dwadzieścia cztery na siedem + +# ── Symbols ──────────────────────────────────────────────────────────────── +/(\d+)\s*%/ = \1 procent +/(\d+)\s*°C/ = \1 stopni Celsjusza +& = i + +# ── Your own rules below ─────────────────────────────────────────────────── diff --git a/docs/README.md b/docs/README.md index be801a4..32716cc 100644 --- a/docs/README.md +++ b/docs/README.md @@ -17,6 +17,7 @@ stream-copied; only the new voiceover track is encoded. | [usage.md](usage.md) | Every CLI flag, the working modes, and practical recipes | | [architecture.md](architecture.md) | How the pipeline works internally, end to end | | [troubleshooting.md](troubleshooting.md) | Diagnosed failure modes and their fixes | +| [../normalizer/README.md](../normalizer/README.md) | The Polish number normalizer, its own component | ## Quick start @@ -35,10 +36,11 @@ python3 lektor_ai_comfy.py "input/film.mkv" --embedded-subs The result is `output/film_lektor.mkv` with two audio tracks — **Original** and **Polish** — plus every subtitle track from the source. -## The two halves of this project +## The three parts of this project -The pieces are deliberately decoupled and it helps to keep them apart when -diagnosing anything: +The pieces are deliberately decoupled — each in its own environment, talking +over a process boundary — and it helps to keep them apart when diagnosing +anything: **ComfyUI + Higgs Audio V3** — a local HTTP service on port 8189 that turns text into speech. Heavy: ~22 GB on disk, needs an NVIDIA GPU, lives in its own conda @@ -49,8 +51,15 @@ ComfyUI over HTTP, and does all timing and audio/video assembly with ffmpeg. It is pure Python standard library, runs on the system Python, and needs no conda environment or pip packages of its own. +**`normalizer/`** — an optional component that spells digits out as Polish words +before they reach the TTS. Its own venv (~6.9 GB, mostly CUDA libraries) and +its own 1.1 GB model, because it needs `transformers >= 5` and the ComfyUI +environment must not be disturbed. Driven as a subprocess for one stage of each +run, then closed. + So a TTS problem (bad pronunciation, hallucination, mumbling) belongs to the -first half, and a timing, mixing, or muxing problem belongs to the second. +first part, a timing, mixing or muxing problem to the second, and a number read +in the wrong case to the third. ## Conventions in these docs diff --git a/docs/architecture.md b/docs/architecture.md index 22b37f0..24a269b 100644 --- a/docs/architecture.md +++ b/docs/architecture.md @@ -37,23 +37,117 @@ must repeat the same `--start`/`--end`. and `<...>` HTML tags (``, ``). 2. **Detect dialogue.** Lines beginning `- ` mean two speakers and are kept separate; ordinary wrapped lines are joined with a space. -3. **Sentence-case ALL-CAPS blocks**, which the model would otherwise shout. +3. **Title-case ALL-CAPS blocks**, which the model would otherwise shout. + Every word is capitalised rather than lowercasing all but the first letter, + because the obvious version also lowercases the proper nouns inside a + caption: `NASA – MARGO MADISON` became `Nasa – margo madison`, which is wrong + to read aloud and invites the number normalizer to "correct" the name (it + produced `marga`). A/B clips showed the TTS reads both casings the same, so + keeping the names costs nothing. 4. **Capitalise a lowercase first letter**, since a fragment continuing from the previous subtitle confuses the model. -5. **Add terminal punctuation** so the model produces a natural falling tail. +5. **Apply the replacement dictionary** — `km/h` → `kilometrów na godzinę` and + anything else the model reads out wrong (see below). +6. **Add terminal punctuation** so the model produces a natural falling tail. Step 1 must come first. `ALL CAPS` looks mixed-case while the tag is still present — the `i` in `` is a lowercase letter — so the caps detection in step 3 would silently miss it. +Step 5 must come after step 3 for the same reason in reverse: a lowercase +replacement dropped into `JEDZIE 200 KM/H` would make the block look mixed-case +and the shouting would survive. It comes before step 6 so a rule can supply +punctuation of its own. + +### The replacement dictionary + +`Dictionary.load()` reads `dictionary.txt` (`[paths] dictionary`, or +`--dictionary`) into `pattern = replacement` rules — literal and +case-insensitive, or a regular expression when the pattern is wrapped in +slashes. Every rule is compiled into **one alternation** applied in a single +`re.sub` pass, which buys three properties worth keeping: + +- Replacement text is never rescanned, so rules cannot chain or loop. +- Literals are ordered longest-first inside the alternation, so `km/h` beats a + plain `km` rule regardless of the order in the file. +- Rules cost one pass over the text no matter how many there are. + +Literal patterns get `(?=5`, which cannot be installed beside +ComfyUI's pinned dependencies, so it lives in `normalizer/` with its own venv and +runs as a **separate process**. `NumberNormalizer` is the client half: + +- One worker per run, not per block — the 1.1 GB load dominates everything else. + It is started only if some block actually contains a number worth expanding, + and closed before step 2, so it is not holding VRAM while Higgs generates. +- **ComfyUI is asked to unload first** (`free_comfy_vram`). Higgs stays resident + between runs and leaves too little VRAM; a normalizer that does not fit is + spilled to RAM by accelerate *silently*, at about 5× the time per block. A + `POST /free` is cheaper than a restart — the server stays warm — and the wait + is on `torch_vram_total` in `/system_stats` rather than a fixed sleep, because + the flag is consumed by ComfyUI's worker loop rather than by the request. +- JSON lines over stdin/stdout, strictly one request per response. stdout is the + protocol; logs go to stderr and are kept in a 40-line ring for diagnostics. +- Reads are on a queue fed by a reader thread, which is what makes a per-block + timeout possible at all — a wedged worker cannot block the run. +- **Nothing here is fatal.** Missing venv, dead worker, timeout, out-of-order + reply: each prints a warning, disables the stage, and lets the digits through + untouched. A number read badly is a bad clip; a crash here is a lost film. + +- **Two models behind one interface.** Plain integers go to num2words for the + value and a general Polish LLM for the case; anything with a time, date, + decimal or identifier goes to the 0.6B fine-tune. Measured on 40 real subtitle + lines, each is clearly better than the other on its half and clearly worse on + the other's. The worker reports which leg answered, and the run prints the + split. +- **Guards, not trust.** Both models can emit something worse than nothing — a + repetition loop to the token ceiling, a dropped minus that inverts a + temperature, a vanished noun, a number changed from 8250 to 8205. Each answer + is checked against the input and falls back to num2words when it fails. + +`needs_number_normalization()` duplicates the worker's own selection rule so +blocks that need nothing never cross the process boundary. The two must stay in +step; the worker's copy is the authoritative one, and a test in the normalizer +suite imports the caller and compares both on a shared sample. + The result is written to `tts_input.srt` in that video's job folder (`output/clips/