lukas/lektor-ai
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

Add subtitle dictionaries, Polish number normalizer, ComfyUI lifecycle control

Browse Source 
Text processing before TTS:

- dictionary.txt rewrites words and phrases (km/h, np., units). Rules compile
  into one alternation applied in a single pass, so replacements never chain
  and longest-literal wins. Case-sensitive, whole-word.
- A per-queue second dictionary, written by the GUI into the first video's job
  folder and merged with the shared one (--dictionary-extra).
- ALL-CAPS blocks become Title Case rather than sentence case: lowercasing
  destroyed proper nouns inside captions ("NASA - MARGO MADISON") and invited
  the normalizer to "correct" the name. Paired clips in output/clips/all-caps
  showed the TTS reads both casings the same.

normalizer/ — digits to spoken Polish, in its own venv because it needs
transformers>=5, which must not touch ComfyUI's pinned versions:

- Two backends. Plain integers go to num2words for the value and a general
  Polish LLM for the case; times, dates, decimals and identifiers go to a 0.6B
  fine-tune. Measured on 40 real subtitle lines, each is clearly better than
  the other on its half. 74% of numbered blocks take the first leg.
- Guards on every answer: runaway length, repetition loops, invalid bytes,
  leftover digits, a lost minus sign, a changed number, a lost word. A failure
  falls back to num2words, which is nominative but never wrong about the value.
- ComfyUI is asked to unload before normalizing; a normalizer that does not fit
  in VRAM is spilled to RAM silently at 5x the cost.

Backend lifecycle:

- ComfyUI is found by port through /proc, so a backend started by hand is
  managed too. SIGTERM escalates to SIGKILL, and a suspended process is killed
  outright since it never runs its handlers.
- A clip that kills the backend is retried three times, each with a different
  seed (the aborts are seed-dependent, not text-dependent), then skipped and
  reported so an overnight queue keeps moving.
- start_comfyui.sh execs python, so the script's PID is ComfyUI's.
- GUI gains ComfyUI and Queue dictionary buttons; quitting stops the backend.

--resume-from now reuses the job folder's tts_input.srt instead of reparsing
the subtitles, so block N still pairs with clip_NNNN.flac.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
This commit is contained in:
claude
2026-08-12 21:55:31 +02:00
co-authored by Claude Opus 5
parent 7e2cee02e5
commit e201aadc3e
21 changed files with 3759 additions and 52 deletions
Download Patch File Download Diff File Expand all files Collapse all files
.gitignore
+9
View File
@@ -2,6 +2,12 @@
input/ input/
output/ output/
# Number normalizer: its own venv, and 1.1 GB of gated model weights that must
# be fetched with the user's own HF token. Recreate with normalizer/setup.sh.
normalizer/venv/
normalizer/model/
normalizer/models/
# Python # Python
__pycache__/ __pycache__/
*.py[cod] *.py[cod]
@@ -9,6 +15,9 @@ __pycache__/
.venv/ .venv/
venv/ venv/
# Server credentials — never commit these
gitea-server-credentials.md
# Machine-local Claude Code permissions (intentionally not shared) # Machine-local Claude Code permissions (intentionally not shared)
.claude/settings.local.json .claude/settings.local.json
BUILD_BRIEF_pl_number_normalizer.md
+253
View File
@@ -0,0 +1,253 @@
# Build brief — Polish number normalizer (offline `transformers` component)
## Scope of this document
This brief covers **only the normalizer component**: installing it, running it
offline with plain `transformers`, and what it does. **Integration is out of
scope** — the calling app (`lektor_ai`) handles SRT parsing, decides which
subtitle blocks to normalize, and invokes this component. The user will specify
that wiring separately.
Assume the caller sends **one sentence / subtitle block at a time**, and only
sends blocks that already contain at least one digit. This component's job is to
return that text with numbers converted to spoken Polish words, per the rules
below.
## Purpose
The downstream TTS (Higgs Audio v3) mispronounces raw digits — especially long
numbers — sometimes emitting garbled non-words. This component removes that
failure by converting digits to fully spelled-out Polish words **before** the
text reaches the TTS, with correct grammatical case (the reason a neural
normalizer is used rather than a lookup table: Polish numerals decline by case,
gender, and the noun they govern).
Behaviour rules:
1. **Skip single-digit numbers.** A standalone one-digit integer (e.g. `3` in
`Mam 3 koty`) is read correctly by the TTS and must be **left as a digit**.
2. **Expand two-or-more-digit numbers** to words (`25`, `2020`, `1 350 000`).
3. Complex numeric expressions — decimals (`3,5`), times (`14:30`), dates
(`3 maja`), ratios/addresses (`12/45`), currency, units — are handled by the
model. They are not "single-digit numbers" and are not skipped.
4. A block that (after removing single digits) contains no multi-digit number is
returned unchanged.
## The model
- **Repo:** `Folx/qwen3-0.6b-pl-text-normalization` (Hugging Face). ~1.1 GB,
bfloat16, 600M params, `Qwen3ForCausalLM`. Fine-tuned specifically for Polish
TTS text normalization (numbers, dates, times, units, abbreviations).
- **GATED REPO.** The user must accept the terms on the model page and provide an
`HF_TOKEN` for the **one-time** download. The build agent cannot accept the
licence. Load the model from a **local path / local cache** at run time — never
re-download.
- **CRITICAL version requirement:** `transformers >= 5.0.0` and `torch >= 2.1.0`.
Older transformers (e.g. 4.57) tokenize the Qwen3 chat template differently and
silently produce wrong output even under greedy decode. Pin `transformers>=5.0`.
- **System prompt (fixed, required):** `Zamieniasz tekst na formę mówioną.`
- **Context:** max 512 tokens; quality degrades past ~300 characters. Input is one
short block per call, so this is fine — warn (don't truncate) on longer inputs.
- **Decoding:** greedy (`do_sample=False`), with **constrained decoding** that
bans digit tokens from the output so no un-normalized number can leak through.
## Environment & installation
Install into a **dedicated, isolated venv/conda env**, separate from the ComfyUI
/ Higgs TTS environment. Rationale: this model hard-requires `transformers>=5.0`,
and installing that into the existing working TTS environment risks disturbing
its pinned dependencies. Keep them apart; this component is invoked as its own
process.
Pinned dependencies:
- `transformers>=5.0.0`
- `torch>=2.1.0` — CUDA build matching the host (target: RTX 3090, 24 GB)
- `accelerate` — for `device_map`
- `num2words` — used only by the fallback path
One-time model fetch (network + token required):
```bash
HF_TOKEN=... huggingface-cli download Folx/qwen3-0.6b-pl-text-normalization \
--local-dir ./model
```
Run time is **offline**. Set `HF_HUB_OFFLINE=1` and `TRANSFORMERS_OFFLINE=1`, and
load the model from `./model` (or the local HF cache). Add a check that a full run
succeeds with networking disabled.
Load the model, tokenizer, and banned-token list **once** (module level) and reuse
across calls — model load dominates cost.
## Normalization logic
For a given input string:
1. **Protect single-digit numbers.** Replace each standalone one-digit integer
with a **digit-free sentinel**, remembering the mapping. The boundary set
excludes adjacent digits and numeric punctuation so decimals/times/dates are
not caught:
```python
KEEP_RE = re.compile(r'(?<![\d.,:/\-])\d(?![\d.,:/\-])')
```
Sentinels must contain **no digits** (digit tokens are banned in output, and
the check in step 2 looks for leftover digits). Use letter-indexed sentinels,
e.g. `KEEPNUMA`, `KEEPNUMB`, … .
2. **Decide.** If the protected text still contains a digit, that digit belongs
to a 2+ digit / complex number → call the model. If no digit remains, skip the
model and return the text with sentinels restored (only single digits were
present).
3. **Model call.** Build the chat prompt (fixed system prompt + protected text),
greedy decode with digit tokens banned, decode the completion.
4. **Restore** sentinels to their original single digits.
5. On any generation error, use the fallback instead of raising.
### Reference: model wrapper (adapt, then verify against the gated model)
```python
import re, torch
from transformers import AutoModelForCausalLM, AutoTokenizer
MODEL = "./model" # local, offline
SYS = "Zamieniasz tekst na formę mówioną."
tok = AutoTokenizer.from_pretrained(MODEL)
model = AutoModelForCausalLM.from_pretrained(
MODEL, torch_dtype=torch.bfloat16, device_map="auto"
).eval()
def _banned_ids(tok):
banned = set()
for tid in range(tok.vocab_size):
d = tok.decode([tid]).strip()
if d and d.isdigit():
banned.add(tid)
for sym in ["$","%","€","£","¥","°","§","@","#","&","×","÷","±",
"©","®","™","→","←","↑","↓","√","∞","≈","≠","≤","≥"]:
ids = tok.encode(sym, add_special_tokens=False)
if len(ids) == 1:
banned.add(ids[0])
return sorted(banned)
BANNED = _banned_ids(tok)
def _model_normalize(text: str) -> str:
messages = [{"role": "system", "content": SYS},
{"role": "user", "content": text}]
prompt = tok.apply_chat_template(messages, tokenize=False,
add_generation_prompt=True)
inputs = tok(prompt, return_tensors="pt").to(model.device)
n = inputs["input_ids"].shape[1]
with torch.no_grad():
out = model.generate(**inputs, max_new_tokens=400, do_sample=False,
pad_token_id=tok.pad_token_id,
suppress_tokens=BANNED)
return tok.decode(out[0][n:], skip_special_tokens=True).strip()
```
### Reference: skip / decide / restore + public function
```python
KEEP_RE = re.compile(r'(?<![\d.,:/\-])\d(?![\d.,:/\-])')
_LETTERS = "ABCDEFGHIJKLMNOPQRSTUVWXYZ"
def _sentinel(i: int) -> str:
s = ""; i += 1
while i:
i, r = divmod(i - 1, 26); s = _LETTERS[r] + s
return f"KEEPNUM{s}"
def normalize(text: str) -> str:
"""Return `text` with 2+ digit numbers spelled out in Polish; single-digit
numbers left as digits. Safe to call on any string."""
mapping = {}
def protect(m):
key = _sentinel(len(mapping)); mapping[key] = m.group(0); return key
protected = KEEP_RE.sub(protect, text)
if re.search(r'\d', protected):
try:
result = _model_normalize(protected)
except Exception:
result = _fallback(protected)
else:
result = protected
for key, digit in mapping.items():
result = result.replace(key, digit)
return result
```
### Reference: fallback (model unavailable / generation error)
Guarantees the component never raises. Degraded: nominative case only, bare
multi-digit integers only (leaves decimals/times/dates untouched rather than
mangle them). Log a warning when used.
```python
from num2words import num2words
_INT_RE = re.compile(r'(?<![\d.,:/\-])\d{2,}(?:[\u00a0 .]\d{3})*(?![\d.,:/\-])')
def _fallback(text: str) -> str:
def repl(m):
digits = re.sub(r'[\u00a0 .]', '', m.group(0))
return num2words(int(digits), lang="pl")
return _INT_RE.sub(repl, text)
```
## Error handling
- Input longer than ~300 chars: log a warning (quality drops), still process,
never truncate silently.
- Model load failure: fall back to `num2words` and log clearly that the neural
normalizer is unavailable — never crash.
- Per-call generation error: use the fallback for that call only.
## Testing / acceptance
The model produces valid case variants, so assert on **properties**, not exact
strings, except where noted.
- **Single digit preserved:** `Mam 3 koty.` → still contains `3`.
- **Multi-digit expanded:** `Wygrał 1247 złotych.` → no `1247`; contains `tysiąc`.
- **Large number:** `populacja 1 350 000 osób` → no digits from that number;
contains `milion`.
- **Mixed (key sentinel round-trip):** `Mam 3 koty i 25 psów.` → keeps `3`,
expands `25` → `dwadzieścia pięć`.
- **Only single digits:** model is not called; text returned unchanged (assert
via a spy on `_model_normalize`).
- **No digits leak:** expanded output contains no digit characters except
restored single digits.
- **Sentinel round-trip:** `KEEPNUM…` never appears in final output. **If the
model is found to alter/inflect the sentinel, flag it** — the fix is a more
inert sentinel, or dropping protection and documenting that single digits
co-occurring with multi-digit numbers get expanded.
- **Offline:** the suite passes with networking disabled, `HF_HUB_OFFLINE=1`,
`TRANSFORMERS_OFFLINE=1`.
## Deliverables
- `normalizer.py` — model/tokenizer/banned-list loaded once at import; public
`normalize(text: str) -> str`; `_fallback`; skip/decide/restore logic.
- `requirements.txt` — pinned as above.
- `README.md` — env creation, one-time gated download with `HF_TOKEN`, offline
run variables. (No caller/integration details — those are provided separately.)
- `tests/` — the checks above with sample inputs.
## Design decisions (defaults — confirm if you disagree)
1. **Single digit inside a date/time.** `3 maja` keeps its `3` (space-separated
standalone single digit), while `9:30`, `3,5`, `12/45` go to the model and are
expanded. If you want `3 maja` → `trzeciego maja`, that's a one-line change to
the boundary set.
2. **Sentinel robustness.** Small residual risk the model alters the placeholder
instead of passing it through. Encoded as an explicit test with a documented
fallback so it's caught, not shipped silently.
dictionary.txt
+49
View File
@@ -0,0 +1,49 @@
# lektor_ai_comfy — replacement dictionary
#
# Words and phrases rewritten in the subtitles before they are sent to the TTS
# engine, for anything the model reads out wrong: abbreviations, units, symbols,
# names spelled in a way it mispronounces.
#
# One rule per line:
#
# pattern = replacement
#
# * Matching is CASE-SENSITIVE and matches whole words only: "np." leaves
# "znp." alone, and "Joshua" leaves "joshua" alone.
# * The replacement is inserted exactly as written — nothing is re-capitalised
# for you. Note that the first letter of every subtitle block is already
# capitalised before these rules run, so if a word can start a block, add a
# second rule for the capitalised form.
# * Longer patterns win: "km/h" below is used even though a plain "km" rule
# would also fit, whatever order the two are written in.
# * The replacement is never re-scanned, so rules cannot feed into each other.
# * A pattern wrapped in slashes is a regular expression, and \1 \2 … in the
# replacement refer to its groups.
# * A line starting with # or ; is a comment. An empty replacement deletes the
# phrase.
# * "->" and "=>" work as separators too, if you prefer them to "=".
#
# The changes are visible in clips/tts_input.srt after a run — check there if a
# rule is not doing what you expect.
# ── Units ──────────────────────────────────────────────────────────────────
km/h = kilometrów na godzinę
m/s = metrów na sekundę
# ── Abbreviations ──────────────────────────────────────────────────────────
np. = na przykład
tzn. = to znaczy
# These two almost always close a sentence, so the dot is kept; drop it if you
# see an unwanted pause mid-sentence.
itp. = i tym podobne.
itd. = i tak dalej.
m.in. = między innymi
24/7 = dwadzieścia cztery na siedem
# ── Symbols ────────────────────────────────────────────────────────────────
/(\d+)\s*%/ = \1 procent
/(\d+)\s*°C/ = \1 stopni Celsjusza
& = i
# ── Your own rules below ───────────────────────────────────────────────────
docs/README.md
+13 -4
View File
@@ -17,6 +17,7 @@ stream-copied; only the new voiceover track is encoded.
| [usage.md](usage.md) | Every CLI flag, the working modes, and practical recipes | | [usage.md](usage.md) | Every CLI flag, the working modes, and practical recipes |
| [architecture.md](architecture.md) | How the pipeline works internally, end to end | | [architecture.md](architecture.md) | How the pipeline works internally, end to end |
| [troubleshooting.md](troubleshooting.md) | Diagnosed failure modes and their fixes | | [troubleshooting.md](troubleshooting.md) | Diagnosed failure modes and their fixes |
| [../normalizer/README.md](../normalizer/README.md) | The Polish number normalizer, its own component |
## Quick start ## Quick start
@@ -35,10 +36,11 @@ python3 lektor_ai_comfy.py "input/film.mkv" --embedded-subs
The result is `output/film_lektor.mkv` with two audio tracks — **Original** and The result is `output/film_lektor.mkv` with two audio tracks — **Original** and
**Polish** — plus every subtitle track from the source. **Polish** — plus every subtitle track from the source.
## The two halves of this project ## The three parts of this project
The pieces are deliberately decoupled and it helps to keep them apart when The pieces are deliberately decoupled — each in its own environment, talking
diagnosing anything: over a process boundary — and it helps to keep them apart when diagnosing
anything:
**ComfyUI + Higgs Audio V3** — a local HTTP service on port 8189 that turns text **ComfyUI + Higgs Audio V3** — a local HTTP service on port 8189 that turns text
into speech. Heavy: ~22 GB on disk, needs an NVIDIA GPU, lives in its own conda into speech. Heavy: ~22 GB on disk, needs an NVIDIA GPU, lives in its own conda
@@ -49,8 +51,15 @@ ComfyUI over HTTP, and does all timing and audio/video assembly with ffmpeg. It
is pure Python standard library, runs on the system Python, and needs no conda is pure Python standard library, runs on the system Python, and needs no conda
environment or pip packages of its own. environment or pip packages of its own.
**`normalizer/`** — an optional component that spells digits out as Polish words
before they reach the TTS. Its own venv (~6.9 GB, mostly CUDA libraries) and
its own 1.1 GB model, because it needs `transformers >= 5` and the ComfyUI
environment must not be disturbed. Driven as a subprocess for one stage of each
run, then closed.
So a TTS problem (bad pronunciation, hallucination, mumbling) belongs to the So a TTS problem (bad pronunciation, hallucination, mumbling) belongs to the
first half, and a timing, mixing, or muxing problem belongs to the second. first part, a timing, mixing or muxing problem to the second, and a number read
in the wrong case to the third.
## Conventions in these docs ## Conventions in these docs
docs/architecture.md
+96 -2
View File
@@ -37,23 +37,117 @@ must repeat the same `--start`/`--end`.
and `<...>` HTML tags (`<i>`, `<font …>`). and `<...>` HTML tags (`<i>`, `<font …>`).
2. **Detect dialogue.** Lines beginning `- ` mean two speakers and are kept 2. **Detect dialogue.** Lines beginning `- ` mean two speakers and are kept
separate; ordinary wrapped lines are joined with a space. separate; ordinary wrapped lines are joined with a space.
3. **Sentence-case ALL-CAPS blocks**, which the model would otherwise shout. 3. **Title-case ALL-CAPS blocks**, which the model would otherwise shout.
Every word is capitalised rather than lowercasing all but the first letter,
because the obvious version also lowercases the proper nouns inside a
caption: `NASA – MARGO MADISON` became `Nasa – margo madison`, which is wrong
to read aloud and invites the number normalizer to "correct" the name (it
produced `marga`). A/B clips showed the TTS reads both casings the same, so
keeping the names costs nothing.
4. **Capitalise a lowercase first letter**, since a fragment continuing from the 4. **Capitalise a lowercase first letter**, since a fragment continuing from the
previous subtitle confuses the model. previous subtitle confuses the model.
5. **Add terminal punctuation** so the model produces a natural falling tail. 5. **Apply the replacement dictionary** — `km/h` → `kilometrów na godzinę` and
anything else the model reads out wrong (see below).
6. **Add terminal punctuation** so the model produces a natural falling tail.
Step 1 must come first. `<i>ALL CAPS</i>` looks mixed-case while the tag is Step 1 must come first. `<i>ALL CAPS</i>` looks mixed-case while the tag is
still present — the `i` in `<i>` is a lowercase letter — so the caps detection still present — the `i` in `<i>` is a lowercase letter — so the caps detection
in step 3 would silently miss it. in step 3 would silently miss it.
Step 5 must come after step 3 for the same reason in reverse: a lowercase
replacement dropped into `JEDZIE 200 KM/H` would make the block look mixed-case
and the shouting would survive. It comes before step 6 so a rule can supply
punctuation of its own.
### The replacement dictionary
`Dictionary.load()` reads `dictionary.txt` (`[paths] dictionary`, or
`--dictionary`) into `pattern = replacement` rules — literal and
case-insensitive, or a regular expression when the pattern is wrapped in
slashes. Every rule is compiled into **one alternation** applied in a single
`re.sub` pass, which buys three properties worth keeping:
- Replacement text is never rescanned, so rules cannot chain or loop.
- Literals are ordered longest-first inside the alternation, so `km/h` beats a
plain `km` rule regardless of the order in the file.
- Rules cost one pass over the text no matter how many there are.
Literal patterns get `(?<!\w)` / `(?!\w)` guards only on edges that are word
characters, which is what lets `km/h` and `np.` match on their own while `znp.`
does not — a guard on a `/` or `.` edge would stop them matching at all.
Matching is case-sensitive and the replacement is verbatim. The alternative —
match loosely, then infer how to re-case the replacement — was tried and
removed: it silently rewrote `NASA` as `Nasa`, and a rule that fires on a case
you did not write is hard to reason about. The cost is that the first letter of
a block is capitalised before the dictionary runs, so a lowercase rule cannot
fire there; on the seven subtitle files here that changed nothing, but it is why
a word that can open a subtitle needs both forms.
A missing file, a line without a separator and an invalid regex are all
warnings, never fatal — the same rule as `settings.ini`, for the same reason: a
typo should not cost a long run.
`split_dialogue_blocks()` then divides a two-speaker block into one clip per `split_dialogue_blocks()` then divides a two-speaker block into one clip per
line, apportioning the block's duration by character count. line, apportioning the block's duration by character count.
### Number normalization (last stage)
`normalize_block_numbers()` rewrites digits as Polish words — `1247` →
`tysiąc dwieście czterdzieści siedem` — because Higgs Audio misreads long digit
strings and sometimes emits garbled non-words. It is **last** for two reasons:
the dictionary may have produced the words a numeral has to agree with (`16%` →
`16 procent` → `szesnaście procent`), and dialogue splitting decides which clip
a number ends up in.
The model that does it needs `transformers>=5`, which cannot be installed beside
ComfyUI's pinned dependencies, so it lives in `normalizer/` with its own venv and
runs as a **separate process**. `NumberNormalizer` is the client half:
- One worker per run, not per block — the 1.1 GB load dominates everything else.
It is started only if some block actually contains a number worth expanding,
and closed before step 2, so it is not holding VRAM while Higgs generates.
- **ComfyUI is asked to unload first** (`free_comfy_vram`). Higgs stays resident
between runs and leaves too little VRAM; a normalizer that does not fit is
spilled to RAM by accelerate *silently*, at about 5× the time per block. A
`POST /free` is cheaper than a restart — the server stays warm — and the wait
is on `torch_vram_total` in `/system_stats` rather than a fixed sleep, because
the flag is consumed by ComfyUI's worker loop rather than by the request.
- JSON lines over stdin/stdout, strictly one request per response. stdout is the
protocol; logs go to stderr and are kept in a 40-line ring for diagnostics.
- Reads are on a queue fed by a reader thread, which is what makes a per-block
timeout possible at all — a wedged worker cannot block the run.
- **Nothing here is fatal.** Missing venv, dead worker, timeout, out-of-order
reply: each prints a warning, disables the stage, and lets the digits through
untouched. A number read badly is a bad clip; a crash here is a lost film.
- **Two models behind one interface.** Plain integers go to num2words for the
value and a general Polish LLM for the case; anything with a time, date,
decimal or identifier goes to the 0.6B fine-tune. Measured on 40 real subtitle
lines, each is clearly better than the other on its half and clearly worse on
the other's. The worker reports which leg answered, and the run prints the
split.
- **Guards, not trust.** Both models can emit something worse than nothing — a
repetition loop to the token ceiling, a dropped minus that inverts a
temperature, a vanished noun, a number changed from 8250 to 8205. Each answer
is checked against the input and falls back to num2words when it fails.
`needs_number_normalization()` duplicates the worker's own selection rule so
blocks that need nothing never cross the process boundary. The two must stay in
step; the worker's copy is the authoritative one, and a test in the normalizer
suite imports the caller and compares both on a shared sample.
The result is written to `tts_input.srt` in that video's job folder The result is written to `tts_input.srt` in that video's job folder
(`output/clips/<video name>/`). It is the source of truth for block (`output/clips/<video name>/`). It is the source of truth for block
numbering: block *N* there is `clip_NNNN.flac` beside it and the numbering: block *N* there is `clip_NNNN.flac` beside it and the
argument to `--regen N`. argument to `--regen N`.
Which is why `--regen`, `--rebuild` **and `--resume-from`** all read that file
rather than the original SRT. Everything upstream of it — the dictionary, the
normalizer, dialogue splitting — can produce different text than it did on the
first run, and any difference breaks the pairing with the clips already on disk.
Step 1 is skipped entirely in those modes.
## 2. Speech generation ## 2. Speech generation
For each block the script POSTs a workflow graph to `/prompt`, polls For each block the script POSTs a workflow graph to `/prompt`, polls
docs/installation.md
+28 -1
View File
@@ -184,6 +184,27 @@ conda install -n comfy-tts -c conda-forge ffmpeg
Note that this puts ffmpeg inside the conda env, so the script must then be run Note that this puts ffmpeg inside the conda env, so the script must then be run
with that env active. with that env active.
## Number normalizer (optional, recommended)
Digits reach the TTS as words rather than figures, in the correct Polish case.
It has its own venv on purpose — the model needs `transformers >= 5`, which must
not be forced on the ComfyUI environment installed above.
```bash
cd ~/Programy/lektor_ai
normalizer/setup.sh # venv + torch (cu128) + transformers
HF_TOKEN=hf_... normalizer/setup.sh --model # the weights, ~1.1 GB
```
The weights are gated: accept the terms at
<https://huggingface.co/Folx/qwen3-0.6b-pl-text-normalization> while signed in,
then take a token from <https://huggingface.co/settings/tokens>. This is the only
step that touches the network; runs are offline afterwards.
Skipping this is fine — the pipeline falls back to `num2words` (nominative case
only) and says so. Turn the whole stage off with `numbers = no` under
`[normalize]` in `settings.ini`. See [normalizer/README.md](../normalizer/README.md).
## Verifying ## Verifying
```bash ```bash
@@ -198,7 +219,11 @@ python -c "import torch; print(torch.__version__, torch.cuda.is_available())"
# 3. Model present # 3. Model present
ls -la ~/comfy-tts-app/ComfyUI/models/TTS/higgs_audio_v3/higgs-audio-v3-tts-4b/model.safetensors ls -la ~/comfy-tts-app/ComfyUI/models/TTS/higgs_audio_v3/higgs-audio-v3-tts-4b/model.safetensors
# 4. End-to-end on 60 seconds of a real file # 4. Number normalizer, if installed
normalizer/venv/bin/python normalizer/normalizer.py --status
# -> "mode": "model" ("fallback" means the weights are missing)
# 5. End-to-end on 60 seconds of a real file
cd ~/Programy/lektor_ai cd ~/Programy/lektor_ai
python3 lektor_ai_comfy.py "input/film.mkv" --embedded-subs --end 01:00 python3 lektor_ai_comfy.py "input/film.mkv" --embedded-subs --end 01:00
``` ```
@@ -213,5 +238,7 @@ Nothing was installed system-wide, so removal is just deleting directories:
```bash ```bash
conda env remove -n comfy-tts conda env remove -n comfy-tts
rm -rf ~/comfy-tts-app rm -rf ~/comfy-tts-app
rm -rf ~/Programy/lektor_ai/normalizer/venv # number normalizer env
rm -rf ~/Programy/lektor_ai/normalizer/model # and its weights
rm -rf ~/miniforge3 # only if nothing else uses it rm -rf ~/miniforge3 # only if nothing else uses it
``` ```
docs/requirements.md
+32 -4
View File
@@ -6,7 +6,7 @@
|---|---|---| |---|---|---|
| GPU | NVIDIA, 12 GB VRAM | Developed on an RTX 4070 Ti SUPER (16 GB) | | GPU | NVIDIA, 12 GB VRAM | Developed on an RTX 4070 Ti SUPER (16 GB) |
| Driver | 525+ | Needed by the CUDA 12.8 wheels | | Driver | 525+ | Needed by the CUDA 12.8 wheels |
| Disk | ~35 GB free | ~22 GB installed, plus working room for output | | Disk | ~35 GB free | ~22 GB installed (+8 GB with the number normalizer) |
| RAM | 16 GB | The final mux briefly peaks around 2.6 GB | | RAM | 16 GB | The final mux briefly peaks around 2.6 GB |
The CUDA **toolkit** is not required. The PyTorch cu128 wheels bundle their own The CUDA **toolkit** is not required. The PyTorch cu128 wheels bundle their own
@@ -25,6 +25,9 @@ feature-length film.
└── models/TTS 8.8 GB Higgs Audio V3 weights └── models/TTS 8.8 GB Higgs Audio V3 weights
------ ------
21.7 GB 21.7 GB
lektor_ai/normalizer/venv 6.9 GB optional: number normalizer env
lektor_ai/normalizer/model 1.1 GB optional: its weights
``` ```
Output is roughly the size of the source video, since video is stream-copied. A Output is roughly the size of the source video, since video is stream-copied. A
@@ -49,13 +52,13 @@ regenerated without redoing a whole run (see
## Dependency surface ## Dependency surface
This is the point most worth internalising, because it makes debugging much This is the point most worth internalising, because it makes debugging much
simpler: **the two halves have almost nothing in common.** simpler: **the three parts have almost nothing in common.**
### `lektor_ai_comfy.py` — standard library only ### `lektor_ai_comfy.py` — standard library only
```python ```python
argparse json random re shutil subprocess sys threading time uuid argparse configparser json os queue random re shlex shutil subprocess
pathlib.Path urllib.request sys threading time uuid pathlib.Path urllib.request
``` ```
No third-party packages. No conda environment. No virtualenv. It runs on the No third-party packages. No conda environment. No virtualenv. It runs on the
@@ -72,6 +75,25 @@ reachable ComfyUI instance. It talks to ComfyUI over plain HTTP with
> **`requirements.txt` in the project root does not apply here.** It belongs to > **`requirements.txt` in the project root does not apply here.** It belongs to
> the older `lektor_ai.py`, which used the hosted Boson AI API and needed > the older `lektor_ai.py`, which used the hosted Boson AI API and needed
> `openai>=1.0.0`. `lektor_ai_comfy.py` requires nothing from it. > `openai>=1.0.0`. `lektor_ai_comfy.py` requires nothing from it.
> `normalizer/requirements.txt` is a different file and does apply — to the
> normalizer's own venv, never to the system Python.
### `normalizer/` — optional third environment
```
torch 2.11.0+cu128 transformers 5.15.0 accelerate 1.14.0
num2words 0.5.x (fallback path only)
```
Its own venv at `normalizer/venv` (~6.9 GB — the cu128 wheels bundle their
own CUDA runtime, again) plus `normalizer/model` (1.1 GB).
Separate from ComfyUI's environment because it hard-requires `transformers >= 5`
— 4.x tokenizes the Qwen3 chat template differently and silently produces wrong
output. `lektor_ai_comfy.py` never imports it; it starts it as a subprocess and
talks JSON lines over a pipe, so the version conflict never has to be resolved.
Entirely optional: without it the pipeline uses a `num2words` fallback, and with
`numbers = no` it is not started at all.
### ComfyUI — a large, isolated environment ### ComfyUI — a large, isolated environment
@@ -130,3 +152,9 @@ API keys, no accounts, no per-use cost.
The model is published under a research / non-commercial licence and may require The model is published under a research / non-commercial licence and may require
accepting its terms on huggingface.co, then `hf auth login`, before it will accepting its terms on huggingface.co, then `hf auth login`, before it will
download. download.
The number normalizer's model is **gated** in the same way and needs an
`HF_TOKEN` for its one-time download; see
[installation.md](installation.md#number-normalizer-optional-recommended). Runs
set `HF_HUB_OFFLINE=1` and `TRANSFORMERS_OFFLINE=1`, so it never reaches the
network afterwards.
docs/troubleshooting.md
+144
View File
@@ -203,6 +203,93 @@ Handled by `_clean_text()`, but only after tags are removed. `<i>HELLO</i>`
looked mixed-case while the tag was present, because `i` is a lowercase letter. looked mixed-case while the tag was present, because `i` is a lowercase letter.
If shouting persists, check for an unhandled tag type. If shouting persists, check for an unhandled tag type.
Such blocks are converted to Title Case (`Nasa – Margo Madison`), not to
sentence case. Lowercasing everything but the first letter also destroys the
proper nouns in a caption, and the number normalizer then tries to repair the
lowercase name — it turned `margo` into `marga`. Paired clips in
`output/clips/all-caps/` show the TTS reads both casings identically, so the
names are kept.
### A clip of an ALL-CAPS caption is cut off mid-word
Check whether the block still contains digits. Higgs truncates on raw digits far
more often than on words: the same caption generated eight times, four with
`21 MAJA 1983` and four with the date spelled out, was cut in 3 of 4 digit
versions and 0 of 4 word versions — and the digit clips ran barely half as long
as the speech required. Casing made no difference in that test.
That is the number normalizer's job, so if you see it, the block reached the TTS
without being normalized: check `numbers = yes` and the `Normalizer ready` line
in step 1.
### An abbreviation, unit or name is read out wrong
Put it in `dictionary.txt`: `km/h = kilometrów na godzinę`. The replacement
happens before the text reaches the model, and the result is visible in the job
folder's `tts_input.srt`. See [usage.md](usage.md#replacement-dictionary).
If a rule appears to do nothing, check the run's `Dictionary:` line — a rule
that failed to parse was reported there and skipped, and a rule cannot match
across a line break because each dialogue line is treated separately.
`--regen` and `--rebuild` read `tts_input.srt`, which already has the
replacements baked in, so adding a rule afterwards needs either a fresh run
from the SRT or a manual edit of `tts_input.srt` before `--regen`.
### Numbers are read as garbled non-words
That is what the number normalizer is for; check what the run printed in step 1.
- `Normalizer ready (model on cuda:0)` — it ran. Look at `tts_input.srt`: if the
digits are still there, the block may hold only standalone single digits,
which are left alone deliberately.
- `Normalizer ready (fallback)` — the model is missing or failed to load, and
`num2words` is standing in: bare integers only, nominative case only, times
and decimals untouched. Install the model, see
[usage.md](usage.md#number-normalization).
- `normalizer interpreter not found` — run `normalizer/setup.sh`.
- Nothing at all — the stage is off (`numbers = no`, or
`--no-normalize-numbers`).
For the underlying cause of a bad load, run the component directly; it prints
the real error instead of a summary:
```bash
normalizer/venv/bin/python normalizer/normalizer.py --status
normalizer/venv/bin/python normalizer/normalizer.py "Wygrał 1247 złotych."
```
### A number is expanded in the wrong grammatical case
Nominative everywhere (`dwadzieścia pięć psach` instead of `dwudziestu pięciu
psach`) means the `num2words` fallback is running, not the model — see above.
The neural model is the only path that declines correctly.
### The normalizer is unexpectedly slow
Check which device the run reported. `Normalizer ready (model on cpu)` means it
never got the GPU. If it says `cuda:0` and is still slow, it may have only got
part of it: with Higgs resident there is not enough VRAM, and `accelerate`
spills the rest to RAM without raising anything — about 5× slower.
The script prevents this by asking ComfyUI to unload before normalizing. If you
do not see `Freed N GB` in step 1, either `free_comfy_vram = no`, or ComfyUI had
nothing loaded (normal on a freshly started backend), or the call failed and
said so.
### The normalizer times out on every block
`timeout` under `[normalize]` is per block, and a block takes well under a second
on a GPU. Timing out means the model landed on the CPU or the worker is wedged.
Check `torch.cuda.is_available()` in that venv:
```bash
normalizer/venv/bin/python -c "import torch; print(torch.cuda.is_available())"
```
The stage disables itself after the first timeout rather than repeating the wait
on every remaining block, so a run costs one `timeout` at worst.
### `--embedded-subs` shows no usable tracks ### `--embedded-subs` shows no usable tracks
The video only carries image-based subtitles (`hdmv_pgs_subtitle`, The video only carries image-based subtitles (`hdmv_pgs_subtitle`,
@@ -226,6 +313,58 @@ ComfyUI is not running. Start it and wait for the model to load:
curl -s http://127.0.0.1:8189/system_stats | head -c 200 curl -s http://127.0.0.1:8189/system_stats | head -c 200
``` ```
### ComfyUI dies partway through a long queue
The run recovers by itself now, and says so:
```
[2035/2073] 02:22:59 'Joshua.'
ComfyUI went away ([Errno 104] Connection reset by peer) [1/3 for this clip].
Stopping ComfyUI (pid 530039, suspended)…
Starting ComfyUI in a new gnome-terminal window…
ComfyUI died on this clip (1/3) — retrying with a different seed.
```
The stale process is stopped before the restart, because a wedged or suspended
backend still holds the port and the GPU and the new one then cannot start. A
process in state `T` is killed outright — it never runs its signal handlers, so
SIGTERM is accepted and ignored.
**Each retry uses a different seed**, which matters: these aborts are
seed-dependent, not text-dependent. The block that killed ComfyUI three times in
a 2073-clip run generated perfectly on its own with another seed.
After three deaths on the same clip the clip is skipped and the queue continues,
with a report at the end:
```
2 clip(s) crashed ComfyUI and were skipped — the voiceover is silent there.
Edit their text in tts_input.srt, then:
--regen 2035 4180
```
Skipping only happens when the backend comes back healthy. If it does not, the
run waits for it instead — otherwise a backend that is simply not running would
cause every remaining clip to be "skipped" and produce a silent film.
### The clip that crashes ComfyUI
The abort looks like this in `comfyui.log`:
```
/pytorch/aten/src/ATen/native/cuda/Indexing.cu:1515: indexSelectSmallIndex:
Assertion `srcIndex < srcSelectDimSize` failed.
Aborted (core dumped)
```
An out-of-range index in an embedding lookup inside the Higgs audio tokenizer.
It is a CUDA device-side assert, so it cannot be caught in Python — the process
dies outright, which is why it looks like ComfyUI "will not start" when a client
is resubmitting the same clip on every restart.
`--regen N` with a different seed is the fix; rewording the block helps if it
persists.
### ComfyUI crashed and the GPU is stuck ### ComfyUI crashed and the GPU is stuck
```bash ```bash
@@ -244,6 +383,11 @@ python3 lektor_ai_comfy.py film.mkv --resume-from 312
Nothing else should be using the GPU during a run. Restart ComfyUI to clear Nothing else should be using the GPU during a run. Restart ComfyUI to clear
fragmentation. The 4B model needs roughly 10–12 GB. fragmentation. The 4B model needs roughly 10–12 GB.
The number normalizer adds ~1.5 GB, but only during step 1 — its process exits
before TTS generation starts, so the two never hold VRAM at the same time. If it
cannot fit anyway, it logs the CUDA error and finishes the run on the `num2words`
fallback rather than failing.
### `torch.cuda.is_available()` is False ### `torch.cuda.is_available()` is False
```bash ```bash
docs/usage.md
+256
View File
@@ -111,6 +111,25 @@ python3 lektor_ai_comfy.py input/film.mkv --resume-from 47
The console prints `[47/693]` style progress, so the number to resume from is The console prints `[47/693]` style progress, so the number to resume from is
the last one that completed. the last one that completed.
**It does not reprocess the subtitles.** The blocks come from the job folder's
existing `tts_input.srt`, so the dictionary and the number normalizer are not
run again:
```
[1/4] Resuming from block 47 — reusing tts_input.srt…
693 blocks, subtitles not reprocessed — the numbering still matches the clips on disk.
```
That is a correctness requirement, not an optimisation. `--resume-from` pairs
block *N* with `clip_NNNN.flac`, and re-deriving the text can change it — a
dictionary rule added since, a different normalizer result, a changed `--start`.
Same block count with different words is the dangerous case, because nothing
would look wrong until you listened.
It also means `--start`/`--end` need not be repeated for the block list, since
the saved file already holds the adjusted timings. If the job folder has no
`tts_input.srt`, the run says so and falls back to parsing the SRT.
### `--skip-tts` ### `--skip-tts`
Uses a pre-assembled audio file and goes straight to the mux. Uses a pre-assembled audio file and goes straight to the mux.
@@ -143,6 +162,14 @@ tts_volume = 1.8
[paths] [paths]
output_dir = output output_dir = output
dictionary = dictionary.txt
[normalize]
numbers = yes
python = normalizer/venv/bin/python
llm = normalizer/models/PLLuM-4B-instruct-2512
timeout = 60
free_comfy_vram = yes
[comfy] [comfy]
comfy_url = http://127.0.0.1:8189 comfy_url = http://127.0.0.1:8189
@@ -251,8 +278,237 @@ Typical starting point for dialogue-heavy material: `--orig-volume 0.5
| Flag | Default | Meaning | | Flag | Default | Meaning |
|---|---|---| |---|---|---|
| `--output-dir DIR` | `output` | Where everything is written | | `--output-dir DIR` | `output` | Where everything is written |
| `--dictionary FILE` | `dictionary.txt` | Word/phrase replacements, `""` disables |
| `--normalize-numbers` / `--no-normalize-numbers` | on | Spell digits out in Polish |
| `--comfy-url URL` | `http://127.0.0.1:8189` | Backend address | | `--comfy-url URL` | `http://127.0.0.1:8189` | Backend address |
## Replacement dictionary
`dictionary.txt` next to the script rewrites words and phrases in the subtitles
**before** they reach the TTS engine. It exists for everything the model reads
out wrong on its own: abbreviations, units, symbols, names with an unusual
spelling.
One rule per line:
```
km/h = kilometrów na godzinę
np. = na przykład
m.in. = między innymi
/(\d+)\s*%/ = \1 procent
```
| Rule | Behaviour |
|---|---|
| Matching | **Case-sensitive**, whole words only — `np.` does not match inside `znp.`, and `Joshua` does not match `joshua` |
| Capitals | The replacement goes in exactly as written; nothing is re-cased for you |
| Priority | Longer literals win: `km/h` beats a plain `km` rule whichever order they are written in |
| Chaining | Replacement text is never rescanned, so rules cannot feed each other or loop |
| Regex | A pattern in slashes is a regular expression; `\1 \2 …` refer to its groups |
| Comments | Lines starting with `#` or `;`; `->` and `=>` also work as separators |
| Deleting | An empty replacement removes the phrase |
Each run reports what it loaded and how much it changed:
```
Dictionary: 14 rule(s) from /home/lukas/Programy/lektor_ai/dictionary.txt
6 dictionary replacement(s) applied.
```
The replaced text is what lands in the job folder's `tts_input.srt` — check
there when a rule is not doing what you expect. Because `--regen` and
`--rebuild` work from that file, they do **not** re-apply the dictionary: edit
`tts_input.srt` and `--regen` the affected blocks, or rerun from the SRT.
A relative name is looked for in the current directory first and then next to
the script, so one film can keep its own list without hiding the shared one.
A missing file, a line with no separator and an invalid regex are all warnings,
never fatal.
### The queue dictionary
`dictionary.txt` holds rules that are true of Polish generally. A batch often
needs its own — a character's name the model mangles, a recurring
mistranslation — and those do not belong in the shared file.
The GUI's **Queue dictionary** button opens a plain text editor with the same
format, one rule per line:
```
Joshua = Dżoszua
P12 = pozycja dwunasta
# comments work here too
```
Saving writes `queue_dictionary.txt` into the **first queued video's job
folder**, and every video in the queue is run with it. The dialog reports how
many rules parsed, so a typo shows up when you save rather than as a rule that
silently does nothing during the run. **Clear** empties the editor only —
nothing is written until you press Save.
The two files are **merged into one rule set**, not chained: a replacement made
by one still cannot be rewritten by the other, and longest-literal-wins applies
across both. A run lists them separately:
```
Dictionary: 10 rule(s) from /home/lukas/Programy/lektor_ai/dictionary.txt
Dictionary: 2 rule(s) from output/clips/S02E01/queue_dictionary.txt
3 dictionary replacement(s) applied.
```
On the command line it is `--dictionary-extra FILE`, which works with any file
you like.
## Number normalization
Higgs Audio misreads long digit strings and sometimes emits garbled non-words,
so the **last** subtitle stage rewrites numbers as Polish words:
```
Wygrał 1247 złotych. → Wygrał tysiąc dwieście czterdzieści siedem złotych.
Mówimy o 25 psach. → Mówimy o dwudziestu pięciu psach.
Mam 3 koty. → Mam 3 koty.
```
The case matters — `25 psów` / `o 25 psach` / `z 25 psami` are three different
forms of the same numeral — which is why this is a small fine-tuned model rather
than a lookup table. Standalone single digits are left as digits: the TTS reads
those correctly, and expanding them only makes the clip longer than the subtitle
window it has to fit.
It runs **after** the dictionary, so a rule that turns `16%` into `16 procent`
feeds `szesnaście procent`, and after dialogue splitting, so the numbers are
normalized in the clip that actually says them.
| Flag | Default | Meaning |
|---|---|---|
| `--normalize-numbers` | on | Spell digits out |
| `--no-normalize-numbers` | | Leave digits alone for this run |
or in `settings.ini`:
```ini
[normalize]
numbers = no
```
### Install
The model needs `transformers >= 5`, which must not go into the ComfyUI
environment, so it lives in `normalizer/` with its own venv:
```bash
normalizer/setup.sh # venv + torch + transformers
HF_TOKEN=hf_... normalizer/setup.sh --model # the weights
```
The weights are **gated**: accept the terms at
<https://huggingface.co/Folx/qwen3-0.6b-pl-text-normalization> with your Hugging
Face account, then create a token at <https://huggingface.co/settings/tokens>.
That is the only step that needs the network; runs are offline afterwards.
Full details, including the JSON-lines protocol and the deviations from the
build brief, are in [normalizer/README.md](../normalizer/README.md).
### Two models, and which one gets a block
Numbers fail in two different ways and no single model handles both:
| block contains | goes to | because |
|---|---|---|
| plain integers only (74%) | num2words → inflection model | num2words cannot miscount; the model only fixes the case |
| times, dates, decimals, `P10`, `LO 3842` (26%) | the 0.6B fine-tune | it was trained on those formats; the other leg destroys them |
The split is why `Mówili o 250000 uchodźcach.` comes out as *o dwustu
pięćdziesięciu tysiącach uchodźców* — the fine-tune alone renders that as
*o pięciu tysiącach*, losing the number — while `Teraz na P12.` still comes out
as *P dwanaście* rather than *Pdziesięć*.
Every answer is then checked before use: runaway length, repetition loops,
invalid characters, leftover digits, a lost minus sign, a changed number, a lost
word. A block that fails falls back to num2words — nominative, but never wrong
about the value. The run reports how often that happened.
Set `llm =` (empty) to use the fine-tune for everything.
### What a run prints
```
5 block(s) contain numbers — normalizing…
Asking ComfyUI to unload its models (7.9 GB)…
Freed 7.9 GB.
Normalizer ready (model on cuda:0, two models).
5 of 5 block(s) rewritten (2 complex, 3 plain).
1 block(s) failed a safety check and fell back (word lost x1).
```
`Normalizer ready (fallback)` means the model is missing or failed to load and
`num2words` is standing in — bare integers in the nominative case only, with
times, dates and decimals left as digits. The run says so and continues.
### Sharing the GPU with ComfyUI
Higgs stays resident between runs — about 8.4 GB of a 16 GB card — which leaves
too little for the normalizer. That failure is silent rather than loud:
`accelerate` answers a shortage by spilling layers to RAM, so the run continues
with no error and roughly **5× the time per block** (measured: 1.19 s → 5.96 s).
So before starting the normalizer the script asks ComfyUI to unload:
```
5 block(s) contain numbers — normalizing…
Asking ComfyUI to unload its models (7.9 GB)…
Freed 7.9 GB.
Normalizer ready (model on cuda:0).
```
ComfyUI reloads Higgs on the first clip of step 2 — 20–30 s, once per run. It is
a `POST /free` on ComfyUI's own API, not a restart: the server stays up, with
its custom nodes loaded. The unload is asynchronous, so the script waits until
`torch_vram_total` in `/system_stats` actually drops before continuing.
`--regen` and `--rebuild` work from `tts_input.srt`, which is already
normalized, so they never normalize and never pay the reload. On a freshly
started ComfyUI the call is a no-op and prints nothing.
Set `free_comfy_vram = no` to skip it — worth doing only if you have VRAM to
spare, or if you have turned the neural normalizer off.
Nothing about this stage is fatal. A missing venv, a worker that dies, a block
that takes longer than `timeout` — each is a warning, and the affected text goes
to the TTS with its digits intact. The worker is started once per run and exits
before TTS generation begins, so it is not holding VRAM while Higgs works.
## The backend's lifecycle
ComfyUI is normally started once and left running. The script manages it when it
has to:
| situation | what happens |
|---|---|
| not running when a clip is submitted | started via `start_comfyui.sh`, waited for, the clip resubmitted |
| running but wedged or suspended | stopped (SIGKILL if suspended) before the restart |
| dies on a clip | up to 3 retries, each with a **different seed** |
| dies 3 times on the same clip | that clip is skipped, the queue continues, all skips reported at the end |
| queue finished | models unloaded, backend left running — it reloads on the next request |
| GUI closed | backend stopped, so no idle process is left holding the GPU |
Its output goes to `comfyui.log` next to the script as well as to the terminal
window, so a crash can be diagnosed after the fact.
The GUI has a **ComfyUI** button showing the same state: `Start ComfyUI` when it
is down, `ComfyUI ●` when it is answering, `ComfyUI ◍` when it holds the port but
does not answer. Pressing it starts or restarts.
Quitting the GUI stops the backend, including one you started by hand in a
terminal — it is found by port, not by remembering what was launched. Between
runs it stays up on purpose: the queue only unloads its models, so the next job
does not pay for a cold start.
`start_comfyui.sh` ends with `exec python main.py`, so the script's PID *is*
ComfyUI's. Without that the launcher only ever sees a wrapper shell, and killing
it orphans the python still holding 8 GB of VRAM.
## Output ## Output
Everything belonging to one video is kept together in a **job folder** named Everything belonging to one video is kept together in a **job folder** named
lektor_ai_comfy.py
+905 -34
View File
File diff suppressed because it is too large Load Diff
normalizer/README.md
+179
View File
@@ -0,0 +1,179 @@
# Polish number normalizer
Rewrites digits as spoken Polish words before the text reaches the TTS:
```
Wygrał 1247 złotych. → Wygrał tysiąc dwieście czterdzieści siedem złotych.
Mówimy o 25 psach. → Mówimy o dwudziestu pięciu psach.
Mam 3 koty. → Mam 3 koty. (nothing worth sending)
Brakuje nam 15000 podpisów. → Brakuje nam piętnastu tysięcy podpisów.
```
Higgs Audio v3 mispronounces raw digits — especially long ones — and sometimes
emits garbled non-words. A lookup table cannot fix that, because Polish numerals
decline by case, gender and the noun they govern (`25 psów` / `o 25 psach` /
`z 25 psami`), so this uses a small fine-tuned model.
It runs in **its own venv**, separate from ComfyUI: the models require
`transformers >= 5`, and forcing that on the working TTS install would risk its
pinned dependencies. `lektor_ai_comfy.py` drives it as a subprocess, and asks
ComfyUI to unload Higgs first so there is VRAM to work with.
## Install
```bash
normalizer/setup.sh # venv + torch (cu128) + transformers + num2words
```
Then the model, which is **gated** — accept the terms at
<https://huggingface.co/Folx/qwen3-0.6b-pl-text-normalization> with your Hugging
Face account first, then:
```bash
HF_TOKEN=hf_... normalizer/setup.sh --model
```
The inflection model is separate and **not gated**:
```bash
normalizer/venv/bin/hf download CYFRAGOVPL/PLLuM-4B-instruct-2512 \
--local-dir normalizer/models/PLLuM-4B-instruct-2512
```
Those are the only steps that need the network. Everything afterwards runs
offline (`HF_HUB_OFFLINE=1`, `TRANSFORMERS_OFFLINE=1`, both set by the caller).
Without the inflection model every block goes to the fine-tune, which is what
the pipeline did before it existed.
Without the model the component still works, on a `num2words` fallback that
handles bare integers in the nominative case only. It says so in the log, and
`lektor_ai_comfy.py` prints a warning for the run.
## Check it
```bash
normalizer/venv/bin/python normalizer/normalizer.py --status
normalizer/venv/bin/python normalizer/normalizer.py "Wygrał 1247 złotych."
normalizer/venv/bin/python -m unittest discover -s normalizer/tests -v
```
The tests assert on properties (`no 1247 left`, `contains tysiąc`) rather than
exact strings, because several case variants are correct. The ones that need the
model skip themselves when it is not downloaded.
## What it does
1. **Skip what needs nothing.** A block whose only numbers are standalone single
digits is never sent: the TTS says "pięć minut" for "5 minut" by itself, so
there is nothing to gain and a whole sentence of rewriting to risk.
2. **Route.** Two models with opposite weaknesses, measured on 40 real subtitle
lines and a 32-case battery:
| | good at | bad at |
|---|---|---|
| fine-tune (0.6B) | `6:05`, `P10`, `LO 3842`, `95,7`, dates | loses a large number's value while declining it |
| num2words + LLM | the value, always; the case | formats — a flight number becomes a quantity |
So a block goes to the second leg only when **every** number in it is a plain
integer — 74% of numbered blocks on the sample. Everything else goes to the
fine-tune.
3. **Generate.** The fine-tune gets the text as-is. The other leg gets text with
no digits at all — num2words has already produced correct nominative words —
so its only job is inflection, and it cannot get a value wrong because it
never reads a number. Greedy decoding, digit tokens banned, either way.
4. **Guard.** The answer is checked before it is used (below). A block that
fails falls back to num2words: dull, nominative, never wrong about the value.
## The guards
Both models can produce output worse than doing nothing. Each of these fired on
a real failure during development:
| check | the failure it caught |
|---|---|
| runaway length | a 10-word line came back as 200 words |
| repetition loop | `łącznie łącznie łącznie…` to the token ceiling |
| invalid characters | a wall of `�` replacement bytes |
| digits left | an un-normalized number slipped through |
| minus sign lost | `-12 stopni` → *dwanaście stopni*, inverting a temperature |
| numbers changed | `Z 8250 osobami` → *osiem tysięcy **dwustu pięciu*** (8205), and `20 lat` → *dwadzieścia **pięć** lat* (25) |
| word lost | `38 000 000 mieszkańców` → *trzydzieści osiem milionów* |
The value check compares against what num2words would say, **in both
directions** — a missing numeral and an added one are equally wrong, and a
subset check passed `dwadzieścia` → `dwadzieścia pięć` while turning 20 into 25.
Numerals are matched through a written-out canonical map rather than string
similarity — `dwieście`
and `dwustu` are one number and must match, `trzy` and `trzysta` differ by a
hundred and must not, and ordinals like `piąty` count as `pięć`. Consonant
folding was tried first and rejected good output.
It is skipped for identifiers: `LO 3842` is correctly read digit by digit, and
demanding num2words' rendering would reject a right answer.
### Deviations from the build brief, and why
- **Single digits are no longer shielded.** The brief asks for standalone
one-digit numbers to be hidden behind `KEEPNUM…` placeholders. Measured, that
scored *worse* — 6/9 against 7/9 on the model card's own examples — and it
produced an outright pronunciation error, because `3 maja` left as a digit is
read *"trzy maja"* where Polish wants *"trzeciego maja"*. It also bought
nothing on length: the TTS speaks `5` as *pięć* either way. What survives is
the cheaper half — a block containing **only** standalone single digits is
never sent at all.
- **`KEEP_RE` has extra guards.** The brief's version protects the leading `1`
of `1 350 000` as if it were standalone, and its boundary set `(?![\d.,:/-])`
rejects any number followed by a comma or full stop — on real subtitles that
skipped most of them (`Przód 85, tył 75.`). Numeric punctuation now only
continues a number when a digit follows it.
- **Two models, not one.** The brief scopes a single normalizer. The 0.6B
fine-tune performs to its advertised 85.7%, but its tail includes losing a
number's value while declining it, which no amount of prompting fixes. See
the routing table above.
- **The model loads lazily, not at import.** Same "load once, reuse" effect, but
`--help`, `--status` and the non-neural half of the suite do not pay for a
1.1 GB load they never use.
- **`num2words` errors are caught broadly.** It raises `KeyError`, not
`ValueError`, past its largest scale word.
### Known limits
Measured, and not fixable at this layer:
| input | output | |
|---|---|---|
| `Mieszka pod 12/45.` | *sto czternaście łamane przez trzy* | an address has no reliable reading without context |
| `Zadzwoń pod 601 234 567.` | read as one 9-digit number | a phone number is indistinguishable from a grouped number |
| `Zyskują 0,3 sekundy` | *trzy setne* | should be *dziesiąte*; the value guard cannot see inside a decimal |
| `To dzieło z XIX wieku.` | unchanged | Roman numerals are not handled — use `dictionary.txt` |
| `Prędkość 0,75 macha.` | unchanged | the model hallucinated a unit conversion; the guard caught it and fell back |
## Files
| File | |
|---|---|
| `normalizer.py` | The component: routing, both backends, guards, `normalize()` |
| `worker.py` | JSON-lines service `lektor_ai_comfy.py` talks to (models loaded once per run) |
| `bench.py` | Compares models on the same battery — `--compare a.json b.json` |
| `setup.sh` | venv, dependencies, one-time gated model download |
| `tests/` | Routing, numerals, guards, and the brief's acceptance checks |
| `model/`, `models/`, `venv/` | Weights and environment — not in git |
## Interface
```python
from normalizer import normalize, needs_normalization
needs_normalization("Mam 3 koty.") # False — nothing worth sending
normalize("Wygrał 1247 złotych.") # never raises, never returns None
```
`worker.py` speaks one JSON object per line on stdin/stdout, with logs on
stderr:
```
← {"ready": true, "mode": "model", "device": "cuda:0", "llm": true}
→ {"id": 1, "text": "Wygrał 1247 złotych."}
← {"id": 1, "text": "Wygrał tysiąc dwieście…", "changed": true,
"leg": "complex", "guard": "", "error": null}
```
normalizer/bench.py
+251
View File
@@ -0,0 +1,251 @@
#!/usr/bin/env python3
"""Compare normalizer models on the same sentences.
The shipped model is a 0.6B task-specific fine-tune that needs no prompt but
miscounts large numbers and occasionally drops a word. This runs any causal LM
over the same battery so the trade is measurable rather than argued about:
python bench.py # the fine-tune
python bench.py --model models/PLLuM-4B-instruct-2512 --prompted
python bench.py --model … --prompted --save pllum.json
python bench.py --compare fine-tune.json pllum.json
`--prompted` switches from the fine-tune's bare "text in, text out" contract to
an instruction + few-shot prompt, which a general instruct model needs. Digit
tokens are banned from the output either way, and decoding is greedy, so a
difference between two runs is the model and not the sampling.
Automatic flags are cheap heuristics, not a grade: `digits` means an
un-normalized digit survived, `lost` means a word from the input has no
plausible inflected match in the output. Read the text before believing either.
"""
import argparse
import json
import re
import sys
import time
from pathlib import Path
HERE = Path(__file__).resolve().parent
sys.path.insert(0, str(HERE))
import normalizer as nz # noqa: E402 (reuses its protect/restore and banned list)
SYSTEM_PROMPTED = (
"Jesteś narzędziem do normalizacji tekstu dla syntezatora mowy. "
"Zamieniasz wszystkie liczby zapisane cyframi na słowa, w poprawnej formie "
"gramatycznej wynikającej z kontekstu zdania. Nie zmieniasz, nie dodajesz "
"ani nie usuwasz żadnych innych słów. Zwracasz wyłącznie przekształcone "
"zdanie, bez komentarza."
)
# Few-shot examples chosen to pin down the three things a general model gets
# wrong without them: the case must follow the governing word, the rest of the
# sentence must survive verbatim, and the answer must be the sentence alone.
FEWSHOT = [
("Koncert dla 8250 osób.",
"Koncert dla ośmiu tysięcy dwustu pięćdziesięciu osób."),
("Populacja Polski to 38 000 000 mieszkańców.",
"Populacja Polski to trzydzieści osiem milionów mieszkańców."),
("Mówimy o 25 psach.",
"Mówimy o dwudziestu pięciu psach."),
("Wygrał 1247 złotych.",
"Wygrał tysiąc dwieście czterdzieści siedem złotych."),
]
BATTERY = [
"Sprawa dotyczy 1 350 000 ludzi.",
"Koncert dla 8250 osób.",
"Budżet wynosi 12 500 000 złotych.",
"Populacja Polski to 38 000 000 mieszkańców.",
"Zapłacili 1 500 000 000 dolarów.",
"Zostało 999 999 sztuk.",
"Numer seryjny 1234567.",
"Ponad 100 000 widzów na stadionie.",
"Mówimy o 1 350 000 ludziach.",
"Z 8250 osobami nie da się dyskutować.",
"Firma zatrudnia 2 500 pracowników.",
"Dystans 42 195 metrów.",
"Cena spadła z 15 700 000 do 8 250 000.",
"Było tam 25 psów.",
"Mówimy o 25 psach.",
"Przyjechał z 25 psami.",
"Przyglądał się 25 psom.",
"Nie ma tu 25 psów.",
"Urodził się 3 maja 1791 roku.",
"Wzrost o 16 procent w 2026 roku.",
"Spotkanie o 14:30.",
"Godzina 23:56.",
"Radio 95,7 FM.",
"Około 3,5 metra.",
"Mieszka pod 12/45.",
"Wygrał 1247 złotych.",
"To kosztuje 1999 dolarów i 99 centów.",
"Mam 3 koty i 25 psów.",
"Zostało 5 minut.",
"Temperatura opon w porządku. Przód 85, tył 75.",
"Louis Chiron miał 57 lat, kiedy wygrał Grand Prix Monako.",
"Zwykłe zdanie bez żadnych liczb.",
]
_WORD_RE = re.compile(r'[^\W\d_]+', re.UNICODE)
def lost_words(src: str, out: str) -> list:
"""Input words with no plausible match in the output.
Polish inflects, so an exact comparison is useless — `ludziach` legitimately
becomes `ludziach` and nothing else, but `psach`/`psami` share a stem. A
five-character prefix is a crude stand-in for a lemma and is good enough to
notice a noun that vanished entirely, which is the failure worth catching.
"""
def stems(text):
return {w.lower()[:5] for w in _WORD_RE.findall(text)}
missing = stems(src) - stems(out)
return sorted(w for w in missing if len(w) >= 4)
def load_prompted(model_dir: str, gpu_gib: float = None,
system: str = None, fewshot: list = None):
"""A general instruct model: system prompt, few-shot, greedy, digits banned.
`gpu_gib` caps GPU use and spills the rest to RAM. Needed for the int4
"awq" builds: transformers decompresses compressed-tensors weights back to
bfloat16 at load, so an 11B model wants ~22 GB whatever the file size says.
Offloading is slow but it is the difference between measuring a model and
guessing about it.
"""
import torch
from transformers import AutoModelForCausalLM, AutoTokenizer
tok = AutoTokenizer.from_pretrained(model_dir)
kwargs = {"dtype": torch.bfloat16, "device_map": "auto"}
if gpu_gib == 0:
# Everything in RAM. Slow, but the only way to run a compressed-tensors
# int4 build in transformers: its decompression hook expands weights to
# bfloat16 in place, and it cannot read the meta tensors that accelerate
# leaves behind when max_memory forces a partial offload.
kwargs["device_map"] = {"": "cpu"}
elif gpu_gib:
kwargs["max_memory"] = {0: f"{gpu_gib}GiB", "cpu": "80GiB"}
model = AutoModelForCausalLM.from_pretrained(model_dir, **kwargs).eval()
banned = set()
for tid in range(len(tok)):
piece = tok.decode([tid]).strip()
if piece and piece.isdigit():
banned.add(tid)
banned = sorted(banned)
print(f" {len(banned)} digit tokens banned; device "
f"{next(model.parameters()).device}", file=sys.stderr)
def run(text: str) -> str:
messages = [{"role": "system", "content": system or SYSTEM_PROMPTED}]
for user, assistant in (fewshot or FEWSHOT):
messages.append({"role": "user", "content": user})
messages.append({"role": "assistant", "content": assistant})
messages.append({"role": "user", "content": text})
prompt = tok.apply_chat_template(messages, tokenize=False,
add_generation_prompt=True)
inputs = tok(prompt, return_tensors="pt").to(model.device)
n = inputs["input_ids"].shape[1]
with torch.no_grad():
out = model.generate(**inputs, max_new_tokens=nz.MAX_NEW_TOKENS,
do_sample=False,
pad_token_id=tok.pad_token_id or tok.eos_token_id,
suppress_tokens=banned)
answer = tok.decode(out[0][n:], skip_special_tokens=True).strip()
# Instruct models like to introduce themselves; keep the last non-empty
# line, which is the sentence in every observed case.
lines = [l.strip() for l in answer.splitlines() if l.strip()]
return lines[-1] if lines else answer
return run
def main() -> int:
ap = argparse.ArgumentParser(description=__doc__,
formatter_class=argparse.RawDescriptionHelpFormatter)
ap.add_argument("--model", default=None,
help="Model directory (default: the shipped fine-tune)")
ap.add_argument("--prompted", action="store_true",
help="Use an instruction + few-shot prompt (general models)")
ap.add_argument("--gpu-gib", type=float, default=None, metavar="N",
help="Cap GPU memory and spill the rest to RAM (slow, but "
"the only way to run a model that does not fit)")
ap.add_argument("--no-protect", action="store_true",
help="Do not shield standalone single digits — normalize the "
"text exactly as the model card does")
ap.add_argument("--save", metavar="FILE", help="Write results as JSON")
ap.add_argument("--compare", nargs=2, metavar=("A", "B"),
help="Print two saved runs side by side and exit")
args = ap.parse_args()
if args.compare:
a, b = (json.loads(Path(f).read_text(encoding="utf-8")) for f in args.compare)
agree = 0
for ra, rb in zip(a["results"], b["results"]):
if ra["out"] == rb["out"]:
agree += 1
continue
print(f" in : {ra['in']}")
print(f" A : {ra['out']}")
print(f" B : {rb['out']}\n")
print(f"identical: {agree}/{len(a['results'])} "
f"A={a['model']} B={b['model']}")
return 0
if args.prompted:
if not args.model:
ap.error("--prompted needs --model")
run = load_prompted(args.model, args.gpu_gib)
label = args.model
else:
if args.model:
nz.MODEL_DIR = Path(args.model)
if not nz.load_model():
print("model unavailable; refusing to benchmark the fallback",
file=sys.stderr)
return 1
run = nz.normalize
label = str(nz.MODEL_DIR)
if args.no_protect:
nz.KEEP_RE = re.compile(r'(?!x)x') # matches nothing
label += " [no single-digit protection]"
results, t0 = [], time.time()
for text in BATTERY:
t = time.time()
out = run(text)
results.append({"in": text, "out": out, "sec": round(time.time() - t, 2),
"digits": bool(nz.DIGIT_RE.search(nz.KEEP_RE.sub("", out))),
"lost": lost_words(text, out)})
total = time.time() - t0
for r in results:
flags = []
if r["digits"]:
flags.append("DIGITS LEFT")
if r["lost"]:
flags.append("LOST: " + ", ".join(r["lost"]))
print(f" in : {r['in']}")
print(f" out: {r['out']}" + (f" [{'; '.join(flags)}]" if flags else ""))
print()
n_dig = sum(r["digits"] for r in results)
n_lost = sum(bool(r["lost"]) for r in results)
print(f"{label}\n {len(results)} sentences in {total:.1f}s "
f"({total/len(results):.2f}s each)\n"
f" {n_dig} with digits left, {n_lost} with a word apparently lost")
if args.save:
Path(args.save).write_text(json.dumps(
{"model": label, "prompted": args.prompted, "seconds": round(total, 1),
"results": results}, ensure_ascii=False, indent=1), encoding="utf-8")
print(f" saved to {args.save}")
return 0
if __name__ == "__main__":
sys.exit(main())
normalizer/normalizer.py
Executable
+645
View File
@@ -0,0 +1,645 @@
#!/usr/bin/env python3
"""Polish number normalizer — digits to spoken Polish words, for TTS.
Higgs Audio v3 mispronounces raw digits, especially long ones, and sometimes
emits garbled non-words. This component rewrites them as fully spelled-out
Polish, in the right grammatical case — which is why it is a neural model and
not a lookup table: Polish numerals decline by case, gender and the noun they
govern.
>>> normalize("Wygrał 1247 złotych.")
'Wygrał tysiąc dwieście czterdzieści siedem złotych.'
>>> normalize("Mam 3 koty.") # nothing worth sending: left alone
'Mam 3 koty.'
Model: Folx/qwen3-0.6b-pl-text-normalization, loaded from a local directory and
run offline. If it is unavailable the component degrades to a num2words
fallback and says so in the log — it never raises and never crashes the caller.
Run `python normalizer.py "some text"` for a quick check, or `python worker.py`
for the JSON-lines service that lektor_ai_comfy.py talks to.
"""
import json
import logging
import os
import re
import sys
from pathlib import Path
LOG = logging.getLogger("pl_normalizer")
HERE = Path(__file__).resolve().parent
MODEL_DIR = Path(os.environ.get("PL_NORMALIZER_MODEL", HERE / "model"))
# "auto" uses the GPU when one is free, "cpu" forces RAM. Forcing CPU is a real
# use case, not a fallback: ComfyUI keeps ~8.4 GB of VRAM resident between runs,
# so a normalizer that insists on the GPU either fights it or has to wait.
DEVICE = os.environ.get("PL_NORMALIZER_DEVICE", "auto").strip().lower()
# The second model. The 0.6B fine-tune reads odd formats well — clock times,
# race positions, call signs — but loses the value of a large number while
# declining it ("z 8250 osobami" came back as 8205). A general Polish LLM has
# the opposite profile. Neither is asked to do the other's job; see route().
LLM_DIR = Path(os.environ.get("PL_NORMALIZER_LLM",
HERE / "models" / "PLLuM-4B-instruct-2512"))
# Fixed by the fine-tune — the model was trained with this exact system prompt.
SYSTEM_PROMPT = "Zamieniasz tekst na formę mówioną."
MAX_NEW_TOKENS = 400
LONG_INPUT_CHARS = 300 # quality degrades past this; warn, never truncate
# Where a number begins and ends. Numeric punctuation only continues a number
# when a digit follows it, which is what separates "3,5" and "14:30" (one number
# each, for the model to handle) from "Przód 85, tył 75." (two, each ending at
# ordinary sentence punctuation).
#
# The build brief writes these as (?<![\d.,:/\-]) and (?![\d.,:/\-]), which also
# rejects every number followed by a comma or a full stop — on real subtitles
# that silently skipped most of them.
_NUM_START = r'(?<!\d)(?<!\d[.,:/\-])'
_NUM_END = r'(?!\d)(?![.,:/\-]\d)'
# A standalone one-digit integer is pronounced correctly by the TTS, so it stays
# a digit; anything longer or more complex goes to the model.
#
# The two space guards are not in the brief either, and are there because
# without them "1 350 000" has its leading 1 protected as if it were standalone:
# the model then sees "KEEPNUMA 350 000" and returns "1 trzysta pięćdziesiąt
# tysięcy" — the million silently lost. A digit that sits next to a group of
# exactly three is part of a grouped number, not a number of its own.
KEEP_RE = re.compile(_NUM_START + r'(?<!\d[\u00a0\u202f ])'
r'\d'
+ _NUM_END + r'(?![\u00a0\u202f ]\d{3}(?!\d))')
DIGIT_RE = re.compile(r'\d')
# Fallback path only: integers of 2+ digits, or any number written in groups of
# three. Decimals, times and dates are deliberately left alone rather than
# mangled without the model's grammar.
_INT_RE = re.compile(_NUM_START +
r'(?:\d{1,3}(?:[\u00a0\u202f .]\d{3})+|\d{2,})'
+ _NUM_END)
# Banned alongside digits so the model cannot emit a symbol the TTS would read
# as a word of the wrong language, or not read at all.
_BANNED_SYMBOLS = ["$", "%", "€", "£", "¥", "°", "§", "@", "#", "&", "×", "÷", "±",
"©", "®", "™", "→", "←", "↑", "↓", "√", "∞", "≈", "≠", "≤", "≥"]
# The instruction the general model gets. It never sees a digit — num2words has
# already produced correct nominative words — so its only job is to inflect
# them. That is why it cannot get a value wrong: it is not reading numbers.
SYSTEM_INFLECT = (
"Otrzymujesz polskie zdanie, w którym liczebniki są już zapisane słowami, "
"ale w mianowniku. Popraw wyłącznie formę gramatyczną tych liczebników tak, "
"aby pasowały do zdania (przypadek, rodzaj, liczba). Nie zmieniaj, nie "
"dodawaj ani nie usuwaj żadnych innych słów. Zwróć samo zdanie."
)
# Pins down the three things it gets wrong unprompted: the case must follow the
# governing word, everything else must survive verbatim, and the answer is the
# sentence alone. The last example is a no-op on purpose — without it the model
# invents changes when none are needed.
FEWSHOT_INFLECT = [
("Mówili o dwieście pięćdziesiąt tysięcy uchodźcach.",
"Mówili o dwustu pięćdziesięciu tysiącach uchodźców."),
("Koncert dla osiem tysięcy dwieście pięćdziesiąt osób.",
"Koncert dla ośmiu tysięcy dwustu pięćdziesięciu osób."),
("Przyjechał z dwadzieścia pięć psami.",
"Przyjechał z dwudziestoma pięcioma psami."),
("Kosztuje tysiąc dwieście czterdzieści siedem złotych.",
"Kosztuje tysiąc dwieście czterdzieści siedem złotych."),
]
_state = {"loaded": False, "tok": None, "model": None, "banned": None, "error": None}
_llm = {"loaded": False, "tok": None, "model": None, "banned": None, "error": None}
# Counted per process and reported by the worker, so a run can say which leg did
# the work and how often a guard had to step in.
STATS = {"plain": 0, "complex": 0, "guarded": 0, "reasons": {}}
# ---------------------------------------------------------------------------
# Model
# ---------------------------------------------------------------------------
def _banned_ids(tok, cache_dir: Path = None) -> list:
"""Token ids that may never appear in the output.
Scanning the whole vocabulary means one decode call per token — a few
seconds for Qwen3's 150k entries — so the result is cached beside the model.
"""
cache = (cache_dir or MODEL_DIR) / "banned_tokens.json"
size = len(tok)
if cache.is_file():
try:
data = json.loads(cache.read_text(encoding="utf-8"))
if data.get("vocab_size") == size:
return data["ids"]
except (OSError, ValueError, KeyError):
pass # stale or corrupt — just recompute
banned = set()
for tid in range(size):
piece = tok.decode([tid]).strip()
if piece and piece.isdigit():
banned.add(tid)
for sym in _BANNED_SYMBOLS:
ids = tok.encode(sym, add_special_tokens=False)
if len(ids) == 1:
banned.add(ids[0])
ids = sorted(banned)
try:
cache.write_text(json.dumps({"vocab_size": size, "ids": ids}), encoding="utf-8")
except OSError:
pass # cache is an optimisation, not a requirement
return ids
def load_model() -> bool:
"""Load model, tokenizer and banned-token list once, and keep them.
Returns True when the neural path is available. A failure here is not an
error: it is logged and the component runs on the fallback for the rest of
the process. Loading is lazy rather than at import so that tests, --help and
fallback-only runs do not pay for a 1.1 GB model they never use.
"""
if _state["loaded"]:
return _state["model"] is not None
_state["loaded"] = True
if not (MODEL_DIR / "config.json").is_file():
_state["error"] = (f"no model in {MODEL_DIR} — see README.md for the "
f"one-time download")
LOG.warning("Neural normalizer unavailable (%s). Falling back to num2words: "
"numbers will be spelled out in the nominative case only.",
_state["error"])
return False
try:
import torch
from transformers import AutoModelForCausalLM, AutoTokenizer
tok = AutoTokenizer.from_pretrained(str(MODEL_DIR))
placement = {"": "cpu"} if DEVICE == "cpu" else "auto"
try:
model = AutoModelForCausalLM.from_pretrained(
str(MODEL_DIR), dtype=torch.bfloat16, device_map=placement)
except TypeError:
# transformers < 5 spells the same argument torch_dtype.
model = AutoModelForCausalLM.from_pretrained(
str(MODEL_DIR), torch_dtype=torch.bfloat16, device_map=placement)
model.eval()
_state["tok"], _state["model"] = tok, model
_state["banned"] = _banned_ids(tok)
LOG.info("Model loaded from %s on %s (%d banned tokens)",
MODEL_DIR, next(model.parameters()).device, len(_state["banned"]))
return True
except Exception as exc: # missing torch, no VRAM, corrupt download…
_state["tok"] = _state["model"] = None
_state["error"] = f"{type(exc).__name__}: {exc}"
LOG.warning("Could not load the neural normalizer (%s). Falling back to "
"num2words: numbers will be spelled out in the nominative "
"case only.", _state["error"])
return False
def model_available() -> bool:
return load_model()
def status() -> dict:
"""What the caller needs to report: which path is live, and why."""
ok = load_model()
device = ""
if ok:
try:
device = str(next(_state["model"].parameters()).device)
except Exception:
device = "unknown"
return {"mode": "model" if ok else "fallback",
"model_dir": str(MODEL_DIR),
"device": device,
"error": _state["error"]}
def _model_normalize(text: str) -> str:
import torch
tok, model = _state["tok"], _state["model"]
messages = [{"role": "system", "content": SYSTEM_PROMPT},
{"role": "user", "content": text}]
prompt = tok.apply_chat_template(messages, tokenize=False,
add_generation_prompt=True)
inputs = tok(prompt, return_tensors="pt").to(model.device)
n = inputs["input_ids"].shape[1]
with torch.no_grad():
out = model.generate(**inputs,
max_new_tokens=MAX_NEW_TOKENS,
do_sample=False,
pad_token_id=tok.pad_token_id or tok.eos_token_id,
suppress_tokens=_state["banned"])
return tok.decode(out[0][n:], skip_special_tokens=True).strip()
def load_llm() -> bool:
"""Load the general Polish LLM used for the inflection leg.
Optional: without it every block goes to the fine-tune, which is what the
pipeline did before. Reported, never fatal.
"""
if _llm["loaded"]:
return _llm["model"] is not None
_llm["loaded"] = True
if not (LLM_DIR / "config.json").is_file():
_llm["error"] = f"no model in {LLM_DIR}"
LOG.warning("Inflection model unavailable (%s) — large numbers will be "
"normalized by the fine-tune alone.", _llm["error"])
return False
try:
import torch
from transformers import AutoModelForCausalLM, AutoTokenizer
tok = AutoTokenizer.from_pretrained(str(LLM_DIR))
placement = {"": "cpu"} if DEVICE == "cpu" else "auto"
model = AutoModelForCausalLM.from_pretrained(
str(LLM_DIR), dtype=torch.bfloat16, device_map=placement).eval()
_llm["tok"], _llm["model"] = tok, model
_llm["banned"] = _banned_ids(tok, cache_dir=LLM_DIR)
LOG.info("Inflection model loaded from %s on %s",
LLM_DIR, next(model.parameters()).device)
return True
except Exception as exc:
_llm["tok"] = _llm["model"] = None
_llm["error"] = f"{type(exc).__name__}: {exc}"
LOG.warning("Could not load the inflection model (%s) — large numbers "
"will be normalized by the fine-tune alone.", _llm["error"])
return False
def _llm_inflect(text: str) -> str:
import torch
tok, model = _llm["tok"], _llm["model"]
messages = [{"role": "system", "content": SYSTEM_INFLECT}]
for user, assistant in FEWSHOT_INFLECT:
messages.append({"role": "user", "content": user})
messages.append({"role": "assistant", "content": assistant})
messages.append({"role": "user", "content": text})
prompt = tok.apply_chat_template(messages, tokenize=False,
add_generation_prompt=True)
inputs = tok(prompt, return_tensors="pt").to(model.device)
n = inputs["input_ids"].shape[1]
with torch.no_grad():
out = model.generate(**inputs, max_new_tokens=MAX_NEW_TOKENS,
do_sample=False,
pad_token_id=tok.pad_token_id or tok.eos_token_id,
suppress_tokens=_llm["banned"])
answer = tok.decode(out[0][n:], skip_special_tokens=True).strip()
# Instruct models like to introduce their answer; the sentence is the last
# non-empty line in every case observed.
lines = [l.strip() for l in answer.splitlines() if l.strip()]
return lines[-1] if lines else answer
# ---------------------------------------------------------------------------
# Fallback
# ---------------------------------------------------------------------------
def _fallback(text: str) -> str:
"""Model unavailable. Nominative case only, bare integers only — wrong case
is still far better than the TTS garbling a long digit string."""
try:
from num2words import num2words
except ImportError:
LOG.warning("num2words is not installed — leaving %r unchanged", text)
return text
# A minus that stays a dash is a minus the TTS will not read, and -12
# spoken as 12 is the wrong temperature.
text = re.sub(r'(?<![\w])-(?=\d)', 'minus ', text)
def repl(m):
digits = re.sub(r'[\u00a0\u202f .]', '', m.group(0))
try:
return num2words(int(digits), lang="pl")
except Exception:
# num2words raises KeyError, not ValueError, past its largest scale
# word (10^66). Leaving the digits is bad; crashing a run is worse.
LOG.warning("num2words could not spell out %r — leaving it", m.group(0))
return m.group(0)
return _INT_RE.sub(repl, text)
# ---------------------------------------------------------------------------
# Routing
# ---------------------------------------------------------------------------
#
# The two models fail in opposite directions, measured on 40 real subtitle lines
# and a 32-case battery:
#
# fine-tune good: 6:05, P10, LO 3842, 95,7, 21.00, dates
# bad: loses the value of a large number while declining it
# ("z 8250 osobami" -> 8205), drops words
# num2words good: the value, always, whatever the digit grouping
# + LLM bad: destroys formats — a flight number becomes a quantity,
# "P10" becomes "Pdziesięć", times fall apart
#
# So a block goes to the second leg only when every number in it is a plain
# integer. On the sample, that is 77% of numbered blocks.
_MONTHS = ("stycznia|lutego|marca|kwietnia|maja|czerwca|lipca|sierpnia|"
"września|października|listopada|grudnia")
_COMPLEX_RE = re.compile(
r'\d[.,:/]\d' # 3,5 14:30 12/45 1.350
r'|\d\s*[-–—]\s*\d' # 10-15
r'|[^\W\d_]\d|\d[^\W\d_]' # P10, 1998cm3 — an identifier, not a count
r'|\b[A-ZĄĆĘŁŃÓŚŹŻ]{2,3}\s\d{2,}\b' # LO 3842 — call sign, read digit by digit
r'|\d+\s+(?i:' + _MONTHS + r')\b' # 13 maja — ordinal
r'|\d{4}\s+(?i:r(?:oku|\.))', # 2026 roku — ordinal
re.UNICODE)
# Case matters here, which is why IGNORECASE is scoped to the two alternatives
# that need it. Applied globally it turned the call-sign rule into "any short
# word before a number", routing "za 150 milionów" and "Przód 85" to the wrong
# leg — a third of the sample.
def route(text: str) -> str:
""""plain" when every number is a plain integer, else "complex"."""
return "complex" if _COMPLEX_RE.search(text) else "plain"
# ---------------------------------------------------------------------------
# Guards
# ---------------------------------------------------------------------------
#
# Both models can produce output that is worse than doing nothing: a repetition
# loop that runs to the token ceiling, a dropped minus sign that inverts a
# temperature, a vanished noun. These are cheap structural checks against the
# input — no second model pass — and a block that fails one falls back to
# num2words, which is dull but never wrong about the value.
_WORD_RE = re.compile(r'[^\W\d_]+', re.UNICODE)
_VOWELS = str.maketrans("", "", "aeiouyąęó")
_FOLD = str.maketrans("ąćęłńóśźż", "acelnoszz")
def _skeleton(word: str) -> str:
"""Consonants of a word, diacritics folded — a poor man's lemma, used to ask
whether a word survived generation at all. Three consonants is deliberate:
ludziach/ludzi and uchodźcach/uchodźców are the same word inflected, and a
longer key would report them as lost."""
return word.lower().translate(_FOLD).translate(_VOWELS)[:3]
# Numerals need identity, not similarity: "dwieście" and "dwustu" are the same
# number and must match, while "trzy" and "trzysta" differ by a factor of a
# hundred and must not. No consonant rule does both, so the closed class is
# written out. Patterns are tried in order, most specific first, against the
# lowercased and diacritic-folded word.
_NUMERAL_CANON = [
(r'zero', "0"),
(r'dwadziesc|dwudziest', "20"), (r'dwanasc|dwunast', "12"),
(r'dwiesc|dwust', "200"), (r'dw', "2"),
(r'trzynasc|trzynast', "13"), (r'trzydzies', "30"),
(r'trzyst', "300"), (r'trz', "3"),
(r'czternasc|czternast', "14"), (r'czterdzies', "40"),
(r'czteryst|czterest', "400"), (r'czter', "4"),
(r'pietnasc|pietnast', "15"), (r'piecdziesi', "50"),
(r'piecset', "500"), (r'pie[cs]', "5"),
(r'szesnasc|szesnast', "16"), (r'szescdziesi', "60"),
(r'szescset', "600"), (r'szes', "6"),
(r'siedemnasc|siedemnast', "17"), (r'siedemdziesi', "70"),
(r'siedemset', "700"), (r'siedem|siedmi', "7"),
(r'osiemnasc|osiemnast', "18"), (r'osiemdziesi', "80"),
(r'osiemset', "800"), (r'osiem|osm', "8"),
(r'dziewietnasc|dziewietnast', "19"), (r'dziewiecdziesi', "90"),
(r'dziewiecset', "900"), (r'dziewi', "9"),
(r'dziesi', "10"), (r'jedenasc|jedenast', "11"), (r'jed', "1"),
# Ordinals denote the same number and are often the better reading —
# "na pięćdziesiątym piątym okrążeniu". Without them the guard rejects
# correct output.
(r'pierwsz', "1"), (r'drug', "2"), (r'czwart', "4"), (r'piat', "5"),
(r'szost', "6"), (r'siodm', "7"), (r'setn', "100"),
(r'st[ou]$', "100"), (r'tysi', "1000"),
(r'miliard', "1e9"), (r'milion', "1e6"),
]
_NUMERAL_CANON = [(re.compile("^(?:" + pat + ")"), value)
for pat, value in _NUMERAL_CANON]
def _numeral(word: str) -> str:
"""The number a word denotes, whatever case it is in, or "" if it is not a
numeral."""
folded = word.lower().translate(_FOLD)
for pattern, value in _NUMERAL_CANON:
if pattern.match(folded):
return value
return ""
# Same as _INT_RE but from one digit up. Only ever applied to blocks the router
# called "plain", where every number is a bare integer — and there the model
# expands single digits too, so the expectation has to include them.
_ANY_INT_RE = re.compile(_NUM_START +
r'(?:\d{1,3}(?:[\u00a0\u202f .]\d{3})+|\d+)'
+ _NUM_END)
def _expected_numerals(text: str) -> list:
"""The numerals num2words would produce for `text`, canonicalised."""
try:
from num2words import num2words
except ImportError:
return []
def repl(m):
digits = re.sub(r'[\u00a0\u202f .]', '', m.group(0))
try:
return num2words(int(digits), lang="pl")
except Exception:
return m.group(0)
spelled = _ANY_INT_RE.sub(repl, text)
return sorted(n for n in map(_numeral, _WORD_RE.findall(spelled)) if n)
def _repeats(text: str) -> bool:
"""A word five times over, or a phrase three times — a generation loop.
PLLuM produced 'łącznie' eighty times on one input; the fine-tune produced
a wall of replacement characters on another."""
words = text.lower().split()
run = 1
for a, b in zip(words, words[1:]):
run = run + 1 if a == b else 1
if run >= 4:
return True
for size in (2, 3):
seen = {}
for i in range(len(words) - size + 1):
gram = tuple(words[i:i + size])
seen[gram] = seen.get(gram, 0) + 1
if seen[gram] >= 3:
return True
return False
def _guard(original: str, candidate: str, check_value: bool) -> str:
"""Return the reason `candidate` is unusable, or "" if it passes."""
if not candidate.strip():
return "empty"
if len(candidate) > 3 * len(original) + 60:
return "runaway length"
if _repeats(candidate):
return "repetition loop"
if "�" in candidate:
return "invalid characters"
if DIGIT_RE.search(candidate):
return "digits left in output"
if re.search(r'(?<![\w])-\s?\d', original) and "minus" not in candidate.lower():
return "minus sign lost"
if check_value:
# The plain leg's model is given text with no digits in it and asked
# only to inflect, so the numerals it returns must be exactly the ones
# num2words put there — same values, same count. A one-directional
# check is not enough: "dwadzieścia" -> "dwadzieścia pięć" keeps every
# expected word and still turns 20 into 25.
#
# The expectation comes from the full num2words rendering rather than
# from the digits alone, so numerals already written as words in the
# subtitle ("350 tysięcy") are counted too.
want = _expected_numerals(original)
got = sorted(n for n in map(_numeral, _WORD_RE.findall(candidate)) if n)
if want != got:
missing = [n for n in want if n not in got]
extra = [n for n in got if n not in want]
detail = (f"missing {','.join(missing)}" if missing else "") + \
(" " if missing and extra else "") + \
(f"added {','.join(extra)}" if extra else "reordered")
return f"numbers changed ({detail})"
# A word that vanished entirely. Short words are skipped because the
# dictionary legitimately expands them (ul. -> ulicy, zł -> złote).
have = [_skeleton(w) for w in _WORD_RE.findall(candidate)]
for word in _WORD_RE.findall(original):
if len(word) >= 5 and _skeleton(word) not in have:
return f"word lost ({word})"
return ""
# ---------------------------------------------------------------------------
# Decide / normalize
# ---------------------------------------------------------------------------
def needs_normalization(text: str) -> bool:
"""True when `text` holds a number the TTS would get wrong — anything with
a digit, once standalone single digits are discounted.
A block whose only numbers are standalone single digits is not sent at all:
the TTS says "pięć minut" for "5 minut" by itself, so there is nothing to
gain and a whole sentence of rewriting to risk. Inside a block that *is*
sent, single digits are normalized along with everything else.
lektor_ai_comfy.py applies the same rule before sending a block, so blocks
that need nothing never cross the process boundary. Keep the two in step.
"""
return bool(DIGIT_RE.search(KEEP_RE.sub("", text)))
def normalize(text: str) -> str:
"""Return `text` with its numbers spelled out in Polish. Never raises."""
return normalize_detail(text)[0]
def normalize_detail(text: str) -> tuple:
"""(result, leg, guard_reason) — the caller reports which leg did the work.
Plain integers go to num2words for the value and the LLM for the grammar,
which is the only arrangement measured to get both right. Everything else
goes to the fine-tune, which understands the formats. Either way the result
is checked before it is returned.
"""
if not text or not DIGIT_RE.search(text):
return text, "skipped", ""
if not needs_normalization(text):
return text, "skipped", "" # only standalone single digits
if len(text) > LONG_INPUT_CHARS:
LOG.warning("Input is %d characters; quality degrades past %d. "
"Processing it anyway.", len(text), LONG_INPUT_CHARS)
leg = route(text)
if leg == "plain" and load_llm():
STATS["plain"] += 1
nominative = _fallback(text) # deterministic: the value cannot be wrong
try:
candidate = _llm_inflect(nominative)
except Exception as exc:
LOG.warning("Inflection failed (%s: %s)", type(exc).__name__, exc)
candidate = nominative
safe, check_value = nominative, True
else:
leg = "complex" # no inflection model: the fine-tune takes it
STATS["complex"] += 1
if load_model():
try:
candidate = _model_normalize(text)
except Exception as exc:
LOG.warning("Generation failed (%s: %s)", type(exc).__name__, exc)
candidate = _fallback(text)
else:
candidate = _fallback(text)
# The value check needs a num2words expectation, which only exists for
# plain integers — "LO 3842" is correctly read digit by digit, and
# demanding "trzy tysiące osiemset" of it would reject a good answer.
safe, check_value = _fallback(text), False
reason = _guard(text, candidate, check_value)
if not reason:
return candidate, leg, ""
STATS["guarded"] += 1
STATS["reasons"][reason.split(" (")[0]] = \
STATS["reasons"].get(reason.split(" (")[0], 0) + 1
LOG.warning("Rejected %r -> %r (%s); using %r", text, candidate, reason, safe)
return (safe if safe.strip() else text), leg, reason
# ---------------------------------------------------------------------------
# CLI
# ---------------------------------------------------------------------------
def main(argv=None) -> int:
import argparse
parser = argparse.ArgumentParser(
description="Spell out Polish numbers for TTS.",
epilog="With no TEXT, reads lines from stdin and writes them normalized.")
parser.add_argument("text", nargs="*", metavar="TEXT")
parser.add_argument("--status", action="store_true",
help="Report which path is live and exit")
parser.add_argument("--fallback-only", action="store_true",
help="Skip the model, use num2words (for testing)")
parser.add_argument("-v", "--verbose", action="store_true")
args = parser.parse_args(argv)
logging.basicConfig(level=logging.INFO if args.verbose else logging.WARNING,
format="%(levelname)s: %(message)s", stream=sys.stderr)
if args.fallback_only:
_state.update(loaded=True, model=None, tok=None,
error="--fallback-only requested")
if args.status:
print(json.dumps(status(), ensure_ascii=False, indent=2))
return 0
if args.text:
print(normalize(" ".join(args.text)))
return 0
for line in sys.stdin:
print(normalize(line.rstrip("\n")), flush=True)
return 0
if __name__ == "__main__":
sys.exit(main())
normalizer/requirements.txt
+13
View File
@@ -0,0 +1,13 @@
# Polish number normalizer — isolated environment.
#
# Deliberately NOT installed into the ComfyUI / Higgs Audio environment: the
# model needs transformers >= 5, and forcing that on the working TTS install
# risks its pinned dependencies. The two run as separate processes.
# 4.x tokenizes the Qwen3 chat template differently and silently produces wrong
# output even under greedy decode. This is a hard requirement, not a preference.
transformers>=5.0.0
torch>=2.1.0 # install the CUDA build matching the host, see README
accelerate>=0.26.0 # device_map="auto"
num2words>=0.5.13 # fallback path only
normalizer/setup.sh
Executable
+98
View File
@@ -0,0 +1,98 @@
#!/usr/bin/env bash
# Create the isolated environment for the Polish number normalizer.
#
# ./setup.sh venv + dependencies
# ./setup.sh --model also download the model (needs HF_TOKEN)
# ./setup.sh --cpu CPU-only torch, for a machine without an NVIDIA GPU
#
# Kept separate from the ComfyUI environment on purpose: this needs
# transformers >= 5, and the TTS install must not be disturbed.
set -euo pipefail
HERE="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)"
VENV="$HERE/venv"
MODEL_DIR="$HERE/model"
MODEL_REPO="Folx/qwen3-0.6b-pl-text-normalization"
TORCH_INDEX="https://download.pytorch.org/whl/cu128" # same wheels as ComfyUI
WANT_MODEL=0
CPU_ONLY=0
for arg in "$@"; do
case "$arg" in
--model) WANT_MODEL=1 ;;
--cpu) CPU_ONLY=1 ;;
-h|--help) sed -n '2,10p' "${BASH_SOURCE[0]}" | sed 's/^# \?//'; exit 0 ;;
*) echo "Unknown option: $arg" >&2; exit 2 ;;
esac
done
say() { printf '\n\033[1;36m==> %s\033[0m\n' "$1"; }
say "Creating $VENV"
python3 -m venv "$VENV"
"$VENV/bin/pip" install --upgrade pip --quiet
say "Installing torch"
if [ "$CPU_ONLY" = 1 ]; then
"$VENV/bin/pip" install torch --index-url https://download.pytorch.org/whl/cpu
else
"$VENV/bin/pip" install torch --index-url "$TORCH_INDEX"
fi
say "Installing transformers, accelerate, num2words"
"$VENV/bin/pip" install "transformers>=5.0.0" "accelerate>=0.26.0" "num2words>=0.5.13"
if [ "$WANT_MODEL" = 1 ]; then
say "Downloading $MODEL_REPO"
"$VENV/bin/pip" install --quiet "huggingface_hub[cli]"
# Either an HF_TOKEN in the environment or a stored `hf auth login` works;
# the CLI picks up both on its own. Only say something when there is
# neither, because the failure then is a confusing "access denied".
if [ -z "${HF_TOKEN:-}" ] && ! "$VENV/bin/hf" auth whoami >/dev/null 2>&1; then
cat >&2 <<EOF
No Hugging Face credentials. The repo is gated (approval is automatic, but it
has to be your account that asks):
1. Open https://huggingface.co/$MODEL_REPO while signed in and accept the terms
2. Either $VENV/bin/hf auth login
or HF_TOKEN=hf_... $0 --model
EOF
exit 1
fi
"$VENV/bin/hf" download "$MODEL_REPO" --local-dir "$MODEL_DIR"
fi
say "Checking the installation"
"$VENV/bin/python" - <<'PY'
import sys
try:
import torch, transformers
print(f" torch {torch.__version__} cuda_available={torch.cuda.is_available()}")
if torch.cuda.is_available():
print(f" device: {torch.cuda.get_device_name(0)}")
print(f" transformers {transformers.__version__}")
if int(transformers.__version__.split('.')[0]) < 5:
print(" WARNING: transformers < 5 produces wrong output with this model",
file=sys.stderr)
except ImportError as exc:
print(f" {exc}", file=sys.stderr); sys.exit(1)
PY
"$VENV/bin/python" "$HERE/normalizer.py" --status
if [ ! -f "$MODEL_DIR/config.json" ]; then
cat <<EOF
The environment is ready, but the model is not downloaded yet, so the
normalizer will run on the degraded num2words fallback.
1. Accept the terms at https://huggingface.co/$MODEL_REPO
2. HF_TOKEN=hf_... $HERE/setup.sh --model
EOF
else
say "Ready. Try: $VENV/bin/python $HERE/normalizer.py 'Wygrał 1247 złotych.'"
fi
normalizer/tests/test_normalizer.py
+375
View File
@@ -0,0 +1,375 @@
#!/usr/bin/env python3
"""Acceptance checks for the Polish number normalizer.
The model produces valid case variants, so these assert on *properties* — a
digit is gone, a word stem is present — never on an exact string, except where
the behaviour genuinely is exact (model-not-called, pass-through).
Run them in the normalizer venv:
normalizer/venv/bin/python -m pytest normalizer/tests -v
They pass with networking disabled. Tests that need the neural path are skipped
automatically when the model has not been downloaded; the rest exercise the
fallback and the protect/decide/restore logic, which is where the integration
bugs actually live.
"""
import os
import re
import sys
import unittest
from pathlib import Path
sys.path.insert(0, str(Path(__file__).resolve().parent.parent))
os.environ.setdefault("HF_HUB_OFFLINE", "1")
os.environ.setdefault("TRANSFORMERS_OFFLINE", "1")
import normalizer # noqa: E402
HAVE_MODEL = normalizer.model_available()
needs_model = unittest.skipUnless(
HAVE_MODEL, "model not downloaded — see normalizer/README.md")
DIGITS = re.compile(r'\d')
class Selection(unittest.TestCase):
"""Which blocks are worth sending at all. lektor_ai_comfy.py mirrors this
rule to keep blocks that need nothing from crossing the process boundary."""
def test_single_digit_is_not_worth_normalizing(self):
self.assertFalse(normalizer.needs_normalization("Mam 3 koty."))
def test_no_digits_at_all(self):
self.assertFalse(normalizer.needs_normalization("Zwykłe zdanie."))
def test_multi_digit(self):
self.assertTrue(normalizer.needs_normalization("Wygrał 1247 złotych."))
def test_complex_expressions_are_not_single_digits(self):
for text in ("Spotkanie o 9:30.", "Około 3,5 metra.", "Mieszka pod 12/45.",
"O 14:30 w środę."):
with self.subTest(text=text):
self.assertTrue(normalizer.needs_normalization(text))
def test_thousands_with_spaces(self):
self.assertTrue(normalizer.needs_normalization("populacja 1 350 000 osób"))
def test_single_digit_before_punctuation_is_still_single(self):
for text in ("Mam 3, potem 4.", "Zostało 5.", "Tylko 7!"):
with self.subTest(text=text):
self.assertFalse(normalizer.needs_normalization(text))
def test_multi_digit_before_punctuation_is_caught(self):
self.assertTrue(normalizer.needs_normalization("Przód 85, tył 75."))
class ClientRuleInSync(unittest.TestCase):
"""lektor_ai_comfy.py keeps its own copy of the selection rule so blocks
that need nothing never start a subprocess. A copy that drifts sends the
wrong blocks — or silently stops sending any — so it is checked here, where
the authoritative rule lives. The caller is pure standard library, so this
venv can import it."""
SAMPLES = [
"Mam 3 koty.", "Zwykłe zdanie.", "Wygrał 1247 złotych.",
"Przód 85, tył 75.", "populacja 1 350 000 osób", "Radio 95,7 FM.",
"Spotkanie o 14:30.", "Mieszka pod 12/45.", "Około 3,5 metra.",
"Formuła 1", "Masz F2, masz F3.", "Zostało 5.", "Mam 3, potem 4.",
"W 2026 roku.", "Tylko 7!", "1 pies, 2 koty, 300 myszy.",
]
def test_same_verdict_on_every_sample(self):
sys.path.insert(0, str(Path(__file__).resolve().parents[2]))
try:
import lektor_ai_comfy
except ImportError as exc: # pragma: no cover - only if moved
self.skipTest(f"caller not importable: {exc}")
for text in self.SAMPLES:
with self.subTest(text=text):
self.assertEqual(lektor_ai_comfy.needs_number_normalization(text),
normalizer.needs_normalization(text))
class Boundaries(unittest.TestCase):
"""KEEP_RE no longer shields anything — it only decides whether a block is
worth sending. These pin down where a number starts and ends."""
def test_grouped_thousands_are_not_split(self):
"""The leading 1 of "1 350 000" is not a standalone single digit; if it
counted as one the block might not be sent at all."""
protected = normalizer.KEEP_RE.sub("X", "populacja 1 350 000 osób")
self.assertIn("1 350 000", protected)
def test_complex_numbers_are_not_single_digits(self):
protected = normalizer.KEEP_RE.sub("X", "Mam 3 koty, 25 psów i 3,5 kg.")
self.assertNotIn(" 3 ", protected) # standalone single digit
self.assertIn("25", protected) # multi-digit untouched
self.assertIn("3,5", protected) # decimal untouched
class Routing(unittest.TestCase):
"""Which leg a block goes to. The two models fail in opposite directions,
so a misrouted block gets the worse of the two."""
PLAIN = ["Wygrał 1247 złotych.", "Mówili o 250000 uchodźcach.",
"Przód 85, tył 75.", "Małpi gaj za 150 milionów dolarów.",
"Bardzo. Ma z 80 lat.", "populacja 1 350 000 osób",
# A range written in words is two plain integers; only a dash
# between digits makes it a compound the fine-tune reads better.
"Ma od 15 do 20 lat."]
COMPLEX = ["Spotkanie o 14:30.", "Radio 95,7 FM.", "Mieszka pod 12/45.",
"Teraz na P12.", "Lot LO 3842 jest opóźniony.",
"Gdzie 13 maja 1950 roku się zaczęło.", "Ma od 10-20 lat.",
"Silnik ma 1998 cm3 pojemności.", "W 2026 roku."]
def test_plain(self):
for text in self.PLAIN:
with self.subTest(text=text):
self.assertEqual(normalizer.route(text), "plain")
def test_complex(self):
for text in self.COMPLEX:
with self.subTest(text=text):
self.assertEqual(normalizer.route(text), "complex")
def test_case_sensitivity_of_the_call_sign_rule(self):
"""Applied case-insensitively this rule matched any short word before a
number and misrouted a third of the sample."""
self.assertEqual(normalizer.route("Lot LO 3842 wystartował."), "complex")
self.assertEqual(normalizer.route("Zapłacił za 150 złotych."), "plain")
class Numerals(unittest.TestCase):
"""The value guard needs numeral identity across inflection, which no
consonant rule provides: dwieście/dwustu are one number, trzy/trzysta are
not."""
SAME = [("dwieście", "dwustu"), ("pięćdziesiąt", "pięćdziesięciu"),
("trzydzieści", "trzydziestu"), ("czterdzieści", "czterdziestu"),
("tysiąc", "tysiącach"), ("milion", "milionach"),
("osiem", "ośmiu"), ("pięć", "piątym"), ("dwadzieścia", "dwudziesty")]
DIFFERENT = [("trzy", "trzysta"), ("pięć", "pięćdziesiąt"),
("dwa", "dwieście"), ("milion", "miliard"),
("sto", "sześćset"), ("cztery", "czternaście")]
def test_same_number(self):
for a, b in self.SAME:
with self.subTest(pair=(a, b)):
self.assertEqual(normalizer._numeral(a), normalizer._numeral(b))
self.assertTrue(normalizer._numeral(a))
def test_different_numbers(self):
for a, b in self.DIFFERENT:
with self.subTest(pair=(a, b)):
self.assertNotEqual(normalizer._numeral(a), normalizer._numeral(b))
def test_ordinary_words_are_not_numerals(self):
for word in ("koty", "uchodźcach", "okrążenie", "prędkość"):
self.assertEqual(normalizer._numeral(word), "")
class Guards(unittest.TestCase):
"""Every check here fired on a real failure during development. The
false-positive cases matter just as much: a guard that rejects good output
silently downgrades the run."""
def test_accepts_correct_output(self):
for original, candidate in [
("Mówili o 250000 uchodźcach.",
"Mówili o dwustu pięćdziesięciu tysiącach uchodźców."),
("To wypis sprzed 30 lat.", "To wypis sprzed trzydziestu lat."),
("kolizja na 55 okrążeniu", "kolizja na pięćdziesiątym piątym okrążeniu"),
("misja to Apollo 25", "misja to Apollo dwudziesty piąty"),
]:
with self.subTest(candidate=candidate):
self.assertEqual(normalizer._guard(original, candidate, True), "")
def test_value_changed(self):
reason = normalizer._guard("Z 8250 osobami.",
"Z osiem tysięcy dwustu pięciu osobami.", True)
self.assertIn("numbers changed", reason)
def test_added_numeral(self):
"""The check must run both ways. "dwadzieścia" -> "dwadzieścia pięć"
keeps every expected word and still turns 20 into 25 — a one-directional
subset check passed this and shipped 25 years of service as 20."""
reason = normalizer._guard("Nagroda Za 20 Lat Pracy.",
"Nagroda Za dwadzieścia Pięć Lat Pracy.", True)
self.assertIn("added", reason)
def test_single_digits_count_toward_the_expectation(self):
"""num2words skips one-digit numbers but the model expands them, so the
expectation must cover them or "trzy" looks like an invention."""
self.assertEqual(normalizer._guard("Mam 3 koty i 25 psów.",
"Mam trzy koty i dwadzieścia pięć psów.",
True), "")
def test_word_lost(self):
reason = normalizer._guard("Populacja to 38000000 mieszkańców.",
"Populacja to trzydzieści osiem milionów.", True)
self.assertIn("word lost", reason)
def test_minus_lost(self):
reason = normalizer._guard("Na dworze jest -12 stopni.",
"Na dworze jest dwanaście stopni.", True)
self.assertIn("minus", reason)
def test_repetition_loop(self):
reason = normalizer._guard("Zapłaciłem 1234,56 zł.",
"Zapłaciłem " + "łącznie " * 40 + "tysiąc.", False)
self.assertTrue(reason)
def test_invalid_characters(self):
self.assertIn("invalid", normalizer._guard("Dnia 15 września 1631 roku.",
"Dnia " + "�" * 30, False))
def test_digits_left(self):
self.assertIn("digits", normalizer._guard("Ma 25 lat.", "Ma 25 lat.", False))
def test_value_check_is_skipped_for_identifiers(self):
""""LO 3842" is correctly read digit by digit, so demanding num2words'
rendering of it would reject a good answer."""
self.assertEqual(
normalizer._guard("Lot LO 3842 jest opóźniony.",
"Lot LO trzy osiem cztery dwa jest opóźniony.",
False), "")
class Fallback(unittest.TestCase):
"""Degraded path: nominative only, bare integers only. It must still never
lose a number or raise."""
def test_expands_multi_digit(self):
out = normalizer._fallback("Wygrał 1247 złotych.")
self.assertNotIn("1247", out)
self.assertIn("tysiąc", out)
def test_keeps_single_digit(self):
self.assertEqual(normalizer._fallback("Mam 3 koty."), "Mam 3 koty.")
def test_large_number_with_separators(self):
out = normalizer._fallback("populacja 1 350 000 osób")
self.assertNotIn("350", out)
self.assertIn("milion", out)
def test_leaves_decimals_and_times_alone(self):
for text in ("Około 3,5 metra.", "Spotkanie o 9:30.", "Radio 95,7 FM."):
self.assertEqual(normalizer._fallback(text), text)
def test_number_followed_by_sentence_punctuation(self):
"""A comma or full stop ends a number; it does not disqualify it. Real
subtitles are full of "Przód 85, tył 75." and the brief's boundary set
skipped every one of them."""
out = normalizer._fallback("Przód 85, tył 75.")
self.assertNotIn("85", out)
self.assertNotIn("75", out)
self.assertIn("osiemdziesiąt pięć", out)
self.assertIn("siedemdziesiąt pięć", out)
def test_never_raises(self):
for text in ("", "———", "9" * 400, "1/0", "0000"):
with self.subTest(text=text[:20]):
normalizer._fallback(text)
class NormalizeContract(unittest.TestCase):
"""Properties that hold on both paths — this is what the caller relies on."""
def test_no_digits_returns_input_unchanged(self):
text = "Zwykłe zdanie bez liczb."
self.assertEqual(normalizer.normalize(text), text)
def test_only_single_digits_skips_the_model(self):
calls = []
original = normalizer._model_normalize
normalizer._model_normalize = lambda t: calls.append(t) or t
try:
self.assertEqual(normalizer.normalize("Mam 3 koty."), "Mam 3 koty.")
finally:
normalizer._model_normalize = original
self.assertEqual(calls, [], "model must not be called for single digits")
def test_block_of_only_single_digits_is_returned_untouched(self):
"""The cheap half of the brief's idea, kept: nothing to gain from
sending it, a whole sentence of rewriting to risk."""
for text in ("Mam 3 koty.", "Zostało 5 minut.", "Tylko 7!"):
with self.subTest(text=text):
self.assertEqual(normalizer.normalize(text), text)
def test_empty_and_odd_inputs(self):
self.assertEqual(normalizer.normalize(""), "")
self.assertEqual(normalizer.normalize(" "), " ")
def test_generation_error_falls_back(self):
original = normalizer._model_normalize
def boom(_):
raise RuntimeError("CUDA out of memory")
normalizer._model_normalize = boom
try:
out = normalizer.normalize("Wygrał 1247 złotych.")
finally:
normalizer._model_normalize = original
self.assertNotIn("1247", out)
self.assertIn("tysiąc", out)
@needs_model
class Neural(unittest.TestCase):
"""The acceptance list from the build brief, against the real model."""
def test_multi_digit_expanded(self):
out = normalizer.normalize("Wygrał 1247 złotych.")
self.assertNotIn("1247", out)
self.assertIn("tysiąc", out.lower())
def test_large_number(self):
out = normalizer.normalize("Miasto ma populację 1 350 000 osób.")
self.assertNotIn("350", out)
self.assertIn("milion", out.lower())
def test_mixed_single_and_multi(self):
"""A block sent for its multi-digit number gets its single digits
normalized too — that is the point of dropping the placeholders, and
it is what makes "3 maja" come out as "trzeciego maja"."""
out = normalizer.normalize("Mam 3 koty i 25 psów.").lower()
self.assertNotIn("3", out)
self.assertIn("trzy", out)
self.assertIn("dwadzieścia pięć", out)
def test_leg_is_reported(self):
self.assertEqual(normalizer.normalize_detail("Mam 3 koty.")[1], "skipped")
self.assertEqual(normalizer.normalize_detail("Jest dopiero 21.00.")[1],
"complex")
def test_ordinal_date_is_fixed(self):
"""The reason the placeholders had to go: left as a digit, the TTS
reads "3 maja" as "trzy maja"."""
out = normalizer.normalize("Urodził się 3 maja 1791 roku.").lower()
self.assertIn("trzeciego maja", out)
def test_no_digits_leak(self):
out = normalizer.normalize("Pędzi 300 kilometrów na godzinę.")
self.assertFalse(DIGITS.search(out))
def test_case_declension(self):
"""The whole reason for a neural normalizer: 'o 25 psach' is locative,
so the numeral must not come out nominative."""
out = normalizer.normalize("Mówimy o 25 psach.").lower()
self.assertNotIn("25", out)
self.assertNotIn("dwadzieścia pięć psach", out)
def test_time(self):
out = normalizer.normalize("Spotkanie o 14:30.")
self.assertFalse(DIGITS.search(out))
def test_offline(self):
self.assertEqual(os.environ.get("HF_HUB_OFFLINE"), "1")
self.assertFalse(DIGITS.search(normalizer.normalize("Ma 99 lat.")))
if __name__ == "__main__":
print(f"model available: {HAVE_MODEL}", file=sys.stderr)
unittest.main(verbosity=2)
normalizer/worker.py
Executable
+74
View File
@@ -0,0 +1,74 @@
#!/usr/bin/env python3
"""JSON-lines front end for normalizer.py.
The normalizer needs `transformers>=5`, which must not be installed into the
ComfyUI/Higgs environment, so it lives in its own venv and is driven as a
separate process. Loading a 1.1 GB model per subtitle block would be absurd, so
this is a long-lived worker: the caller starts it once per run, streams blocks
through it, and closes stdin when done.
Protocol — one JSON object per line, in both directions:
← {"ready": true, "mode": "model", "device": "cuda:0", "llm": true, …}
→ {"id": 1, "text": "Wygrał 1247 złotych."}
← {"id": 1, "text": "Wygrał tysiąc dwieście…", "changed": true,
"leg": "plain", "guard": "", "error": null}
→ {"quit": true} (or just close stdin)
`leg` is which model answered — "plain" for num2words plus the inflection
model, "complex" for the fine-tune, "skipped" when the block needed nothing.
`guard` names the check that rejected the model's answer, empty when it passed.
stdout carries the protocol and nothing else; every log line, warning and
traceback goes to stderr. A request that fails is answered with the original
text and an `error` field — the worker does not die on one bad block.
"""
import json
import logging
import sys
import normalizer
def _send(obj) -> None:
sys.stdout.write(json.dumps(obj, ensure_ascii=False) + "\n")
sys.stdout.flush()
def main() -> int:
logging.basicConfig(level=logging.INFO, format="%(levelname)s: %(message)s",
stream=sys.stderr)
state = normalizer.status() # loads the model, or reports why not
state["llm"] = normalizer.load_llm() # the inflection leg is optional
state["llm_dir"] = str(normalizer.LLM_DIR)
_send({"ready": True, **state})
for line in sys.stdin:
line = line.strip()
if not line:
continue
try:
req = json.loads(line)
except ValueError as exc:
_send({"id": None, "text": None, "error": f"bad request: {exc}"})
continue
if req.get("quit"):
break
text = req.get("text") or ""
leg, guard = "", ""
try:
result, leg, guard = normalizer.normalize_detail(text)
error = None
except Exception as exc: # normalize_detail() is not meant to raise
logging.exception("normalize_detail() raised")
result, error = text, f"{type(exc).__name__}: {exc}"
_send({"id": req.get("id"), "text": result, "changed": result != text,
"leg": leg, "guard": guard, "error": error})
return 0
if __name__ == "__main__":
sys.exit(main())
readme.txt
+70 -3
View File
@@ -30,7 +30,7 @@ Settings
-------- --------
Defaults live in settings.ini next to the script: Defaults live in settings.ini next to the script:
voice, speed, temperature, max_tokens, strip_leading_ms, trailing_fade_ms, voice, speed, temperature, max_tokens, strip_leading_ms, trailing_fade_ms,
orig_volume, tts_volume, output_dir, comfy_url orig_volume, tts_volume, output_dir, dictionary, numbers, comfy_url
Precedence: command-line flag > settings.ini > built-in default. Precedence: command-line flag > settings.ini > built-in default.
Use --settings FILE to select a different profile. Use --settings FILE to select a different profile.
@@ -59,10 +59,12 @@ Both external SRT files and embedded subtitle tracks go through the same cleanin
- HTML tags stripped: <i>, </i>, <b>, <font color="…">, etc. - HTML tags stripped: <i>, </i>, <b>, <font color="…">, etc.
- ASS override codes stripped: {\an8}, {\pos(x,y)}, {\c&H…&}, etc. - ASS override codes stripped: {\an8}, {\pos(x,y)}, {\c&H…&}, etc.
- ALL-CAPS blocks lowercased and converted to sentence case - ALL-CAPS blocks converted to Title Case (keeps proper nouns readable)
- Sentence fragments starting with a lowercase letter are capitalised - Sentence fragments starting with a lowercase letter are capitalised
- Missing terminal punctuation added (… or . depending on context) - Missing terminal punctuation added (… or . depending on context)
- Dictionary replacements applied (see below)
- Dialogue lines (prefixed with -) are split into separate clips with proportional timing - Dialogue lines (prefixed with -) are split into separate clips with proportional timing
- Numbers spelled out as Polish words, in the right case (see below)
With --embedded-subs the extracted subtitles are saved to the job folder as embedded_sub.srt With --embedded-subs the extracted subtitles are saved to the job folder as embedded_sub.srt
so you can inspect them before or after the run. so you can inspect them before or after the run.
@@ -72,6 +74,63 @@ The final MKV preserves all subtitle tracks from the source video — text-based
stream copy with no conversion. stream copy with no conversion.
Replacement Dictionary
----------------------
dictionary.txt next to the script rewrites words and phrases before they reach
the TTS engine — abbreviations, units, symbols, names the model mispronounces.
One rule per line:
km/h = kilometrów na godzinę
np. = na przykład
/(\d+)\s*%/ = \1 procent
Matching is case-sensitive and whole-word only, so "np." leaves "znp." alone and
"Joshua" leaves "joshua" alone. The replacement is inserted exactly as written.
The first letter of each block is capitalised before these rules run, so add a
capitalised variant for words that can start a subtitle.
Longer patterns win over shorter ones regardless of the order they are written
in, and replacement text is never rescanned, so rules cannot feed each other.
A pattern wrapped in slashes is a regular expression, with \1 \2 … referring to
its groups. Lines starting with # or ; are comments.
The result is visible in the job folder's tts_input.srt — check there if a rule
is not doing what you expect. --regen and --rebuild work from that file, so they
do not re-apply the dictionary; edit tts_input.srt directly, or rerun.
Use another list with --dictionary FILE (a relative name is looked for in the
current directory first, then next to the script) or turn it off with
--dictionary "". A missing file is not an error.
A second list can be merged in with --dictionary-extra FILE. The GUI writes one
per queue — the "Queue dictionary" button — into the first queued video's job
folder, for rules that belong to one batch rather than to Polish in general.
Number Normalization
--------------------
The TTS misreads long digit strings and sometimes turns them into non-words, so
after the dictionary every block containing a number is rewritten with the
numbers spelled out in Polish — in the correct grammatical case, which is why
this is a small neural model and not a lookup table:
Wygrał 1247 złotych. -> Wygrał tysiąc dwieście czterdzieści siedem złotych.
Mówimy o 25 psach. -> Mówimy o dwudziestu pięciu psach.
Mam 3 koty. -> Mam 3 koty. (single digits are read fine)
It runs in its own environment (transformers>=5, which must not be installed
next to ComfyUI's pinned versions). One-time install:
normalizer/setup.sh # venv + dependencies
HF_TOKEN=hf_... normalizer/setup.sh --model # gated weights, accept terms first
Terms: https://huggingface.co/Folx/qwen3-0.6b-pl-text-normalization
Everything after that runs offline. See normalizer/README.md.
Turn it off with numbers = no under [normalize] in settings.ini, or with
--no-normalize-numbers for one run. If the environment or the model is missing
the run continues with the digits untouched — it is never fatal.
Output Files Output Files
------------ ------------
Everything belonging to one video is kept together in a job folder named after Everything belonging to one video is kept together in a job folder named after
@@ -130,6 +189,13 @@ All Options
--max-tokens N Max audio tokens the model may generate per clip (default: 4096) --max-tokens N Max audio tokens the model may generate per clip (default: 4096)
Raise this if long clips are cut off mid-word. Raise this if long clips are cut off mid-word.
--dictionary FILE Word/phrase replacements applied to the subtitles before
TTS (default: dictionary.txt). Pass "" to disable.
--normalize-numbers Spell digits out as Polish words before TTS (default: on)
--no-normalize-numbers
Leave digits as digits. The TTS often misreads them.
--orig-volume 0-1 Volume of the original audio in the Polish lektor mix (default: 0.60) --orig-volume 0-1 Volume of the original audio in the Polish lektor mix (default: 0.60)
--tts-volume 0-2 Volume of the TTS voiceover in the Polish lektor mix (default: 1.8) --tts-volume 0-2 Volume of the TTS voiceover in the Polish lektor mix (default: 1.8)
Note: these affect Track 1 only. Track 0 is always the raw original. Note: these affect Track 1 only. Track 0 is always the raw original.
@@ -151,7 +217,8 @@ All Options
Recovery and Fixing Bad Clips Recovery and Fixing Bad Clips
------------------------------ ------------------------------
Resume a crashed run from block N (reuses existing clips 1..N-1): Resume a crashed run from block N (reuses existing clips 1..N-1 and the job
folder's tts_input.srt — subtitles are not reprocessed):
python3 lektor_ai_comfy.py input/film.srt input/film.mkv --resume-from 47 python3 lektor_ai_comfy.py input/film.srt input/film.mkv --resume-from 47
settings.ini
+42 -1
View File
@@ -23,7 +23,7 @@ speed = 1.2
# Sampling temperature. 1.0 makes the model occasionally recite the reference # Sampling temperature. 1.0 makes the model occasionally recite the reference
# transcript instead of the subtitle; 0.9 largely prevents that. Lower is more # transcript instead of the subtitle; 0.9 largely prevents that. Lower is more
# deterministic but flatter. # deterministic but flatter.
temperature = 0.6 temperature = 0.8
# Ceiling on audio tokens per clip. Raise only if long clips are cut off # Ceiling on audio tokens per clip. Raise only if long clips are cut off
# mid-word. # mid-word.
@@ -55,6 +55,47 @@ tts_volume = 1.4
# Relative paths are resolved from wherever you run the script. # Relative paths are resolved from wherever you run the script.
output_dir = output output_dir = output
# Word and phrase replacements applied to the subtitles before they are spoken,
# e.g. "km/h = kilometrów na godzinę". One rule per line in that file; see the
# comments at the top of it. A relative name is looked for in the current
# directory first, then next to lektor_ai_comfy.py, so a single film can have
# its own list. A missing file is not an error.
dictionary = dictionary.txt
[normalize]
# Spell digits out as Polish words before the text reaches the TTS, in the right
# grammatical case: "Wygrał 1247 złotych" -> "Wygrał tysiąc dwieście czterdzieści
# siedem złotych". Higgs Audio misreads long digit strings and sometimes turns
# them into non-words. Standalone single digits are read correctly and are left
# alone. Runs last, on the text that goes into tts_input.srt. yes/no.
numbers = yes
# Interpreter of the normalizer's own environment — it needs transformers>=5,
# which must not be installed next to ComfyUI's pinned versions. Created by
# normalizer/setup.sh. Relative paths are resolved next to lektor_ai_comfy.py.
# If it is missing the run continues with the digits untouched.
python = normalizer/venv/bin/python
# Second model, used for the grammar of plain integers. num2words supplies the
# value (it cannot miscount) and this model puts the numeral into the right
# case: "o 250000 uchodzcach" -> "o dwustu piecdziesieciu tysiacach uchodzcow".
# The fine-tune alone gets that wrong, and sometimes loses the number itself.
# Blocks with times, dates, decimals or identifiers skip it — it is worse than
# the fine-tune on those. Leave empty to use the fine-tune for everything.
llm = normalizer/models/PLLuM-4B-instruct-2512
# Seconds to wait for one block. On a GPU a block takes well under a second;
# this only exists so a wedged worker cannot stall the whole run.
timeout = 60
# Ask ComfyUI to unload Higgs before normalizing, so the normalizer gets the
# GPU. Higgs stays resident between runs and leaves too little VRAM; a
# normalizer that does not fit spills to RAM silently and runs about 5x slower.
# ComfyUI reloads on the first clip of step 2 (20-30s, once per run). --regen
# and --rebuild never normalize, so they never pay it. yes/no.
free_comfy_vram = yes
[comfy] [comfy]
# Address of the ComfyUI backend running Higgs Audio V3. # Address of the ComfyUI backend running Higgs Audio V3.
start_comfyui.sh
+4 -1
View File
@@ -1,4 +1,7 @@
#!/bin/bash #!/bin/bash
source ~/miniforge3/etc/profile.d/conda.sh source ~/miniforge3/etc/profile.d/conda.sh
conda activate comfy-tts conda activate comfy-tts
cd ~/comfy-tts-app/ComfyUI && python main.py --port 8189 # exec: the shell is replaced by ComfyUI, so this script's PID is
# ComfyUI's PID. Without it the launcher only ever sees a wrapper,
# and killing that orphans the python still holding the GPU.
cd ~/comfy-tts-app/ComfyUI && exec python main.py --port 8189
ui.py
+223 -2
View File
@@ -228,11 +228,18 @@ SETTING_LABELS = {
"orig_volume": "sound volume", "orig_volume": "sound volume",
"tts_volume": "voiceover volume", "tts_volume": "voiceover volume",
"output_dir": "output folder", "output_dir": "output folder",
"dictionary": "replacement dictionary",
"normalize_numbers": "spell out numbers",
"normalizer_python": "normalizer interpreter",
"normalizer_llm": "inflection model",
"normalizer_timeout": "normalizer timeout (s)",
"free_comfy_vram": "unload Higgs to normalize",
"comfy_url": "ComfyUI URL", "comfy_url": "ComfyUI URL",
"comfy_autostart": "start ComfyUI if down", "comfy_autostart": "start ComfyUI if down",
"comfy_start_command": "start script", "comfy_start_command": "start script",
} }
SECTION_TITLES = {"tts": "Speech", "mix": "Mix", "paths": "Paths", "comfy": "Backend"} SECTION_TITLES = {"tts": "Speech", "mix": "Mix", "paths": "Paths",
"normalize": "Numbers", "comfy": "Backend"}
def raw_settings(path): def raw_settings(path):
@@ -315,6 +322,100 @@ def save_settings(path, values):
tmp.replace(path) # atomic, so a crash cannot truncate it tmp.replace(path) # atomic, so a crash cannot truncate it
class QueueDictionaryDialog(tk.Toplevel):
"""Plain text editor for the queue's own replacement rules.
Deliberately a text box rather than a table: the file format is one
`pattern = replacement` per line, and typing that is faster than filling in
two columns. The same parser reads it as the shared dictionary, so regex
rules and `#` comments work here too.
"""
def __init__(self, app):
super().__init__(app)
self.app = app
F = app.font_family
self.title("Queue dictionary")
self.configure(bg=BG)
self.transient(app)
path = app.queue_dictionary_path()
where = str(path) if path else "(add a video to the queue first)"
tk.Label(self, text="Replacements applied to every video in this queue,",
bg=BG, fg=FG, font=(F, 10)).pack(anchor="w", padx=22, pady=(16, 0))
tk.Label(self, text="in addition to dictionary.txt. One rule per line, "
"case-sensitive, whole words:",
bg=BG, fg=FG, font=(F, 10)).pack(anchor="w", padx=22)
tk.Label(self, text=" Joshua = Dżoszua", bg=BG, fg="#8fb8d6",
font=(F, 9)).pack(anchor="w", padx=22, pady=(6, 0))
tk.Label(self, text=where, bg=BG, fg=MUTED, font=(F, 8)
).pack(anchor="w", padx=22, pady=(8, 4))
frame = tk.Frame(self, bg=BG)
frame.pack(fill="both", expand=True, padx=22)
bar = tk.Scrollbar(frame)
bar.pack(side="right", fill="y")
self.text = tk.Text(frame, width=64, height=16, bg=FIELD, fg=FIELD_FG,
insertbackground=FIELD_FG, relief="flat", bd=0,
font=(F, 10), yscrollcommand=bar.set, undo=True)
self.text.pack(side="left", fill="both", expand=True)
bar.config(command=self.text.yview)
if path is not None and path.is_file():
try:
self.text.insert("1.0", path.read_text(encoding="utf-8"))
except OSError as exc:
messagebox.showerror("Queue dictionary", str(exc), parent=self)
self.status = tk.Label(self, text="", bg=BG, fg=MUTED, font=(F, 8))
self.status.pack(anchor="w", padx=22, pady=(6, 0))
buttons = tk.Frame(self, bg=BG)
buttons.pack(fill="x", padx=22, pady=16)
RoundButton(buttons, "Close", self.destroy, bg="#6f6f6f", fg="#f0f0f0",
hover="#828282", font=(F, 9, "bold"), padx=16, pady=6
).pack(side="right")
RoundButton(buttons, "Save", self._save, font=(F, 9, "bold"),
padx=22, pady=6).pack(side="right", padx=(0, 8))
RoundButton(buttons, "Clear", self._clear, bg="#4a4a4a", fg=MUTED,
hover="#5c5c5c", font=(F, 9), padx=14, pady=6).pack(side="left")
self.bind("<Escape>", lambda _: self.destroy())
self.update_idletasks()
x = app.winfo_rootx() + (app.winfo_width() - self.winfo_width()) // 2
self.geometry(f"+{max(0, x)}+{app.winfo_rooty() + 60}")
self.grab_set()
def _clear(self):
"""Empties the box only. Nothing is written until Save, so a misclick
costs a Ctrl+Z rather than the file."""
self.text.delete("1.0", "end")
self.status.config(text="Cleared — press Save to write the empty file.")
def _save(self):
path = self.app.queue_dictionary_path()
if path is None:
messagebox.showinfo(
"Queue dictionary",
"Add a video to the queue first — the file is written to the "
"first video's job folder.", parent=self)
return
body = self.text.get("1.0", "end").rstrip() + "\n"
try:
path.parent.mkdir(parents=True, exist_ok=True)
path.write_text(body, encoding="utf-8")
except OSError as exc:
messagebox.showerror("Queue dictionary", str(exc), parent=self)
return
# Report what the pipeline will actually make of it, so a typo shows up
# now rather than as a rule that silently does nothing during the run.
loaded = core.Dictionary.load(path)
self.status.config(text=f"Saved — {len(loaded)} rule(s) parsed.")
self.app.log(f"Queue dictionary saved to {path} ({len(loaded)} rules).")
class SettingsDialog(tk.Toplevel): class SettingsDialog(tk.Toplevel):
"""Editor for settings.ini — the defaults new job blocks start from.""" """Editor for settings.ini — the defaults new job blocks start from."""
@@ -342,7 +443,7 @@ class SettingsDialog(tk.Toplevel):
if core.VOICE_PROMPTS_DIR.is_dir() else [] if core.VOICE_PROMPTS_DIR.is_dir() else []
row = 0 row = 0
for section in ("tts", "mix", "paths", "comfy"): for section in ("tts", "mix", "paths", "normalize", "comfy"):
keys = [k for k, spec in core._DEFAULTS.items() if spec[0] == section] keys = [k for k, spec in core._DEFAULTS.items() if spec[0] == section]
if not keys: if not keys:
continue continue
@@ -774,6 +875,16 @@ class JobBlock(tk.Frame):
cmd += self.common_args() cmd += self.common_args()
# The whole queue shares one extra dictionary, kept with the first
# video. --regen and --rebuild work purely from tts_input.srt, so the
# flag would do nothing there. A resume normally reuses that file too,
# but falls back to parsing when it is missing — and then the queue's
# rules must still apply, so it is passed.
if mode in ("generate", "resume"):
extra = self.app.queue_dictionary_path()
if extra is not None and extra.is_file():
cmd += ["--dictionary-extra", str(extra)]
if mode == "rebuild": if mode == "rebuild":
cmd.append("--rebuild") cmd.append("--rebuild")
elif mode == "regen": elif mode == "regen":
@@ -985,6 +1096,14 @@ class LektorUI(tk.Tk):
RoundButton(top, "Settings", self.open_settings, bg="#3a3a3a", fg=MUTED, RoundButton(top, "Settings", self.open_settings, bg="#3a3a3a", fg=MUTED,
hover="#4a4a4a", font=(F, 9), padx=13, pady=6 hover="#4a4a4a", font=(F, 9), padx=13, pady=6
).pack(side="right") ).pack(side="right")
RoundButton(top, "Queue dictionary", self.open_queue_dictionary,
bg="#3a3a3a", fg=MUTED, hover="#4a4a4a", font=(F, 9),
padx=13, pady=6).pack(side="right", padx=(0, 8))
self.comfy_btn = RoundButton(top, "ComfyUI", self.toggle_comfy,
bg="#3a3a3a", fg=MUTED, hover="#4a4a4a",
font=(F, 9), padx=13, pady=6)
self.comfy_btn.pack(side="right", padx=(0, 8))
self.after(500, self._refresh_comfy_button)
# scrollable list of job blocks # scrollable list of job blocks
mid = tk.Frame(self, bg=BG) mid = tk.Frame(self, bg=BG)
@@ -1053,6 +1172,89 @@ class LektorUI(tk.Tk):
self.stop_btn.pack(side="right") self.stop_btn.pack(side="right")
self.stop_btn.set_enabled(False) self.stop_btn.set_enabled(False)
# ── queue dictionary ────────────────────────────────────────────────────
#
# Replacements that apply to this batch only — a character's name the model
# mangles, a recurring mistranslation — without editing the shared
# dictionary.txt, which holds the rules that are true of Polish generally.
# It lives in the first queued video's job folder so it sits with the
# material it belongs to, and every job in the queue is passed it.
QUEUE_DICT_NAME = "queue_dictionary.txt"
def queue_dictionary_path(self):
"""Path in the first queued video's job folder, or None if the queue is
empty — there is nowhere to put it until a video is chosen."""
for blk in self.blocks:
folder = blk.job_folder()
if folder is not None:
return folder / self.QUEUE_DICT_NAME
return None
def open_queue_dictionary(self):
QueueDictionaryDialog(self)
# ── ComfyUI lifecycle ───────────────────────────────────────────────────
#
# The backend is normally started by hand in a terminal and left running,
# which is fine until it crashes or gets suspended: it then holds the GPU
# while answering nothing, and a queue started overnight goes nowhere. All
# of the work is in lektor_ai_comfy so the terminal script and this button
# behave identically.
def _comfy_state(self):
"""(running, healthy) — separate, because a backend can hold the port
and the GPU while no longer answering."""
pid = core.comfy_pid()
if pid is None:
return False, False
try:
core._comfy_get("/system_stats")
return True, True
except Exception:
return True, False
def _refresh_comfy_button(self):
running, healthy = self._comfy_state()
if healthy:
self.comfy_btn.set_text("ComfyUI ●")
elif running:
self.comfy_btn.set_text("ComfyUI ◍") # up but not answering
else:
self.comfy_btn.set_text("Start ComfyUI")
self.after(5000, self._refresh_comfy_button)
def toggle_comfy(self):
"""Start it, or restart it when it is up but wedged."""
core.COMFY_URL = self.defaults["comfy_url"]
running, healthy = self._comfy_state()
if healthy:
if not messagebox.askyesno(
"ComfyUI", "ComfyUI is running and answering.\n\n"
"Restart it anyway?", parent=self):
return
self.comfy_btn.set_enabled(False)
threading.Thread(target=self._comfy_worker, args=(running,),
daemon=True).start()
def _comfy_worker(self, running):
try:
if running:
self.log("Stopping ComfyUI…")
core.stop_comfy_backend()
self.log("Starting ComfyUI…")
if not core.start_comfy_backend(core.COMFY_START_SCRIPT):
self.log("Could not start ComfyUI — see the terminal window.")
return
if core.wait_for_backend(180.0):
self.log("ComfyUI is up.")
else:
self.log(f"ComfyUI did not answer within 180s. "
f"Its output is in {core.COMFY_LOG}.")
finally:
self.after(0, lambda: self.comfy_btn.set_enabled(True))
self.after(0, self._refresh_comfy_button)
def open_settings(self): def open_settings(self):
SettingsDialog(self) SettingsDialog(self)
@@ -1322,8 +1524,27 @@ class LektorUI(tk.Tk):
return return
if self.proc and self.proc.poll() is None: if self.proc and self.proc.poll() is None:
self.proc.kill() self.proc.kill()
# Leaving ComfyUI behind means ~8 GB of VRAM held by a backend nobody is
# driving. It is stopped by port, so one started by hand in a terminal
# goes too — which is the case that actually strands the GPU.
self._stop_comfy_on_exit()
self.destroy() self.destroy()
def _stop_comfy_on_exit(self):
"""Best effort: quitting must not be blocked by a backend that will not
die, so failures are reported and swallowed."""
core.COMFY_URL = self.defaults["comfy_url"]
try:
if core.comfy_pid() is None:
return
self.log("Stopping ComfyUI…")
self.update_idletasks() # let the message paint before we block
if not core.stop_comfy_backend():
self.log("ComfyUI did not stop — check for a leftover process.")
except Exception as exc: # noqa: BLE001
self.log(f"Could not stop ComfyUI: {exc}")
if __name__ == "__main__": if __name__ == "__main__":
if not CLI.is_file(): if not CLI.is_file():