lukas/lektor-ai
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

Add subtitle dictionaries, Polish number normalizer, ComfyUI lifecycle control

Browse Source 
Text processing before TTS:

- dictionary.txt rewrites words and phrases (km/h, np., units). Rules compile
  into one alternation applied in a single pass, so replacements never chain
  and longest-literal wins. Case-sensitive, whole-word.
- A per-queue second dictionary, written by the GUI into the first video's job
  folder and merged with the shared one (--dictionary-extra).
- ALL-CAPS blocks become Title Case rather than sentence case: lowercasing
  destroyed proper nouns inside captions ("NASA - MARGO MADISON") and invited
  the normalizer to "correct" the name. Paired clips in output/clips/all-caps
  showed the TTS reads both casings the same.

normalizer/ — digits to spoken Polish, in its own venv because it needs
transformers>=5, which must not touch ComfyUI's pinned versions:

- Two backends. Plain integers go to num2words for the value and a general
  Polish LLM for the case; times, dates, decimals and identifiers go to a 0.6B
  fine-tune. Measured on 40 real subtitle lines, each is clearly better than
  the other on its half. 74% of numbered blocks take the first leg.
- Guards on every answer: runaway length, repetition loops, invalid bytes,
  leftover digits, a lost minus sign, a changed number, a lost word. A failure
  falls back to num2words, which is nominative but never wrong about the value.
- ComfyUI is asked to unload before normalizing; a normalizer that does not fit
  in VRAM is spilled to RAM silently at 5x the cost.

Backend lifecycle:

- ComfyUI is found by port through /proc, so a backend started by hand is
  managed too. SIGTERM escalates to SIGKILL, and a suspended process is killed
  outright since it never runs its handlers.
- A clip that kills the backend is retried three times, each with a different
  seed (the aborts are seed-dependent, not text-dependent), then skipped and
  reported so an overnight queue keeps moving.
- start_comfyui.sh execs python, so the script's PID is ComfyUI's.
- GUI gains ComfyUI and Queue dictionary buttons; quitting stops the backend.

--resume-from now reuses the job folder's tts_input.srt instead of reparsing
the subtitles, so block N still pairs with clip_NNNN.flac.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
This commit is contained in:
claude
2026-08-12 21:55:31 +02:00
co-authored by Claude Opus 5
parent 7e2cee02e5
commit e201aadc3e
21 changed files with 3759 additions and 52 deletions
Download Patch File Download Diff File Expand all files Collapse all files
normalizer/README.md
+179
View File
@@ -0,0 +1,179 @@
# Polish number normalizer
Rewrites digits as spoken Polish words before the text reaches the TTS:
```
Wygrał 1247 złotych. → Wygrał tysiąc dwieście czterdzieści siedem złotych.
Mówimy o 25 psach. → Mówimy o dwudziestu pięciu psach.
Mam 3 koty. → Mam 3 koty. (nothing worth sending)
Brakuje nam 15000 podpisów. → Brakuje nam piętnastu tysięcy podpisów.
```
Higgs Audio v3 mispronounces raw digits — especially long ones — and sometimes
emits garbled non-words. A lookup table cannot fix that, because Polish numerals
decline by case, gender and the noun they govern (`25 psów` / `o 25 psach` /
`z 25 psami`), so this uses a small fine-tuned model.
It runs in **its own venv**, separate from ComfyUI: the models require
`transformers >= 5`, and forcing that on the working TTS install would risk its
pinned dependencies. `lektor_ai_comfy.py` drives it as a subprocess, and asks
ComfyUI to unload Higgs first so there is VRAM to work with.
## Install
```bash
normalizer/setup.sh # venv + torch (cu128) + transformers + num2words
```
Then the model, which is **gated** — accept the terms at
<https://huggingface.co/Folx/qwen3-0.6b-pl-text-normalization> with your Hugging
Face account first, then:
```bash
HF_TOKEN=hf_... normalizer/setup.sh --model
```
The inflection model is separate and **not gated**:
```bash
normalizer/venv/bin/hf download CYFRAGOVPL/PLLuM-4B-instruct-2512 \
--local-dir normalizer/models/PLLuM-4B-instruct-2512
```
Those are the only steps that need the network. Everything afterwards runs
offline (`HF_HUB_OFFLINE=1`, `TRANSFORMERS_OFFLINE=1`, both set by the caller).
Without the inflection model every block goes to the fine-tune, which is what
the pipeline did before it existed.
Without the model the component still works, on a `num2words` fallback that
handles bare integers in the nominative case only. It says so in the log, and
`lektor_ai_comfy.py` prints a warning for the run.
## Check it
```bash
normalizer/venv/bin/python normalizer/normalizer.py --status
normalizer/venv/bin/python normalizer/normalizer.py "Wygrał 1247 złotych."
normalizer/venv/bin/python -m unittest discover -s normalizer/tests -v
```
The tests assert on properties (`no 1247 left`, `contains tysiąc`) rather than
exact strings, because several case variants are correct. The ones that need the
model skip themselves when it is not downloaded.
## What it does
1. **Skip what needs nothing.** A block whose only numbers are standalone single
digits is never sent: the TTS says "pięć minut" for "5 minut" by itself, so
there is nothing to gain and a whole sentence of rewriting to risk.
2. **Route.** Two models with opposite weaknesses, measured on 40 real subtitle
lines and a 32-case battery:
| | good at | bad at |
|---|---|---|
| fine-tune (0.6B) | `6:05`, `P10`, `LO 3842`, `95,7`, dates | loses a large number's value while declining it |
| num2words + LLM | the value, always; the case | formats — a flight number becomes a quantity |
So a block goes to the second leg only when **every** number in it is a plain
integer — 74% of numbered blocks on the sample. Everything else goes to the
fine-tune.
3. **Generate.** The fine-tune gets the text as-is. The other leg gets text with
no digits at all — num2words has already produced correct nominative words —
so its only job is inflection, and it cannot get a value wrong because it
never reads a number. Greedy decoding, digit tokens banned, either way.
4. **Guard.** The answer is checked before it is used (below). A block that
fails falls back to num2words: dull, nominative, never wrong about the value.
## The guards
Both models can produce output worse than doing nothing. Each of these fired on
a real failure during development:
| check | the failure it caught |
|---|---|
| runaway length | a 10-word line came back as 200 words |
| repetition loop | `łącznie łącznie łącznie…` to the token ceiling |
| invalid characters | a wall of `�` replacement bytes |
| digits left | an un-normalized number slipped through |
| minus sign lost | `-12 stopni` → *dwanaście stopni*, inverting a temperature |
| numbers changed | `Z 8250 osobami` → *osiem tysięcy **dwustu pięciu*** (8205), and `20 lat` → *dwadzieścia **pięć** lat* (25) |
| word lost | `38 000 000 mieszkańców` → *trzydzieści osiem milionów* |
The value check compares against what num2words would say, **in both
directions** — a missing numeral and an added one are equally wrong, and a
subset check passed `dwadzieścia` → `dwadzieścia pięć` while turning 20 into 25.
Numerals are matched through a written-out canonical map rather than string
similarity — `dwieście`
and `dwustu` are one number and must match, `trzy` and `trzysta` differ by a
hundred and must not, and ordinals like `piąty` count as `pięć`. Consonant
folding was tried first and rejected good output.
It is skipped for identifiers: `LO 3842` is correctly read digit by digit, and
demanding num2words' rendering would reject a right answer.
### Deviations from the build brief, and why
- **Single digits are no longer shielded.** The brief asks for standalone
one-digit numbers to be hidden behind `KEEPNUM…` placeholders. Measured, that
scored *worse* — 6/9 against 7/9 on the model card's own examples — and it
produced an outright pronunciation error, because `3 maja` left as a digit is
read *"trzy maja"* where Polish wants *"trzeciego maja"*. It also bought
nothing on length: the TTS speaks `5` as *pięć* either way. What survives is
the cheaper half — a block containing **only** standalone single digits is
never sent at all.
- **`KEEP_RE` has extra guards.** The brief's version protects the leading `1`
of `1 350 000` as if it were standalone, and its boundary set `(?![\d.,:/-])`
rejects any number followed by a comma or full stop — on real subtitles that
skipped most of them (`Przód 85, tył 75.`). Numeric punctuation now only
continues a number when a digit follows it.
- **Two models, not one.** The brief scopes a single normalizer. The 0.6B
fine-tune performs to its advertised 85.7%, but its tail includes losing a
number's value while declining it, which no amount of prompting fixes. See
the routing table above.
- **The model loads lazily, not at import.** Same "load once, reuse" effect, but
`--help`, `--status` and the non-neural half of the suite do not pay for a
1.1 GB load they never use.
- **`num2words` errors are caught broadly.** It raises `KeyError`, not
`ValueError`, past its largest scale word.
### Known limits
Measured, and not fixable at this layer:
| input | output | |
|---|---|---|
| `Mieszka pod 12/45.` | *sto czternaście łamane przez trzy* | an address has no reliable reading without context |
| `Zadzwoń pod 601 234 567.` | read as one 9-digit number | a phone number is indistinguishable from a grouped number |
| `Zyskują 0,3 sekundy` | *trzy setne* | should be *dziesiąte*; the value guard cannot see inside a decimal |
| `To dzieło z XIX wieku.` | unchanged | Roman numerals are not handled — use `dictionary.txt` |
| `Prędkość 0,75 macha.` | unchanged | the model hallucinated a unit conversion; the guard caught it and fell back |
## Files
| File | |
|---|---|
| `normalizer.py` | The component: routing, both backends, guards, `normalize()` |
| `worker.py` | JSON-lines service `lektor_ai_comfy.py` talks to (models loaded once per run) |
| `bench.py` | Compares models on the same battery — `--compare a.json b.json` |
| `setup.sh` | venv, dependencies, one-time gated model download |
| `tests/` | Routing, numerals, guards, and the brief's acceptance checks |
| `model/`, `models/`, `venv/` | Weights and environment — not in git |
## Interface
```python
from normalizer import normalize, needs_normalization
needs_normalization("Mam 3 koty.") # False — nothing worth sending
normalize("Wygrał 1247 złotych.") # never raises, never returns None
```
`worker.py` speaks one JSON object per line on stdin/stdout, with logs on
stderr:
```
← {"ready": true, "mode": "model", "device": "cuda:0", "llm": true}
→ {"id": 1, "text": "Wygrał 1247 złotych."}
← {"id": 1, "text": "Wygrał tysiąc dwieście…", "changed": true,
"leg": "complex", "guard": "", "error": null}
```
normalizer/bench.py
+251
View File
@@ -0,0 +1,251 @@
#!/usr/bin/env python3
"""Compare normalizer models on the same sentences.
The shipped model is a 0.6B task-specific fine-tune that needs no prompt but
miscounts large numbers and occasionally drops a word. This runs any causal LM
over the same battery so the trade is measurable rather than argued about:
python bench.py # the fine-tune
python bench.py --model models/PLLuM-4B-instruct-2512 --prompted
python bench.py --model … --prompted --save pllum.json
python bench.py --compare fine-tune.json pllum.json
`--prompted` switches from the fine-tune's bare "text in, text out" contract to
an instruction + few-shot prompt, which a general instruct model needs. Digit
tokens are banned from the output either way, and decoding is greedy, so a
difference between two runs is the model and not the sampling.
Automatic flags are cheap heuristics, not a grade: `digits` means an
un-normalized digit survived, `lost` means a word from the input has no
plausible inflected match in the output. Read the text before believing either.
"""
import argparse
import json
import re
import sys
import time
from pathlib import Path
HERE = Path(__file__).resolve().parent
sys.path.insert(0, str(HERE))
import normalizer as nz # noqa: E402 (reuses its protect/restore and banned list)
SYSTEM_PROMPTED = (
"Jesteś narzędziem do normalizacji tekstu dla syntezatora mowy. "
"Zamieniasz wszystkie liczby zapisane cyframi na słowa, w poprawnej formie "
"gramatycznej wynikającej z kontekstu zdania. Nie zmieniasz, nie dodajesz "
"ani nie usuwasz żadnych innych słów. Zwracasz wyłącznie przekształcone "
"zdanie, bez komentarza."
)
# Few-shot examples chosen to pin down the three things a general model gets
# wrong without them: the case must follow the governing word, the rest of the
# sentence must survive verbatim, and the answer must be the sentence alone.
FEWSHOT = [
("Koncert dla 8250 osób.",
"Koncert dla ośmiu tysięcy dwustu pięćdziesięciu osób."),
("Populacja Polski to 38 000 000 mieszkańców.",
"Populacja Polski to trzydzieści osiem milionów mieszkańców."),
("Mówimy o 25 psach.",
"Mówimy o dwudziestu pięciu psach."),
("Wygrał 1247 złotych.",
"Wygrał tysiąc dwieście czterdzieści siedem złotych."),
]
BATTERY = [
"Sprawa dotyczy 1 350 000 ludzi.",
"Koncert dla 8250 osób.",
"Budżet wynosi 12 500 000 złotych.",
"Populacja Polski to 38 000 000 mieszkańców.",
"Zapłacili 1 500 000 000 dolarów.",
"Zostało 999 999 sztuk.",
"Numer seryjny 1234567.",
"Ponad 100 000 widzów na stadionie.",
"Mówimy o 1 350 000 ludziach.",
"Z 8250 osobami nie da się dyskutować.",
"Firma zatrudnia 2 500 pracowników.",
"Dystans 42 195 metrów.",
"Cena spadła z 15 700 000 do 8 250 000.",
"Było tam 25 psów.",
"Mówimy o 25 psach.",
"Przyjechał z 25 psami.",
"Przyglądał się 25 psom.",
"Nie ma tu 25 psów.",
"Urodził się 3 maja 1791 roku.",
"Wzrost o 16 procent w 2026 roku.",
"Spotkanie o 14:30.",
"Godzina 23:56.",
"Radio 95,7 FM.",
"Około 3,5 metra.",
"Mieszka pod 12/45.",
"Wygrał 1247 złotych.",
"To kosztuje 1999 dolarów i 99 centów.",
"Mam 3 koty i 25 psów.",
"Zostało 5 minut.",
"Temperatura opon w porządku. Przód 85, tył 75.",
"Louis Chiron miał 57 lat, kiedy wygrał Grand Prix Monako.",
"Zwykłe zdanie bez żadnych liczb.",
]
_WORD_RE = re.compile(r'[^\W\d_]+', re.UNICODE)
def lost_words(src: str, out: str) -> list:
"""Input words with no plausible match in the output.
Polish inflects, so an exact comparison is useless — `ludziach` legitimately
becomes `ludziach` and nothing else, but `psach`/`psami` share a stem. A
five-character prefix is a crude stand-in for a lemma and is good enough to
notice a noun that vanished entirely, which is the failure worth catching.
"""
def stems(text):
return {w.lower()[:5] for w in _WORD_RE.findall(text)}
missing = stems(src) - stems(out)
return sorted(w for w in missing if len(w) >= 4)
def load_prompted(model_dir: str, gpu_gib: float = None,
system: str = None, fewshot: list = None):
"""A general instruct model: system prompt, few-shot, greedy, digits banned.
`gpu_gib` caps GPU use and spills the rest to RAM. Needed for the int4
"awq" builds: transformers decompresses compressed-tensors weights back to
bfloat16 at load, so an 11B model wants ~22 GB whatever the file size says.
Offloading is slow but it is the difference between measuring a model and
guessing about it.
"""
import torch
from transformers import AutoModelForCausalLM, AutoTokenizer
tok = AutoTokenizer.from_pretrained(model_dir)
kwargs = {"dtype": torch.bfloat16, "device_map": "auto"}
if gpu_gib == 0:
# Everything in RAM. Slow, but the only way to run a compressed-tensors
# int4 build in transformers: its decompression hook expands weights to
# bfloat16 in place, and it cannot read the meta tensors that accelerate
# leaves behind when max_memory forces a partial offload.
kwargs["device_map"] = {"": "cpu"}
elif gpu_gib:
kwargs["max_memory"] = {0: f"{gpu_gib}GiB", "cpu": "80GiB"}
model = AutoModelForCausalLM.from_pretrained(model_dir, **kwargs).eval()
banned = set()
for tid in range(len(tok)):
piece = tok.decode([tid]).strip()
if piece and piece.isdigit():
banned.add(tid)
banned = sorted(banned)
print(f" {len(banned)} digit tokens banned; device "
f"{next(model.parameters()).device}", file=sys.stderr)
def run(text: str) -> str:
messages = [{"role": "system", "content": system or SYSTEM_PROMPTED}]
for user, assistant in (fewshot or FEWSHOT):
messages.append({"role": "user", "content": user})
messages.append({"role": "assistant", "content": assistant})
messages.append({"role": "user", "content": text})
prompt = tok.apply_chat_template(messages, tokenize=False,
add_generation_prompt=True)
inputs = tok(prompt, return_tensors="pt").to(model.device)
n = inputs["input_ids"].shape[1]
with torch.no_grad():
out = model.generate(**inputs, max_new_tokens=nz.MAX_NEW_TOKENS,
do_sample=False,
pad_token_id=tok.pad_token_id or tok.eos_token_id,
suppress_tokens=banned)
answer = tok.decode(out[0][n:], skip_special_tokens=True).strip()
# Instruct models like to introduce themselves; keep the last non-empty
# line, which is the sentence in every observed case.
lines = [l.strip() for l in answer.splitlines() if l.strip()]
return lines[-1] if lines else answer
return run
def main() -> int:
ap = argparse.ArgumentParser(description=__doc__,
formatter_class=argparse.RawDescriptionHelpFormatter)
ap.add_argument("--model", default=None,
help="Model directory (default: the shipped fine-tune)")
ap.add_argument("--prompted", action="store_true",
help="Use an instruction + few-shot prompt (general models)")
ap.add_argument("--gpu-gib", type=float, default=None, metavar="N",
help="Cap GPU memory and spill the rest to RAM (slow, but "
"the only way to run a model that does not fit)")
ap.add_argument("--no-protect", action="store_true",
help="Do not shield standalone single digits — normalize the "
"text exactly as the model card does")
ap.add_argument("--save", metavar="FILE", help="Write results as JSON")
ap.add_argument("--compare", nargs=2, metavar=("A", "B"),
help="Print two saved runs side by side and exit")
args = ap.parse_args()
if args.compare:
a, b = (json.loads(Path(f).read_text(encoding="utf-8")) for f in args.compare)
agree = 0
for ra, rb in zip(a["results"], b["results"]):
if ra["out"] == rb["out"]:
agree += 1
continue
print(f" in : {ra['in']}")
print(f" A : {ra['out']}")
print(f" B : {rb['out']}\n")
print(f"identical: {agree}/{len(a['results'])} "
f"A={a['model']} B={b['model']}")
return 0
if args.prompted:
if not args.model:
ap.error("--prompted needs --model")
run = load_prompted(args.model, args.gpu_gib)
label = args.model
else:
if args.model:
nz.MODEL_DIR = Path(args.model)
if not nz.load_model():
print("model unavailable; refusing to benchmark the fallback",
file=sys.stderr)
return 1
run = nz.normalize
label = str(nz.MODEL_DIR)
if args.no_protect:
nz.KEEP_RE = re.compile(r'(?!x)x') # matches nothing
label += " [no single-digit protection]"
results, t0 = [], time.time()
for text in BATTERY:
t = time.time()
out = run(text)
results.append({"in": text, "out": out, "sec": round(time.time() - t, 2),
"digits": bool(nz.DIGIT_RE.search(nz.KEEP_RE.sub("", out))),
"lost": lost_words(text, out)})
total = time.time() - t0
for r in results:
flags = []
if r["digits"]:
flags.append("DIGITS LEFT")
if r["lost"]:
flags.append("LOST: " + ", ".join(r["lost"]))
print(f" in : {r['in']}")
print(f" out: {r['out']}" + (f" [{'; '.join(flags)}]" if flags else ""))
print()
n_dig = sum(r["digits"] for r in results)
n_lost = sum(bool(r["lost"]) for r in results)
print(f"{label}\n {len(results)} sentences in {total:.1f}s "
f"({total/len(results):.2f}s each)\n"
f" {n_dig} with digits left, {n_lost} with a word apparently lost")
if args.save:
Path(args.save).write_text(json.dumps(
{"model": label, "prompted": args.prompted, "seconds": round(total, 1),
"results": results}, ensure_ascii=False, indent=1), encoding="utf-8")
print(f" saved to {args.save}")
return 0
if __name__ == "__main__":
sys.exit(main())
normalizer/normalizer.py
Executable
+645
View File
@@ -0,0 +1,645 @@
#!/usr/bin/env python3
"""Polish number normalizer — digits to spoken Polish words, for TTS.
Higgs Audio v3 mispronounces raw digits, especially long ones, and sometimes
emits garbled non-words. This component rewrites them as fully spelled-out
Polish, in the right grammatical case — which is why it is a neural model and
not a lookup table: Polish numerals decline by case, gender and the noun they
govern.
>>> normalize("Wygrał 1247 złotych.")
'Wygrał tysiąc dwieście czterdzieści siedem złotych.'
>>> normalize("Mam 3 koty.") # nothing worth sending: left alone
'Mam 3 koty.'
Model: Folx/qwen3-0.6b-pl-text-normalization, loaded from a local directory and
run offline. If it is unavailable the component degrades to a num2words
fallback and says so in the log — it never raises and never crashes the caller.
Run `python normalizer.py "some text"` for a quick check, or `python worker.py`
for the JSON-lines service that lektor_ai_comfy.py talks to.
"""
import json
import logging
import os
import re
import sys
from pathlib import Path
LOG = logging.getLogger("pl_normalizer")
HERE = Path(__file__).resolve().parent
MODEL_DIR = Path(os.environ.get("PL_NORMALIZER_MODEL", HERE / "model"))
# "auto" uses the GPU when one is free, "cpu" forces RAM. Forcing CPU is a real
# use case, not a fallback: ComfyUI keeps ~8.4 GB of VRAM resident between runs,
# so a normalizer that insists on the GPU either fights it or has to wait.
DEVICE = os.environ.get("PL_NORMALIZER_DEVICE", "auto").strip().lower()
# The second model. The 0.6B fine-tune reads odd formats well — clock times,
# race positions, call signs — but loses the value of a large number while
# declining it ("z 8250 osobami" came back as 8205). A general Polish LLM has
# the opposite profile. Neither is asked to do the other's job; see route().
LLM_DIR = Path(os.environ.get("PL_NORMALIZER_LLM",
HERE / "models" / "PLLuM-4B-instruct-2512"))
# Fixed by the fine-tune — the model was trained with this exact system prompt.
SYSTEM_PROMPT = "Zamieniasz tekst na formę mówioną."
MAX_NEW_TOKENS = 400
LONG_INPUT_CHARS = 300 # quality degrades past this; warn, never truncate
# Where a number begins and ends. Numeric punctuation only continues a number
# when a digit follows it, which is what separates "3,5" and "14:30" (one number
# each, for the model to handle) from "Przód 85, tył 75." (two, each ending at
# ordinary sentence punctuation).
#
# The build brief writes these as (?<![\d.,:/\-]) and (?![\d.,:/\-]), which also
# rejects every number followed by a comma or a full stop — on real subtitles
# that silently skipped most of them.
_NUM_START = r'(?<!\d)(?<!\d[.,:/\-])'
_NUM_END = r'(?!\d)(?![.,:/\-]\d)'
# A standalone one-digit integer is pronounced correctly by the TTS, so it stays
# a digit; anything longer or more complex goes to the model.
#
# The two space guards are not in the brief either, and are there because
# without them "1 350 000" has its leading 1 protected as if it were standalone:
# the model then sees "KEEPNUMA 350 000" and returns "1 trzysta pięćdziesiąt
# tysięcy" — the million silently lost. A digit that sits next to a group of
# exactly three is part of a grouped number, not a number of its own.
KEEP_RE = re.compile(_NUM_START + r'(?<!\d[\u00a0\u202f ])'
r'\d'
+ _NUM_END + r'(?![\u00a0\u202f ]\d{3}(?!\d))')
DIGIT_RE = re.compile(r'\d')
# Fallback path only: integers of 2+ digits, or any number written in groups of
# three. Decimals, times and dates are deliberately left alone rather than
# mangled without the model's grammar.
_INT_RE = re.compile(_NUM_START +
r'(?:\d{1,3}(?:[\u00a0\u202f .]\d{3})+|\d{2,})'
+ _NUM_END)
# Banned alongside digits so the model cannot emit a symbol the TTS would read
# as a word of the wrong language, or not read at all.
_BANNED_SYMBOLS = ["$", "%", "€", "£", "¥", "°", "§", "@", "#", "&", "×", "÷", "±",
"©", "®", "™", "→", "←", "↑", "↓", "√", "∞", "≈", "≠", "≤", "≥"]
# The instruction the general model gets. It never sees a digit — num2words has
# already produced correct nominative words — so its only job is to inflect
# them. That is why it cannot get a value wrong: it is not reading numbers.
SYSTEM_INFLECT = (
"Otrzymujesz polskie zdanie, w którym liczebniki są już zapisane słowami, "
"ale w mianowniku. Popraw wyłącznie formę gramatyczną tych liczebników tak, "
"aby pasowały do zdania (przypadek, rodzaj, liczba). Nie zmieniaj, nie "
"dodawaj ani nie usuwaj żadnych innych słów. Zwróć samo zdanie."
)
# Pins down the three things it gets wrong unprompted: the case must follow the
# governing word, everything else must survive verbatim, and the answer is the
# sentence alone. The last example is a no-op on purpose — without it the model
# invents changes when none are needed.
FEWSHOT_INFLECT = [
("Mówili o dwieście pięćdziesiąt tysięcy uchodźcach.",
"Mówili o dwustu pięćdziesięciu tysiącach uchodźców."),
("Koncert dla osiem tysięcy dwieście pięćdziesiąt osób.",
"Koncert dla ośmiu tysięcy dwustu pięćdziesięciu osób."),
("Przyjechał z dwadzieścia pięć psami.",
"Przyjechał z dwudziestoma pięcioma psami."),
("Kosztuje tysiąc dwieście czterdzieści siedem złotych.",
"Kosztuje tysiąc dwieście czterdzieści siedem złotych."),
]
_state = {"loaded": False, "tok": None, "model": None, "banned": None, "error": None}
_llm = {"loaded": False, "tok": None, "model": None, "banned": None, "error": None}
# Counted per process and reported by the worker, so a run can say which leg did
# the work and how often a guard had to step in.
STATS = {"plain": 0, "complex": 0, "guarded": 0, "reasons": {}}
# ---------------------------------------------------------------------------
# Model
# ---------------------------------------------------------------------------
def _banned_ids(tok, cache_dir: Path = None) -> list:
"""Token ids that may never appear in the output.
Scanning the whole vocabulary means one decode call per token — a few
seconds for Qwen3's 150k entries — so the result is cached beside the model.
"""
cache = (cache_dir or MODEL_DIR) / "banned_tokens.json"
size = len(tok)
if cache.is_file():
try:
data = json.loads(cache.read_text(encoding="utf-8"))
if data.get("vocab_size") == size:
return data["ids"]
except (OSError, ValueError, KeyError):
pass # stale or corrupt — just recompute
banned = set()
for tid in range(size):
piece = tok.decode([tid]).strip()
if piece and piece.isdigit():
banned.add(tid)
for sym in _BANNED_SYMBOLS:
ids = tok.encode(sym, add_special_tokens=False)
if len(ids) == 1:
banned.add(ids[0])
ids = sorted(banned)
try:
cache.write_text(json.dumps({"vocab_size": size, "ids": ids}), encoding="utf-8")
except OSError:
pass # cache is an optimisation, not a requirement
return ids
def load_model() -> bool:
"""Load model, tokenizer and banned-token list once, and keep them.
Returns True when the neural path is available. A failure here is not an
error: it is logged and the component runs on the fallback for the rest of
the process. Loading is lazy rather than at import so that tests, --help and
fallback-only runs do not pay for a 1.1 GB model they never use.
"""
if _state["loaded"]:
return _state["model"] is not None
_state["loaded"] = True
if not (MODEL_DIR / "config.json").is_file():
_state["error"] = (f"no model in {MODEL_DIR} — see README.md for the "
f"one-time download")
LOG.warning("Neural normalizer unavailable (%s). Falling back to num2words: "
"numbers will be spelled out in the nominative case only.",
_state["error"])
return False
try:
import torch
from transformers import AutoModelForCausalLM, AutoTokenizer
tok = AutoTokenizer.from_pretrained(str(MODEL_DIR))
placement = {"": "cpu"} if DEVICE == "cpu" else "auto"
try:
model = AutoModelForCausalLM.from_pretrained(
str(MODEL_DIR), dtype=torch.bfloat16, device_map=placement)
except TypeError:
# transformers < 5 spells the same argument torch_dtype.
model = AutoModelForCausalLM.from_pretrained(
str(MODEL_DIR), torch_dtype=torch.bfloat16, device_map=placement)
model.eval()
_state["tok"], _state["model"] = tok, model
_state["banned"] = _banned_ids(tok)
LOG.info("Model loaded from %s on %s (%d banned tokens)",
MODEL_DIR, next(model.parameters()).device, len(_state["banned"]))
return True
except Exception as exc: # missing torch, no VRAM, corrupt download…
_state["tok"] = _state["model"] = None
_state["error"] = f"{type(exc).__name__}: {exc}"
LOG.warning("Could not load the neural normalizer (%s). Falling back to "
"num2words: numbers will be spelled out in the nominative "
"case only.", _state["error"])
return False
def model_available() -> bool:
return load_model()
def status() -> dict:
"""What the caller needs to report: which path is live, and why."""
ok = load_model()
device = ""
if ok:
try:
device = str(next(_state["model"].parameters()).device)
except Exception:
device = "unknown"
return {"mode": "model" if ok else "fallback",
"model_dir": str(MODEL_DIR),
"device": device,
"error": _state["error"]}
def _model_normalize(text: str) -> str:
import torch
tok, model = _state["tok"], _state["model"]
messages = [{"role": "system", "content": SYSTEM_PROMPT},
{"role": "user", "content": text}]
prompt = tok.apply_chat_template(messages, tokenize=False,
add_generation_prompt=True)
inputs = tok(prompt, return_tensors="pt").to(model.device)
n = inputs["input_ids"].shape[1]
with torch.no_grad():
out = model.generate(**inputs,
max_new_tokens=MAX_NEW_TOKENS,
do_sample=False,
pad_token_id=tok.pad_token_id or tok.eos_token_id,
suppress_tokens=_state["banned"])
return tok.decode(out[0][n:], skip_special_tokens=True).strip()
def load_llm() -> bool:
"""Load the general Polish LLM used for the inflection leg.
Optional: without it every block goes to the fine-tune, which is what the
pipeline did before. Reported, never fatal.
"""
if _llm["loaded"]:
return _llm["model"] is not None
_llm["loaded"] = True
if not (LLM_DIR / "config.json").is_file():
_llm["error"] = f"no model in {LLM_DIR}"
LOG.warning("Inflection model unavailable (%s) — large numbers will be "
"normalized by the fine-tune alone.", _llm["error"])
return False
try:
import torch
from transformers import AutoModelForCausalLM, AutoTokenizer
tok = AutoTokenizer.from_pretrained(str(LLM_DIR))
placement = {"": "cpu"} if DEVICE == "cpu" else "auto"
model = AutoModelForCausalLM.from_pretrained(
str(LLM_DIR), dtype=torch.bfloat16, device_map=placement).eval()
_llm["tok"], _llm["model"] = tok, model
_llm["banned"] = _banned_ids(tok, cache_dir=LLM_DIR)
LOG.info("Inflection model loaded from %s on %s",
LLM_DIR, next(model.parameters()).device)
return True
except Exception as exc:
_llm["tok"] = _llm["model"] = None
_llm["error"] = f"{type(exc).__name__}: {exc}"
LOG.warning("Could not load the inflection model (%s) — large numbers "
"will be normalized by the fine-tune alone.", _llm["error"])
return False
def _llm_inflect(text: str) -> str:
import torch
tok, model = _llm["tok"], _llm["model"]
messages = [{"role": "system", "content": SYSTEM_INFLECT}]
for user, assistant in FEWSHOT_INFLECT:
messages.append({"role": "user", "content": user})
messages.append({"role": "assistant", "content": assistant})
messages.append({"role": "user", "content": text})
prompt = tok.apply_chat_template(messages, tokenize=False,
add_generation_prompt=True)
inputs = tok(prompt, return_tensors="pt").to(model.device)
n = inputs["input_ids"].shape[1]
with torch.no_grad():
out = model.generate(**inputs, max_new_tokens=MAX_NEW_TOKENS,
do_sample=False,
pad_token_id=tok.pad_token_id or tok.eos_token_id,
suppress_tokens=_llm["banned"])
answer = tok.decode(out[0][n:], skip_special_tokens=True).strip()
# Instruct models like to introduce their answer; the sentence is the last
# non-empty line in every case observed.
lines = [l.strip() for l in answer.splitlines() if l.strip()]
return lines[-1] if lines else answer
# ---------------------------------------------------------------------------
# Fallback
# ---------------------------------------------------------------------------
def _fallback(text: str) -> str:
"""Model unavailable. Nominative case only, bare integers only — wrong case
is still far better than the TTS garbling a long digit string."""
try:
from num2words import num2words
except ImportError:
LOG.warning("num2words is not installed — leaving %r unchanged", text)
return text
# A minus that stays a dash is a minus the TTS will not read, and -12
# spoken as 12 is the wrong temperature.
text = re.sub(r'(?<![\w])-(?=\d)', 'minus ', text)
def repl(m):
digits = re.sub(r'[\u00a0\u202f .]', '', m.group(0))
try:
return num2words(int(digits), lang="pl")
except Exception:
# num2words raises KeyError, not ValueError, past its largest scale
# word (10^66). Leaving the digits is bad; crashing a run is worse.
LOG.warning("num2words could not spell out %r — leaving it", m.group(0))
return m.group(0)
return _INT_RE.sub(repl, text)
# ---------------------------------------------------------------------------
# Routing
# ---------------------------------------------------------------------------
#
# The two models fail in opposite directions, measured on 40 real subtitle lines
# and a 32-case battery:
#
# fine-tune good: 6:05, P10, LO 3842, 95,7, 21.00, dates
# bad: loses the value of a large number while declining it
# ("z 8250 osobami" -> 8205), drops words
# num2words good: the value, always, whatever the digit grouping
# + LLM bad: destroys formats — a flight number becomes a quantity,
# "P10" becomes "Pdziesięć", times fall apart
#
# So a block goes to the second leg only when every number in it is a plain
# integer. On the sample, that is 77% of numbered blocks.
_MONTHS = ("stycznia|lutego|marca|kwietnia|maja|czerwca|lipca|sierpnia|"
"września|października|listopada|grudnia")
_COMPLEX_RE = re.compile(
r'\d[.,:/]\d' # 3,5 14:30 12/45 1.350
r'|\d\s*[-–—]\s*\d' # 10-15
r'|[^\W\d_]\d|\d[^\W\d_]' # P10, 1998cm3 — an identifier, not a count
r'|\b[A-ZĄĆĘŁŃÓŚŹŻ]{2,3}\s\d{2,}\b' # LO 3842 — call sign, read digit by digit
r'|\d+\s+(?i:' + _MONTHS + r')\b' # 13 maja — ordinal
r'|\d{4}\s+(?i:r(?:oku|\.))', # 2026 roku — ordinal
re.UNICODE)
# Case matters here, which is why IGNORECASE is scoped to the two alternatives
# that need it. Applied globally it turned the call-sign rule into "any short
# word before a number", routing "za 150 milionów" and "Przód 85" to the wrong
# leg — a third of the sample.
def route(text: str) -> str:
""""plain" when every number is a plain integer, else "complex"."""
return "complex" if _COMPLEX_RE.search(text) else "plain"
# ---------------------------------------------------------------------------
# Guards
# ---------------------------------------------------------------------------
#
# Both models can produce output that is worse than doing nothing: a repetition
# loop that runs to the token ceiling, a dropped minus sign that inverts a
# temperature, a vanished noun. These are cheap structural checks against the
# input — no second model pass — and a block that fails one falls back to
# num2words, which is dull but never wrong about the value.
_WORD_RE = re.compile(r'[^\W\d_]+', re.UNICODE)
_VOWELS = str.maketrans("", "", "aeiouyąęó")
_FOLD = str.maketrans("ąćęłńóśźż", "acelnoszz")
def _skeleton(word: str) -> str:
"""Consonants of a word, diacritics folded — a poor man's lemma, used to ask
whether a word survived generation at all. Three consonants is deliberate:
ludziach/ludzi and uchodźcach/uchodźców are the same word inflected, and a
longer key would report them as lost."""
return word.lower().translate(_FOLD).translate(_VOWELS)[:3]
# Numerals need identity, not similarity: "dwieście" and "dwustu" are the same
# number and must match, while "trzy" and "trzysta" differ by a factor of a
# hundred and must not. No consonant rule does both, so the closed class is
# written out. Patterns are tried in order, most specific first, against the
# lowercased and diacritic-folded word.
_NUMERAL_CANON = [
(r'zero', "0"),
(r'dwadziesc|dwudziest', "20"), (r'dwanasc|dwunast', "12"),
(r'dwiesc|dwust', "200"), (r'dw', "2"),
(r'trzynasc|trzynast', "13"), (r'trzydzies', "30"),
(r'trzyst', "300"), (r'trz', "3"),
(r'czternasc|czternast', "14"), (r'czterdzies', "40"),
(r'czteryst|czterest', "400"), (r'czter', "4"),
(r'pietnasc|pietnast', "15"), (r'piecdziesi', "50"),
(r'piecset', "500"), (r'pie[cs]', "5"),
(r'szesnasc|szesnast', "16"), (r'szescdziesi', "60"),
(r'szescset', "600"), (r'szes', "6"),
(r'siedemnasc|siedemnast', "17"), (r'siedemdziesi', "70"),
(r'siedemset', "700"), (r'siedem|siedmi', "7"),
(r'osiemnasc|osiemnast', "18"), (r'osiemdziesi', "80"),
(r'osiemset', "800"), (r'osiem|osm', "8"),
(r'dziewietnasc|dziewietnast', "19"), (r'dziewiecdziesi', "90"),
(r'dziewiecset', "900"), (r'dziewi', "9"),
(r'dziesi', "10"), (r'jedenasc|jedenast', "11"), (r'jed', "1"),
# Ordinals denote the same number and are often the better reading —
# "na pięćdziesiątym piątym okrążeniu". Without them the guard rejects
# correct output.
(r'pierwsz', "1"), (r'drug', "2"), (r'czwart', "4"), (r'piat', "5"),
(r'szost', "6"), (r'siodm', "7"), (r'setn', "100"),
(r'st[ou]$', "100"), (r'tysi', "1000"),
(r'miliard', "1e9"), (r'milion', "1e6"),
]
_NUMERAL_CANON = [(re.compile("^(?:" + pat + ")"), value)
for pat, value in _NUMERAL_CANON]
def _numeral(word: str) -> str:
"""The number a word denotes, whatever case it is in, or "" if it is not a
numeral."""
folded = word.lower().translate(_FOLD)
for pattern, value in _NUMERAL_CANON:
if pattern.match(folded):
return value
return ""
# Same as _INT_RE but from one digit up. Only ever applied to blocks the router
# called "plain", where every number is a bare integer — and there the model
# expands single digits too, so the expectation has to include them.
_ANY_INT_RE = re.compile(_NUM_START +
r'(?:\d{1,3}(?:[\u00a0\u202f .]\d{3})+|\d+)'
+ _NUM_END)
def _expected_numerals(text: str) -> list:
"""The numerals num2words would produce for `text`, canonicalised."""
try:
from num2words import num2words
except ImportError:
return []
def repl(m):
digits = re.sub(r'[\u00a0\u202f .]', '', m.group(0))
try:
return num2words(int(digits), lang="pl")
except Exception:
return m.group(0)
spelled = _ANY_INT_RE.sub(repl, text)
return sorted(n for n in map(_numeral, _WORD_RE.findall(spelled)) if n)
def _repeats(text: str) -> bool:
"""A word five times over, or a phrase three times — a generation loop.
PLLuM produced 'łącznie' eighty times on one input; the fine-tune produced
a wall of replacement characters on another."""
words = text.lower().split()
run = 1
for a, b in zip(words, words[1:]):
run = run + 1 if a == b else 1
if run >= 4:
return True
for size in (2, 3):
seen = {}
for i in range(len(words) - size + 1):
gram = tuple(words[i:i + size])
seen[gram] = seen.get(gram, 0) + 1
if seen[gram] >= 3:
return True
return False
def _guard(original: str, candidate: str, check_value: bool) -> str:
"""Return the reason `candidate` is unusable, or "" if it passes."""
if not candidate.strip():
return "empty"
if len(candidate) > 3 * len(original) + 60:
return "runaway length"
if _repeats(candidate):
return "repetition loop"
if "�" in candidate:
return "invalid characters"
if DIGIT_RE.search(candidate):
return "digits left in output"
if re.search(r'(?<![\w])-\s?\d', original) and "minus" not in candidate.lower():
return "minus sign lost"
if check_value:
# The plain leg's model is given text with no digits in it and asked
# only to inflect, so the numerals it returns must be exactly the ones
# num2words put there — same values, same count. A one-directional
# check is not enough: "dwadzieścia" -> "dwadzieścia pięć" keeps every
# expected word and still turns 20 into 25.
#
# The expectation comes from the full num2words rendering rather than
# from the digits alone, so numerals already written as words in the
# subtitle ("350 tysięcy") are counted too.
want = _expected_numerals(original)
got = sorted(n for n in map(_numeral, _WORD_RE.findall(candidate)) if n)
if want != got:
missing = [n for n in want if n not in got]
extra = [n for n in got if n not in want]
detail = (f"missing {','.join(missing)}" if missing else "") + \
(" " if missing and extra else "") + \
(f"added {','.join(extra)}" if extra else "reordered")
return f"numbers changed ({detail})"
# A word that vanished entirely. Short words are skipped because the
# dictionary legitimately expands them (ul. -> ulicy, zł -> złote).
have = [_skeleton(w) for w in _WORD_RE.findall(candidate)]
for word in _WORD_RE.findall(original):
if len(word) >= 5 and _skeleton(word) not in have:
return f"word lost ({word})"
return ""
# ---------------------------------------------------------------------------
# Decide / normalize
# ---------------------------------------------------------------------------
def needs_normalization(text: str) -> bool:
"""True when `text` holds a number the TTS would get wrong — anything with
a digit, once standalone single digits are discounted.
A block whose only numbers are standalone single digits is not sent at all:
the TTS says "pięć minut" for "5 minut" by itself, so there is nothing to
gain and a whole sentence of rewriting to risk. Inside a block that *is*
sent, single digits are normalized along with everything else.
lektor_ai_comfy.py applies the same rule before sending a block, so blocks
that need nothing never cross the process boundary. Keep the two in step.
"""
return bool(DIGIT_RE.search(KEEP_RE.sub("", text)))
def normalize(text: str) -> str:
"""Return `text` with its numbers spelled out in Polish. Never raises."""
return normalize_detail(text)[0]
def normalize_detail(text: str) -> tuple:
"""(result, leg, guard_reason) — the caller reports which leg did the work.
Plain integers go to num2words for the value and the LLM for the grammar,
which is the only arrangement measured to get both right. Everything else
goes to the fine-tune, which understands the formats. Either way the result
is checked before it is returned.
"""
if not text or not DIGIT_RE.search(text):
return text, "skipped", ""
if not needs_normalization(text):
return text, "skipped", "" # only standalone single digits
if len(text) > LONG_INPUT_CHARS:
LOG.warning("Input is %d characters; quality degrades past %d. "
"Processing it anyway.", len(text), LONG_INPUT_CHARS)
leg = route(text)
if leg == "plain" and load_llm():
STATS["plain"] += 1
nominative = _fallback(text) # deterministic: the value cannot be wrong
try:
candidate = _llm_inflect(nominative)
except Exception as exc:
LOG.warning("Inflection failed (%s: %s)", type(exc).__name__, exc)
candidate = nominative
safe, check_value = nominative, True
else:
leg = "complex" # no inflection model: the fine-tune takes it
STATS["complex"] += 1
if load_model():
try:
candidate = _model_normalize(text)
except Exception as exc:
LOG.warning("Generation failed (%s: %s)", type(exc).__name__, exc)
candidate = _fallback(text)
else:
candidate = _fallback(text)
# The value check needs a num2words expectation, which only exists for
# plain integers — "LO 3842" is correctly read digit by digit, and
# demanding "trzy tysiące osiemset" of it would reject a good answer.
safe, check_value = _fallback(text), False
reason = _guard(text, candidate, check_value)
if not reason:
return candidate, leg, ""
STATS["guarded"] += 1
STATS["reasons"][reason.split(" (")[0]] = \
STATS["reasons"].get(reason.split(" (")[0], 0) + 1
LOG.warning("Rejected %r -> %r (%s); using %r", text, candidate, reason, safe)
return (safe if safe.strip() else text), leg, reason
# ---------------------------------------------------------------------------
# CLI
# ---------------------------------------------------------------------------
def main(argv=None) -> int:
import argparse
parser = argparse.ArgumentParser(
description="Spell out Polish numbers for TTS.",
epilog="With no TEXT, reads lines from stdin and writes them normalized.")
parser.add_argument("text", nargs="*", metavar="TEXT")
parser.add_argument("--status", action="store_true",
help="Report which path is live and exit")
parser.add_argument("--fallback-only", action="store_true",
help="Skip the model, use num2words (for testing)")
parser.add_argument("-v", "--verbose", action="store_true")
args = parser.parse_args(argv)
logging.basicConfig(level=logging.INFO if args.verbose else logging.WARNING,
format="%(levelname)s: %(message)s", stream=sys.stderr)
if args.fallback_only:
_state.update(loaded=True, model=None, tok=None,
error="--fallback-only requested")
if args.status:
print(json.dumps(status(), ensure_ascii=False, indent=2))
return 0
if args.text:
print(normalize(" ".join(args.text)))
return 0
for line in sys.stdin:
print(normalize(line.rstrip("\n")), flush=True)
return 0
if __name__ == "__main__":
sys.exit(main())
normalizer/requirements.txt
+13
View File
@@ -0,0 +1,13 @@
# Polish number normalizer — isolated environment.
#
# Deliberately NOT installed into the ComfyUI / Higgs Audio environment: the
# model needs transformers >= 5, and forcing that on the working TTS install
# risks its pinned dependencies. The two run as separate processes.
# 4.x tokenizes the Qwen3 chat template differently and silently produces wrong
# output even under greedy decode. This is a hard requirement, not a preference.
transformers>=5.0.0
torch>=2.1.0 # install the CUDA build matching the host, see README
accelerate>=0.26.0 # device_map="auto"
num2words>=0.5.13 # fallback path only
normalizer/setup.sh
Executable
+98
View File
@@ -0,0 +1,98 @@
#!/usr/bin/env bash
# Create the isolated environment for the Polish number normalizer.
#
# ./setup.sh venv + dependencies
# ./setup.sh --model also download the model (needs HF_TOKEN)
# ./setup.sh --cpu CPU-only torch, for a machine without an NVIDIA GPU
#
# Kept separate from the ComfyUI environment on purpose: this needs
# transformers >= 5, and the TTS install must not be disturbed.
set -euo pipefail
HERE="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)"
VENV="$HERE/venv"
MODEL_DIR="$HERE/model"
MODEL_REPO="Folx/qwen3-0.6b-pl-text-normalization"
TORCH_INDEX="https://download.pytorch.org/whl/cu128" # same wheels as ComfyUI
WANT_MODEL=0
CPU_ONLY=0
for arg in "$@"; do
case "$arg" in
--model) WANT_MODEL=1 ;;
--cpu) CPU_ONLY=1 ;;
-h|--help) sed -n '2,10p' "${BASH_SOURCE[0]}" | sed 's/^# \?//'; exit 0 ;;
*) echo "Unknown option: $arg" >&2; exit 2 ;;
esac
done
say() { printf '\n\033[1;36m==> %s\033[0m\n' "$1"; }
say "Creating $VENV"
python3 -m venv "$VENV"
"$VENV/bin/pip" install --upgrade pip --quiet
say "Installing torch"
if [ "$CPU_ONLY" = 1 ]; then
"$VENV/bin/pip" install torch --index-url https://download.pytorch.org/whl/cpu
else
"$VENV/bin/pip" install torch --index-url "$TORCH_INDEX"
fi
say "Installing transformers, accelerate, num2words"
"$VENV/bin/pip" install "transformers>=5.0.0" "accelerate>=0.26.0" "num2words>=0.5.13"
if [ "$WANT_MODEL" = 1 ]; then
say "Downloading $MODEL_REPO"
"$VENV/bin/pip" install --quiet "huggingface_hub[cli]"
# Either an HF_TOKEN in the environment or a stored `hf auth login` works;
# the CLI picks up both on its own. Only say something when there is
# neither, because the failure then is a confusing "access denied".
if [ -z "${HF_TOKEN:-}" ] && ! "$VENV/bin/hf" auth whoami >/dev/null 2>&1; then
cat >&2 <<EOF
No Hugging Face credentials. The repo is gated (approval is automatic, but it
has to be your account that asks):
1. Open https://huggingface.co/$MODEL_REPO while signed in and accept the terms
2. Either $VENV/bin/hf auth login
or HF_TOKEN=hf_... $0 --model
EOF
exit 1
fi
"$VENV/bin/hf" download "$MODEL_REPO" --local-dir "$MODEL_DIR"
fi
say "Checking the installation"
"$VENV/bin/python" - <<'PY'
import sys
try:
import torch, transformers
print(f" torch {torch.__version__} cuda_available={torch.cuda.is_available()}")
if torch.cuda.is_available():
print(f" device: {torch.cuda.get_device_name(0)}")
print(f" transformers {transformers.__version__}")
if int(transformers.__version__.split('.')[0]) < 5:
print(" WARNING: transformers < 5 produces wrong output with this model",
file=sys.stderr)
except ImportError as exc:
print(f" {exc}", file=sys.stderr); sys.exit(1)
PY
"$VENV/bin/python" "$HERE/normalizer.py" --status
if [ ! -f "$MODEL_DIR/config.json" ]; then
cat <<EOF
The environment is ready, but the model is not downloaded yet, so the
normalizer will run on the degraded num2words fallback.
1. Accept the terms at https://huggingface.co/$MODEL_REPO
2. HF_TOKEN=hf_... $HERE/setup.sh --model
EOF
else
say "Ready. Try: $VENV/bin/python $HERE/normalizer.py 'Wygrał 1247 złotych.'"
fi
normalizer/tests/test_normalizer.py
+375
View File
@@ -0,0 +1,375 @@
#!/usr/bin/env python3
"""Acceptance checks for the Polish number normalizer.
The model produces valid case variants, so these assert on *properties* — a
digit is gone, a word stem is present — never on an exact string, except where
the behaviour genuinely is exact (model-not-called, pass-through).
Run them in the normalizer venv:
normalizer/venv/bin/python -m pytest normalizer/tests -v
They pass with networking disabled. Tests that need the neural path are skipped
automatically when the model has not been downloaded; the rest exercise the
fallback and the protect/decide/restore logic, which is where the integration
bugs actually live.
"""
import os
import re
import sys
import unittest
from pathlib import Path
sys.path.insert(0, str(Path(__file__).resolve().parent.parent))
os.environ.setdefault("HF_HUB_OFFLINE", "1")
os.environ.setdefault("TRANSFORMERS_OFFLINE", "1")
import normalizer # noqa: E402
HAVE_MODEL = normalizer.model_available()
needs_model = unittest.skipUnless(
HAVE_MODEL, "model not downloaded — see normalizer/README.md")
DIGITS = re.compile(r'\d')
class Selection(unittest.TestCase):
"""Which blocks are worth sending at all. lektor_ai_comfy.py mirrors this
rule to keep blocks that need nothing from crossing the process boundary."""
def test_single_digit_is_not_worth_normalizing(self):
self.assertFalse(normalizer.needs_normalization("Mam 3 koty."))
def test_no_digits_at_all(self):
self.assertFalse(normalizer.needs_normalization("Zwykłe zdanie."))
def test_multi_digit(self):
self.assertTrue(normalizer.needs_normalization("Wygrał 1247 złotych."))
def test_complex_expressions_are_not_single_digits(self):
for text in ("Spotkanie o 9:30.", "Około 3,5 metra.", "Mieszka pod 12/45.",
"O 14:30 w środę."):
with self.subTest(text=text):
self.assertTrue(normalizer.needs_normalization(text))
def test_thousands_with_spaces(self):
self.assertTrue(normalizer.needs_normalization("populacja 1 350 000 osób"))
def test_single_digit_before_punctuation_is_still_single(self):
for text in ("Mam 3, potem 4.", "Zostało 5.", "Tylko 7!"):
with self.subTest(text=text):
self.assertFalse(normalizer.needs_normalization(text))
def test_multi_digit_before_punctuation_is_caught(self):
self.assertTrue(normalizer.needs_normalization("Przód 85, tył 75."))
class ClientRuleInSync(unittest.TestCase):
"""lektor_ai_comfy.py keeps its own copy of the selection rule so blocks
that need nothing never start a subprocess. A copy that drifts sends the
wrong blocks — or silently stops sending any — so it is checked here, where
the authoritative rule lives. The caller is pure standard library, so this
venv can import it."""
SAMPLES = [
"Mam 3 koty.", "Zwykłe zdanie.", "Wygrał 1247 złotych.",
"Przód 85, tył 75.", "populacja 1 350 000 osób", "Radio 95,7 FM.",
"Spotkanie o 14:30.", "Mieszka pod 12/45.", "Około 3,5 metra.",
"Formuła 1", "Masz F2, masz F3.", "Zostało 5.", "Mam 3, potem 4.",
"W 2026 roku.", "Tylko 7!", "1 pies, 2 koty, 300 myszy.",
]
def test_same_verdict_on_every_sample(self):
sys.path.insert(0, str(Path(__file__).resolve().parents[2]))
try:
import lektor_ai_comfy
except ImportError as exc: # pragma: no cover - only if moved
self.skipTest(f"caller not importable: {exc}")
for text in self.SAMPLES:
with self.subTest(text=text):
self.assertEqual(lektor_ai_comfy.needs_number_normalization(text),
normalizer.needs_normalization(text))
class Boundaries(unittest.TestCase):
"""KEEP_RE no longer shields anything — it only decides whether a block is
worth sending. These pin down where a number starts and ends."""
def test_grouped_thousands_are_not_split(self):
"""The leading 1 of "1 350 000" is not a standalone single digit; if it
counted as one the block might not be sent at all."""
protected = normalizer.KEEP_RE.sub("X", "populacja 1 350 000 osób")
self.assertIn("1 350 000", protected)
def test_complex_numbers_are_not_single_digits(self):
protected = normalizer.KEEP_RE.sub("X", "Mam 3 koty, 25 psów i 3,5 kg.")
self.assertNotIn(" 3 ", protected) # standalone single digit
self.assertIn("25", protected) # multi-digit untouched
self.assertIn("3,5", protected) # decimal untouched
class Routing(unittest.TestCase):
"""Which leg a block goes to. The two models fail in opposite directions,
so a misrouted block gets the worse of the two."""
PLAIN = ["Wygrał 1247 złotych.", "Mówili o 250000 uchodźcach.",
"Przód 85, tył 75.", "Małpi gaj za 150 milionów dolarów.",
"Bardzo. Ma z 80 lat.", "populacja 1 350 000 osób",
# A range written in words is two plain integers; only a dash
# between digits makes it a compound the fine-tune reads better.
"Ma od 15 do 20 lat."]
COMPLEX = ["Spotkanie o 14:30.", "Radio 95,7 FM.", "Mieszka pod 12/45.",
"Teraz na P12.", "Lot LO 3842 jest opóźniony.",
"Gdzie 13 maja 1950 roku się zaczęło.", "Ma od 10-20 lat.",
"Silnik ma 1998 cm3 pojemności.", "W 2026 roku."]
def test_plain(self):
for text in self.PLAIN:
with self.subTest(text=text):
self.assertEqual(normalizer.route(text), "plain")
def test_complex(self):
for text in self.COMPLEX:
with self.subTest(text=text):
self.assertEqual(normalizer.route(text), "complex")
def test_case_sensitivity_of_the_call_sign_rule(self):
"""Applied case-insensitively this rule matched any short word before a
number and misrouted a third of the sample."""
self.assertEqual(normalizer.route("Lot LO 3842 wystartował."), "complex")
self.assertEqual(normalizer.route("Zapłacił za 150 złotych."), "plain")
class Numerals(unittest.TestCase):
"""The value guard needs numeral identity across inflection, which no
consonant rule provides: dwieście/dwustu are one number, trzy/trzysta are
not."""
SAME = [("dwieście", "dwustu"), ("pięćdziesiąt", "pięćdziesięciu"),
("trzydzieści", "trzydziestu"), ("czterdzieści", "czterdziestu"),
("tysiąc", "tysiącach"), ("milion", "milionach"),
("osiem", "ośmiu"), ("pięć", "piątym"), ("dwadzieścia", "dwudziesty")]
DIFFERENT = [("trzy", "trzysta"), ("pięć", "pięćdziesiąt"),
("dwa", "dwieście"), ("milion", "miliard"),
("sto", "sześćset"), ("cztery", "czternaście")]
def test_same_number(self):
for a, b in self.SAME:
with self.subTest(pair=(a, b)):
self.assertEqual(normalizer._numeral(a), normalizer._numeral(b))
self.assertTrue(normalizer._numeral(a))
def test_different_numbers(self):
for a, b in self.DIFFERENT:
with self.subTest(pair=(a, b)):
self.assertNotEqual(normalizer._numeral(a), normalizer._numeral(b))
def test_ordinary_words_are_not_numerals(self):
for word in ("koty", "uchodźcach", "okrążenie", "prędkość"):
self.assertEqual(normalizer._numeral(word), "")
class Guards(unittest.TestCase):
"""Every check here fired on a real failure during development. The
false-positive cases matter just as much: a guard that rejects good output
silently downgrades the run."""
def test_accepts_correct_output(self):
for original, candidate in [
("Mówili o 250000 uchodźcach.",
"Mówili o dwustu pięćdziesięciu tysiącach uchodźców."),
("To wypis sprzed 30 lat.", "To wypis sprzed trzydziestu lat."),
("kolizja na 55 okrążeniu", "kolizja na pięćdziesiątym piątym okrążeniu"),
("misja to Apollo 25", "misja to Apollo dwudziesty piąty"),
]:
with self.subTest(candidate=candidate):
self.assertEqual(normalizer._guard(original, candidate, True), "")
def test_value_changed(self):
reason = normalizer._guard("Z 8250 osobami.",
"Z osiem tysięcy dwustu pięciu osobami.", True)
self.assertIn("numbers changed", reason)
def test_added_numeral(self):
"""The check must run both ways. "dwadzieścia" -> "dwadzieścia pięć"
keeps every expected word and still turns 20 into 25 — a one-directional
subset check passed this and shipped 25 years of service as 20."""
reason = normalizer._guard("Nagroda Za 20 Lat Pracy.",
"Nagroda Za dwadzieścia Pięć Lat Pracy.", True)
self.assertIn("added", reason)
def test_single_digits_count_toward_the_expectation(self):
"""num2words skips one-digit numbers but the model expands them, so the
expectation must cover them or "trzy" looks like an invention."""
self.assertEqual(normalizer._guard("Mam 3 koty i 25 psów.",
"Mam trzy koty i dwadzieścia pięć psów.",
True), "")
def test_word_lost(self):
reason = normalizer._guard("Populacja to 38000000 mieszkańców.",
"Populacja to trzydzieści osiem milionów.", True)
self.assertIn("word lost", reason)
def test_minus_lost(self):
reason = normalizer._guard("Na dworze jest -12 stopni.",
"Na dworze jest dwanaście stopni.", True)
self.assertIn("minus", reason)
def test_repetition_loop(self):
reason = normalizer._guard("Zapłaciłem 1234,56 zł.",
"Zapłaciłem " + "łącznie " * 40 + "tysiąc.", False)
self.assertTrue(reason)
def test_invalid_characters(self):
self.assertIn("invalid", normalizer._guard("Dnia 15 września 1631 roku.",
"Dnia " + "�" * 30, False))
def test_digits_left(self):
self.assertIn("digits", normalizer._guard("Ma 25 lat.", "Ma 25 lat.", False))
def test_value_check_is_skipped_for_identifiers(self):
""""LO 3842" is correctly read digit by digit, so demanding num2words'
rendering of it would reject a good answer."""
self.assertEqual(
normalizer._guard("Lot LO 3842 jest opóźniony.",
"Lot LO trzy osiem cztery dwa jest opóźniony.",
False), "")
class Fallback(unittest.TestCase):
"""Degraded path: nominative only, bare integers only. It must still never
lose a number or raise."""
def test_expands_multi_digit(self):
out = normalizer._fallback("Wygrał 1247 złotych.")
self.assertNotIn("1247", out)
self.assertIn("tysiąc", out)
def test_keeps_single_digit(self):
self.assertEqual(normalizer._fallback("Mam 3 koty."), "Mam 3 koty.")
def test_large_number_with_separators(self):
out = normalizer._fallback("populacja 1 350 000 osób")
self.assertNotIn("350", out)
self.assertIn("milion", out)
def test_leaves_decimals_and_times_alone(self):
for text in ("Około 3,5 metra.", "Spotkanie o 9:30.", "Radio 95,7 FM."):
self.assertEqual(normalizer._fallback(text), text)
def test_number_followed_by_sentence_punctuation(self):
"""A comma or full stop ends a number; it does not disqualify it. Real
subtitles are full of "Przód 85, tył 75." and the brief's boundary set
skipped every one of them."""
out = normalizer._fallback("Przód 85, tył 75.")
self.assertNotIn("85", out)
self.assertNotIn("75", out)
self.assertIn("osiemdziesiąt pięć", out)
self.assertIn("siedemdziesiąt pięć", out)
def test_never_raises(self):
for text in ("", "———", "9" * 400, "1/0", "0000"):
with self.subTest(text=text[:20]):
normalizer._fallback(text)
class NormalizeContract(unittest.TestCase):
"""Properties that hold on both paths — this is what the caller relies on."""
def test_no_digits_returns_input_unchanged(self):
text = "Zwykłe zdanie bez liczb."
self.assertEqual(normalizer.normalize(text), text)
def test_only_single_digits_skips_the_model(self):
calls = []
original = normalizer._model_normalize
normalizer._model_normalize = lambda t: calls.append(t) or t
try:
self.assertEqual(normalizer.normalize("Mam 3 koty."), "Mam 3 koty.")
finally:
normalizer._model_normalize = original
self.assertEqual(calls, [], "model must not be called for single digits")
def test_block_of_only_single_digits_is_returned_untouched(self):
"""The cheap half of the brief's idea, kept: nothing to gain from
sending it, a whole sentence of rewriting to risk."""
for text in ("Mam 3 koty.", "Zostało 5 minut.", "Tylko 7!"):
with self.subTest(text=text):
self.assertEqual(normalizer.normalize(text), text)
def test_empty_and_odd_inputs(self):
self.assertEqual(normalizer.normalize(""), "")
self.assertEqual(normalizer.normalize(" "), " ")
def test_generation_error_falls_back(self):
original = normalizer._model_normalize
def boom(_):
raise RuntimeError("CUDA out of memory")
normalizer._model_normalize = boom
try:
out = normalizer.normalize("Wygrał 1247 złotych.")
finally:
normalizer._model_normalize = original
self.assertNotIn("1247", out)
self.assertIn("tysiąc", out)
@needs_model
class Neural(unittest.TestCase):
"""The acceptance list from the build brief, against the real model."""
def test_multi_digit_expanded(self):
out = normalizer.normalize("Wygrał 1247 złotych.")
self.assertNotIn("1247", out)
self.assertIn("tysiąc", out.lower())
def test_large_number(self):
out = normalizer.normalize("Miasto ma populację 1 350 000 osób.")
self.assertNotIn("350", out)
self.assertIn("milion", out.lower())
def test_mixed_single_and_multi(self):
"""A block sent for its multi-digit number gets its single digits
normalized too — that is the point of dropping the placeholders, and
it is what makes "3 maja" come out as "trzeciego maja"."""
out = normalizer.normalize("Mam 3 koty i 25 psów.").lower()
self.assertNotIn("3", out)
self.assertIn("trzy", out)
self.assertIn("dwadzieścia pięć", out)
def test_leg_is_reported(self):
self.assertEqual(normalizer.normalize_detail("Mam 3 koty.")[1], "skipped")
self.assertEqual(normalizer.normalize_detail("Jest dopiero 21.00.")[1],
"complex")
def test_ordinal_date_is_fixed(self):
"""The reason the placeholders had to go: left as a digit, the TTS
reads "3 maja" as "trzy maja"."""
out = normalizer.normalize("Urodził się 3 maja 1791 roku.").lower()
self.assertIn("trzeciego maja", out)
def test_no_digits_leak(self):
out = normalizer.normalize("Pędzi 300 kilometrów na godzinę.")
self.assertFalse(DIGITS.search(out))
def test_case_declension(self):
"""The whole reason for a neural normalizer: 'o 25 psach' is locative,
so the numeral must not come out nominative."""
out = normalizer.normalize("Mówimy o 25 psach.").lower()
self.assertNotIn("25", out)
self.assertNotIn("dwadzieścia pięć psach", out)
def test_time(self):
out = normalizer.normalize("Spotkanie o 14:30.")
self.assertFalse(DIGITS.search(out))
def test_offline(self):
self.assertEqual(os.environ.get("HF_HUB_OFFLINE"), "1")
self.assertFalse(DIGITS.search(normalizer.normalize("Ma 99 lat.")))
if __name__ == "__main__":
print(f"model available: {HAVE_MODEL}", file=sys.stderr)
unittest.main(verbosity=2)
normalizer/worker.py
Executable
+74
View File
@@ -0,0 +1,74 @@
#!/usr/bin/env python3
"""JSON-lines front end for normalizer.py.
The normalizer needs `transformers>=5`, which must not be installed into the
ComfyUI/Higgs environment, so it lives in its own venv and is driven as a
separate process. Loading a 1.1 GB model per subtitle block would be absurd, so
this is a long-lived worker: the caller starts it once per run, streams blocks
through it, and closes stdin when done.
Protocol — one JSON object per line, in both directions:
← {"ready": true, "mode": "model", "device": "cuda:0", "llm": true, …}
→ {"id": 1, "text": "Wygrał 1247 złotych."}
← {"id": 1, "text": "Wygrał tysiąc dwieście…", "changed": true,
"leg": "plain", "guard": "", "error": null}
→ {"quit": true} (or just close stdin)
`leg` is which model answered — "plain" for num2words plus the inflection
model, "complex" for the fine-tune, "skipped" when the block needed nothing.
`guard` names the check that rejected the model's answer, empty when it passed.
stdout carries the protocol and nothing else; every log line, warning and
traceback goes to stderr. A request that fails is answered with the original
text and an `error` field — the worker does not die on one bad block.
"""
import json
import logging
import sys
import normalizer
def _send(obj) -> None:
sys.stdout.write(json.dumps(obj, ensure_ascii=False) + "\n")
sys.stdout.flush()
def main() -> int:
logging.basicConfig(level=logging.INFO, format="%(levelname)s: %(message)s",
stream=sys.stderr)
state = normalizer.status() # loads the model, or reports why not
state["llm"] = normalizer.load_llm() # the inflection leg is optional
state["llm_dir"] = str(normalizer.LLM_DIR)
_send({"ready": True, **state})
for line in sys.stdin:
line = line.strip()
if not line:
continue
try:
req = json.loads(line)
except ValueError as exc:
_send({"id": None, "text": None, "error": f"bad request: {exc}"})
continue
if req.get("quit"):
break
text = req.get("text") or ""
leg, guard = "", ""
try:
result, leg, guard = normalizer.normalize_detail(text)
error = None
except Exception as exc: # normalize_detail() is not meant to raise
logging.exception("normalize_detail() raised")
result, error = text, f"{type(exc).__name__}: {exc}"
_send({"id": req.get("id"), "text": result, "changed": result != text,
"leg": leg, "guard": guard, "error": error})
return 0
if __name__ == "__main__":
sys.exit(main())