lektor-ai/normalizer/README.md

Polish number normalizer

Rewrites digits as spoken Polish words before the text reaches the TTS:

Wygrał 1247 złotych.        →  Wygrał tysiąc dwieście czterdzieści siedem złotych.
Mówimy o 25 psach.          →  Mówimy o dwudziestu pięciu psach.
Mam 3 koty.                 →  Mam 3 koty.          (nothing worth sending)
Brakuje nam 15000 podpisów. →  Brakuje nam piętnastu tysięcy podpisów.

Higgs Audio v3 mispronounces raw digits — especially long ones — and sometimes emits garbled non-words. A lookup table cannot fix that, because Polish numerals decline by case, gender and the noun they govern (25 psów / o 25 psach / z 25 psami), so this uses a small fine-tuned model.

It runs in its own venv, separate from ComfyUI: the models require transformers >= 5, and forcing that on the working TTS install would risk its pinned dependencies. lektor_ai_comfy.py drives it as a subprocess, and asks ComfyUI to unload Higgs first so there is VRAM to work with.

Install

normalizer/setup.sh          # venv + torch (cu128) + transformers + num2words

Then the model, which is gated — accept the terms at https://huggingface.co/Folx/qwen3-0.6b-pl-text-normalization with your Hugging Face account first, then:

HF_TOKEN=hf_... normalizer/setup.sh --model

The inflection model is separate and not gated:

normalizer/venv/bin/hf download CYFRAGOVPL/PLLuM-4B-instruct-2512 \
    --local-dir normalizer/models/PLLuM-4B-instruct-2512

Those are the only steps that need the network. Everything afterwards runs offline (HF_HUB_OFFLINE=1, TRANSFORMERS_OFFLINE=1, both set by the caller). Without the inflection model every block goes to the fine-tune, which is what the pipeline did before it existed.

Without the model the component still works, on a num2words fallback that handles bare integers in the nominative case only. It says so in the log, and lektor_ai_comfy.py prints a warning for the run.

Check it

normalizer/venv/bin/python normalizer/normalizer.py --status
normalizer/venv/bin/python normalizer/normalizer.py "Wygrał 1247 złotych."
normalizer/venv/bin/python -m unittest discover -s normalizer/tests -v

The tests assert on properties (no 1247 left, contains tysiąc) rather than exact strings, because several case variants are correct. The ones that need the model skip themselves when it is not downloaded.

What it does

1. Skip what needs nothing. A block whose only numbers are standalone single digits is never sent: the TTS says "pięć minut" for "5 minut" by itself, so there is nothing to gain and a whole sentence of rewriting to risk.

2. Route. Two models with opposite weaknesses, measured on 40 real subtitle lines and a 32-case battery:

   	good at	bad at
   fine-tune (0.6B)	6:05, P10, LO 3842, 95,7, dates	loses a large number's value while declining it
   num2words + LLM	the value, always; the case	formats — a flight number becomes a quantity

   So a block goes to the second leg only when every number in it is a plain integer — 74% of numbered blocks on the sample. Everything else goes to the fine-tune.

3. Generate. The fine-tune gets the text as-is. The other leg gets text with no digits at all — num2words has already produced correct nominative words — so its only job is inflection, and it cannot get a value wrong because it never reads a number. Greedy decoding, digit tokens banned, either way.

4. Guard. The answer is checked before it is used (below). A block that fails falls back to num2words: dull, nominative, never wrong about the value.

The guards

Both models can produce output worse than doing nothing. Each of these fired on a real failure during development:

check	the failure it caught
runaway length	a 10-word line came back as 200 words
repetition loop	łącznie łącznie łącznie… to the token ceiling
invalid characters	a wall of � replacement bytes
digits left	an un-normalized number slipped through
minus sign lost	-12 stopni → dwanaście stopni, inverting a temperature
numbers changed	Z 8250 osobami → osiem tysięcy dwustu pięciu (8205), and 20 lat → dwadzieścia pięć lat (25)
word lost	38 000 000 mieszkańców → trzydzieści osiem milionów

The value check compares against what num2words would say, in both directions — a missing numeral and an added one are equally wrong, and a subset check passed dwadzieścia → dwadzieścia pięć while turning 20 into 25. Numerals are matched through a written-out canonical map rather than string similarity — dwieście and dwustu are one number and must match, trzy and trzysta differ by a hundred and must not, and ordinals like piąty count as pięć. Consonant folding was tried first and rejected good output.

It is skipped for identifiers: LO 3842 is correctly read digit by digit, and demanding num2words' rendering would reject a right answer.

Deviations from the build brief, and why

• Single digits are no longer shielded. The brief asks for standalone one-digit numbers to be hidden behind KEEPNUM… placeholders. Measured, that scored worse — 6/9 against 7/9 on the model card's own examples — and it produced an outright pronunciation error, because 3 maja left as a digit is read "trzy maja" where Polish wants "trzeciego maja". It also bought nothing on length: the TTS speaks 5 as pięć either way. What survives is the cheaper half — a block containing only standalone single digits is never sent at all.
• KEEP_RE has extra guards. The brief's version protects the leading 1 of 1 350 000 as if it were standalone, and its boundary set (?![\d.,:/-]) rejects any number followed by a comma or full stop — on real subtitles that skipped most of them (Przód 85, tył 75.). Numeric punctuation now only continues a number when a digit follows it.
• Two models, not one. The brief scopes a single normalizer. The 0.6B fine-tune performs to its advertised 85.7%, but its tail includes losing a number's value while declining it, which no amount of prompting fixes. See the routing table above.
• The model loads lazily, not at import. Same "load once, reuse" effect, but --help, --status and the non-neural half of the suite do not pay for a 1.1 GB load they never use.
• num2words errors are caught broadly. It raises KeyError, not ValueError, past its largest scale word.

Known limits

Measured, and not fixable at this layer:

input	output
Mieszka pod 12/45.	sto czternaście łamane przez trzy	an address has no reliable reading without context
Zadzwoń pod 601 234 567.	read as one 9-digit number	a phone number is indistinguishable from a grouped number
Zyskują 0,3 sekundy	trzy setne	should be dziesiąte; the value guard cannot see inside a decimal
To dzieło z XIX wieku.	unchanged	Roman numerals are not handled — use dictionary.txt
Prędkość 0,75 macha.	unchanged	the model hallucinated a unit conversion; the guard caught it and fell back

Files

File
normalizer.py	The component: routing, both backends, guards, normalize()
worker.py	JSON-lines service lektor_ai_comfy.py talks to (models loaded once per run)
bench.py	Compares models on the same battery — --compare a.json b.json
setup.sh	venv, dependencies, one-time gated model download
tests/	Routing, numerals, guards, and the brief's acceptance checks
model/, models/, venv/	Weights and environment — not in git

Interface

from normalizer import normalize, needs_normalization

needs_normalization("Mam 3 koty.")        # False — nothing worth sending
normalize("Wygrał 1247 złotych.")         # never raises, never returns None

worker.py speaks one JSON object per line on stdin/stdout, with logs on stderr:

← {"ready": true, "mode": "model", "device": "cuda:0", "llm": true}
→ {"id": 1, "text": "Wygrał 1247 złotych."}
← {"id": 1, "text": "Wygrał tysiąc dwieście…", "changed": true,
   "leg": "complex", "guard": "", "error": null}