# Polish number normalizer Rewrites digits as spoken Polish words before the text reaches the TTS: ``` Wygrał 1247 złotych. → Wygrał tysiąc dwieście czterdzieści siedem złotych. Mówimy o 25 psach. → Mówimy o dwudziestu pięciu psach. Mam 3 koty. → Mam 3 koty. (nothing worth sending) Brakuje nam 15000 podpisów. → Brakuje nam piętnastu tysięcy podpisów. ``` Higgs Audio v3 mispronounces raw digits — especially long ones — and sometimes emits garbled non-words. A lookup table cannot fix that, because Polish numerals decline by case, gender and the noun they govern (`25 psów` / `o 25 psach` / `z 25 psami`), so this uses a small fine-tuned model. It runs in **its own venv**, separate from ComfyUI: the models require `transformers >= 5`, and forcing that on the working TTS install would risk its pinned dependencies. `lektor_ai_comfy.py` drives it as a subprocess, and asks ComfyUI to unload Higgs first so there is VRAM to work with. ## Install ```bash normalizer/setup.sh # venv + torch (cu128) + transformers + num2words ``` Then the model, which is **gated** — accept the terms at with your Hugging Face account first, then: ```bash HF_TOKEN=hf_... normalizer/setup.sh --model ``` The inflection model is separate and **not gated**: ```bash normalizer/venv/bin/hf download CYFRAGOVPL/PLLuM-4B-instruct-2512 \ --local-dir normalizer/models/PLLuM-4B-instruct-2512 ``` Those are the only steps that need the network. Everything afterwards runs offline (`HF_HUB_OFFLINE=1`, `TRANSFORMERS_OFFLINE=1`, both set by the caller). Without the inflection model every block goes to the fine-tune, which is what the pipeline did before it existed. Without the model the component still works, on a `num2words` fallback that handles bare integers in the nominative case only. It says so in the log, and `lektor_ai_comfy.py` prints a warning for the run. ## Check it ```bash normalizer/venv/bin/python normalizer/normalizer.py --status normalizer/venv/bin/python normalizer/normalizer.py "Wygrał 1247 złotych." normalizer/venv/bin/python -m unittest discover -s normalizer/tests -v ``` The tests assert on properties (`no 1247 left`, `contains tysiąc`) rather than exact strings, because several case variants are correct. The ones that need the model skip themselves when it is not downloaded. ## What it does 1. **Skip what needs nothing.** A block whose only numbers are standalone single digits is never sent: the TTS says "pięć minut" for "5 minut" by itself, so there is nothing to gain and a whole sentence of rewriting to risk. 2. **Route.** Two models with opposite weaknesses, measured on 40 real subtitle lines and a 32-case battery: | | good at | bad at | |---|---|---| | fine-tune (0.6B) | `6:05`, `P10`, `LO 3842`, `95,7`, dates | loses a large number's value while declining it | | num2words + LLM | the value, always; the case | formats — a flight number becomes a quantity | So a block goes to the second leg only when **every** number in it is a plain integer — 74% of numbered blocks on the sample. Everything else goes to the fine-tune. 3. **Generate.** The fine-tune gets the text as-is. The other leg gets text with no digits at all — num2words has already produced correct nominative words — so its only job is inflection, and it cannot get a value wrong because it never reads a number. Greedy decoding, digit tokens banned, either way. 4. **Guard.** The answer is checked before it is used (below). A block that fails falls back to num2words: dull, nominative, never wrong about the value. ## The guards Both models can produce output worse than doing nothing. Each of these fired on a real failure during development: | check | the failure it caught | |---|---| | runaway length | a 10-word line came back as 200 words | | repetition loop | `łącznie łącznie łącznie…` to the token ceiling | | invalid characters | a wall of `�` replacement bytes | | digits left | an un-normalized number slipped through | | minus sign lost | `-12 stopni` → *dwanaście stopni*, inverting a temperature | | numbers changed | `Z 8250 osobami` → *osiem tysięcy **dwustu pięciu*** (8205), and `20 lat` → *dwadzieścia **pięć** lat* (25) | | word lost | `38 000 000 mieszkańców` → *trzydzieści osiem milionów* | The value check compares against what num2words would say, **in both directions** — a missing numeral and an added one are equally wrong, and a subset check passed `dwadzieścia` → `dwadzieścia pięć` while turning 20 into 25. Numerals are matched through a written-out canonical map rather than string similarity — `dwieście` and `dwustu` are one number and must match, `trzy` and `trzysta` differ by a hundred and must not, and ordinals like `piąty` count as `pięć`. Consonant folding was tried first and rejected good output. It is skipped for identifiers: `LO 3842` is correctly read digit by digit, and demanding num2words' rendering would reject a right answer. ### Deviations from the build brief, and why - **Single digits are no longer shielded.** The brief asks for standalone one-digit numbers to be hidden behind `KEEPNUM…` placeholders. Measured, that scored *worse* — 6/9 against 7/9 on the model card's own examples — and it produced an outright pronunciation error, because `3 maja` left as a digit is read *"trzy maja"* where Polish wants *"trzeciego maja"*. It also bought nothing on length: the TTS speaks `5` as *pięć* either way. What survives is the cheaper half — a block containing **only** standalone single digits is never sent at all. - **`KEEP_RE` has extra guards.** The brief's version protects the leading `1` of `1 350 000` as if it were standalone, and its boundary set `(?![\d.,:/-])` rejects any number followed by a comma or full stop — on real subtitles that skipped most of them (`Przód 85, tył 75.`). Numeric punctuation now only continues a number when a digit follows it. - **Two models, not one.** The brief scopes a single normalizer. The 0.6B fine-tune performs to its advertised 85.7%, but its tail includes losing a number's value while declining it, which no amount of prompting fixes. See the routing table above. - **The model loads lazily, not at import.** Same "load once, reuse" effect, but `--help`, `--status` and the non-neural half of the suite do not pay for a 1.1 GB load they never use. - **`num2words` errors are caught broadly.** It raises `KeyError`, not `ValueError`, past its largest scale word. ### Known limits Measured, and not fixable at this layer: | input | output | | |---|---|---| | `Mieszka pod 12/45.` | *sto czternaście łamane przez trzy* | an address has no reliable reading without context | | `Zadzwoń pod 601 234 567.` | read as one 9-digit number | a phone number is indistinguishable from a grouped number | | `Zyskują 0,3 sekundy` | *trzy setne* | should be *dziesiąte*; the value guard cannot see inside a decimal | | `To dzieło z XIX wieku.` | unchanged | Roman numerals are not handled — use `dictionary.txt` | | `Prędkość 0,75 macha.` | unchanged | the model hallucinated a unit conversion; the guard caught it and fell back | ## Files | File | | |---|---| | `normalizer.py` | The component: routing, both backends, guards, `normalize()` | | `worker.py` | JSON-lines service `lektor_ai_comfy.py` talks to (models loaded once per run) | | `bench.py` | Compares models on the same battery — `--compare a.json b.json` | | `setup.sh` | venv, dependencies, one-time gated model download | | `tests/` | Routing, numerals, guards, and the brief's acceptance checks | | `model/`, `models/`, `venv/` | Weights and environment — not in git | ## Interface ```python from normalizer import normalize, needs_normalization needs_normalization("Mam 3 koty.") # False — nothing worth sending normalize("Wygrał 1247 złotych.") # never raises, never returns None ``` `worker.py` speaks one JSON object per line on stdin/stdout, with logs on stderr: ``` ← {"ready": true, "mode": "model", "device": "cuda:0", "llm": true} → {"id": 1, "text": "Wygrał 1247 złotych."} ← {"id": 1, "text": "Wygrał tysiąc dwieście…", "changed": true, "leg": "complex", "guard": "", "error": null} ```