lektor-ai/BUILD_BRIEF_pl_number_normalizer.md

# Build brief — Polish number normalizer (offline `transformers` component)

## Scope of this document

This brief covers **only the normalizer component**: installing it, running it
offline with plain `transformers`, and what it does. **Integration is out of
scope** — the calling app (`lektor_ai`) handles SRT parsing, decides which
subtitle blocks to normalize, and invokes this component. The user will specify
that wiring separately.

Assume the caller sends **one sentence / subtitle block at a time**, and only
sends blocks that already contain at least one digit. This component's job is to
return that text with numbers converted to spoken Polish words, per the rules
below.

## Purpose

The downstream TTS (Higgs Audio v3) mispronounces raw digits — especially long
numbers — sometimes emitting garbled non-words. This component removes that
failure by converting digits to fully spelled-out Polish words **before** the
text reaches the TTS, with correct grammatical case (the reason a neural
normalizer is used rather than a lookup table: Polish numerals decline by case,
gender, and the noun they govern).

Behaviour rules:

1. **Skip single-digit numbers.** A standalone one-digit integer (e.g. `3` in
   `Mam 3 koty`) is read correctly by the TTS and must be **left as a digit**.
2. **Expand two-or-more-digit numbers** to words (`25`, `2020`, `1 350 000`).
3. Complex numeric expressions — decimals (`3,5`), times (`14:30`), dates
   (`3 maja`), ratios/addresses (`12/45`), currency, units — are handled by the
   model. They are not "single-digit numbers" and are not skipped.
4. A block that (after removing single digits) contains no multi-digit number is
   returned unchanged.

## The model

- **Repo:** `Folx/qwen3-0.6b-pl-text-normalization` (Hugging Face). ~1.1 GB,
  bfloat16, 600M params, `Qwen3ForCausalLM`. Fine-tuned specifically for Polish
  TTS text normalization (numbers, dates, times, units, abbreviations).
- **GATED REPO.** The user must accept the terms on the model page and provide an
  `HF_TOKEN` for the **one-time** download. The build agent cannot accept the
  licence. Load the model from a **local path / local cache** at run time — never
  re-download.
- **CRITICAL version requirement:** `transformers >= 5.0.0` and `torch >= 2.1.0`.
  Older transformers (e.g. 4.57) tokenize the Qwen3 chat template differently and
  silently produce wrong output even under greedy decode. Pin `transformers>=5.0`.
- **System prompt (fixed, required):** `Zamieniasz tekst na formę mówioną.`
- **Context:** max 512 tokens; quality degrades past ~300 characters. Input is one
  short block per call, so this is fine — warn (don't truncate) on longer inputs.
- **Decoding:** greedy (`do_sample=False`), with **constrained decoding** that
  bans digit tokens from the output so no un-normalized number can leak through.

## Environment & installation

Install into a **dedicated, isolated venv/conda env**, separate from the ComfyUI
/ Higgs TTS environment. Rationale: this model hard-requires `transformers>=5.0`,
and installing that into the existing working TTS environment risks disturbing
its pinned dependencies. Keep them apart; this component is invoked as its own
process.

Pinned dependencies:

- `transformers>=5.0.0`
- `torch>=2.1.0` — CUDA build matching the host (target: RTX 3090, 24 GB)
- `accelerate` — for `device_map`
- `num2words` — used only by the fallback path

One-time model fetch (network + token required):

```bash
HF_TOKEN=... huggingface-cli download Folx/qwen3-0.6b-pl-text-normalization \
    --local-dir ./model
```

Run time is **offline**. Set `HF_HUB_OFFLINE=1` and `TRANSFORMERS_OFFLINE=1`, and
load the model from `./model` (or the local HF cache). Add a check that a full run
succeeds with networking disabled.

Load the model, tokenizer, and banned-token list **once** (module level) and reuse
across calls — model load dominates cost.

## Normalization logic

For a given input string:

1. **Protect single-digit numbers.** Replace each standalone one-digit integer
   with a **digit-free sentinel**, remembering the mapping. The boundary set
   excludes adjacent digits and numeric punctuation so decimals/times/dates are
   not caught:

   ```python
   KEEP_RE = re.compile(r'(?<![\d.,:/\-])\d(?![\d.,:/\-])')
   ```

   Sentinels must contain **no digits** (digit tokens are banned in output, and
   the check in step 2 looks for leftover digits). Use letter-indexed sentinels,
   e.g. `KEEPNUMA`, `KEEPNUMB`, … .

2. **Decide.** If the protected text still contains a digit, that digit belongs
   to a 2+ digit / complex number → call the model. If no digit remains, skip the
   model and return the text with sentinels restored (only single digits were
   present).

3. **Model call.** Build the chat prompt (fixed system prompt + protected text),
   greedy decode with digit tokens banned, decode the completion.

4. **Restore** sentinels to their original single digits.

5. On any generation error, use the fallback instead of raising.

### Reference: model wrapper (adapt, then verify against the gated model)

```python
import re, torch
from transformers import AutoModelForCausalLM, AutoTokenizer

MODEL = "./model"  # local, offline
SYS = "Zamieniasz tekst na formę mówioną."

tok = AutoTokenizer.from_pretrained(MODEL)
model = AutoModelForCausalLM.from_pretrained(
    MODEL, torch_dtype=torch.bfloat16, device_map="auto"
).eval()

def _banned_ids(tok):
    banned = set()
    for tid in range(tok.vocab_size):
        d = tok.decode([tid]).strip()
        if d and d.isdigit():
            banned.add(tid)
    for sym in ["$","%","€","£","¥","°","§","@","#","&","×","÷","±",
                "©","®","™","→","←","↑","↓","√","∞","≈","≠","≤","≥"]:
        ids = tok.encode(sym, add_special_tokens=False)
        if len(ids) == 1:
            banned.add(ids[0])
    return sorted(banned)

BANNED = _banned_ids(tok)

def _model_normalize(text: str) -> str:
    messages = [{"role": "system", "content": SYS},
                {"role": "user", "content": text}]
    prompt = tok.apply_chat_template(messages, tokenize=False,
                                     add_generation_prompt=True)
    inputs = tok(prompt, return_tensors="pt").to(model.device)
    n = inputs["input_ids"].shape[1]
    with torch.no_grad():
        out = model.generate(**inputs, max_new_tokens=400, do_sample=False,
                             pad_token_id=tok.pad_token_id,
                             suppress_tokens=BANNED)
    return tok.decode(out[0][n:], skip_special_tokens=True).strip()
```

### Reference: skip / decide / restore + public function

```python
KEEP_RE = re.compile(r'(?<![\d.,:/\-])\d(?![\d.,:/\-])')
_LETTERS = "ABCDEFGHIJKLMNOPQRSTUVWXYZ"

def _sentinel(i: int) -> str:
    s = ""; i += 1
    while i:
        i, r = divmod(i - 1, 26); s = _LETTERS[r] + s
    return f"KEEPNUM{s}"

def normalize(text: str) -> str:
    """Return `text` with 2+ digit numbers spelled out in Polish; single-digit
    numbers left as digits. Safe to call on any string."""
    mapping = {}
    def protect(m):
        key = _sentinel(len(mapping)); mapping[key] = m.group(0); return key
    protected = KEEP_RE.sub(protect, text)

    if re.search(r'\d', protected):
        try:
            result = _model_normalize(protected)
        except Exception:
            result = _fallback(protected)
    else:
        result = protected

    for key, digit in mapping.items():
        result = result.replace(key, digit)
    return result
```

### Reference: fallback (model unavailable / generation error)

Guarantees the component never raises. Degraded: nominative case only, bare
multi-digit integers only (leaves decimals/times/dates untouched rather than
mangle them). Log a warning when used.

```python
from num2words import num2words

_INT_RE = re.compile(r'(?<![\d.,:/\-])\d{2,}(?:[\u00a0 .]\d{3})*(?![\d.,:/\-])')

def _fallback(text: str) -> str:
    def repl(m):
        digits = re.sub(r'[\u00a0 .]', '', m.group(0))
        return num2words(int(digits), lang="pl")
    return _INT_RE.sub(repl, text)
```

## Error handling

- Input longer than ~300 chars: log a warning (quality drops), still process,
  never truncate silently.
- Model load failure: fall back to `num2words` and log clearly that the neural
  normalizer is unavailable — never crash.
- Per-call generation error: use the fallback for that call only.

## Testing / acceptance

The model produces valid case variants, so assert on **properties**, not exact
strings, except where noted.

- **Single digit preserved:** `Mam 3 koty.` → still contains `3`.
- **Multi-digit expanded:** `Wygrał 1247 złotych.` → no `1247`; contains `tysiąc`.
- **Large number:** `populacja 1 350 000 osób` → no digits from that number;
  contains `milion`.
- **Mixed (key sentinel round-trip):** `Mam 3 koty i 25 psów.` → keeps `3`,
  expands `25` → `dwadzieścia pięć`.
- **Only single digits:** model is not called; text returned unchanged (assert
  via a spy on `_model_normalize`).
- **No digits leak:** expanded output contains no digit characters except
  restored single digits.
- **Sentinel round-trip:** `KEEPNUM…` never appears in final output. **If the
  model is found to alter/inflect the sentinel, flag it** — the fix is a more
  inert sentinel, or dropping protection and documenting that single digits
  co-occurring with multi-digit numbers get expanded.
- **Offline:** the suite passes with networking disabled, `HF_HUB_OFFLINE=1`,
  `TRANSFORMERS_OFFLINE=1`.

## Deliverables

- `normalizer.py` — model/tokenizer/banned-list loaded once at import; public
  `normalize(text: str) -> str`; `_fallback`; skip/decide/restore logic.
- `requirements.txt` — pinned as above.
- `README.md` — env creation, one-time gated download with `HF_TOKEN`, offline
  run variables. (No caller/integration details — those are provided separately.)
- `tests/` — the checks above with sample inputs.

## Design decisions (defaults — confirm if you disagree)

1. **Single digit inside a date/time.** `3 maja` keeps its `3` (space-separated
   standalone single digit), while `9:30`, `3,5`, `12/45` go to the model and are
   expanded. If you want `3 maja` → `trzeciego maja`, that's a one-line change to
   the boundary set.
2. **Sentinel robustness.** Small residual risk the model alters the placeholder
   instead of passing it through. Encoded as an explicit test with a documented
   fallback so it's caught, not shipped silently.