Install the whole backend from one script; queue log; UI fixes

install_comfyui.sh becomes install_backend.sh and now builds everything the app
needs, not just the speech half:

- Step 7 runs normalizer/setup.sh --model, which fetches both models: the open
  inflection model first, so the install gets as far as it can before anything
  can ask for credentials, then the gated fine-tune.
- A missing Hugging Face account is no longer fatal. Accepting a licence is the
  one thing an installer cannot do, so the rest installs, the verification
  reports "fallback only (gated model missing)", and the summary prints the
  three commands that finish it.
- --skip-normalizer, disk requirement 35 -> 55 GB, removal instructions and the
  step list updated to match.

Queue logging — queue.log in the first queued video's job folder, which for a
single-video run is its own:

- Records what went wrong rather than what happened: warnings, tracebacks,
  ffmpeg errors, guard rejections, clips cut off mid-word or that crashed the
  backend, and the --regen line carrying their numbers.
- ComfyUI failures *and* restarts, so the log shows the recovery rather than
  reading as though the run simply stopped. GUI-initiated backend actions go in
  too; they never pass through a job's pipe.
- A failure also dumps the job's last 15 lines. Keyword matching alone is not
  enough — ffmpeg says "Invalid data found when processing input" and nothing
  that looks like an error.

UI:

- The output pane follows new output only while already at the bottom. Scroll up
  and it stays there; scroll back down and it resumes. Reading an earlier
  warning during a run was impossible while every ffmpeg progress frame yanked
  the view to the end.
- The ComfyUI button's status dot is drawn as its own canvas item, so it can be
  green (answering) or amber (holds the port, does not answer). A "." inside the
  label could only ever be the same grey as the words.

comfyui.log is capped at 120,000 lines, trimmed on backend start and every five
minutes. Rewritten in place, never replaced: tee holds it open in append mode,
so swapping the inode would leave the backend writing to an unlinked file.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>

comfy_installation.txt

@@ -1,5 +1,5 @@
 =============================================================================
-NOTE: These are the original planning notes. Use install_comfyui.sh instead —
+NOTE: These are the original planning notes. Use install_backend.sh instead —
 it reproduces what is ACTUALLY installed, with versions pinned.
 
 Two things below do not match the real installation:
@@ -11,7 +11,7 @@ Two things below do not match the real installation:
     git` line was never actually applied. The environment contains only the 27
     base Python packages from conda-forge. ffmpeg is used from the system
     (/usr/bin/ffmpeg) and espeak-ng is not installed at all. Everything works
-    without them, so install_comfyui.sh does not install them either — it only
+    without them, so install_backend.sh does not install them either — it only
     warns if ffmpeg is missing, since lektor_ai_comfy.py needs it (ComfyUI
     itself does not).
 =============================================================================

docs/installation.md

@@ -5,16 +5,17 @@ and the **script** (which needs nothing installed at all).
 
 ## Automated install
 
-`install_comfyui.sh` in the project root reproduces the exact working
+`install_backend.sh` in the project root reproduces the exact working
 environment. It is idempotent — safe to re-run, and it skips any step already
 satisfied.
 
 ```bash
 cd ~/Programy/lektor_ai
-./install_comfyui.sh
+./install_backend.sh
 ```
 
-Roughly 30–60 minutes, mostly downloading. Expect ~22 GB.
+Roughly 45–90 minutes, almost all of it downloading. Expect ~39 GB: ~22 GB for
+the speech backend and ~17 GB for the normalizer's own environment and models.
 
 ### Options
 
@@ -22,14 +23,15 @@ Roughly 30–60 minutes, mostly downloading. Expect ~22 GB.
 |---|---|
 | *(none)* | Pinned versions — ComfyUI `7cb784e0`, suite `v5.3.0`. Recommended. |
 | `--latest` | Newest ComfyUI and suite instead. You are on your own for breakage. |
-| `--skip-model` | Skip the 8.7 GB download; it happens on first generation anyway. |
+| `--skip-model` | Skip the 8.7 GB TTS download; it happens on first generation anyway. |
+| `--skip-normalizer` | ComfyUI only. Digits then reach the TTS as digits, which it misreads. |
 | `--lock FILE` | Restore an exact package set from a `pip freeze` lockfile. |
 | `--help` | Usage summary. |
 
 ### Changing where things go
 
 ```bash
-CONDA_ROOT=/opt/miniforge3 COMFY_ROOT=/data/comfy ./install_comfyui.sh
+CONDA_ROOT=/opt/miniforge3 COMFY_ROOT=/data/comfy ./install_backend.sh
 ```
 
 | Variable | Default |
@@ -49,7 +51,7 @@ see [Configuring the script](#configuring-the-script).
 byte-identical environment:
 
 ```bash
-./install_comfyui.sh --lock comfyui-requirements.lock.txt
+./install_backend.sh --lock comfyui-requirements.lock.txt
 ```
 
 Regenerate it after intentional changes:
@@ -78,9 +80,13 @@ its own.
    pip install (see [requirements.md](requirements.md#dependency-surface)).
 7. **Model** — `bosonai/higgs-audio-v3-tts-4b` into
    `ComfyUI/models/TTS/higgs_audio_v3/`.
-8. **Launcher** written to `$COMFY_ROOT/start_comfyui.sh`.
+8. **Number normalizer** — `normalizer/setup.sh --model`: its own venv with
-9. **Verification** — imports torch, torchaudio, transformers, safetensors,
+   `transformers>=5`, the open inflection model, and the gated fine-tune if you
-   soundfile, librosa and reports CUDA availability.
+   have credentials. Failure here is reported, not fatal.
+9. **Launcher** written to `$COMFY_ROOT/start_comfyui.sh`.
+10. **Verification** — imports torch, torchaudio, transformers, safetensors,
+    soundfile, librosa, reports CUDA availability, and prints whether the
+    normalizer came up neural or on the fallback.
 
 ## Manual install
 
@@ -184,25 +190,32 @@ conda install -n comfy-tts -c conda-forge ffmpeg
 Note that this puts ffmpeg inside the conda env, so the script must then be run
 with that env active.
 
-## Number normalizer (optional, recommended)
+## Number normalizer
 
-Digits reach the TTS as words rather than figures, in the correct Polish case.
+`install_backend.sh` installs this too — its own venv, the 8 GB inflection
-It has its own venv on purpose — the model needs `transformers >= 5`, which must
+model, and the 1.1 GB fine-tune. Nothing further is needed unless the script
-not be forced on the ComfyUI environment installed above.
+says otherwise.
+
+It gets its own venv on purpose: it needs `transformers >= 5`, which must not be
+forced on the ComfyUI environment installed above. They talk over a pipe.
+
+**The one thing the installer cannot do for you** is accept a licence. The
+fine-tune is gated, so if you have no Hugging Face credentials the script
+installs everything else, reports `fallback only (gated model missing)`, and
+prints these three steps:
 
 ```bash
-cd ~/Programy/lektor_ai
+# 1. accept the terms while signed in:
-normalizer/setup.sh                             # venv + torch (cu128) + transformers
+#    https://huggingface.co/Folx/qwen3-0.6b-pl-text-normalization
-HF_TOKEN=hf_... normalizer/setup.sh --model     # the weights, ~1.1 GB
+normalizer/venv/bin/hf auth login
+normalizer/setup.sh --model
 ```
 
-The weights are gated: accept the terms at
+`HF_TOKEN=hf_...` in the environment works instead of `hf auth login`, and the
-<https://huggingface.co/Folx/qwen3-0.6b-pl-text-normalization> while signed in,
+installer picks up either automatically if you set it up beforehand.
-then take a token from <https://huggingface.co/settings/tokens>. This is the only
-step that touches the network; runs are offline afterwards.
 
-Skipping this is fine — the pipeline falls back to `num2words` (nominative case
+Until then the pipeline runs with `num2words` for times, dates and identifiers,
-only) and says so. Turn the whole stage off with `numbers = no` under
+and says so on every run. Turn the stage off entirely with `numbers = no` under
 `[normalize]` in `settings.ini`. See [normalizer/README.md](../normalizer/README.md).
 
 ## Verifying

docs/usage.md

@@ -479,6 +479,48 @@ that takes longer than `timeout` — each is a warning, and the affected text go
 to the TTS with its digits intact. The worker is started once per run and exits
 before TTS generation begins, so it is not holding VRAM while Higgs works.
 
+## The output pane
+
+It follows the newest output only while you are already at the bottom. Scroll
+up and it stays where you put it, however much arrives in the meantime; scroll
+back down and it resumes following. There is nothing to click either way.
+
+That matters during a run, where output lands several times a second and ffmpeg
+redraws its progress bar continuously — scrolling back to read an earlier
+warning was impossible while every frame yanked the view to the end.
+
+## The queue log
+
+Every GUI run writes `queue.log` into the **first queued video's job folder** —
+which for a single-video run is simply its own. It records what went wrong, not
+what happened: the output pane already carries every clip and placement, and
+none of that helps at 8 a.m. when you want to know why a queue stopped.
+
+```
+===== queue started 2026-08-13 09:09:17 — 2 job(s) =====
+09:09:17  --- generate: good.mkv
+09:09:17  ComfyUI unreachable (<urlopen error [Errno 111] Connection refused>) [1/3 for this clip].
+09:09:48      finished OK
+09:09:48  --- generate: bad.mkv
+09:10:04  ffmpeg error (Mixing lektor audio track):
+09:10:04  /dev/null: Invalid data found when processing input
+09:10:04      FAILED with exit code 1 — 0 job(s) cancelled
+09:10:04      last 15 line(s) before it stopped:
+09:10:04        | …
+```
+
+Each job is bracketed by a start line and its outcome. In between go the lines
+that report a problem — warnings, tracebacks, ffmpeg errors, guard rejections,
+clips that crashed the backend or came back cut off mid-word.
+
+**A failure also dumps the job's last 15 lines.** Keyword matching alone is not
+enough: ffmpeg says *"Invalid data found when processing input"* and nothing
+that looks like an error, so an exit code would otherwise be recorded with no
+explanation.
+
+The file is appended to, never truncated, so it is a record across runs and a
+crash cannot cost you the previous session.
+
 ## The backend's lifecycle
 
 ComfyUI is normally started once and left running. The script manages it when it
@@ -496,6 +538,13 @@ has to:
 Its output goes to `comfyui.log` next to the script as well as to the terminal
 window, so a crash can be diagnosed after the fact.
 
+That file is **capped at 120,000 lines** (roughly 10 MB). Past that the oldest
+lines are dropped, on backend start and every five minutes while the GUI is
+open. It is trimmed in place rather than rotated: there is one file to look at
+after a crash, no `.1`/`.2` to hunt through, and rewriting in place is also what
+keeps `tee` writing to it — replacing the file would leave the backend appending
+to an unlinked inode and every later line would vanish.
+
 The GUI has a **ComfyUI** button showing the same state: `Start ComfyUI` when it
 is down, `ComfyUI ●` when it is answering, `ComfyUI ◍` when it holds the port but
 does not answer. Pressing it starts or restarts.

install_backend.sh

@@ -1,24 +1,35 @@
 #!/usr/bin/env bash
 #
-# install_comfyui.sh — reproduce the ComfyUI + Higgs Audio V3 TTS environment
+# install_backend.sh — build everything lektor_ai_comfy.py needs, on this or any
-# used by lektor_ai_comfy.py, on this or any other Linux machine.
+# other Linux machine: the ComfyUI + Higgs Audio V3 speech backend, and the
+# Polish number normalizer with its own environment and models.
 #
-# Everything lands in two removable directories and nothing touches the system:
+# After this finishes there is nothing left to install by hand, with one
-#   ~/miniforge3      — conda distribution (its own Python)
+# exception the script cannot do for you: one normalizer model is gated behind
-#   ~/comfy-tts-app   — ComfyUI, the TTS-Audio-Suite node, and the models
+# a licence you have to accept with your own Hugging Face account. The script
+# says so and installs everything else regardless.
+#
+# Everything lands in three removable directories and nothing touches the system:
+#   ~/miniforge3        — conda distribution (its own Python)
+#   ~/comfy-tts-app     — ComfyUI, the TTS-Audio-Suite node, and the TTS model
+#   ./normalizer/       — the normalizer venv and its models, beside this script
 #
 # No sudo, no apt, no changes to system Python. `conda init` is never run, so
 # your .bashrc is left alone.
 #
 # Usage:
-#   ./install_comfyui.sh                 # pinned versions (recommended)
+#   ./install_backend.sh                 # pinned versions (recommended)
-#   ./install_comfyui.sh --latest        # newest ComfyUI + suite instead
+#   ./install_backend.sh --latest        # newest ComfyUI + suite instead
-#   ./install_comfyui.sh --skip-model    # don't pre-download the 8.7 GB model
+#   ./install_backend.sh --skip-model    # don't pre-download the 8.7 GB TTS model
-#   ./install_comfyui.sh --lock FILE     # restore exact package set from a lockfile
+#   ./install_backend.sh --skip-normalizer   # ComfyUI only, no number normalizer
-#   ./install_comfyui.sh --help
+#   ./install_backend.sh --lock FILE     # restore exact package set from a lockfile
+#   ./install_backend.sh --help
+#
+# Hugging Face credentials, if you have them, are picked up automatically from
+# `hf auth login` or from HF_TOKEN in the environment.
 #
 # Override locations with environment variables:
-#   CONDA_ROOT=/opt/miniforge3 COMFY_ROOT=/data/comfy ./install_comfyui.sh
+#   CONDA_ROOT=/opt/miniforge3 COMFY_ROOT=/data/comfy ./install_backend.sh
 #
 set -euo pipefail
 
@@ -46,17 +57,21 @@ TORCH_PINS=(torch==2.11.0 torchvision==0.26.0 torchaudio==2.11.0)
 HF_MODEL_REPO="bosonai/higgs-audio-v3-tts-4b"
 MODEL_SUBDIR="models/TTS/higgs_audio_v3/higgs-audio-v3-tts-4b"
 
-REQUIRED_DISK_GB=35   # ~10 env + ~3 ComfyUI + ~9 model + headroom
+# ~10 conda env + ~3 ComfyUI + ~9 TTS model, then the normalizer: ~7 venv
+# (CUDA wheels again, in its own environment) + ~8 inflection model + ~1 gated.
+REQUIRED_DISK_GB=55
 
 # ── Options ──────────────────────────────────────────────────────────────────
 USE_LATEST=0
 SKIP_MODEL=0
+SKIP_NORMALIZER=0
 LOCKFILE=""
 
 while [[ $# -gt 0 ]]; do
     case "$1" in
         --latest)     USE_LATEST=1; shift ;;
         --skip-model) SKIP_MODEL=1; shift ;;
+        --skip-normalizer) SKIP_NORMALIZER=1; shift ;;
         --lock)       LOCKFILE="${2:-}"; shift 2 ;;
         -h|--help)    awk 'NR>1 {if (/^#/) {sub(/^# ?/,""); print} else exit}' "$0"; exit 0 ;;
         *)            echo "Unknown option: $1 (try --help)" >&2; exit 1 ;;
@@ -276,13 +291,35 @@ PY
     fi
 fi
 
-# ── 7. Launcher ──────────────────────────────────────────────────────────────
+# ── 7. Number normalizer ─────────────────────────────────────────────────────
+# Its own venv, deliberately: the normalizer needs transformers>=5 and the TTS
+# environment above is pinned to versions that must not move. They talk over a
+# pipe instead, so the conflict never has to be resolved.
+NORMALIZER_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)/normalizer"
+
+if [[ $SKIP_NORMALIZER -eq 1 ]]; then
+    step "Number normalizer (skipped)"
+    warn "Digits will reach the TTS as digits, which it often misreads."
+elif [[ ! -x "$NORMALIZER_DIR/setup.sh" ]]; then
+    step "Number normalizer"
+    warn "$NORMALIZER_DIR/setup.sh not found — skipping."
+else
+    step "Number normalizer (own venv + models)"
+    if "$NORMALIZER_DIR/setup.sh" --model; then
+        ok "Normalizer environment ready"
+    else
+        warn "Normalizer setup reported a problem — see above."
+        warn "The pipeline still runs; numbers fall back to num2words."
+    fi
+fi
+
+# ── 8. Launcher ──────────────────────────────────────────────────────────────
 step "Launcher script"
 
 LAUNCHER="$COMFY_ROOT/start_comfyui.sh"
 cat > "$LAUNCHER" <<EOF
 #!/bin/bash
-# Generated by install_comfyui.sh — starts ComfyUI on port $COMFY_PORT.
+# Generated by install_backend.sh — starts ComfyUI on port $COMFY_PORT.
 source "$CONDA_ROOT/etc/profile.d/conda.sh"
 conda activate $ENV_NAME
 cd "$COMFY_DIR" && python main.py --port $COMFY_PORT
@@ -290,7 +327,7 @@ EOF
 chmod +x "$LAUNCHER"
 ok "Wrote $LAUNCHER"
 
-# ── 8. Verification ──────────────────────────────────────────────────────────
+# ── 9. Verification ──────────────────────────────────────────────────────────
 step "Verifying installation"
 
 python - <<'PY'
@@ -307,9 +344,23 @@ sys.exit(1 if failed else 0)
 PY
 ok "Core imports succeed"
 
+NORMALIZER_STATE="not installed"
+if [[ -x "$NORMALIZER_DIR/venv/bin/python" ]]; then
+    if "$NORMALIZER_DIR/venv/bin/python" "$NORMALIZER_DIR/normalizer.py" --status \
+            2>/dev/null | grep -q '"mode": "model"'; then
+        NORMALIZER_STATE="ready (neural)"
+        ok "Normalizer ready"
+    else
+        NORMALIZER_STATE="fallback only (gated model missing)"
+        warn "Normalizer installed but running on the num2words fallback"
+    fi
+fi
+
 echo
 echo "${B}${G}Installation complete.${N}"
 echo
+echo "  Normalizer:      $NORMALIZER_STATE"
+echo
 echo "  Start ComfyUI:   $LAUNCHER"
 echo "  Web UI:          http://127.0.0.1:$COMFY_PORT"
 echo "  Then run:        python3 lektor_ai_comfy.py <video.mkv> --embedded-subs"
@@ -325,4 +376,15 @@ echo "    pip freeze > comfyui-requirements.lock.txt"
 echo
 echo "  Remove everything:"
 echo "    conda env remove -n $ENV_NAME && rm -rf $COMFY_ROOT"
+echo "    rm -rf $NORMALIZER_DIR/venv $NORMALIZER_DIR/model $NORMALIZER_DIR/models"
 echo "    rm -rf $CONDA_ROOT   # only if nothing else uses it"
+
+if [[ "$NORMALIZER_STATE" == fallback* ]]; then
+    echo
+    echo "  ${Y}One model still needs you:${N} it is gated behind a licence."
+    echo "    1. Open https://huggingface.co/Folx/qwen3-0.6b-pl-text-normalization"
+    echo "       while signed in, and accept the terms"
+    echo "    2. $NORMALIZER_DIR/venv/bin/hf auth login"
+    echo "    3. $NORMALIZER_DIR/setup.sh --model"
+    echo "  Until then times, dates and identifiers fall back to num2words."
+fi

lektor_ai_comfy.py

@@ -83,6 +83,12 @@ COMFY_AUTOSTART    = True
 COMFY_START_SCRIPT = Path(__file__).resolve().parent / "start_comfyui.sh"
 COMFY_LOG = Path(__file__).resolve().parent / "comfyui.log"
 
+# The backend log is appended to for ever otherwise — it reached 10 MB here in
+# a couple of days. Trimmed to the newest lines rather than rotated, so there is
+# one file to look at after a crash and no .1 .2 .3 to hunt through.
+COMFY_LOG_MAX_LINES = 120_000
+_LOG_TRIM_MIN_BYTES = 2 << 20      # below this it cannot plausibly be over the cap
+
 
 def load_settings(path: Path) -> dict:
     """Read settings.ini, falling back to _DEFAULTS for anything missing.
@@ -1533,6 +1539,7 @@ def start_comfy_backend(script: Path) -> bool:
 
     _autostart["attempts"] += 1
     _autostart["last"] = now
+    trim_comfy_log()                       # before the backend opens it again
     # Pause on exit so a backend that fails to start leaves its error on screen
     # instead of the window vanishing.
     log = COMFY_LOG
@@ -1567,6 +1574,34 @@ def start_comfy_backend(script: Path) -> bool:
     return True
 
 
+def trim_comfy_log(path: Path = None, max_lines: int = COMFY_LOG_MAX_LINES) -> int:
+    """Drop the oldest lines of the backend log, keeping the newest `max_lines`.
+    Returns how many were dropped.
+
+    Rewritten in place rather than replaced. `tee` holds the file open in append
+    mode, so the inode must survive: writing a new file and renaming it over the
+    old one would leave tee appending to an unlinked inode, and every later line
+    would vanish silently. Rewriting in place works because O_APPEND directs
+    each write to the current end of file rather than to a remembered offset.
+    """
+    path = path or COMFY_LOG
+    try:
+        if not path.is_file() or path.stat().st_size < _LOG_TRIM_MIN_BYTES:
+            return 0
+        data = path.read_bytes()
+        if data.count(b"\n") <= max_lines:
+            return 0
+        lines = data.splitlines(keepends=True)
+        dropped = len(lines) - max_lines
+        with path.open("r+b") as handle:
+            handle.write(b"".join(lines[-max_lines:]))
+            handle.truncate()
+        return dropped
+    except OSError as exc:
+        print(f"  Could not trim {path.name}: {exc}", file=sys.stderr)
+        return 0
+
+
 def wait_for_backend(timeout: float = 300.0) -> bool:
     """Block until the backend answers again. After a restart the port is not
     listening for a while, and submitting into it would look like another

normalizer/setup.sh

@@ -2,7 +2,7 @@
 # Create the isolated environment for the Polish number normalizer.
 #
 #   ./setup.sh                 venv + dependencies
-#   ./setup.sh --model         also download the model (needs HF_TOKEN)
+#   ./setup.sh --model         also download both models (see below)
 #   ./setup.sh --cpu           CPU-only torch, for a machine without an NVIDIA GPU
 #
 # Kept separate from the ComfyUI environment on purpose: this needs
@@ -13,7 +13,9 @@ set -euo pipefail
 HERE="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)"
 VENV="$HERE/venv"
 MODEL_DIR="$HERE/model"
-MODEL_REPO="Folx/qwen3-0.6b-pl-text-normalization"
+MODEL_REPO="Folx/qwen3-0.6b-pl-text-normalization"      # gated: times, dates, IDs
+LLM_REPO="CYFRAGOVPL/PLLuM-4B-instruct-2512"            # open: grammar of plain integers
+LLM_DIR="$HERE/models/PLLuM-4B-instruct-2512"
 TORCH_INDEX="https://download.pytorch.org/whl/cu128"   # same wheels as ComfyUI
 
 WANT_MODEL=0
@@ -44,25 +46,43 @@ say "Installing transformers, accelerate, num2words"
 "$VENV/bin/pip" install "transformers>=5.0.0" "accelerate>=0.26.0" "num2words>=0.5.13"
 
 if [ "$WANT_MODEL" = 1 ]; then
-    say "Downloading $MODEL_REPO"
     "$VENV/bin/pip" install --quiet "huggingface_hub[cli]"
 
+    # The inflection model first: it needs no account, so the install gets as
+    # far as it can before anything can ask for credentials.
+    if [ -f "$LLM_DIR/config.json" ]; then
+        say "$LLM_REPO already present"
+    else
+        say "Downloading $LLM_REPO (8 GB, no account needed)"
+        "$VENV/bin/hf" download "$LLM_REPO" --local-dir "$LLM_DIR"
+    fi
+
+    say "Downloading $MODEL_REPO"
+
     # Either an HF_TOKEN in the environment or a stored `hf auth login` works;
     # the CLI picks up both on its own. Only say something when there is
     # neither, because the failure then is a confusing "access denied".
-    if [ -z "${HF_TOKEN:-}" ] && ! "$VENV/bin/hf" auth whoami >/dev/null 2>&1; then
+    if [ -f "$MODEL_DIR/config.json" ]; then
+        say "$MODEL_REPO already present"
+    elif [ -z "${HF_TOKEN:-}" ] && ! "$VENV/bin/hf" auth whoami >/dev/null 2>&1; then
+        # Not fatal. Everything else is installed and useful; this one model
+        # needs a human to accept a licence, which no installer can do.
         cat >&2 <<EOF
-No Hugging Face credentials. The repo is gated (approval is automatic, but it
-has to be your account that asks):
 
-  1. Open https://huggingface.co/$MODEL_REPO while signed in and accept the terms
+  [!] Skipped $MODEL_REPO — no Hugging Face credentials.
-  2. Either   $VENV/bin/hf auth login
+      It is gated (approval is automatic, but your account has to ask):
-     or       HF_TOKEN=hf_... $0 --model
+
+        1. Open https://huggingface.co/$MODEL_REPO while signed in, accept terms
+        2. Either   $VENV/bin/hf auth login
+           or       export HF_TOKEN=hf_...
+        3. Rerun    $0 --model
+
+      Until then times, dates and identifiers fall back to num2words.
 
 EOF
-        exit 1
+    else
+        "$VENV/bin/hf" download "$MODEL_REPO" --local-dir "$MODEL_DIR"
     fi
-    "$VENV/bin/hf" download "$MODEL_REPO" --local-dir "$MODEL_DIR"
 fi
 
 say "Checking the installation"

readme.txt

@@ -9,7 +9,7 @@ FULL DOCUMENTATION: see docs/
   docs/architecture.md     how the pipeline works internally
   docs/troubleshooting.md  diagnosed failure modes and their fixes
 
-Installer: ./install_comfyui.sh   (see docs/installation.md)
+Installer: ./install_backend.sh   (see docs/installation.md)
 
 This file remains as a condensed quick reference.
 
@@ -144,6 +144,7 @@ output/
       tts_assembled.flac — all clips assembled with correct timing (lossless)
       embedded_sub.srt — extracted embedded subtitle (only with --embedded-subs)
       video_fragment.mkv — extracted video segment (only when --end is used)
+      queue.log        — errors and warnings from GUI runs (first video of a queue)
   {stem}_lektor.mkv    — final video with two audio tracks + all original subtitles:
                            Track 0 — Original (source audio, stream-copied as-is)
                            Track 1 — Polish lektor (TTS + original mixed,

ui.py

@@ -16,6 +16,7 @@ Needs tkinter (sudo apt install python3-tk). It also runs under
 to X11 bitmaps, so the window looks dated there; the UI detects this and says so.
 """
 
+import collections
 import configparser
 import os
 import queue
@@ -24,6 +25,7 @@ import shutil
 import subprocess
 import sys
 import threading
+import time
 from pathlib import Path
 
 try:
@@ -61,6 +63,10 @@ PLUS      = "#8f8f8f"
 LOG_BG    = "#242424"
 LOG_FG    = "#d0d0d0"
 
+# Status dot on the ComfyUI button.
+COMFY_UP   = "#4caf50"    # answering
+COMFY_BUSY = "#d9a441"    # holds the port but does not answer
+
 # Rounding radius, in pixels, for fields and buttons.
 RADIUS = 9
 
@@ -113,6 +119,7 @@ class RoundButton(tk.Canvas):
                          highlightthickness=0, bd=0)
         self._shape = round_rect(self, 1, 1, w - 1, h - 1, radius, fill=bg, outline="")
         self._label = self.create_text(w / 2, h / 2, text=text, fill=fg, font=f)
+        self._dot = None
         self.bind("<Enter>", lambda _: self._paint(self._hover))
         self.bind("<Leave>", lambda _: self._paint(self._bg))
         self.bind("<Button-1>", self._click)
@@ -131,6 +138,35 @@ class RoundButton(tk.Canvas):
 
     def set_text(self, text):
         self.itemconfigure(self._label, text=text)
+        if self._dot is not None:
+            self._place_dot()
+
+    def set_indicator(self, colour):
+        """A status dot beside the label, or None for no dot.
+
+        Drawn as its own canvas item rather than a coloured character in the
+        text: a canvas text item has one fill colour for the whole string, so
+        a "●" in the label can only ever be the same grey as the words.
+        """
+        if colour is None:
+            if self._dot is not None:
+                self.delete(self._dot)
+                self._dot = None
+                self.coords(self._label, int(self["width"]) / 2,
+                            int(self["height"]) / 2)
+            return
+        if self._dot is None:
+            self._dot = self.create_oval(0, 0, 0, 0, fill=colour, outline="")
+        self.itemconfigure(self._dot, fill=colour)
+        self._place_dot()
+
+    def _place_dot(self):
+        """Keep label+dot centred together as the text changes length."""
+        w, h, gap, r = int(self["width"]), int(self["height"]), 5, 4
+        self.coords(self._label, w / 2 - (gap + r), h / 2)
+        x1, _y1, x2, _y2 = self.bbox(self._label)
+        cx, cy = x2 + gap + r, h / 2
+        self.coords(self._dot, cx - r, cy - r, cx + r, cy + r)
 
 
 class RoundField(tk.Frame):
@@ -925,7 +961,7 @@ class JobBlock(tk.Frame):
         except ValueError as exc:
             messagebox.showerror("Cannot run", str(exc))
             return
-        self.app.enqueue([(self.label(), mode, cmd)])
+        self.app.enqueue([(self.label(), mode, cmd, self.job_folder())])
 
     def ask_regen(self):
         raw = SimplePrompt(self.app, "Regenerate clips",
@@ -1036,7 +1072,9 @@ class LektorUI(tk.Tk):
         self.blocks   = []
         self._group_seq = 0
         self._syncing = False
-        self.jobs     = []           # pending [(label, mode, cmd), …]
+        self.jobs     = []           # pending [(label, mode, cmd, job_folder), …]
+        self.log_path = None         # queue.log for the batch being run
+        self._job_tail = collections.deque(maxlen=self.TAIL_LINES)
         self.proc     = None
         self.running  = False
         self.out_q    = queue.Queue()
@@ -1099,11 +1137,12 @@ class LektorUI(tk.Tk):
         RoundButton(top, "Queue dictionary", self.open_queue_dictionary,
                     bg="#3a3a3a", fg=MUTED, hover="#4a4a4a", font=(F, 9),
                     padx=13, pady=6).pack(side="right", padx=(0, 8))
-        self.comfy_btn = RoundButton(top, "ComfyUI", self.toggle_comfy,
+        self.comfy_btn = RoundButton(top, "Start ComfyUI", self.toggle_comfy,
                                      bg="#3a3a3a", fg=MUTED, hover="#4a4a4a",
                                      font=(F, 9), padx=13, pady=6)
         self.comfy_btn.pack(side="right", padx=(0, 8))
         self.after(500, self._refresh_comfy_button)
+        self.after(60_000, self._trim_backend_log)
 
         # scrollable list of job blocks
         mid = tk.Frame(self, bg=BG)
@@ -1202,6 +1241,22 @@ class LektorUI(tk.Tk):
     # of the work is in lektor_ai_comfy so the terminal script and this button
     # behave identically.
 
+    def _trim_backend_log(self):
+        """Keep comfyui.log bounded while the backend runs.
+
+        Trimming only at startup is not enough: ComfyUI can stay up for days,
+        and it is the running backend that fills the file. Every five minutes,
+        and the call is cheap — it returns on a stat when the file is small.
+        """
+        try:
+            dropped = core.trim_comfy_log()
+            if dropped:
+                self.log(f"Trimmed {core.COMFY_LOG.name}: dropped {dropped} old "
+                         f"line(s), keeping the newest {core.COMFY_LOG_MAX_LINES}.")
+        except Exception as exc:                        # noqa: BLE001
+            self.log(f"Could not trim {core.COMFY_LOG.name}: {exc}")
+        self.after(300_000, self._trim_backend_log)
+
     def _comfy_state(self):
         """(running, healthy) — separate, because a backend can hold the port
         and the GPU while no longer answering."""
@@ -1217,11 +1272,16 @@ class LektorUI(tk.Tk):
     def _refresh_comfy_button(self):
         running, healthy = self._comfy_state()
         if healthy:
-            self.comfy_btn.set_text("ComfyUI ●")
+            self.comfy_btn.set_text("ComfyUI")
+            self.comfy_btn.set_indicator(COMFY_UP)
         elif running:
-            self.comfy_btn.set_text("ComfyUI ◍")      # up but not answering
+            # Holding the port but not answering — the state that strands the
+            # GPU, so it gets its own colour rather than looking healthy.
+            self.comfy_btn.set_text("ComfyUI")
+            self.comfy_btn.set_indicator(COMFY_BUSY)
         else:
             self.comfy_btn.set_text("Start ComfyUI")
+            self.comfy_btn.set_indicator(None)
         self.after(5000, self._refresh_comfy_button)
 
     def toggle_comfy(self):
@@ -1240,17 +1300,17 @@ class LektorUI(tk.Tk):
     def _comfy_worker(self, running):
         try:
             if running:
-                self.log("Stopping ComfyUI…")
+                self.log_both("Stopping ComfyUI…")
                 core.stop_comfy_backend()
-            self.log("Starting ComfyUI…")
+            self.log_both("Starting ComfyUI…")
             if not core.start_comfy_backend(core.COMFY_START_SCRIPT):
-                self.log("Could not start ComfyUI — see the terminal window.")
+                self.log_both("Could not start ComfyUI — see the terminal window.")
                 return
             if core.wait_for_backend(180.0):
-                self.log("ComfyUI is up.")
+                self.log_both("ComfyUI is up.")
             else:
-                self.log(f"ComfyUI did not answer within 180s. "
+                self.log_both(f"ComfyUI did not answer within 180s. "
-                         f"Its output is in {core.COMFY_LOG}.")
+                              f"Its output is in {core.COMFY_LOG}.")
         finally:
             self.after(0, lambda: self.comfy_btn.set_enabled(True))
             self.after(0, self._refresh_comfy_button)
@@ -1353,13 +1413,83 @@ class LektorUI(tk.Tk):
         jobs = []
         for b in self.blocks:
             try:
-                jobs.append((b.label(), "generate", b.build_command("generate")))
+                jobs.append((b.label(), "generate", b.build_command("generate"),
+                             b.job_folder()))
             except ValueError as exc:
                 messagebox.showerror("Cannot start", f"{b.label()}:\n\n{exc}")
                 return
         self.enqueue(jobs)
 
+    LOG_NAME = "queue.log"
+
+    # What counts as an issue. The pipeline is deliberately chatty on stdout —
+    # every clip, every placement — and none of that belongs in a file you open
+    # to find out what went wrong overnight. Matched case-insensitively because
+    # the interesting lines come from three different sources: this script,
+    # Python tracebacks, and ffmpeg, which capitalises nothing.
+    ISSUE_RE = re.compile(
+        r"error|warning|traceback|exception|fail|"
+        r"cannot|could not|not found|no such|invalid|denied|"
+        r"crash|killed comfyui|died|went away|unreachable|did not|"
+        # Backend lifecycle: a restart is not an error, but a log that shows
+        # the crash without the recovery reads like the run simply stopped.
+        r"start(?:ing|ed) comfyui|stopping comfyui|comfyui is back|"
+        r"comfyui is up|not answering|resubmitting|waiting for it|"
+        r"rejected |safety check|skipping this clip|mid-word|"
+        # The --regen line carries the clip numbers, and it follows its summary
+        # on a line of its own — without it the log says three clips are broken
+        # but not which three.
+        r"--regen|"
+        r"^\s*!", re.IGNORECASE)
+
+    # Kept per job so a failure can be explained even when its last words match
+    # nothing above — an exit code with no context is not worth logging.
+    TAIL_LINES = 15
+
+    def _open_log(self, jobs):
+        """Point the log at the first job of this batch.
+
+        One rule covers both cases the same way: the file goes in the first
+        queued video's folder, which for a single-video run is simply its own.
+        """
+        folder = next((f for _l, _m, _c, f in jobs if f is not None), None)
+        if folder is None:
+            self.log_path = None
+            return
+        self.log_path = folder / self.LOG_NAME
+        try:
+            self.log_path.parent.mkdir(parents=True, exist_ok=True)
+            with self.log_path.open("a", encoding="utf-8") as fh:
+                fh.write(f"\n===== queue started {time.strftime('%Y-%m-%d %H:%M:%S')}"
+                         f" — {len(jobs)} job(s) =====\n")
+        except OSError as exc:
+            self.log(f"Could not open {self.log_path}: {exc}")
+            self.log_path = None
+
+    def _log_tail(self):
+        """The job's last lines, so a failure is explained even when its error
+        text matches no keyword — ffmpeg in particular says things like
+        "Invalid data found when processing input" and nothing else."""
+        if not self.log_path or not self._job_tail:
+            return
+        self._log_issue(f"    last {len(self._job_tail)} line(s) before it stopped:")
+        for line in self._job_tail:
+            self._log_issue(f"      | {line}")
+
+    def _log_issue(self, text):
+        """Append one line. Appends rather than truncates, so the file is a
+        record across runs and a crash never costs the previous session."""
+        if not self.log_path:
+            return
+        try:
+            with self.log_path.open("a", encoding="utf-8") as fh:
+                fh.write(f"{time.strftime('%H:%M:%S')}  {text.rstrip()}\n")
+        except OSError:
+            self.log_path = None            # a broken log must not stop a run
+
     def enqueue(self, jobs):
+        if not self.running:
+            self._open_log(jobs)
         self.jobs.extend(jobs)
         self.log(f"Queued {len(jobs)} job{'' if len(jobs) == 1 else 's'} "
                  f"({len(self.jobs)} pending).")
@@ -1372,13 +1502,16 @@ class LektorUI(tk.Tk):
             self._buttons_idle()
             self.status.configure(text="Idle")
             self.log("Queue finished.\n")
+            self._log_issue("===== queue finished =====")
             return
-        label, mode, cmd = self.jobs.pop(0)
+        label, mode, cmd, folder = self.jobs.pop(0)
         self.running = True
         self.gen_btn.set_enabled(False)
         self.stop_btn.set_enabled(True)
         self.status.configure(text=f"{mode}: {label}   ({len(self.jobs)} left)")
         self.log(f"\n=== {mode.upper()}: {label} ===")
+        self._log_issue(f"--- {mode}: {label}")
+        self._job_tail.clear()
         self.log("$ " + " ".join(f'"{c}"' if " " in c else c for c in cmd[1:]))
         threading.Thread(target=self._run, args=(cmd,), daemon=True).start()
 
@@ -1441,11 +1574,17 @@ class LektorUI(tk.Tk):
                     self.proc = None
                     if payload == 0:
                         self.log("  OK - finished")
+                        self._log_issue("    finished OK")
                     elif payload is not None and payload < 0:
                         self.log(f"  STOPPED (signal {-payload})")
+                        self._log_issue(f"    STOPPED by signal {-payload} — "
+                                        f"{len(self.jobs)} job(s) cancelled")
                         self.jobs.clear()
                     else:
                         self.log(f"  FAILED (exit {payload})")
+                        self._log_issue(f"    FAILED with exit code {payload} — "
+                                        f"{len(self.jobs)} job(s) cancelled")
+                        self._log_tail()
                         self.jobs.clear()
                         self.after(0, lambda c=payload: messagebox.showerror(
                             "Job failed",
@@ -1457,19 +1596,45 @@ class LektorUI(tk.Tk):
             pass
         self.after(80, self._drain_output)
 
+    def _at_bottom(self):
+        """True when the view is already showing the end of the log.
+
+        Scrolling to the end on every insert makes the pane unreadable during a
+        run: output arrives several times a second, so any attempt to look back
+        is yanked to the bottom before a line can be read. Following the tail
+        only while the user is already at it means scrolling up pauses the
+        follow, and scrolling back down resumes it, with no button to press.
+        """
+        try:
+            return self.log_txt.yview()[1] >= 0.999
+        except tk.TclError:                   # widget gone during shutdown
+            return True
+
     def _append(self, text):
+        follow = self._at_bottom()
         self.log_txt.configure(state="normal")
         self.log_txt.insert("end", text)
-        self.log_txt.see("end")
         self.log_txt.configure(state="disabled")
+        if follow:
+            self.log_txt.see("end")
 
     def _replace_last(self, text):
         """Overwrite the last line, the way a terminal handles a carriage return."""
+        follow = self._at_bottom()
         self.log_txt.configure(state="normal")
         self.log_txt.delete("end-1c linestart", "end-1c")
         self.log_txt.insert("end-1c", text)
-        self.log_txt.see("end")
         self.log_txt.configure(state="disabled")
+        if follow:
+            self.log_txt.see("end")
+
+    def _record_if_issue(self, text):
+        line = text.strip()
+        if not line:
+            return
+        self._job_tail.append(line)
+        if self.ISSUE_RE.search(line):
+            self._log_issue(line)
 
     def _emit_line(self, text):
         """A \n-terminated segment.
@@ -1479,6 +1644,7 @@ class LektorUI(tk.Tk):
         to overwrite the line rather than start a new one, or every update is
         appended and the bar marches down the pane instead of animating.
         """
+        self._record_if_issue(text)
         if self._in_redraw:
             self._replace_last(text)
             self._append("\n")
@@ -1493,6 +1659,12 @@ class LektorUI(tk.Tk):
     def log(self, msg):
         self._append(msg + "\n")
 
+    def log_both(self, msg):
+        """Output pane and queue.log. For things the GUI does itself — starting
+        or stopping the backend — which never pass through a job's pipe."""
+        self.log(msg)
+        self._log_issue(msg)
+
     def _buttons_idle(self):
         self.gen_btn.set_enabled(True)
         self.stop_btn.set_enabled(False)
@@ -1538,12 +1710,12 @@ class LektorUI(tk.Tk):
         try:
             if core.comfy_pid() is None:
                 return
-            self.log("Stopping ComfyUI…")
+            self.log_both("Stopping ComfyUI on exit…")
             self.update_idletasks()          # let the message paint before we block
             if not core.stop_comfy_backend():
-                self.log("ComfyUI did not stop — check for a leftover process.")
+                self.log_both("ComfyUI did not stop — check for a leftover process.")
         except Exception as exc:             # noqa: BLE001
-            self.log(f"Could not stop ComfyUI: {exc}")
+            self.log_both(f"Could not stop ComfyUI: {exc}")
 
 
 if __name__ == "__main__":