Initial commit: Polish AI dubbing pipeline (ComfyUI + Higgs Audio V3)

Converts SRT subtitles into a Polish lektor-style voiceover using a local
ComfyUI instance running Higgs Audio V3, then muxes the result back into the
source video as a second audio track. Video and original audio are always
stream-copied; only the new voiceover track is encoded.

Contents:
  lektor_ai_comfy.py    main pipeline (standard library only, needs ffmpeg)
  lektor_ai_comfy_v2.py batched-context experiment (timing unresolved)
  lektor_ai.py          earlier version using the hosted Boson AI API
  install_comfyui.sh    reproduces the ComfyUI environment, versions pinned
  docs/                 requirements, installation, usage, architecture,
                        troubleshooting
  gudowski/             voice references (.wav) with their transcripts (.txt)

Source media and generated video are gitignored.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>

docs/architecture.md

@@ -0,0 +1,215 @@
+# Architecture
+
+How `lektor_ai_comfy.py` turns subtitles into a finished video, and why each
+stage works the way it does. Most of the non-obvious choices here exist because
+the straightforward version was tried first and broke.
+
+```
+SRT / embedded track
+   │  parse, clean, split dialogue
+   ▼
+subtitle blocks ──────────────► output/tts_input.srt
+   │  one HTTP job per block
+   ▼
+ComfyUI + Higgs Audio V3
+   │  per-clip atempo + fade
+   ▼
+clips/clip_NNNN.flac
+   │  adelay + amix, batched
+   ▼
+tts_assembled.flac
+   │  pass 1: mix with original audio
+   │  pass 2: stream-copy mux
+   ▼
+{stem}_lektor.mkv
+```
+
+## 1. Subtitle parsing
+
+`parse_subtitle_blocks()` reads the SRT with `utf-8-sig` (tolerating a BOM),
+splits on blank lines, and keeps blocks whose start falls inside the requested
+range. Times become **relative to `--start`**, which is why a later `--rebuild`
+must repeat the same `--start`/`--end`.
+
+`_clean_text()` then normalises each block, in this order:
+
+1. **Strip formatting tags.** `{...}` ASS overrides (`{\an8}`, `{\pos(x,y)}`)
+   and `<...>` HTML tags (`<i>`, `<font …>`).
+2. **Detect dialogue.** Lines beginning `- ` mean two speakers and are kept
+   separate; ordinary wrapped lines are joined with a space.
+3. **Sentence-case ALL-CAPS blocks**, which the model would otherwise shout.
+4. **Capitalise a lowercase first letter**, since a fragment continuing from the
+   previous subtitle confuses the model.
+5. **Add terminal punctuation** so the model produces a natural falling tail.
+
+Step 1 must come first. `<i>ALL CAPS</i>` looks mixed-case while the tag is
+still present — the `i` in `<i>` is a lowercase letter — so the caps detection
+in step 3 would silently miss it.
+
+`split_dialogue_blocks()` then divides a two-speaker block into one clip per
+line, apportioning the block's duration by character count.
+
+The result is written to `output/tts_input.srt`. That file is the source of
+truth for block numbering: block *N* there is `clips/clip_NNNN.flac` and the
+argument to `--regen N`.
+
+## 2. Speech generation
+
+For each block the script POSTs a workflow graph to `/prompt`, polls
+`/history/{id}` every 3 s, and reads the resulting file directly from ComfyUI's
+output directory.
+
+The graph has five nodes:
+
+| Node | Class | Role |
+|---|---|---|
+| 1 | `HiggsAudioV3EngineNode` | Model, temperature, top_p 0.95, top_k 50 |
+| 2 | `LoadAudio` | Voice reference WAV from ComfyUI's input dir |
+| 3 | `CharacterVoicesNode` | Pairs that audio with its transcript |
+| 4 | `UnifiedTTSTextNode` | The text, the seed, `enable_audio_cache: False` |
+| 5 | `SaveAudio` | Writes FLAC |
+
+Three details matter:
+
+- **`enable_audio_cache: False`** is essential. With caching on, ComfyUI returns
+  a previous result for repeated text and every `--regen` is a no-op.
+- **The seed is `base_seed + block_index`**, so a fixed `--seed` is fully
+  reproducible while each clip still differs.
+- **No language tag is sent.** Higgs Audio V3 detects Polish from the text
+  itself; its tokenizer has no `<|lang:pl|>` token, and a `[pl:]` prefix is
+  stripped by the character parser before it reaches the model. A `lang: en`
+  line in the ComfyUI log is cosmetic.
+
+Each finished clip is optionally head-trimmed (`--strip-leading-ms`), then
+sped up with `atempo` and given a short fade-out — **per clip, before
+assembly**. Applying `atempo` to the assembled track instead would speed up the
+silences between clips and shift every subsequent timestamp.
+
+## 3. Placement
+
+Clips are positioned with a push-forward cursor:
+
+```python
+actual_start = max(t_start, cursor)
+cursor       = actual_start + duration
+```
+
+A clip starts at its subtitle time, or when the previous clip finished if that
+is later. Clips are **never trimmed** to their subtitle window — cutting speech
+mid-sentence sounds far worse than running a little long. Overrun cascades
+forward and then resolves by itself at the next natural gap in the subtitles,
+because a gap lets the cursor fall back below the next `t_start`.
+
+The console prints `[+1.20s behind SRT]` when drift occurs. Steady growth means
+`--speed` is too low for the material.
+
+## 4. Assembly
+
+Each clip becomes an `adelay` filter input and everything is mixed with `amix`:
+
+```
+[0:a]adelay=1340:all=1[a0]; [1:a]adelay=4820:all=1[a1]; …
+[a0][a1]…amix=inputs=N:duration=longest:normalize=0[out]
+```
+
+`normalize=0` is important — `amix` otherwise divides by the input count, which
+would make every clip inaudibly quiet across hundreds of inputs.
+
+Above 500 clips the work is **batched**, because one ffmpeg process cannot hold
+thousands of open inputs. Each batch is assembled to its own temporary FLAC and
+the batches are then mixed together.
+
+Batches use **relative** delays: the first clip's start time is subtracted from
+every clip in the batch, and re-applied when the batches are combined. Without
+that subtraction, batch 7 would begin with 40 minutes of silence, wasting time
+and making the progress bar useless.
+
+## 5. Muxing
+
+`merge_voiceover()` runs **two separate ffmpeg passes**. This is deliberate and
+should not be collapsed back into one.
+
+### Pass 1 — build the lektor mix
+
+```
+[0:a:0]<norm>,aformat=channel_layouts=stereo,volume=<orig>[orig_mix];
+[1:a]aformat=channel_layouts=stereo,volume=<tts>[tts];
+[orig_mix][tts]amix=inputs=2:duration=first:normalize=0[lektor]
+```
+
+Written to a temporary `*_mix.m4a`, AAC 192k, and deleted in a `finally` block.
+
+**Stereo, not 5.1.** ffmpeg's AAC encoder does not signal a 5.1 channel layout —
+the track comes back tagged `unknown` in every container tested, and players
+that cannot route the channels play silence. Stereo is tagged correctly, plays
+everywhere, and spends the 192 kbit/s on two channels instead of six. Track 0
+keeps the untouched 5.1, so no surround information is lost.
+
+**PTS normalisation.** MKV rips often start their audio at a non-zero PTS (one
+source here begins at 4.043 s). `amix` cannot emit output before both inputs
+have frames, so the voiceover would be delayed by that offset. When ffprobe
+reports an offset the filter becomes:
+
+```
+asetpts=PTS-STARTPTS,adelay=<offset_ms>:all=1,aresample=async=1
+```
+
+`asetpts` rebases the stream to zero and `adelay` reinserts the offset as real
+silence, so the background audio stays where it belongs while `amix` receives
+frames from both inputs starting at zero.
+
+`duration=first` with `orig_mix` listed first means the mix runs as long as the
+film, not as long as the voiceover.
+
+### Pass 2 — stream-copy mux
+
+Video, original audio, the mix, and every subtitle track are copied into the
+final MKV with `-c copy`. Nothing is re-encoded, and it runs at several hundred
+times real time.
+
+Two flags carry the weight here:
+
+**`-max_interleave_delta 0`**, placed **after the maps and before the output
+path**. It is an output-muxer option; before `-i` it silently configures the
+demuxer and does nothing. Without it, sparse subtitle tracks make the muxer
+abandon interleaving after its default 10-second window, and the lektor audio
+ends up written hundreds of megabytes away from the video it belongs with —
+measurably fine to ffmpeg, but silent in any player that reads sequentially.
+
+**MKV output**, not MP4. MKV copies subtitles of any codec without conversion,
+carries DTS and TrueHD in track 0, and avoids the `mov_text` conversion that
+made large multi-subtitle files abort mid-mux.
+
+### Why two passes
+
+Doing the filtering and the subtitle muxing in one command fails on sources with
+many subtitle tracks. The sparse subtitle streams stall the muxer while the
+audio filter graph keeps producing, and ffmpeg aborts partway through with:
+
+```
+Error while filtering: Invalid argument
+Failed to inject frame into filter network: Invalid argument
+```
+
+The message points at audio, but audio is not the cause. Splitting the work
+keeps the filter graph free of subtitle streams and makes the final mux a pure
+copy. It is also about ten times faster.
+
+## Progress reporting
+
+`_run_progress()` adds `-progress pipe:1 -nostats` before the output path, reads
+`out_time_us=` lines from stdout, and rewrites a single line with `\r`. A daemon
+thread drains stderr concurrently, so ffmpeg cannot deadlock on a full stderr
+pipe while the main thread reads stdout.
+
+## Design rules
+
+Worth preserving, since each was learned by breaking it:
+
+1. **Per-clip processing before assembly**, never on the assembled track.
+2. **Never trim clips** to the subtitle window.
+3. **Keep every clip file** — `--regen` depends on them.
+4. **FLAC throughout**, encoding only the final voiceover track.
+5. **Two audio tracks**, with the original stream-copied and untouched.
+6. **Verify muxed output by byte position**, not only by volume. A track can
+   measure perfectly and still be unplayable.