# Architecture How `lektor_ai_comfy.py` turns subtitles into a finished video, and why each stage works the way it does. Most of the non-obvious choices here exist because the straightforward version was tried first and broke. ``` SRT / embedded track │ parse, clean, split dialogue ▼ subtitle blocks ──────────────► output/tts_input.srt │ one HTTP job per block ▼ ComfyUI + Higgs Audio V3 │ per-clip atempo + fade ▼ clips/clip_NNNN.flac │ adelay + amix, batched ▼ tts_assembled.flac │ pass 1: mix with original audio │ pass 2: stream-copy mux ▼ {stem}_lektor.mkv ``` ## 1. Subtitle parsing `parse_subtitle_blocks()` reads the SRT with `utf-8-sig` (tolerating a BOM), splits on blank lines, and keeps blocks whose start falls inside the requested range. Times become **relative to `--start`**, which is why a later `--rebuild` must repeat the same `--start`/`--end`. `_clean_text()` then normalises each block, in this order: 1. **Strip formatting tags.** `{...}` ASS overrides (`{\an8}`, `{\pos(x,y)}`) and `<...>` HTML tags (``, ``). 2. **Detect dialogue.** Lines beginning `- ` mean two speakers and are kept separate; ordinary wrapped lines are joined with a space. 3. **Sentence-case ALL-CAPS blocks**, which the model would otherwise shout. 4. **Capitalise a lowercase first letter**, since a fragment continuing from the previous subtitle confuses the model. 5. **Add terminal punctuation** so the model produces a natural falling tail. Step 1 must come first. `ALL CAPS` looks mixed-case while the tag is still present — the `i` in `` is a lowercase letter — so the caps detection in step 3 would silently miss it. `split_dialogue_blocks()` then divides a two-speaker block into one clip per line, apportioning the block's duration by character count. The result is written to `output/tts_input.srt`. That file is the source of truth for block numbering: block *N* there is `clips/clip_NNNN.flac` and the argument to `--regen N`. ## 2. Speech generation For each block the script POSTs a workflow graph to `/prompt`, polls `/history/{id}` every 3 s, and reads the resulting file directly from ComfyUI's output directory. The graph has five nodes: | Node | Class | Role | |---|---|---| | 1 | `HiggsAudioV3EngineNode` | Model, temperature, top_p 0.95, top_k 50 | | 2 | `LoadAudio` | Voice reference WAV from ComfyUI's input dir | | 3 | `CharacterVoicesNode` | Pairs that audio with its transcript | | 4 | `UnifiedTTSTextNode` | The text, the seed, `enable_audio_cache: False` | | 5 | `SaveAudio` | Writes FLAC | Three details matter: - **`enable_audio_cache: False`** is essential. With caching on, ComfyUI returns a previous result for repeated text and every `--regen` is a no-op. - **The seed is `base_seed + block_index`**, so a fixed `--seed` is fully reproducible while each clip still differs. - **No language tag is sent.** Higgs Audio V3 detects Polish from the text itself; its tokenizer has no `<|lang:pl|>` token, and a `[pl:]` prefix is stripped by the character parser before it reaches the model. A `lang: en` line in the ComfyUI log is cosmetic. Each finished clip is optionally head-trimmed (`--strip-leading-ms`), then sped up with `atempo` and given a short fade-out — **per clip, before assembly**. Applying `atempo` to the assembled track instead would speed up the silences between clips and shift every subsequent timestamp. ## 3. Placement Clips are positioned with a push-forward cursor: ```python actual_start = max(t_start, cursor) cursor = actual_start + duration ``` A clip starts at its subtitle time, or when the previous clip finished if that is later. Clips are **never trimmed** to their subtitle window — cutting speech mid-sentence sounds far worse than running a little long. Overrun cascades forward and then resolves by itself at the next natural gap in the subtitles, because a gap lets the cursor fall back below the next `t_start`. The console prints `[+1.20s behind SRT]` when drift occurs. Steady growth means `--speed` is too low for the material. ## 4. Assembly Each clip becomes an `adelay` filter input and everything is mixed with `amix`: ``` [0:a]adelay=1340:all=1[a0]; [1:a]adelay=4820:all=1[a1]; … [a0][a1]…amix=inputs=N:duration=longest:normalize=0[out] ``` `normalize=0` is important — `amix` otherwise divides by the input count, which would make every clip inaudibly quiet across hundreds of inputs. Above 500 clips the work is **batched**, because one ffmpeg process cannot hold thousands of open inputs. Each batch is assembled to its own temporary FLAC and the batches are then mixed together. Batches use **relative** delays: the first clip's start time is subtracted from every clip in the batch, and re-applied when the batches are combined. Without that subtraction, batch 7 would begin with 40 minutes of silence, wasting time and making the progress bar useless. ## 5. Muxing `merge_voiceover()` runs **two separate ffmpeg passes**. This is deliberate and should not be collapsed back into one. ### Pass 1 — build the lektor mix ``` [0:a:0],aformat=channel_layouts=stereo,volume=[orig_mix]; [1:a]aformat=channel_layouts=stereo,volume=[tts]; [orig_mix][tts]amix=inputs=2:duration=first:normalize=0[lektor] ``` Written to a temporary `*_mix.m4a`, AAC 192k, and deleted in a `finally` block. **Stereo, not 5.1.** ffmpeg's AAC encoder does not signal a 5.1 channel layout — the track comes back tagged `unknown` in every container tested, and players that cannot route the channels play silence. Stereo is tagged correctly, plays everywhere, and spends the 192 kbit/s on two channels instead of six. Track 0 keeps the untouched 5.1, so no surround information is lost. **PTS normalisation.** MKV rips often start their audio at a non-zero PTS (one source here begins at 4.043 s). `amix` cannot emit output before both inputs have frames, so the voiceover would be delayed by that offset. When ffprobe reports an offset the filter becomes: ``` asetpts=PTS-STARTPTS,adelay=:all=1,aresample=async=1 ``` `asetpts` rebases the stream to zero and `adelay` reinserts the offset as real silence, so the background audio stays where it belongs while `amix` receives frames from both inputs starting at zero. `duration=first` with `orig_mix` listed first means the mix runs as long as the film, not as long as the voiceover. ### Pass 2 — stream-copy mux Video, original audio, the mix, and every subtitle track are copied into the final MKV with `-c copy`. Nothing is re-encoded, and it runs at several hundred times real time. Two flags carry the weight here: **`-max_interleave_delta 0`**, placed **after the maps and before the output path**. It is an output-muxer option; before `-i` it silently configures the demuxer and does nothing. Without it, sparse subtitle tracks make the muxer abandon interleaving after its default 10-second window, and the lektor audio ends up written hundreds of megabytes away from the video it belongs with — measurably fine to ffmpeg, but silent in any player that reads sequentially. **MKV output**, not MP4. MKV copies subtitles of any codec without conversion, carries DTS and TrueHD in track 0, and avoids the `mov_text` conversion that made large multi-subtitle files abort mid-mux. ### Why two passes Doing the filtering and the subtitle muxing in one command fails on sources with many subtitle tracks. The sparse subtitle streams stall the muxer while the audio filter graph keeps producing, and ffmpeg aborts partway through with: ``` Error while filtering: Invalid argument Failed to inject frame into filter network: Invalid argument ``` The message points at audio, but audio is not the cause. Splitting the work keeps the filter graph free of subtitle streams and makes the final mux a pure copy. It is also about ten times faster. ## Progress reporting `_run_progress()` adds `-progress pipe:1 -nostats` before the output path, reads `out_time_us=` lines from stdout, and rewrites a single line with `\r`. A daemon thread drains stderr concurrently, so ffmpeg cannot deadlock on a full stderr pipe while the main thread reads stdout. ## Design rules Worth preserving, since each was learned by breaking it: 1. **Per-clip processing before assembly**, never on the assembled track. 2. **Never trim clips** to the subtitle window. 3. **Keep every clip file** — `--regen` depends on them. 4. **FLAC throughout**, encoding only the final voiceover track. 5. **Two audio tracks**, with the original stream-copied and untouched. 6. **Verify muxed output by byte position**, not only by volume. A track can measure perfectly and still be unplayable.