lektor-ai/docs/architecture.md

Architecture

How lektor_ai_comfy.py turns subtitles into a finished video, and why each stage works the way it does. Most of the non-obvious choices here exist because the straightforward version was tried first and broke.

SRT / embedded track
   │  parse, clean, split dialogue
   ▼
subtitle blocks ──────────────► <job>/tts_input.srt
   │  one HTTP job per block
   ▼
ComfyUI + Higgs Audio V3
   │  per-clip atempo + fade
   ▼
<job>/clip_NNNN.flac
   │  adelay + amix, batched
   ▼
tts_assembled.flac
   │  pass 1: mix with original audio
   │  pass 2: stream-copy mux
   ▼
{stem}_lektor.mkv

1. Subtitle parsing

parse_subtitle_blocks() reads the SRT with utf-8-sig (tolerating a BOM), splits on blank lines, and keeps blocks whose start falls inside the requested range. Times become relative to --start, which is why a later --rebuild must repeat the same --start/--end.

_clean_text() then normalises each block, in this order:

1. Strip formatting tags. {...} ASS overrides ({\an8}, {\pos(x,y)}) and <...> HTML tags (<i>, <font …>).
2. Detect dialogue. Lines beginning - mean two speakers and are kept separate; ordinary wrapped lines are joined with a space.
3. Sentence-case ALL-CAPS blocks, which the model would otherwise shout.
4. Capitalise a lowercase first letter, since a fragment continuing from the previous subtitle confuses the model.
5. Add terminal punctuation so the model produces a natural falling tail.

Step 1 must come first. <i>ALL CAPS</i> looks mixed-case while the tag is still present — the i in <i> is a lowercase letter — so the caps detection in step 3 would silently miss it.

split_dialogue_blocks() then divides a two-speaker block into one clip per line, apportioning the block's duration by character count.

The result is written to tts_input.srt in that video's job folder (output/clips/<video name>/). It is the source of truth for block numbering: block N there is clip_NNNN.flac beside it and the argument to --regen N.

2. Speech generation

For each block the script POSTs a workflow graph to /prompt, polls /history/{id} until it completes, and reads the resulting file directly from ComfyUI's output directory.

Polling cadence is a performance decision, not a detail. A clip that finishes just after a poll waits for the next one, so the interval — not the GPU — sets the floor on throughput. It was originally a flat 3 s, which quantised every clip to a 3 s boundary: measured across 2075 real clips, 91% of the gaps landed exactly on a multiple of 3 s.

wait_for_completion() now starts at 0.25 s and backs off to 2 s once a job passes 10 s, so the common case is picked up almost immediately while a stuck generation does not sit at 4 requests/second indefinitely. Measured over 34 clips:

	mean	median	34 clips
Actual generation	4.75 s	4.60 s	162 s
Old flat 3 s poll	6.35 s	6.00 s	216 s

That is 1.60 s wasted per clip, a 25% reduction in TTS wait — about 18 minutes on a 693-clip episode. The run also now prints real generation time (done (4.2s)), which is what makes this measurable at all.

ComfyUI also pushes completion events over a WebSocket, which would remove polling entirely. It is not used because Python has no standard-library WebSocket client, and that would cost this script its only real virtue as a dependency-free file. Polling at 0.25 s recovers most of the benefit.

The graph has five nodes:

Node	Class	Role
1	HiggsAudioV3EngineNode	Model, temperature, top_p 0.95, top_k 50
2	LoadAudio	Voice reference WAV from ComfyUI's input dir
3	CharacterVoicesNode	Pairs that audio with its transcript
4	UnifiedTTSTextNode	The text, the seed, enable_audio_cache: False
5	SaveAudio	Writes FLAC

Three details matter:

• enable_audio_cache: False is essential. With caching on, ComfyUI returns a previous result for repeated text and every --regen is a no-op.
• The seed is base_seed + block_index, so a fixed --seed is fully reproducible while each clip still differs.
• No language tag is sent. Higgs Audio V3 detects Polish from the text itself; its tokenizer has no <|lang:pl|> token, and a [pl:] prefix is stripped by the character parser before it reaches the model. A lang: en line in the ComfyUI log is cosmetic.

Each finished clip is optionally head-trimmed (--strip-leading-ms), then sped up with atempo and given a short fade-out — per clip, before assembly. Applying atempo to the assembled track instead would speed up the silences between clips and shift every subsequent timestamp.

3. Placement

Clips are positioned with a push-forward cursor:

actual_start = max(t_start, cursor)
cursor       = actual_start + duration

A clip starts at its subtitle time, or when the previous clip finished if that is later. Clips are never trimmed to their subtitle window — cutting speech mid-sentence sounds far worse than running a little long. Overrun cascades forward and then resolves by itself at the next natural gap in the subtitles, because a gap lets the cursor fall back below the next t_start.

The console prints [+1.20s behind SRT] when drift occurs. Steady growth means --speed is too low for the material.

4. Assembly

Each clip becomes an adelay filter input and everything is mixed with amix:

[0:a]adelay=1340:all=1[a0]; [1:a]adelay=4820:all=1[a1]; …
[a0][a1]…amix=inputs=N:duration=longest:normalize=0[out]

normalize=0 is important — amix otherwise divides by the input count, which would make every clip inaudibly quiet across hundreds of inputs.

Above 500 clips the work is batched, because one ffmpeg process cannot hold thousands of open inputs. Each batch is assembled to its own temporary FLAC and the batches are then mixed together.

Batches use relative delays: the first clip's start time is subtracted from every clip in the batch, and re-applied when the batches are combined. Without that subtraction, batch 7 would begin with 40 minutes of silence, wasting time and making the progress bar useless.

5. Muxing

merge_voiceover() runs two separate ffmpeg passes. This is deliberate and should not be collapsed back into one.

Pass 1 — build the lektor mix

[0:a:0]<norm>,aformat=channel_layouts=stereo,volume=<orig>[orig_mix];
[1:a]aformat=channel_layouts=stereo,volume=<tts>[tts];
[orig_mix][tts]amix=inputs=2:duration=first:normalize=0[lektor]

Written to a temporary *_mix.m4a, AAC 192k, and deleted in a finally block.

Stereo, not 5.1. ffmpeg's AAC encoder does not signal a 5.1 channel layout — the track comes back tagged unknown in every container tested, and players that cannot route the channels play silence. Stereo is tagged correctly, plays everywhere, and spends the 192 kbit/s on two channels instead of six. Track 0 keeps the untouched 5.1, so no surround information is lost.

PTS normalisation. MKV rips often start their audio at a non-zero PTS (one source here begins at 4.043 s). amix cannot emit output before both inputs have frames, so the voiceover would be delayed by that offset. When ffprobe reports an offset the filter becomes:

asetpts=PTS-STARTPTS,adelay=<offset_ms>:all=1,aresample=async=1

asetpts rebases the stream to zero and adelay reinserts the offset as real silence, so the background audio stays where it belongs while amix receives frames from both inputs starting at zero.

duration=first with orig_mix listed first means the mix runs as long as the film, not as long as the voiceover.

Pass 2 — stream-copy mux

Video, original audio, the mix, and every subtitle track are copied into the final MKV with -c copy. Nothing is re-encoded, and it runs at several hundred times real time.

Two flags carry the weight here:

-max_interleave_delta 0, placed after the maps and before the output path. It is an output-muxer option; before -i it silently configures the demuxer and does nothing. Without it, sparse subtitle tracks make the muxer abandon interleaving after its default 10-second window, and the lektor audio ends up written hundreds of megabytes away from the video it belongs with — measurably fine to ffmpeg, but silent in any player that reads sequentially.

MKV output, not MP4. MKV copies subtitles of any codec without conversion, carries DTS and TrueHD in track 0, and avoids the mov_text conversion that made large multi-subtitle files abort mid-mux.

Why two passes

Doing the filtering and the subtitle muxing in one command fails on sources with many subtitle tracks. The sparse subtitle streams stall the muxer while the audio filter graph keeps producing, and ffmpeg aborts partway through with:

Error while filtering: Invalid argument
Failed to inject frame into filter network: Invalid argument

The message points at audio, but audio is not the cause. Splitting the work keeps the filter graph free of subtitle streams and makes the final mux a pure copy. It is also about ten times faster.

Progress reporting

_run_progress() adds -progress pipe:1 -nostats before the output path, reads out_time_us= lines from stdout, and rewrites a single line with \r. A daemon thread drains stderr concurrently, so ffmpeg cannot deadlock on a full stderr pipe while the main thread reads stdout.

Design rules

Worth preserving, since each was learned by breaking it:

1. Per-clip processing before assembly, never on the assembled track.
2. Never trim clips to the subtitle window.
3. Keep every clip file — --regen depends on them.
4. FLAC throughout, encoding only the final voiceover track.
5. Two audio tracks, with the original stream-copied and untouched.
6. Verify muxed output by byte position, not only by volume. A track can measure perfectly and still be unplayable.