lektor-ai/docs/usage.md

Usage

ComfyUI must be running before any command that generates speech (~/comfy-tts-app/start_comfyui.sh). --rebuild is the exception — it only touches ffmpeg and works with the backend stopped.

Invocation

python3 lektor_ai_comfy.py [SRT] VIDEO [options]

Both positionals are optional in the argparse sense, because several modes do not need an SRT file. The rule:

# External subtitle file — pass both, SRT first
python3 lektor_ai_comfy.py input/film.pl.srt input/film.mkv

# Subtitles inside the video — pass the video alone
python3 lektor_ai_comfy.py input/film.mkv --embedded-subs

# --rebuild and --regen also need only the video
python3 lektor_ai_comfy.py input/film.mkv --rebuild

When a single positional is given in a mode that needs no SRT, the script moves it into the video slot automatically. Always quote paths containing spaces or parentheses:

python3 lektor_ai_comfy.py "input/Star City (2026)/S01_E01 The Eyes.mkv" --embedded-subs

Working modes

Full generation (default)

Parses subtitles, generates one TTS clip per block, assembles, muxes.

python3 lektor_ai_comfy.py input/film.mkv --embedded-subs

Roughly real-time-ish on a 4070 Ti SUPER: a 60-minute episode of ~700 subtitle blocks takes on the order of an hour. Each clip is saved as it completes, so a crash never loses more than the clip in flight.

--embedded-subs

Lists the video's subtitle tracks and prompts for one:

  Available subtitle tracks in S01_E01 The Eyes.mkv:
    [1] Stream #2: subrip [eng] — Forced
    [2] Stream #3: subrip [eng]
    [3] Stream #4: subrip [eng] — SDH
   ...
    [30] Stream #31: subrip [pol]
  Select track number: 30

The chosen track is extracted to embedded_sub.srt in the job folder and processed exactly like an external file. Image-based tracks are listed but marked unusable — they hold bitmaps, not text.

--rebuild

Reassembles audio from existing clips and rebuilds the video, with no TTS at all. This is the fast iteration loop — seconds to minutes rather than an hour.

python3 lektor_ai_comfy.py input/film.mkv --rebuild
python3 lektor_ai_comfy.py input/film.mkv --rebuild --orig-volume 0.5 --tts-volume 2.0

Use it after --regen, or to adjust the mix. It reads that video's job folder (output/clips/<video name>/) and needs its tts_input.srt plus a complete set of clip_NNNN.flac, reporting which are missing if any are.

If the original run used --end, pass it again so the same video fragment is used.

--regen

Regenerates specific clips by block number, leaving everything else alone.

python3 lektor_ai_comfy.py input/film.mkv --regen 12 22 41
python3 lektor_ai_comfy.py input/film.mkv --rebuild        # then apply

Block numbers match that job's tts_input.srt and its clip_NNNN.flac files. Only the selected FLAC files are replaced — --rebuild afterwards is what actually updates the video.

This is the intended fix for a bad clip. There is no automatic retry; when the model hallucinates, mumbles, or truncates, note the number and regenerate it. A different seed usually resolves it, and --seed 0 (the default) randomises every run.

--resume-from

Continues a crashed run, loading clips 1..N-1 from disk and generating from N onward.

python3 lektor_ai_comfy.py input/film.mkv --resume-from 47

The console prints [47/693] style progress, so the number to resume from is the last one that completed.

It does not reprocess the subtitles. The blocks come from the job folder's existing tts_input.srt, so the dictionary and the number normalizer are not run again:

[1/4] Resuming from block 47 — reusing tts_input.srt…
      693 blocks, subtitles not reprocessed — the numbering still matches the clips on disk.

That is a correctness requirement, not an optimisation. --resume-from pairs block N with clip_NNNN.flac, and re-deriving the text can change it — a dictionary rule added since, a different normalizer result, a changed --start. Same block count with different words is the dangerous case, because nothing would look wrong until you listened.

It also means --start/--end need not be repeated for the block list, since the saved file already holds the adjusted timings. If the job folder has no tts_input.srt, the run says so and falls back to parsing the SRT.

--skip-tts

Uses a pre-assembled audio file and goes straight to the mux.

python3 lektor_ai_comfy.py input/film.mkv --skip-tts "output/clips/film/tts_assembled.flac"

Settings file

Defaults live in settings.ini next to the script, so the values you always want no longer need to be typed on every run. Precedence:

command-line flag   >   settings.ini   >   built-in default

[tts]
voice = gudowski_short
speed = 1.2
temperature = 0.9
max_tokens = 4096
strip_leading_ms = 0
trailing_fade_ms = 80

[mix]
orig_volume = 0.60
tts_volume = 1.8

[paths]
output_dir = output
dictionary = dictionary.txt

[normalize]
numbers = yes
python = normalizer/venv/bin/python
llm = normalizer/models/PLLuM-4B-instruct-2512
timeout = 60
free_comfy_vram = yes

[comfy]
comfy_url = http://127.0.0.1:8189

Every one of these has a matching flag that overrides it for a single run:

# settings.ini says tts_volume = 1.8; this run uses 2.5
python3 lektor_ai_comfy.py film.mkv --embedded-subs --tts-volume 2.5

Each run prints which file it used, so there is never any doubt:

  Settings: /home/lukas/Programy/lektor_ai/settings.ini

Alternative profiles — keep per-show settings side by side and select one:

python3 lektor_ai_comfy.py film.mkv --embedded-subs --settings profiles/docu.ini

Behaviour when things are missing or wrong, none of which is fatal:

Situation	Result
No settings.ini	Built-in defaults; the run says so
Key absent from the file	That key falls back to its built-in default
Value unparseable (speed = fast)	Warning, that key falls back, run continues
File is not valid INI	Warning, all built-in defaults, run continues

Comments use # or ;, inline comments included. Deleting a line is a valid way to return it to its built-in default.

Not in the file, by design: --start, --end, --seed, --regen, --rebuild, --resume-from, --embedded-subs, --skip-tts. These describe a particular run rather than your setup, so pinning them in config would be a trap.

Options

Every option below can also be set in settings.ini, except those noted as run-specific.

Range

Flag	Default	Meaning
--start MM:SS	00:00	Start of the range to process
--end MM:SS	(end)	End of the range

With --end, a matching fragment is extracted to video_fragment.mkv in the job folder and all TTS timestamps become relative to --start. Pass the same values to any later --rebuild, or the audio will not line up.

--end 01:00 is the quickest way to sanity-check a new voice or speed setting.

Voice and model

Flag	Default	Meaning
--voice NAME|PATH	gudowski_short	Name in voices/, or a path to a WAV
--seed N	0	Base seed; 0 randomises each run
--temperature T	0.9	Sampling temperature
--max-tokens N	4096	Token ceiling per clip

--temperature is worth understanding. At 1.0 the model sometimes recites the reference transcript instead of the subtitle. 0.9 largely prevents that; lower is more deterministic but flatter. A per-clip seed of base_seed + index means a fixed --seed reproduces a run exactly.

Raise --max-tokens only if long clips are cut off mid-word.

Timing and cleanup

Flag	Default	Meaning
--speed FACTOR	1.2	Per-clip speed-up via atempo
--strip-leading-ms MS	0	Trim the start of each clip
--trailing-fade-ms MS	80	Fade the end of each clip

--speed compresses speech so a clip is more likely to fit its subtitle window. It is applied per clip, before assembly — never to the assembled track, which would shift every timestamp by speeding up the silences too.

--strip-leading-ms removes an inhale artifact at clip starts; try 80–150 if you hear one. --trailing-fade-ms suppresses tail pops; 0 disables it.

Mixing

Flag	Default	Meaning
--orig-volume 0-1	0.60	Original audio level inside the lektor mix
--tts-volume 0-2	1.8	Voiceover level inside the lektor mix

These affect Track 1 only. Track 0 is always the untouched original, so nothing is lost by mixing aggressively. Changing them needs only --rebuild.

Typical starting point for dialogue-heavy material: --orig-volume 0.5 --tts-volume 2.0. For quiet sources, --orig-volume 1.0 --tts-volume 1.4.

Paths

Flag	Default	Meaning
--output-dir DIR	output	Where everything is written
--dictionary FILE	dictionary.txt	Word/phrase replacements, "" disables
--normalize-numbers / --no-normalize-numbers	on	Spell digits out in Polish
--comfy-url URL	http://127.0.0.1:8189	Backend address

Replacement dictionary

dictionary.txt next to the script rewrites words and phrases in the subtitles before they reach the TTS engine. It exists for everything the model reads out wrong on its own: abbreviations, units, symbols, names with an unusual spelling.

One rule per line:

km/h = kilometrów na godzinę
np. = na przykład
m.in. = między innymi
/(\d+)\s*%/ = \1 procent

Rule	Behaviour
Matching	Case-sensitive, whole words only — np. does not match inside znp., and Joshua does not match joshua
Capitals	The replacement goes in exactly as written; nothing is re-cased for you
Priority	Longer literals win: km/h beats a plain km rule whichever order they are written in
Chaining	Replacement text is never rescanned, so rules cannot feed each other or loop
Regex	A pattern in slashes is a regular expression; \1 \2 … refer to its groups
Comments	Lines starting with # or ;; -> and => also work as separators
Deleting	An empty replacement removes the phrase

Each run reports what it loaded and how much it changed:

  Dictionary: 14 rule(s) from /home/lukas/Programy/lektor_ai/dictionary.txt
      6 dictionary replacement(s) applied.

The replaced text is what lands in the job folder's tts_input.srt — check there when a rule is not doing what you expect. Because --regen and --rebuild work from that file, they do not re-apply the dictionary: edit tts_input.srt and --regen the affected blocks, or rerun from the SRT.

A relative name is looked for in the current directory first and then next to the script, so one film can keep its own list without hiding the shared one. A missing file, a line with no separator and an invalid regex are all warnings, never fatal.

The queue dictionary

dictionary.txt holds rules that are true of Polish generally. A batch often needs its own — a character's name the model mangles, a recurring mistranslation — and those do not belong in the shared file.

The GUI's Queue dictionary button opens a plain text editor with the same format, one rule per line:

Joshua = Dżoszua
P12 = pozycja dwunasta
# comments work here too

Saving writes queue_dictionary.txt into the first queued video's job folder, and every video in the queue is run with it. The dialog reports how many rules parsed, so a typo shows up when you save rather than as a rule that silently does nothing during the run. Clear empties the editor only — nothing is written until you press Save.

The two files are merged into one rule set, not chained: a replacement made by one still cannot be rewritten by the other, and longest-literal-wins applies across both. A run lists them separately:

  Dictionary: 10 rule(s) from /home/lukas/Programy/lektor_ai/dictionary.txt
  Dictionary: 2 rule(s) from output/clips/S02E01/queue_dictionary.txt
      3 dictionary replacement(s) applied.

On the command line it is --dictionary-extra FILE, which works with any file you like.

Number normalization

Higgs Audio misreads long digit strings and sometimes emits garbled non-words, so the last subtitle stage rewrites numbers as Polish words:

Wygrał 1247 złotych.   →  Wygrał tysiąc dwieście czterdzieści siedem złotych.
Mówimy o 25 psach.     →  Mówimy o dwudziestu pięciu psach.
Mam 3 koty.            →  Mam 3 koty.

The case matters — 25 psów / o 25 psach / z 25 psami are three different forms of the same numeral — which is why this is a small fine-tuned model rather than a lookup table. Standalone single digits are left as digits: the TTS reads those correctly, and expanding them only makes the clip longer than the subtitle window it has to fit.

It runs after the dictionary, so a rule that turns 16% into 16 procent feeds szesnaście procent, and after dialogue splitting, so the numbers are normalized in the clip that actually says them.

Flag	Default	Meaning
--normalize-numbers	on	Spell digits out
--no-normalize-numbers		Leave digits alone for this run

or in settings.ini:

[normalize]
numbers = no

Install

The model needs transformers >= 5, which must not go into the ComfyUI environment, so it lives in normalizer/ with its own venv:

normalizer/setup.sh                             # venv + torch + transformers
HF_TOKEN=hf_... normalizer/setup.sh --model     # the weights

The weights are gated: accept the terms at https://huggingface.co/Folx/qwen3-0.6b-pl-text-normalization with your Hugging Face account, then create a token at https://huggingface.co/settings/tokens. That is the only step that needs the network; runs are offline afterwards.

Full details, including the JSON-lines protocol and the deviations from the build brief, are in normalizer/README.md.

Two models, and which one gets a block

Numbers fail in two different ways and no single model handles both:

block contains	goes to	because
plain integers only (74%)	num2words → inflection model	num2words cannot miscount; the model only fixes the case
times, dates, decimals, P10, LO 3842 (26%)	the 0.6B fine-tune	it was trained on those formats; the other leg destroys them

The split is why Mówili o 250000 uchodźcach. comes out as o dwustu pięćdziesięciu tysiącach uchodźców — the fine-tune alone renders that as o pięciu tysiącach, losing the number — while Teraz na P12. still comes out as P dwanaście rather than Pdziesięć.

Every answer is then checked before use: runaway length, repetition loops, invalid characters, leftover digits, a lost minus sign, a changed number, a lost word. A block that fails falls back to num2words — nominative, but never wrong about the value. The run reports how often that happened.

Set llm = (empty) to use the fine-tune for everything.

What a run prints

      5 block(s) contain numbers — normalizing…
      Asking ComfyUI to unload its models (7.9 GB)…
      Freed 7.9 GB.
      Normalizer ready (model on cuda:0, two models).
      5 of 5 block(s) rewritten (2 complex, 3 plain).
      1 block(s) failed a safety check and fell back (word lost x1).

Normalizer ready (fallback) means the model is missing or failed to load and num2words is standing in — bare integers in the nominative case only, with times, dates and decimals left as digits. The run says so and continues.

Sharing the GPU with ComfyUI

Higgs stays resident between runs — about 8.4 GB of a 16 GB card — which leaves too little for the normalizer. That failure is silent rather than loud: accelerate answers a shortage by spilling layers to RAM, so the run continues with no error and roughly 5× the time per block (measured: 1.19 s → 5.96 s).

So before starting the normalizer the script asks ComfyUI to unload:

      5 block(s) contain numbers — normalizing…
      Asking ComfyUI to unload its models (7.9 GB)…
      Freed 7.9 GB.
      Normalizer ready (model on cuda:0).

ComfyUI reloads Higgs on the first clip of step 2 — 20–30 s, once per run. It is a POST /free on ComfyUI's own API, not a restart: the server stays up, with its custom nodes loaded. The unload is asynchronous, so the script waits until torch_vram_total in /system_stats actually drops before continuing.

--regen and --rebuild work from tts_input.srt, which is already normalized, so they never normalize and never pay the reload. On a freshly started ComfyUI the call is a no-op and prints nothing.

Set free_comfy_vram = no to skip it — worth doing only if you have VRAM to spare, or if you have turned the neural normalizer off.

Nothing about this stage is fatal. A missing venv, a worker that dies, a block that takes longer than timeout — each is a warning, and the affected text goes to the TTS with its digits intact. The worker is started once per run and exits before TTS generation begins, so it is not holding VRAM while Higgs works.

The backend's lifecycle

ComfyUI is normally started once and left running. The script manages it when it has to:

situation	what happens
not running when a clip is submitted	started via start_comfyui.sh, waited for, the clip resubmitted
running but wedged or suspended	stopped (SIGKILL if suspended) before the restart
dies on a clip	up to 3 retries, each with a different seed
dies 3 times on the same clip	that clip is skipped, the queue continues, all skips reported at the end
queue finished	models unloaded, backend left running — it reloads on the next request
GUI closed	backend stopped, so no idle process is left holding the GPU

Its output goes to comfyui.log next to the script as well as to the terminal window, so a crash can be diagnosed after the fact.

The GUI has a ComfyUI button showing the same state: Start ComfyUI when it is down, ComfyUI ● when it is answering, ComfyUI ◍ when it holds the port but does not answer. Pressing it starts or restarts.

Quitting the GUI stops the backend, including one you started by hand in a terminal — it is found by port, not by remembering what was launched. Between runs it stays up on purpose: the queue only unloads its models, so the next job does not pay for a cold start.

start_comfyui.sh ends with exec python main.py, so the script's PID is ComfyUI's. Without that the launcher only ever sees a wrapper shell, and killing it orphans the python still holding 8 GB of VRAM.

Output

Everything belonging to one video is kept together in a job folder named after the video file. Only the finished video is written to the top level.

output/
  clips/
    S01_E01 The Eyes/          job folder, named after the video file
      clip_NNNN.flac           one lossless clip per block, kept for --regen
      tts_input.srt            processed text actually sent to TTS
      tts_assembled.flac       all clips at their final positions, lossless
      embedded_sub.srt         extracted subtitle (--embedded-subs only)
      video_fragment.mkv       extracted segment (--end only)
    F1 The Movie 2025/
      ...
  S01_E01 The Eyes_lektor.mkv  the finished video
  F1 The Movie 2025_lektor.mkv

Each run prints the folder it is using:

  Job folder: output/clips/S01_E01 The Eyes

This is what makes --rebuild and --regen safe across films. tts_input.srt and the clips are a matched pair — block N in the SRT is clip_NNNN.flac — so both live in the job folder. They previously sat at the top level and every new job overwrote the last one's, which meant rebuilding an older film paired its clips with the newest film's text. Any past job can now be rebuilt at any time.

The folder is derived from the video filename, so --rebuild and --regen locate it automatically; just pass the same video path.

The finished file contains:

Track	Content
Video	Stream-copied, untouched
Audio 0	Original — stream-copied, full 5.1 if the source had it
Audio 1	Polish — lektor mix, AAC 192k stereo, language=pol
Subtitles	Every track from the source, stream-copied

The lektor track is stereo deliberately; see troubleshooting.md.

FLAC is used for every intermediate so repeated processing stays lossless. Only the final voiceover track is ever encoded.

Recipes

Check a voice before committing to a full run

python3 lektor_ai_comfy.py input/film.mkv --embedded-subs --end 02:00 --voice gudowski

Tune the mix without regenerating speech

python3 lektor_ai_comfy.py input/film.mkv --rebuild --orig-volume 0.45 --tts-volume 2.1

Fix a handful of bad clips

# read output/clips/<video name>/tts_input.srt, note the offending block numbers
python3 lektor_ai_comfy.py input/film.mkv --regen 88 91 104
python3 lektor_ai_comfy.py input/film.mkv --rebuild

Reproduce a run exactly

python3 lektor_ai_comfy.py input/film.mkv --embedded-subs --seed 123456

Speech running long and overlapping the next line

python3 lektor_ai_comfy.py input/film.mkv --embedded-subs --speed 1.35

Reading the console

[1/4] Parsing subtitles 00:00–end…
      693 entries after dialogue splitting.
[2/4] Generating TTS (693 clips, speed 1.2×)…
  [12/693] 00:01:34,120  'Nie wiem, o czym mówisz.'
    Waiting for a3f8b201… done.
    → placed at 00:01:34,120, duration 1.84s

→ placed at reveals timing drift. A clip that overruns its window pushes the next one forward, shown as [+1.20s behind SRT]. Small values are normal and self-correct at the next natural gap in the subtitles; values that keep growing mean --speed is too low for the material.

Assembly then shows a progress bar per batch (clips are assembled 500 at a time to stay under the open-file limit), followed by the two-pass mux.