lektor-ai/readme.txt

lektor_ai_comfy — Polish AI Dubbing Tool
=========================================

FULL DOCUMENTATION: see docs/
  docs/README.md           start here — overview and quick start
  docs/requirements.md     hardware, software, dependencies
  docs/installation.md     installing ComfyUI + the TTS engine
  docs/usage.md            every CLI flag, working modes, recipes
  docs/architecture.md     how the pipeline works internally
  docs/troubleshooting.md  diagnosed failure modes and their fixes

Installer: ./install_backend.sh   (see docs/installation.md)

This file remains as a condensed quick reference.


Converts SRT subtitles to AI voiceover (lektor style) using ComfyUI + Higgs Audio V3 TTS,
then mixes the result with the original video audio into a two-track MKV.
All subtitle tracks in the source video are carried over to the output.

Requirements
------------
- ComfyUI running at http://127.0.0.1:8189 with TTS-Audio-Suite custom node
- Model: higgs-audio-v3-tts-4b (bosonai/higgs-audio-v3)
- ffmpeg and ffprobe available on PATH
- Voice reference: a WAV file + matching .txt transcript in the voice directory
  (default: voices/gudowski_short.wav + voices/gudowski_short.txt)

Settings
--------
Defaults live in settings.ini next to the script:
  voice, speed, temperature, max_tokens, strip_leading_ms, trailing_fade_ms,
  orig_volume, tts_volume, output_dir, dictionary, numbers, comfy_url

Precedence: command-line flag > settings.ini > built-in default.
Use --settings FILE to select a different profile.
See docs/usage.md for the full description.


Basic Usage
-----------

Full run with an external SRT file:

    python3 lektor_ai_comfy.py input/film.srt input/film.mkv

Use subtitles embedded in the video (lists tracks, prompts for a choice):

    python3 lektor_ai_comfy.py input/film.mkv --embedded-subs

Process only a time range (extracts a video fragment automatically):

    python3 lektor_ai_comfy.py input/film.srt input/film.mkv --start 01:20 --end 05:40


Subtitle Handling
-----------------
Both external SRT files and embedded subtitle tracks go through the same cleaning:

- HTML tags stripped:          <i>, </i>, <b>, <font color="…">, etc.
- ASS override codes stripped: {\an8}, {\pos(x,y)}, {\c&H…&}, etc.
- ALL-CAPS blocks converted to Title Case (keeps proper nouns readable)
- Sentence fragments starting with a lowercase letter are capitalised
- Missing terminal punctuation added (… or . depending on context)
- Dictionary replacements applied (see below)
- Dialogue lines (prefixed with -) are split into separate clips with proportional timing
- Numbers spelled out as Polish words, in the right case (see below)

With --embedded-subs the extracted subtitles are saved to the job folder as embedded_sub.srt
so you can inspect them before or after the run.

The final MKV preserves all subtitle tracks from the source video — text-based
(SRT, ASS, SSA, WebVTT) and image-based (PGS Blu-ray, VOBSUB) alike — using
stream copy with no conversion.


Replacement Dictionary
----------------------
dictionary.txt next to the script rewrites words and phrases before they reach
the TTS engine — abbreviations, units, symbols, names the model mispronounces.
One rule per line:

    km/h = kilometrów na godzinę
    np. = na przykład
    /(\d+)\s*%/ = \1 procent

Matching is case-sensitive and whole-word only, so "np." leaves "znp." alone and
"Joshua" leaves "joshua" alone. The replacement is inserted exactly as written.
The first letter of each block is capitalised before these rules run, so add a
capitalised variant for words that can start a subtitle.
Longer patterns win over shorter ones regardless of the order they are written
in, and replacement text is never rescanned, so rules cannot feed each other.
A pattern wrapped in slashes is a regular expression, with \1 \2 … referring to
its groups. Lines starting with # or ; are comments.

The result is visible in the job folder's tts_input.srt — check there if a rule
is not doing what you expect. --regen and --rebuild work from that file, so they
do not re-apply the dictionary; edit tts_input.srt directly, or rerun.

Use another list with --dictionary FILE (a relative name is looked for in the
current directory first, then next to the script) or turn it off with
--dictionary "". A missing file is not an error.

A second list can be merged in with --dictionary-extra FILE. The GUI writes one
per queue — the "Queue dictionary" button — into the first queued video's job
folder, for rules that belong to one batch rather than to Polish in general.


Number Normalization
--------------------
The TTS misreads long digit strings and sometimes turns them into non-words, so
after the dictionary every block containing a number is rewritten with the
numbers spelled out in Polish — in the correct grammatical case, which is why
this is a small neural model and not a lookup table:

    Wygrał 1247 złotych.     ->  Wygrał tysiąc dwieście czterdzieści siedem złotych.
    Mówimy o 25 psach.       ->  Mówimy o dwudziestu pięciu psach.
    Mam 3 koty.              ->  Mam 3 koty.        (single digits are read fine)

It runs in its own environment (transformers>=5, which must not be installed
next to ComfyUI's pinned versions). One-time install:

    normalizer/setup.sh                          # venv + dependencies
    HF_TOKEN=hf_... normalizer/setup.sh --model  # gated weights, accept terms first

Terms: https://huggingface.co/Folx/qwen3-0.6b-pl-text-normalization
Everything after that runs offline. See normalizer/README.md.

Turn it off with numbers = no under [normalize] in settings.ini, or with
--no-normalize-numbers for one run. If the environment or the model is missing
the run continues with the digits untouched — it is never fatal.


Output Files
------------
Everything belonging to one video is kept together in a job folder named after
the video file. Only the finished video is written to the top level.

output/
  clips/
    <video name>/      — job folder for one video:
      clip_NNNN.flac   — individual TTS clips, one per block (lossless)
      tts_input.srt    — processed subtitle text sent to TTS (inspect to verify)
      tts_assembled.flac — all clips assembled with correct timing (lossless)
      embedded_sub.srt — extracted embedded subtitle (only with --embedded-subs)
      video_fragment.mkv — extracted video segment (only when --end is used)
      queue.log        — errors and warnings from GUI runs (first video of a queue)
  {stem}_lektor.mkv    — final video with two audio tracks + all original subtitles:
                           Track 0 — Original (source audio, stream-copied as-is)
                           Track 1 — Polish lektor (TTS + original mixed,
                                     AAC 192k stereo)

Each run prints the job folder it uses. tts_input.srt and the clips are a
matched pair (block N is clip_NNNN.flac), which is why they live together —
previously each new job overwrote the last one's, so --rebuild on an older
film silently paired its clips with the newest film's text.


All Options
-----------

  srt                   Input SRT subtitle file (omit when using --embedded-subs)
  video                 Input video file (MKV, MP4, ...)

  --embedded-subs       Use a subtitle track embedded in the video instead of an
                        external SRT file. The script lists all subtitle tracks
                        found in the video and prompts you to choose one.
                        The video path can be given as the only positional argument:
                            python3 lektor_ai_comfy.py film.mkv --embedded-subs

  --start MM:SS         Start time of the range to process (default: 00:00)
  --end MM:SS           End time of the range to process (default: end of file)

  --voice NAME|PATH     Voice prompt: name of a subdirectory under the voice prompts
                        directory, or a direct path to a WAV file.
                        The directory must contain a .wav and matching .txt transcript.
                        (default: gudowski_short)

  --speed FACTOR        Per-clip playback speed via atempo (default: 1.2)
                        Values above 1.0 speed up speech; 1.0 = no change.

  --seed N              Base generation seed. 0 = random each run. (default: 0)
                        A fixed seed produces deterministic output.

  --temperature T       TTS sampling temperature (default: 0.9)
                        Lower = more deterministic; higher = more expressive but
                        risks hallucination (1.0 can cause the model to recite
                        the reference text instead of the subtitle).

  --max-tokens N        Max audio tokens the model may generate per clip (default: 4096)
                        Raise this if long clips are cut off mid-word.

  --dictionary FILE     Word/phrase replacements applied to the subtitles before
                        TTS (default: dictionary.txt). Pass "" to disable.

  --normalize-numbers   Spell digits out as Polish words before TTS (default: on)
  --no-normalize-numbers
                        Leave digits as digits. The TTS often misreads them.

  --orig-volume 0-1     Volume of the original audio in the Polish lektor mix (default: 0.60)
  --tts-volume 0-2      Volume of the TTS voiceover in the Polish lektor mix (default: 1.8)
                        Note: these affect Track 1 only. Track 0 is always the raw original.

  --strip-leading-ms MS Strip N milliseconds from the start of each clip to remove
                        leading breath or noise artifacts (default: 0, try 80-150)

  --trailing-fade-ms MS Fade out the last N milliseconds of each clip to suppress
                        tail pops or noise (default: 80, set 0 to disable)

  --output-dir DIR      Output directory (default: output/)

  --comfy-url URL       ComfyUI server URL (default: http://127.0.0.1:8189)

  --skip-tts AUDIO      Skip TTS generation entirely and use this pre-assembled
                        audio file instead. Goes straight to video merge.


Recovery and Fixing Bad Clips
------------------------------

Resume a crashed run from block N (reuses existing clips 1..N-1 and the job
folder's tts_input.srt — subtitles are not reprocessed):

    python3 lektor_ai_comfy.py input/film.srt input/film.mkv --resume-from 47

Regenerate specific clips by block number (same numbering as tts_input.srt):

    python3 lektor_ai_comfy.py input/film.mkv --regen 12 22 41

    Only replaces the selected clip_NNNN.flac files in the job folder — does not reassemble
    or rebuild the video. Run --rebuild afterwards to apply the changes.
    Useful for fixing hallucinations, mumbling, or cut-off clips.

Reassemble audio from all existing clips and rebuild the final video
(no TTS generation — use this after --regen, or to adjust volumes):

    python3 lektor_ai_comfy.py input/film.mkv --rebuild

    python3 lektor_ai_comfy.py input/film.mkv --rebuild --orig-volume 0.5 --tts-volume 2.0

    For --rebuild and --regen the SRT file is not needed — just pass the video path.
    When --end is given, rebuilds from the same video fragment as the original run
    (re-extracts it if needed). Without --end, uses the full video file.


Tips
----

- Check output/clips/<video name>/tts_input.srt to see the exact text sent to TTS.
  Block numbers in that file match the clip_NNNN.flac filenames.

- If a clip sounds wrong (hallucination, wrong language, cut off), note its block
  number from tts_input.srt and use --regen N to replace just that clip,
  then run --rebuild to reassemble the video.

- If ComfyUI crashes mid-run, note the last completed block number and use
  --resume-from N to continue without regenerating finished clips.

- The final MP4 has two audio tracks. Most players let you switch between them:
    Track 0 — Original soundtrack
    Track 1 — Polish lektor (AI voiceover mixed with quiet original)

- Track 0 (Original) is stream-copied without re-encoding, so source quality is preserved.
  The MKV container supports all common audio codecs (AAC, AC3, EAC3, DTS, TrueHD),
  so no workarounds are needed for the original audio track.

- Videos with audio starting at a non-zero PTS offset (common in MKV rips) are handled
  automatically — the original audio background in the lektor mix is normalised so it
  stays in sync with the video.

- The lektor track (Track 1) is stereo by design, while Track 0 keeps the original
  5.1 untouched. ffmpeg's AAC encoder does not signal a 5.1 channel layout — it ends up
  tagged "unknown" in every container, and players that cannot route the channels play
  the track as silence. Stereo is tagged correctly and plays everywhere. It also spends
  the 192k on two channels instead of spreading it thinly across six. The 5.1 original
  is downmixed with ffmpeg's standard coefficients and the mono TTS is centred.

- The final video is built in two passes: first the lektor audio mix is rendered to a
  temporary file, then everything is stream-copied into the MKV. This is deliberate.
  Doing both at once fails on sources with many subtitle tracks (some releases carry
  40+) — the sparse subtitle streams stall the muxer while the audio filter keeps
  producing, and ffmpeg aborts with "Failed to inject frame into filter network".
  The two-pass approach also makes the final mux about 10x faster.

- The final mux passes -max_interleave_delta 0 as an OUTPUT option (it must come after
  the maps and before the output path; placed before -i it configures the demuxer and
  silently does nothing). Without it, sparse subtitle tracks make the muxer abandon
  interleaving after its default 10-second window, and the lektor audio ends up written
  hundreds of megabytes away from the video it belongs with. The file measures perfectly
  fine with ffmpeg, but players read MKV sequentially and never reach that audio, so the
  Polish track plays as complete silence. Setting it to 0 buffers until every stream has
  a packet, costing a couple of GB of transient RAM for a few seconds.

- If a lektor track ever plays silent again, check interleaving before suspecting the
  audio. Compare byte positions at the same timestamp:
    ffprobe -read_intervals 00:20:00%+1 -select_streams v   -show_entries packet=pos file.mkv
    ffprobe -read_intervals 00:20:00%+1 -select_streams a:1 -show_entries packet=pos file.mkv
  They should be within a few MB of each other. An existing file can be repaired without
  regenerating anything:
    ffmpeg -i in.mkv -map 0 -c copy -max_interleave_delta 0 out.mkv

- ComfyUI crash recovery:
    sudo fuser -k /dev/nvidia*
    sudo nvidia-smi --gpu-reset