lektor_ai_comfy — Polish AI Dubbing Tool ========================================= FULL DOCUMENTATION: see docs/ docs/README.md start here — overview and quick start docs/requirements.md hardware, software, dependencies docs/installation.md installing ComfyUI + the TTS engine docs/usage.md every CLI flag, working modes, recipes docs/architecture.md how the pipeline works internally docs/troubleshooting.md diagnosed failure modes and their fixes Installer: ./install_comfyui.sh (see docs/installation.md) This file remains as a condensed quick reference. Converts SRT subtitles to AI voiceover (lektor style) using ComfyUI + Higgs Audio V3 TTS, then mixes the result with the original video audio into a two-track MKV. All subtitle tracks in the source video are carried over to the output. Requirements ------------ - ComfyUI running at http://127.0.0.1:8189 with TTS-Audio-Suite custom node - Model: higgs-audio-v3-tts-4b (bosonai/higgs-audio-v3) - ffmpeg and ffprobe available on PATH - Voice reference: a WAV file + matching .txt transcript in the voice directory (default: gudowski/gudowski.wav + gudowski/gudowski.txt) Basic Usage ----------- Full run with an external SRT file: python3 lektor_ai_comfy.py input/film.srt input/film.mkv Use subtitles embedded in the video (lists tracks, prompts for a choice): python3 lektor_ai_comfy.py input/film.mkv --embedded-subs Process only a time range (extracts a video fragment automatically): python3 lektor_ai_comfy.py input/film.srt input/film.mkv --start 01:20 --end 05:40 Subtitle Handling ----------------- Both external SRT files and embedded subtitle tracks go through the same cleaning: - HTML tags stripped: , , , , etc. - ASS override codes stripped: {\an8}, {\pos(x,y)}, {\c&H…&}, etc. - ALL-CAPS blocks lowercased and converted to sentence case - Sentence fragments starting with a lowercase letter are capitalised - Missing terminal punctuation added (… or . depending on context) - Dialogue lines (prefixed with -) are split into separate clips with proportional timing With --embedded-subs the extracted subtitles are saved to output/embedded_sub.srt so you can inspect them before or after the run. The final MKV preserves all subtitle tracks from the source video — text-based (SRT, ASS, SSA, WebVTT) and image-based (PGS Blu-ray, VOBSUB) alike — using stream copy with no conversion. Output Files ------------ output/ tts_input.srt — processed subtitle text sent to TTS (inspect to verify) embedded_sub.srt — extracted embedded subtitle (only when --embedded-subs) clips/clip_NNNN.flac — individual TTS clips, one per subtitle block (lossless) tts_assembled.flac — all clips assembled with correct timing (lossless) video_fragment.mkv — extracted video segment (only when --end is used) {stem}_lektor.mkv — final video with two audio tracks + all original subtitles: Track 0 — Original (source audio, stream-copied as-is) Track 1 — Polish lektor (TTS + original mixed, AAC 192k stereo) All Options ----------- srt Input SRT subtitle file (omit when using --embedded-subs) video Input video file (MKV, MP4, ...) --embedded-subs Use a subtitle track embedded in the video instead of an external SRT file. The script lists all subtitle tracks found in the video and prompts you to choose one. The video path can be given as the only positional argument: python3 lektor_ai_comfy.py film.mkv --embedded-subs --start MM:SS Start time of the range to process (default: 00:00) --end MM:SS End time of the range to process (default: end of file) --voice NAME|PATH Voice prompt: name of a subdirectory under the voice prompts directory, or a direct path to a WAV file. The directory must contain a .wav and matching .txt transcript. (default: gudowski_short) --speed FACTOR Per-clip playback speed via atempo (default: 1.2) Values above 1.0 speed up speech; 1.0 = no change. --seed N Base generation seed. 0 = random each run. (default: 0) A fixed seed produces deterministic output. --temperature T TTS sampling temperature (default: 0.9) Lower = more deterministic; higher = more expressive but risks hallucination (1.0 can cause the model to recite the reference text instead of the subtitle). --max-tokens N Max audio tokens the model may generate per clip (default: 4096) Raise this if long clips are cut off mid-word. --orig-volume 0-1 Volume of the original audio in the Polish lektor mix (default: 0.60) --tts-volume 0-2 Volume of the TTS voiceover in the Polish lektor mix (default: 1.8) Note: these affect Track 1 only. Track 0 is always the raw original. --strip-leading-ms MS Strip N milliseconds from the start of each clip to remove leading breath or noise artifacts (default: 0, try 80-150) --trailing-fade-ms MS Fade out the last N milliseconds of each clip to suppress tail pops or noise (default: 80, set 0 to disable) --output-dir DIR Output directory (default: output/) --comfy-url URL ComfyUI server URL (default: http://127.0.0.1:8189) --skip-tts AUDIO Skip TTS generation entirely and use this pre-assembled audio file instead. Goes straight to video merge. Recovery and Fixing Bad Clips ------------------------------ Resume a crashed run from block N (reuses existing clips 1..N-1): python3 lektor_ai_comfy.py input/film.srt input/film.mkv --resume-from 47 Regenerate specific clips by block number (same numbering as tts_input.srt): python3 lektor_ai_comfy.py input/film.mkv --regen 12 22 41 Only replaces the selected clips/clip_NNNN.flac files — does not reassemble or rebuild the video. Run --rebuild afterwards to apply the changes. Useful for fixing hallucinations, mumbling, or cut-off clips. Reassemble audio from all existing clips and rebuild the final video (no TTS generation — use this after --regen, or to adjust volumes): python3 lektor_ai_comfy.py input/film.mkv --rebuild python3 lektor_ai_comfy.py input/film.mkv --rebuild --orig-volume 0.5 --tts-volume 2.0 For --rebuild and --regen the SRT file is not needed — just pass the video path. When --end is given, rebuilds from the same video fragment as the original run (re-extracts it if needed). Without --end, uses the full video file. Tips ---- - Check output/tts_input.srt before or after a run to see the exact text sent to TTS. Block numbers in that file match the clip_NNNN.flac filenames. - If a clip sounds wrong (hallucination, wrong language, cut off), note its block number from tts_input.srt and use --regen N to replace just that clip, then run --rebuild to reassemble the video. - If ComfyUI crashes mid-run, note the last completed block number and use --resume-from N to continue without regenerating finished clips. - The final MP4 has two audio tracks. Most players let you switch between them: Track 0 — Original soundtrack Track 1 — Polish lektor (AI voiceover mixed with quiet original) - Track 0 (Original) is stream-copied without re-encoding, so source quality is preserved. The MKV container supports all common audio codecs (AAC, AC3, EAC3, DTS, TrueHD), so no workarounds are needed for the original audio track. - Videos with audio starting at a non-zero PTS offset (common in MKV rips) are handled automatically — the original audio background in the lektor mix is normalised so it stays in sync with the video. - The lektor track (Track 1) is stereo by design, while Track 0 keeps the original 5.1 untouched. ffmpeg's AAC encoder does not signal a 5.1 channel layout — it ends up tagged "unknown" in every container, and players that cannot route the channels play the track as silence. Stereo is tagged correctly and plays everywhere. It also spends the 192k on two channels instead of spreading it thinly across six. The 5.1 original is downmixed with ffmpeg's standard coefficients and the mono TTS is centred. - The final video is built in two passes: first the lektor audio mix is rendered to a temporary file, then everything is stream-copied into the MKV. This is deliberate. Doing both at once fails on sources with many subtitle tracks (some releases carry 40+) — the sparse subtitle streams stall the muxer while the audio filter keeps producing, and ffmpeg aborts with "Failed to inject frame into filter network". The two-pass approach also makes the final mux about 10x faster. - The final mux passes -max_interleave_delta 0 as an OUTPUT option (it must come after the maps and before the output path; placed before -i it configures the demuxer and silently does nothing). Without it, sparse subtitle tracks make the muxer abandon interleaving after its default 10-second window, and the lektor audio ends up written hundreds of megabytes away from the video it belongs with. The file measures perfectly fine with ffmpeg, but players read MKV sequentially and never reach that audio, so the Polish track plays as complete silence. Setting it to 0 buffers until every stream has a packet, costing a couple of GB of transient RAM for a few seconds. - If a lektor track ever plays silent again, check interleaving before suspecting the audio. Compare byte positions at the same timestamp: ffprobe -read_intervals 00:20:00%+1 -select_streams v -show_entries packet=pos file.mkv ffprobe -read_intervals 00:20:00%+1 -select_streams a:1 -show_entries packet=pos file.mkv They should be within a few MB of each other. An existing file can be repaired without regenerating anything: ffmpeg -i in.mkv -map 0 -c copy -max_interleave_delta 0 out.mkv - ComfyUI crash recovery: sudo fuser -k /dev/nvidia* sudo nvidia-smi --gpu-reset