claudeandClaude Opus 5
fc4b761f52
Converts SRT subtitles into a Polish lektor-style voiceover using a local
ComfyUI instance running Higgs Audio V3, then muxes the result back into the
source video as a second audio track. Video and original audio are always
stream-copied; only the new voiceover track is encoded.
Contents:
lektor_ai_comfy.py main pipeline (standard library only, needs ffmpeg)
lektor_ai_comfy_v2.py batched-context experiment (timing unresolved)
lektor_ai.py earlier version using the hosted Boson AI API
install_comfyui.sh reproduces the ComfyUI environment, versions pinned
docs/ requirements, installation, usage, architecture,
troubleshooting
gudowski/ voice references (.wav) with their transcripts (.txt)
Source media and generated video are gitignored.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
217 lines
10 KiB
Plaintext
217 lines
10 KiB
Plaintext
lektor_ai_comfy — Polish AI Dubbing Tool
|
|
=========================================
|
|
|
|
FULL DOCUMENTATION: see docs/
|
|
docs/README.md start here — overview and quick start
|
|
docs/requirements.md hardware, software, dependencies
|
|
docs/installation.md installing ComfyUI + the TTS engine
|
|
docs/usage.md every CLI flag, working modes, recipes
|
|
docs/architecture.md how the pipeline works internally
|
|
docs/troubleshooting.md diagnosed failure modes and their fixes
|
|
|
|
Installer: ./install_comfyui.sh (see docs/installation.md)
|
|
|
|
This file remains as a condensed quick reference.
|
|
|
|
|
|
Converts SRT subtitles to AI voiceover (lektor style) using ComfyUI + Higgs Audio V3 TTS,
|
|
then mixes the result with the original video audio into a two-track MKV.
|
|
All subtitle tracks in the source video are carried over to the output.
|
|
|
|
Requirements
|
|
------------
|
|
- ComfyUI running at http://127.0.0.1:8189 with TTS-Audio-Suite custom node
|
|
- Model: higgs-audio-v3-tts-4b (bosonai/higgs-audio-v3)
|
|
- ffmpeg and ffprobe available on PATH
|
|
- Voice reference: a WAV file + matching .txt transcript in the voice directory
|
|
(default: gudowski/gudowski.wav + gudowski/gudowski.txt)
|
|
|
|
|
|
Basic Usage
|
|
-----------
|
|
|
|
Full run with an external SRT file:
|
|
|
|
python3 lektor_ai_comfy.py input/film.srt input/film.mkv
|
|
|
|
Use subtitles embedded in the video (lists tracks, prompts for a choice):
|
|
|
|
python3 lektor_ai_comfy.py input/film.mkv --embedded-subs
|
|
|
|
Process only a time range (extracts a video fragment automatically):
|
|
|
|
python3 lektor_ai_comfy.py input/film.srt input/film.mkv --start 01:20 --end 05:40
|
|
|
|
|
|
Subtitle Handling
|
|
-----------------
|
|
Both external SRT files and embedded subtitle tracks go through the same cleaning:
|
|
|
|
- HTML tags stripped: <i>, </i>, <b>, <font color="…">, etc.
|
|
- ASS override codes stripped: {\an8}, {\pos(x,y)}, {\c&H…&}, etc.
|
|
- ALL-CAPS blocks lowercased and converted to sentence case
|
|
- Sentence fragments starting with a lowercase letter are capitalised
|
|
- Missing terminal punctuation added (… or . depending on context)
|
|
- Dialogue lines (prefixed with -) are split into separate clips with proportional timing
|
|
|
|
With --embedded-subs the extracted subtitles are saved to output/embedded_sub.srt
|
|
so you can inspect them before or after the run.
|
|
|
|
The final MKV preserves all subtitle tracks from the source video — text-based
|
|
(SRT, ASS, SSA, WebVTT) and image-based (PGS Blu-ray, VOBSUB) alike — using
|
|
stream copy with no conversion.
|
|
|
|
|
|
Output Files
|
|
------------
|
|
output/
|
|
tts_input.srt — processed subtitle text sent to TTS (inspect to verify)
|
|
embedded_sub.srt — extracted embedded subtitle (only when --embedded-subs)
|
|
clips/clip_NNNN.flac — individual TTS clips, one per subtitle block (lossless)
|
|
tts_assembled.flac — all clips assembled with correct timing (lossless)
|
|
video_fragment.mkv — extracted video segment (only when --end is used)
|
|
{stem}_lektor.mkv — final video with two audio tracks + all original subtitles:
|
|
Track 0 — Original (source audio, stream-copied as-is)
|
|
Track 1 — Polish lektor (TTS + original mixed,
|
|
AAC 192k stereo)
|
|
|
|
|
|
All Options
|
|
-----------
|
|
|
|
srt Input SRT subtitle file (omit when using --embedded-subs)
|
|
video Input video file (MKV, MP4, ...)
|
|
|
|
--embedded-subs Use a subtitle track embedded in the video instead of an
|
|
external SRT file. The script lists all subtitle tracks
|
|
found in the video and prompts you to choose one.
|
|
The video path can be given as the only positional argument:
|
|
python3 lektor_ai_comfy.py film.mkv --embedded-subs
|
|
|
|
--start MM:SS Start time of the range to process (default: 00:00)
|
|
--end MM:SS End time of the range to process (default: end of file)
|
|
|
|
--voice NAME|PATH Voice prompt: name of a subdirectory under the voice prompts
|
|
directory, or a direct path to a WAV file.
|
|
The directory must contain a .wav and matching .txt transcript.
|
|
(default: gudowski_short)
|
|
|
|
--speed FACTOR Per-clip playback speed via atempo (default: 1.2)
|
|
Values above 1.0 speed up speech; 1.0 = no change.
|
|
|
|
--seed N Base generation seed. 0 = random each run. (default: 0)
|
|
A fixed seed produces deterministic output.
|
|
|
|
--temperature T TTS sampling temperature (default: 0.9)
|
|
Lower = more deterministic; higher = more expressive but
|
|
risks hallucination (1.0 can cause the model to recite
|
|
the reference text instead of the subtitle).
|
|
|
|
--max-tokens N Max audio tokens the model may generate per clip (default: 4096)
|
|
Raise this if long clips are cut off mid-word.
|
|
|
|
--orig-volume 0-1 Volume of the original audio in the Polish lektor mix (default: 0.60)
|
|
--tts-volume 0-2 Volume of the TTS voiceover in the Polish lektor mix (default: 1.8)
|
|
Note: these affect Track 1 only. Track 0 is always the raw original.
|
|
|
|
--strip-leading-ms MS Strip N milliseconds from the start of each clip to remove
|
|
leading breath or noise artifacts (default: 0, try 80-150)
|
|
|
|
--trailing-fade-ms MS Fade out the last N milliseconds of each clip to suppress
|
|
tail pops or noise (default: 80, set 0 to disable)
|
|
|
|
--output-dir DIR Output directory (default: output/)
|
|
|
|
--comfy-url URL ComfyUI server URL (default: http://127.0.0.1:8189)
|
|
|
|
--skip-tts AUDIO Skip TTS generation entirely and use this pre-assembled
|
|
audio file instead. Goes straight to video merge.
|
|
|
|
|
|
Recovery and Fixing Bad Clips
|
|
------------------------------
|
|
|
|
Resume a crashed run from block N (reuses existing clips 1..N-1):
|
|
|
|
python3 lektor_ai_comfy.py input/film.srt input/film.mkv --resume-from 47
|
|
|
|
Regenerate specific clips by block number (same numbering as tts_input.srt):
|
|
|
|
python3 lektor_ai_comfy.py input/film.mkv --regen 12 22 41
|
|
|
|
Only replaces the selected clips/clip_NNNN.flac files — does not reassemble
|
|
or rebuild the video. Run --rebuild afterwards to apply the changes.
|
|
Useful for fixing hallucinations, mumbling, or cut-off clips.
|
|
|
|
Reassemble audio from all existing clips and rebuild the final video
|
|
(no TTS generation — use this after --regen, or to adjust volumes):
|
|
|
|
python3 lektor_ai_comfy.py input/film.mkv --rebuild
|
|
|
|
python3 lektor_ai_comfy.py input/film.mkv --rebuild --orig-volume 0.5 --tts-volume 2.0
|
|
|
|
For --rebuild and --regen the SRT file is not needed — just pass the video path.
|
|
When --end is given, rebuilds from the same video fragment as the original run
|
|
(re-extracts it if needed). Without --end, uses the full video file.
|
|
|
|
|
|
Tips
|
|
----
|
|
|
|
- Check output/tts_input.srt before or after a run to see the exact text sent to TTS.
|
|
Block numbers in that file match the clip_NNNN.flac filenames.
|
|
|
|
- If a clip sounds wrong (hallucination, wrong language, cut off), note its block
|
|
number from tts_input.srt and use --regen N to replace just that clip,
|
|
then run --rebuild to reassemble the video.
|
|
|
|
- If ComfyUI crashes mid-run, note the last completed block number and use
|
|
--resume-from N to continue without regenerating finished clips.
|
|
|
|
- The final MP4 has two audio tracks. Most players let you switch between them:
|
|
Track 0 — Original soundtrack
|
|
Track 1 — Polish lektor (AI voiceover mixed with quiet original)
|
|
|
|
- Track 0 (Original) is stream-copied without re-encoding, so source quality is preserved.
|
|
The MKV container supports all common audio codecs (AAC, AC3, EAC3, DTS, TrueHD),
|
|
so no workarounds are needed for the original audio track.
|
|
|
|
- Videos with audio starting at a non-zero PTS offset (common in MKV rips) are handled
|
|
automatically — the original audio background in the lektor mix is normalised so it
|
|
stays in sync with the video.
|
|
|
|
- The lektor track (Track 1) is stereo by design, while Track 0 keeps the original
|
|
5.1 untouched. ffmpeg's AAC encoder does not signal a 5.1 channel layout — it ends up
|
|
tagged "unknown" in every container, and players that cannot route the channels play
|
|
the track as silence. Stereo is tagged correctly and plays everywhere. It also spends
|
|
the 192k on two channels instead of spreading it thinly across six. The 5.1 original
|
|
is downmixed with ffmpeg's standard coefficients and the mono TTS is centred.
|
|
|
|
- The final video is built in two passes: first the lektor audio mix is rendered to a
|
|
temporary file, then everything is stream-copied into the MKV. This is deliberate.
|
|
Doing both at once fails on sources with many subtitle tracks (some releases carry
|
|
40+) — the sparse subtitle streams stall the muxer while the audio filter keeps
|
|
producing, and ffmpeg aborts with "Failed to inject frame into filter network".
|
|
The two-pass approach also makes the final mux about 10x faster.
|
|
|
|
- The final mux passes -max_interleave_delta 0 as an OUTPUT option (it must come after
|
|
the maps and before the output path; placed before -i it configures the demuxer and
|
|
silently does nothing). Without it, sparse subtitle tracks make the muxer abandon
|
|
interleaving after its default 10-second window, and the lektor audio ends up written
|
|
hundreds of megabytes away from the video it belongs with. The file measures perfectly
|
|
fine with ffmpeg, but players read MKV sequentially and never reach that audio, so the
|
|
Polish track plays as complete silence. Setting it to 0 buffers until every stream has
|
|
a packet, costing a couple of GB of transient RAM for a few seconds.
|
|
|
|
- If a lektor track ever plays silent again, check interleaving before suspecting the
|
|
audio. Compare byte positions at the same timestamp:
|
|
ffprobe -read_intervals 00:20:00%+1 -select_streams v -show_entries packet=pos file.mkv
|
|
ffprobe -read_intervals 00:20:00%+1 -select_streams a:1 -show_entries packet=pos file.mkv
|
|
They should be within a few MB of each other. An existing file can be repaired without
|
|
regenerating anything:
|
|
ffmpeg -i in.mkv -map 0 -c copy -max_interleave_delta 0 out.mkv
|
|
|
|
- ComfyUI crash recovery:
|
|
sudo fuser -k /dev/nvidia*
|
|
sudo nvidia-smi --gpu-reset
|