lukas/lektor-ai
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

Initial commit: Polish AI dubbing pipeline (ComfyUI + Higgs Audio V3)

Browse Source 
Converts SRT subtitles into a Polish lektor-style voiceover using a local
ComfyUI instance running Higgs Audio V3, then muxes the result back into the
source video as a second audio track. Video and original audio are always
stream-copied; only the new voiceover track is encoded.

Contents:
  lektor_ai_comfy.py    main pipeline (standard library only, needs ffmpeg)
  lektor_ai_comfy_v2.py batched-context experiment (timing unresolved)
  lektor_ai.py          earlier version using the hosted Boson AI API
  install_comfyui.sh    reproduces the ComfyUI environment, versions pinned
  docs/                 requirements, installation, usage, architecture,
                        troubleshooting
  gudowski/             voice references (.wav) with their transcripts (.txt)

Source media and generated video are gitignored.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
This commit is contained in:
claude
2026-08-09 14:29:53 +02:00
co-authored by Claude Opus 5
commit fc4b761f52
23 changed files with 4452 additions and 0 deletions
Download Patch File Download Diff File Expand all files Collapse all files
docs/usage.md
+266
View File
@@ -0,0 +1,266 @@
# Usage
ComfyUI must be running before any command that generates speech
(`~/comfy-tts-app/start_comfyui.sh`). `--rebuild` is the exception — it only
touches ffmpeg and works with the backend stopped.
## Invocation
```
python3 lektor_ai_comfy.py [SRT] VIDEO [options]
```
Both positionals are optional in the argparse sense, because several modes do
not need an SRT file. The rule:
```bash
# External subtitle file — pass both, SRT first
python3 lektor_ai_comfy.py input/film.pl.srt input/film.mkv
# Subtitles inside the video — pass the video alone
python3 lektor_ai_comfy.py input/film.mkv --embedded-subs
# --rebuild and --regen also need only the video
python3 lektor_ai_comfy.py input/film.mkv --rebuild
```
When a single positional is given in a mode that needs no SRT, the script moves
it into the video slot automatically. Always quote paths containing spaces or
parentheses:
```bash
python3 lektor_ai_comfy.py "input/Star City (2026)/S01_E01 The Eyes.mkv" --embedded-subs
```
## Working modes
### Full generation (default)
Parses subtitles, generates one TTS clip per block, assembles, muxes.
```bash
python3 lektor_ai_comfy.py input/film.mkv --embedded-subs
```
Roughly real-time-ish on a 4070 Ti SUPER: a 60-minute episode of ~700 subtitle
blocks takes on the order of an hour. Each clip is saved as it completes, so a
crash never loses more than the clip in flight.
### `--embedded-subs`
Lists the video's subtitle tracks and prompts for one:
```
Available subtitle tracks in S01_E01 The Eyes.mkv:
[1] Stream #2: subrip [eng] — Forced
[2] Stream #3: subrip [eng]
[3] Stream #4: subrip [eng] — SDH
...
[30] Stream #31: subrip [pol]
Select track number: 30
```
The chosen track is extracted to `output/embedded_sub.srt` and then processed
exactly like an external file. Image-based tracks are listed but marked
unusable — they hold bitmaps, not text.
### `--rebuild`
Reassembles audio from existing clips and rebuilds the video, with no TTS at
all. This is the fast iteration loop — seconds to minutes rather than an hour.
```bash
python3 lektor_ai_comfy.py input/film.mkv --rebuild
python3 lektor_ai_comfy.py input/film.mkv --rebuild --orig-volume 0.5 --tts-volume 2.0
```
Use it after `--regen`, or to adjust the mix. It requires `output/tts_input.srt`
and a complete set of `output/clips/clip_NNNN.flac`, and reports which are
missing if any are.
If the original run used `--end`, pass it again so the same video fragment is
used.
### `--regen`
Regenerates specific clips by block number, leaving everything else alone.
```bash
python3 lektor_ai_comfy.py input/film.mkv --regen 12 22 41
python3 lektor_ai_comfy.py input/film.mkv --rebuild # then apply
```
Block numbers match `output/tts_input.srt` and the `clip_NNNN.flac` filenames.
Only the selected FLAC files are replaced — `--rebuild` afterwards is what
actually updates the video.
This is the intended fix for a bad clip. There is no automatic retry; when the
model hallucinates, mumbles, or truncates, note the number and regenerate it. A
different seed usually resolves it, and `--seed 0` (the default) randomises
every run.
### `--resume-from`
Continues a crashed run, loading clips `1..N-1` from disk and generating from
`N` onward.
```bash
python3 lektor_ai_comfy.py input/film.mkv --resume-from 47
```
The console prints `[47/693]` style progress, so the number to resume from is
the last one that completed.
### `--skip-tts`
Uses a pre-assembled audio file and goes straight to the mux.
```bash
python3 lektor_ai_comfy.py input/film.mkv --skip-tts output/tts_assembled.flac
```
## Options
### Range
| Flag | Default | Meaning |
|---|---|---|
| `--start MM:SS` | `00:00` | Start of the range to process |
| `--end MM:SS` | *(end)* | End of the range |
With `--end`, a matching fragment is extracted to `output/video_fragment.mkv`
and all TTS timestamps become relative to `--start`. Pass the same values to any
later `--rebuild`, or the audio will not line up.
`--end 01:00` is the quickest way to sanity-check a new voice or speed setting.
### Voice and model
| Flag | Default | Meaning |
|---|---|---|
| `--voice NAME\|PATH` | `gudowski_short` | Name in `gudowski/`, or a path to a WAV |
| `--seed N` | `0` | Base seed; `0` randomises each run |
| `--temperature T` | `0.9` | Sampling temperature |
| `--max-tokens N` | `4096` | Token ceiling per clip |
`--temperature` is worth understanding. At `1.0` the model sometimes recites the
reference transcript instead of the subtitle. `0.9` largely prevents that;
lower is more deterministic but flatter. A per-clip seed of `base_seed + index`
means a fixed `--seed` reproduces a run exactly.
Raise `--max-tokens` only if long clips are cut off mid-word.
### Timing and cleanup
| Flag | Default | Meaning |
|---|---|---|
| `--speed FACTOR` | `1.2` | Per-clip speed-up via `atempo` |
| `--strip-leading-ms MS` | `0` | Trim the start of each clip |
| `--trailing-fade-ms MS` | `80` | Fade the end of each clip |
`--speed` compresses speech so a clip is more likely to fit its subtitle window.
It is applied **per clip, before assembly** — never to the assembled track,
which would shift every timestamp by speeding up the silences too.
`--strip-leading-ms` removes an inhale artifact at clip starts; try 80–150 if
you hear one. `--trailing-fade-ms` suppresses tail pops; `0` disables it.
### Mixing
| Flag | Default | Meaning |
|---|---|---|
| `--orig-volume 0-1` | `0.60` | Original audio level inside the lektor mix |
| `--tts-volume 0-2` | `1.8` | Voiceover level inside the lektor mix |
These affect **Track 1 only**. Track 0 is always the untouched original, so
nothing is lost by mixing aggressively. Changing them needs only `--rebuild`.
Typical starting point for dialogue-heavy material: `--orig-volume 0.5
--tts-volume 2.0`. For quiet sources, `--orig-volume 1.0 --tts-volume 1.4`.
### Paths
| Flag | Default | Meaning |
|---|---|---|
| `--output-dir DIR` | `output` | Where everything is written |
| `--comfy-url URL` | `http://127.0.0.1:8189` | Backend address |
## Output
```
output/
tts_input.srt processed text actually sent to TTS — inspect this
embedded_sub.srt extracted embedded subtitle (--embedded-subs only)
clips/clip_NNNN.flac one lossless clip per block, kept for --regen
tts_assembled.flac all clips at their final positions, lossless
video_fragment.mkv extracted segment (--end only)
{stem}_lektor.mkv the finished video
```
The finished file contains:
| Track | Content |
|---|---|
| Video | Stream-copied, untouched |
| Audio 0 | **Original** — stream-copied, full 5.1 if the source had it |
| Audio 1 | **Polish** — lektor mix, AAC 192k stereo, `language=pol` |
| Subtitles | Every track from the source, stream-copied |
The lektor track is stereo deliberately; see
[troubleshooting.md](troubleshooting.md#the-lektor-track-is-silent-in-the-player).
FLAC is used for every intermediate so repeated processing stays lossless. Only
the final voiceover track is ever encoded.
## Recipes
**Check a voice before committing to a full run**
```bash
python3 lektor_ai_comfy.py input/film.mkv --embedded-subs --end 02:00 --voice gudowski
```
**Tune the mix without regenerating speech**
```bash
python3 lektor_ai_comfy.py input/film.mkv --rebuild --orig-volume 0.45 --tts-volume 2.1
```
**Fix a handful of bad clips**
```bash
# read output/tts_input.srt, note the offending block numbers
python3 lektor_ai_comfy.py input/film.mkv --regen 88 91 104
python3 lektor_ai_comfy.py input/film.mkv --rebuild
```
**Reproduce a run exactly**
```bash
python3 lektor_ai_comfy.py input/film.mkv --embedded-subs --seed 123456
```
**Speech running long and overlapping the next line**
```bash
python3 lektor_ai_comfy.py input/film.mkv --embedded-subs --speed 1.35
```
## Reading the console
```
[1/4] Parsing subtitles 00:00–end…
693 entries after dialogue splitting.
[2/4] Generating TTS (693 clips, speed 1.2×)…
[12/693] 00:01:34,120 'Nie wiem, o czym mówisz.'
Waiting for a3f8b201… done.
→ placed at 00:01:34,120, duration 1.84s
```
`→ placed at` reveals timing drift. A clip that overruns its window pushes the
next one forward, shown as `[+1.20s behind SRT]`. Small values are normal and
self-correct at the next natural gap in the subtitles; values that keep growing
mean `--speed` is too low for the material.
Assembly then shows a progress bar per batch (clips are assembled 500 at a time
to stay under the open-file limit), followed by the two-pass mux.