Settings file, per-job clip folders, faster polling, voices/ rename
Poll interval wait_for_completion() polled every 3 s, which quantised every clip to a 3 s boundary — across 2075 existing clips, 91% of the gaps landed exactly on a multiple of 3 s. It now starts at 0.25 s and backs off to 2 s after 10 s, so short clips are picked up immediately without hammering the server on long ones. Measured over 34 clips: 4.75 s actual vs 6.35 s under the old poll, a 25% cut in TTS wait (~18 min on a 693-clip episode). Runs now print real generation time, e.g. "done (4.2s)". A WebSocket would remove polling entirely but needs a third-party client, which would cost this script its dependency-free property. settings.ini voice, speed, temperature, max_tokens, strip_leading_ms, trailing_fade_ms, orig_volume, tts_volume, output_dir and comfy_url moved to an INI file next to the script. Precedence is CLI flag > settings.ini > built-in default, wired by feeding loaded values in as argparse defaults. --settings FILE selects an alternative profile. INI rather than TOML (tomllib is 3.11+, this runs on system Python 3.10) or JSON (no comments). A missing, partial, malformed or corrupt file degrades to built-in defaults with a warning instead of killing a long run. Run-specific flags are deliberately excluded: --start, --end, --seed, --regen, --rebuild, --resume-from, --embedded-subs, --skip-tts. Per-job folders All state for one video now lives in <output_dir>/clips/<video name>/ — clips, tts_input.srt, tts_assembled.flac, embedded_sub.srt and video_fragment.mkv. Only <stem>_lektor.mkv stays at the top level. tts_input.srt moved with the clips because they are a matched pair: block N in the SRT is clip_NNNN.flac. Previously both sat at the top level and each new job overwrote the last one's, so rebuilding an older film silently paired its clips with the newest film's text. Any past job can now be rebuilt. A run against the old flat layout gets an explanatory message. gudowski/ -> voices/ VOICE_PROMPTS_DIR is now resolved relative to the script rather than hardcoded, so moving the project no longer breaks it. Voice filenames are unchanged, so --voice gudowski_short still works. Docs updated throughout for the new layout, settings file and poll behaviour. Known issue, not addressed here: with a non-zero --start the voiceover plays early. extract_video_fragment() stream-copies, so it can only cut on a keyframe and seeks backwards to it, while TTS timestamps are relative to the requested start. On S01_E08 at --start 07:40 the nearest earlier keyframe is 458.375 s, putting the voiceover 1.625 s ahead of the picture. Runs with --start 00:00 are unaffected, since 0 is always a keyframe. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
This commit is contained in:
Claude Opus 5
+32
-6
@@ -8,13 +8,13 @@ the straightforward version was tried first and broke.
|
||||
SRT / embedded track
|
||||
│ parse, clean, split dialogue
|
||||
▼
|
||||
subtitle blocks ──────────────► output/tts_input.srt
|
||||
subtitle blocks ──────────────► <job>/tts_input.srt
|
||||
│ one HTTP job per block
|
||||
▼
|
||||
ComfyUI + Higgs Audio V3
|
||||
│ per-clip atempo + fade
|
||||
▼
|
||||
clips/clip_NNNN.flac
|
||||
<job>/clip_NNNN.flac
|
||||
│ adelay + amix, batched
|
||||
▼
|
||||
tts_assembled.flac
|
||||
@@ -49,15 +49,41 @@ in step 3 would silently miss it.
|
||||
`split_dialogue_blocks()` then divides a two-speaker block into one clip per
|
||||
line, apportioning the block's duration by character count.
|
||||
|
||||
The result is written to `output/tts_input.srt`. That file is the source of
|
||||
truth for block numbering: block *N* there is `clips/clip_NNNN.flac` and the
|
||||
The result is written to `tts_input.srt` in that video's job folder
|
||||
(`output/clips/<video name>/`). It is the source of truth for block
|
||||
numbering: block *N* there is `clip_NNNN.flac` beside it and the
|
||||
argument to `--regen N`.
|
||||
|
||||
## 2. Speech generation
|
||||
|
||||
For each block the script POSTs a workflow graph to `/prompt`, polls
|
||||
`/history/{id}` every 3 s, and reads the resulting file directly from ComfyUI's
|
||||
output directory.
|
||||
`/history/{id}` until it completes, and reads the resulting file directly from
|
||||
ComfyUI's output directory.
|
||||
|
||||
**Polling cadence is a performance decision, not a detail.** A clip that
|
||||
finishes just after a poll waits for the next one, so the interval — not the
|
||||
GPU — sets the floor on throughput. It was originally a flat 3 s, which
|
||||
quantised every clip to a 3 s boundary: measured across 2075 real clips, 91% of
|
||||
the gaps landed exactly on a multiple of 3 s.
|
||||
|
||||
`wait_for_completion()` now starts at 0.25 s and backs off to 2 s once a job
|
||||
passes 10 s, so the common case is picked up almost immediately while a stuck
|
||||
generation does not sit at 4 requests/second indefinitely. Measured over 34
|
||||
clips:
|
||||
|
||||
| | mean | median | 34 clips |
|
||||
|---|---|---|---|
|
||||
| Actual generation | 4.75 s | 4.60 s | 162 s |
|
||||
| Old flat 3 s poll | 6.35 s | 6.00 s | 216 s |
|
||||
|
||||
That is 1.60 s wasted per clip, a **25% reduction in TTS wait** — about 18
|
||||
minutes on a 693-clip episode. The run also now prints real generation time
|
||||
(`done (4.2s)`), which is what makes this measurable at all.
|
||||
|
||||
ComfyUI also pushes completion events over a WebSocket, which would remove
|
||||
polling entirely. It is not used because Python has no standard-library
|
||||
WebSocket client, and that would cost this script its only real virtue as a
|
||||
dependency-free file. Polling at 0.25 s recovers most of the benefit.
|
||||
|
||||
The graph has five nodes:
|
||||
|
||||
|
||||
+21
-11
@@ -144,20 +144,30 @@ is not needed; the script drives everything over HTTP.
|
||||
|
||||
## Configuring the script
|
||||
|
||||
`lektor_ai_comfy.py` has four constants at the top. If you installed anywhere
|
||||
other than the defaults, edit them:
|
||||
Most settings live in **`settings.ini`** next to the script — see
|
||||
[usage.md](usage.md#settings-file). The backend address in particular:
|
||||
|
||||
```python
|
||||
COMFY_URL = "http://127.0.0.1:8189"
|
||||
COMFY_OUTPUT_DIR = Path("/home/lukas/comfy-tts-app/ComfyUI/output")
|
||||
COMFY_INPUT_DIR = Path("/home/lukas/comfy-tts-app/ComfyUI/input")
|
||||
VOICE_PROMPTS_DIR = Path("/home/lukas/Programy/lektor_ai/gudowski")
|
||||
```ini
|
||||
[comfy]
|
||||
comfy_url = http://127.0.0.1:8189
|
||||
```
|
||||
|
||||
`COMFY_URL` can also be overridden per-run with `--comfy-url`. The two directory
|
||||
constants cannot — the script reads generated audio directly from ComfyUI's
|
||||
output directory and copies voice references into its input directory, so both
|
||||
must be real paths on the same filesystem.
|
||||
Two paths remain constants in `lektor_ai_comfy.py`, because the script touches
|
||||
ComfyUI's own directories directly rather than over HTTP:
|
||||
|
||||
```python
|
||||
COMFY_OUTPUT_DIR = Path("/home/lukas/comfy-tts-app/ComfyUI/output")
|
||||
COMFY_INPUT_DIR = Path("/home/lukas/comfy-tts-app/ComfyUI/input")
|
||||
```
|
||||
|
||||
Generated audio is read straight out of the output directory and voice
|
||||
references are copied into the input directory, so both must be real paths
|
||||
reachable from wherever the script runs. **If you installed ComfyUI anywhere
|
||||
other than `~/comfy-tts-app`, edit these two lines.** They are also what would
|
||||
have to change first to drive a ComfyUI instance on another machine.
|
||||
|
||||
A third path, `VOICE_PROMPTS_DIR`, resolves to `voices/` beside the script and
|
||||
needs no editing as long as the project stays intact.
|
||||
|
||||
The script itself needs no installation. Ensure ffmpeg is present:
|
||||
|
||||
|
||||
@@ -110,7 +110,7 @@ text and cannot drive TTS without OCR; the `--embedded-subs` picker marks them
|
||||
as unusable. They are still copied to the output untouched.
|
||||
|
||||
**Voice reference** — a `.wav` plus a `.txt` holding its exact transcript, both
|
||||
in `gudowski/`. Three are bundled:
|
||||
in `voices/`. Three are bundled:
|
||||
|
||||
| Voice | Length | Format |
|
||||
|---|---|---|
|
||||
|
||||
@@ -212,7 +212,7 @@ instead — the image tracks are still copied to the output.
|
||||
### Wrong language selected
|
||||
|
||||
The picker lists language tags but they are only as good as the source's
|
||||
metadata. `output/embedded_sub.srt` holds exactly what was extracted; check it
|
||||
metadata. `embedded_sub.srt` in the job folder holds exactly what was extracted; check it
|
||||
before committing to a full run.
|
||||
|
||||
## Backend problems
|
||||
|
||||
+107
-15
@@ -60,7 +60,7 @@ Lists the video's subtitle tracks and prompts for one:
|
||||
Select track number: 30
|
||||
```
|
||||
|
||||
The chosen track is extracted to `output/embedded_sub.srt` and then processed
|
||||
The chosen track is extracted to `embedded_sub.srt` in the job folder and processed
|
||||
exactly like an external file. Image-based tracks are listed but marked
|
||||
unusable — they hold bitmaps, not text.
|
||||
|
||||
@@ -74,9 +74,9 @@ python3 lektor_ai_comfy.py input/film.mkv --rebuild
|
||||
python3 lektor_ai_comfy.py input/film.mkv --rebuild --orig-volume 0.5 --tts-volume 2.0
|
||||
```
|
||||
|
||||
Use it after `--regen`, or to adjust the mix. It requires `output/tts_input.srt`
|
||||
and a complete set of `output/clips/clip_NNNN.flac`, and reports which are
|
||||
missing if any are.
|
||||
Use it after `--regen`, or to adjust the mix. It reads that video's job folder
|
||||
(`output/clips/<video name>/`) and needs its `tts_input.srt` plus a complete set
|
||||
of `clip_NNNN.flac`, reporting which are missing if any are.
|
||||
|
||||
If the original run used `--end`, pass it again so the same video fragment is
|
||||
used.
|
||||
@@ -90,7 +90,7 @@ python3 lektor_ai_comfy.py input/film.mkv --regen 12 22 41
|
||||
python3 lektor_ai_comfy.py input/film.mkv --rebuild # then apply
|
||||
```
|
||||
|
||||
Block numbers match `output/tts_input.srt` and the `clip_NNNN.flac` filenames.
|
||||
Block numbers match that job's `tts_input.srt` and its `clip_NNNN.flac` files.
|
||||
Only the selected FLAC files are replaced — `--rebuild` afterwards is what
|
||||
actually updates the video.
|
||||
|
||||
@@ -116,11 +116,79 @@ the last one that completed.
|
||||
Uses a pre-assembled audio file and goes straight to the mux.
|
||||
|
||||
```bash
|
||||
python3 lektor_ai_comfy.py input/film.mkv --skip-tts output/tts_assembled.flac
|
||||
python3 lektor_ai_comfy.py input/film.mkv --skip-tts "output/clips/film/tts_assembled.flac"
|
||||
```
|
||||
|
||||
## Settings file
|
||||
|
||||
Defaults live in **`settings.ini`** next to the script, so the values you always
|
||||
want no longer need to be typed on every run. Precedence:
|
||||
|
||||
```
|
||||
command-line flag > settings.ini > built-in default
|
||||
```
|
||||
|
||||
```ini
|
||||
[tts]
|
||||
voice = gudowski_short
|
||||
speed = 1.2
|
||||
temperature = 0.9
|
||||
max_tokens = 4096
|
||||
strip_leading_ms = 0
|
||||
trailing_fade_ms = 80
|
||||
|
||||
[mix]
|
||||
orig_volume = 0.60
|
||||
tts_volume = 1.8
|
||||
|
||||
[paths]
|
||||
output_dir = output
|
||||
|
||||
[comfy]
|
||||
comfy_url = http://127.0.0.1:8189
|
||||
```
|
||||
|
||||
Every one of these has a matching flag that overrides it for a single run:
|
||||
|
||||
```bash
|
||||
# settings.ini says tts_volume = 1.8; this run uses 2.5
|
||||
python3 lektor_ai_comfy.py film.mkv --embedded-subs --tts-volume 2.5
|
||||
```
|
||||
|
||||
Each run prints which file it used, so there is never any doubt:
|
||||
|
||||
```
|
||||
Settings: /home/lukas/Programy/lektor_ai/settings.ini
|
||||
```
|
||||
|
||||
**Alternative profiles** — keep per-show settings side by side and select one:
|
||||
|
||||
```bash
|
||||
python3 lektor_ai_comfy.py film.mkv --embedded-subs --settings profiles/docu.ini
|
||||
```
|
||||
|
||||
**Behaviour when things are missing or wrong**, none of which is fatal:
|
||||
|
||||
| Situation | Result |
|
||||
|---|---|
|
||||
| No `settings.ini` | Built-in defaults; the run says so |
|
||||
| Key absent from the file | That key falls back to its built-in default |
|
||||
| Value unparseable (`speed = fast`) | Warning, that key falls back, run continues |
|
||||
| File is not valid INI | Warning, all built-in defaults, run continues |
|
||||
|
||||
Comments use `#` or `;`, inline comments included. Deleting a line is a valid
|
||||
way to return it to its built-in default.
|
||||
|
||||
**Not in the file, by design**: `--start`, `--end`, `--seed`, `--regen`,
|
||||
`--rebuild`, `--resume-from`, `--embedded-subs`, `--skip-tts`. These describe a
|
||||
particular run rather than your setup, so pinning them in config would be a
|
||||
trap.
|
||||
|
||||
## Options
|
||||
|
||||
Every option below can also be set in `settings.ini`, except those noted as
|
||||
run-specific.
|
||||
|
||||
### Range
|
||||
|
||||
| Flag | Default | Meaning |
|
||||
@@ -128,7 +196,7 @@ python3 lektor_ai_comfy.py input/film.mkv --skip-tts output/tts_assembled.flac
|
||||
| `--start MM:SS` | `00:00` | Start of the range to process |
|
||||
| `--end MM:SS` | *(end)* | End of the range |
|
||||
|
||||
With `--end`, a matching fragment is extracted to `output/video_fragment.mkv`
|
||||
With `--end`, a matching fragment is extracted to `video_fragment.mkv` in the job folder
|
||||
and all TTS timestamps become relative to `--start`. Pass the same values to any
|
||||
later `--rebuild`, or the audio will not line up.
|
||||
|
||||
@@ -138,7 +206,7 @@ later `--rebuild`, or the audio will not line up.
|
||||
|
||||
| Flag | Default | Meaning |
|
||||
|---|---|---|
|
||||
| `--voice NAME\|PATH` | `gudowski_short` | Name in `gudowski/`, or a path to a WAV |
|
||||
| `--voice NAME\|PATH` | `gudowski_short` | Name in `voices/`, or a path to a WAV |
|
||||
| `--seed N` | `0` | Base seed; `0` randomises each run |
|
||||
| `--temperature T` | `0.9` | Sampling temperature |
|
||||
| `--max-tokens N` | `4096` | Token ceiling per clip |
|
||||
@@ -187,16 +255,40 @@ Typical starting point for dialogue-heavy material: `--orig-volume 0.5
|
||||
|
||||
## Output
|
||||
|
||||
Everything belonging to one video is kept together in a **job folder** named
|
||||
after the video file. Only the finished video is written to the top level.
|
||||
|
||||
```
|
||||
output/
|
||||
tts_input.srt processed text actually sent to TTS — inspect this
|
||||
embedded_sub.srt extracted embedded subtitle (--embedded-subs only)
|
||||
clips/clip_NNNN.flac one lossless clip per block, kept for --regen
|
||||
tts_assembled.flac all clips at their final positions, lossless
|
||||
video_fragment.mkv extracted segment (--end only)
|
||||
{stem}_lektor.mkv the finished video
|
||||
clips/
|
||||
S01_E01 The Eyes/ job folder, named after the video file
|
||||
clip_NNNN.flac one lossless clip per block, kept for --regen
|
||||
tts_input.srt processed text actually sent to TTS
|
||||
tts_assembled.flac all clips at their final positions, lossless
|
||||
embedded_sub.srt extracted subtitle (--embedded-subs only)
|
||||
video_fragment.mkv extracted segment (--end only)
|
||||
F1 The Movie 2025/
|
||||
...
|
||||
S01_E01 The Eyes_lektor.mkv the finished video
|
||||
F1 The Movie 2025_lektor.mkv
|
||||
```
|
||||
|
||||
Each run prints the folder it is using:
|
||||
|
||||
```
|
||||
Job folder: output/clips/S01_E01 The Eyes
|
||||
```
|
||||
|
||||
This is what makes `--rebuild` and `--regen` safe across films. `tts_input.srt`
|
||||
and the clips are a matched pair — block *N* in the SRT is `clip_NNNN.flac` —
|
||||
so both live in the job folder. They previously sat at the top level and every
|
||||
new job overwrote the last one's, which meant rebuilding an older film paired
|
||||
its clips with the newest film's text. Any past job can now be rebuilt at any
|
||||
time.
|
||||
|
||||
The folder is derived from the video filename, so `--rebuild` and `--regen`
|
||||
locate it automatically; just pass the same video path.
|
||||
|
||||
The finished file contains:
|
||||
|
||||
| Track | Content |
|
||||
@@ -229,7 +321,7 @@ python3 lektor_ai_comfy.py input/film.mkv --rebuild --orig-volume 0.45 --tts-vol
|
||||
**Fix a handful of bad clips**
|
||||
|
||||
```bash
|
||||
# read output/tts_input.srt, note the offending block numbers
|
||||
# read output/clips/<video name>/tts_input.srt, note the offending block numbers
|
||||
python3 lektor_ai_comfy.py input/film.mkv --regen 88 91 104
|
||||
python3 lektor_ai_comfy.py input/film.mkv --rebuild
|
||||
```
|
||||
|
||||
+182
-47
@@ -9,6 +9,7 @@ Requires ComfyUI with TTS-Audio-Suite running at http://127.0.0.1:8189
|
||||
"""
|
||||
|
||||
import argparse
|
||||
import configparser
|
||||
import json
|
||||
import random
|
||||
import re
|
||||
@@ -25,10 +26,63 @@ from urllib.request import urlopen, Request
|
||||
# Config
|
||||
# ---------------------------------------------------------------------------
|
||||
|
||||
COMFY_URL = "http://127.0.0.1:8189"
|
||||
COMFY_OUTPUT_DIR = Path("/home/lukas/comfy-tts-app/ComfyUI/output")
|
||||
COMFY_INPUT_DIR = Path("/home/lukas/comfy-tts-app/ComfyUI/input")
|
||||
VOICE_PROMPTS_DIR = Path("/home/lukas/Programy/lektor_ai/gudowski")
|
||||
VOICE_PROMPTS_DIR = Path(__file__).resolve().parent / "voices"
|
||||
|
||||
SETTINGS_FILE = Path(__file__).resolve().parent / "settings.ini"
|
||||
|
||||
# Fallback values, used when settings.ini is absent or omits a key. Keeping them
|
||||
# here means the script still runs correctly with no settings file at all.
|
||||
# Each entry is (ini_section, ini_key, type, value).
|
||||
_DEFAULTS = {
|
||||
"voice": ("tts", "voice", str, "gudowski_short"),
|
||||
"speed": ("tts", "speed", float, 1.2),
|
||||
"temperature": ("tts", "temperature", float, 0.9),
|
||||
"max_tokens": ("tts", "max_tokens", int, 4096),
|
||||
"strip_leading_ms": ("tts", "strip_leading_ms", float, 0.0),
|
||||
"trailing_fade_ms": ("tts", "trailing_fade_ms", float, 80.0),
|
||||
"orig_volume": ("mix", "orig_volume", float, 0.60),
|
||||
"tts_volume": ("mix", "tts_volume", float, 1.8),
|
||||
"output_dir": ("paths", "output_dir", str, "output"),
|
||||
"comfy_url": ("comfy", "comfy_url", str, "http://127.0.0.1:8189"),
|
||||
}
|
||||
|
||||
COMFY_URL = _DEFAULTS["comfy_url"][3] # rebound from settings/CLI in main()
|
||||
|
||||
|
||||
def load_settings(path: Path) -> dict:
|
||||
"""Read settings.ini, falling back to _DEFAULTS for anything missing.
|
||||
|
||||
Values here become argparse's defaults, so precedence ends up as
|
||||
CLI flag > settings.ini > _DEFAULTS without any extra bookkeeping.
|
||||
A malformed or unreadable file is reported and then ignored rather than
|
||||
being fatal — a typo in a config file should not cost you a long run.
|
||||
"""
|
||||
values = {key: spec[3] for key, spec in _DEFAULTS.items()}
|
||||
if not path.is_file():
|
||||
return values
|
||||
|
||||
parser = configparser.ConfigParser(inline_comment_prefixes=("#", ";"))
|
||||
try:
|
||||
parser.read(path, encoding="utf-8")
|
||||
except (configparser.Error, OSError) as exc:
|
||||
print(f" WARNING: could not read {path.name} ({exc}) — using built-in defaults",
|
||||
file=sys.stderr)
|
||||
return values
|
||||
|
||||
for key, (section, option, cast, fallback) in _DEFAULTS.items():
|
||||
if not parser.has_option(section, option):
|
||||
continue
|
||||
raw = parser.get(section, option).strip()
|
||||
if not raw:
|
||||
continue
|
||||
try:
|
||||
values[key] = cast(raw)
|
||||
except ValueError:
|
||||
print(f" WARNING: {path.name} [{section}] {option}={raw!r} is not a valid "
|
||||
f"{cast.__name__} — using {fallback!r}", file=sys.stderr)
|
||||
return values
|
||||
|
||||
# Subtitle codecs that carry real text and can therefore be extracted and fed
|
||||
# to the TTS engine. Used only by the --embedded-subs picker; image-based
|
||||
@@ -642,14 +696,31 @@ def submit_workflow(workflow: dict) -> str:
|
||||
return resp["prompt_id"]
|
||||
|
||||
|
||||
def wait_for_completion(prompt_id: str, poll_interval: int = 3) -> dict:
|
||||
def wait_for_completion(prompt_id: str, poll_interval: float = 0.25,
|
||||
max_interval: float = 2.0, backoff_after: float = 10.0) -> dict:
|
||||
"""Block until ComfyUI reports the job finished, then return its history entry.
|
||||
|
||||
The poll interval, not the GPU, governs perceived generation speed: a clip
|
||||
that finishes 100 ms after a poll still waits for the next one. This used to
|
||||
be a flat 3 s, which quantised every clip to a 3 s boundary — measured across
|
||||
2075 real clips, 91% of the gaps landed exactly on a multiple of 3 s, and
|
||||
roughly a third of each run was spent sleeping on already-completed work.
|
||||
|
||||
Polling starts tight so the common case (most clips finish in 3-6 s) is
|
||||
picked up almost immediately, then backs off once a job looks slow so a
|
||||
stuck or very long generation does not sit at 4 requests/second forever.
|
||||
"""
|
||||
print(f" Waiting for {prompt_id[:8]}…", end="", flush=True)
|
||||
start = time.monotonic()
|
||||
interval = poll_interval
|
||||
next_dot = 1.0
|
||||
|
||||
while True:
|
||||
try:
|
||||
history = _comfy_get(f"/history/{prompt_id}")
|
||||
except Exception as e:
|
||||
print(f"\n Poll error: {e}", file=sys.stderr)
|
||||
time.sleep(poll_interval)
|
||||
time.sleep(min(interval, 1.0))
|
||||
continue
|
||||
|
||||
if prompt_id in history:
|
||||
@@ -660,11 +731,17 @@ def wait_for_completion(prompt_id: str, poll_interval: int = 3) -> dict:
|
||||
print(f"\nComfyUI error: {msg_data}", file=sys.stderr)
|
||||
sys.exit(1)
|
||||
if status.get("completed"):
|
||||
print(" done.")
|
||||
print(f" done ({time.monotonic() - start:.1f}s).")
|
||||
return entry
|
||||
|
||||
print(".", end="", flush=True)
|
||||
time.sleep(poll_interval)
|
||||
elapsed = time.monotonic() - start
|
||||
# One dot per second regardless of poll rate, or the line fills with dots.
|
||||
if elapsed >= next_dot:
|
||||
print(".", end="", flush=True)
|
||||
next_dot += 1.0
|
||||
time.sleep(interval)
|
||||
if elapsed > backoff_after:
|
||||
interval = min(interval * 1.5, max_interval)
|
||||
|
||||
|
||||
def extract_output_audio(history_entry: dict) -> Path:
|
||||
@@ -762,18 +839,45 @@ def generate_clips(blocks: list, voice_filename: str, voice_ref_text: str,
|
||||
# Rebuild / Regen helpers
|
||||
# ---------------------------------------------------------------------------
|
||||
|
||||
def job_dir(out_dir: Path, video_path: str) -> Path:
|
||||
"""Per-job working directory: <output_dir>/clips/<video filename without extension>/
|
||||
|
||||
Everything that belongs to one video lives here — the clips, the processed
|
||||
tts_input.srt, the assembled audio, any extracted subtitle or video
|
||||
fragment. Only the finished <stem>_lektor.mkv is written to <output_dir>.
|
||||
|
||||
Without this separation every job overwrote the previous one's clip_NNNN.flac
|
||||
and tts_input.srt, so --rebuild and --regen were only ever safe on the most
|
||||
recent run. The clips and tts_input.srt are a matched pair — block N in the
|
||||
SRT is clip_NNNN.flac — which is why the SRT lives here too rather than at
|
||||
the top level.
|
||||
"""
|
||||
return out_dir / "clips" / Path(video_path).stem
|
||||
|
||||
|
||||
def _legacy_hint(out_dir: Path, job: Path) -> str:
|
||||
"""Point at the pre-per-job layout if that is what the user actually has."""
|
||||
if (out_dir / "tts_input.srt").is_file():
|
||||
return (f"\n\nA tts_input.srt exists at {out_dir} in the old shared layout. "
|
||||
f"Clips are now kept per job in {job}. Move that run's files there, "
|
||||
f"or regenerate it.")
|
||||
return ""
|
||||
|
||||
|
||||
def _rebuild_video(args, out_dir: Path):
|
||||
"""Re-assemble audio from existing clips and rebuild the final video.
|
||||
No TTS generation — use this to change volumes or fix assembly without
|
||||
waiting for the model to run again.
|
||||
"""
|
||||
tts_srt = out_dir / "tts_input.srt"
|
||||
job = job_dir(out_dir, args.video)
|
||||
tts_srt = job / "tts_input.srt"
|
||||
if not tts_srt.exists():
|
||||
sys.exit(f"No tts_input.srt found in {out_dir} — run a full generation first.")
|
||||
sys.exit(f"No tts_input.srt found in {job} — run a full generation first."
|
||||
+ _legacy_hint(out_dir, job))
|
||||
|
||||
blocks = load_tts_input_srt(tts_srt)
|
||||
total = len(blocks)
|
||||
clips_dir = out_dir / "clips"
|
||||
clips_dir = job
|
||||
|
||||
missing = [
|
||||
clips_dir / f"clip_{i+1:04d}.flac"
|
||||
@@ -793,7 +897,7 @@ def _rebuild_video(args, out_dir: Path):
|
||||
clips.append((actual_start, clip_path))
|
||||
cursor = actual_start + duration
|
||||
|
||||
assembled_tts = out_dir / "tts_assembled.flac"
|
||||
assembled_tts = job / "tts_assembled.flac"
|
||||
print(f"\n Assembling {total} clips…")
|
||||
assemble_timed_audio(clips, str(assembled_tts))
|
||||
print(f" Assembled: {assembled_tts} ({assembled_tts.stat().st_size // 1024} KB)")
|
||||
@@ -803,7 +907,7 @@ def _rebuild_video(args, out_dir: Path):
|
||||
if args.end:
|
||||
start_sec = _parse_mmss(args.start)
|
||||
end_sec = _parse_mmss(args.end)
|
||||
video_fragment = out_dir / "video_fragment.mkv"
|
||||
video_fragment = job / "video_fragment.mkv"
|
||||
if not video_fragment.exists():
|
||||
extract_video_fragment(args.video, start_sec, end_sec, str(video_fragment))
|
||||
video_src = str(video_fragment)
|
||||
@@ -824,15 +928,17 @@ def _regen_clips(args, out_dir: Path):
|
||||
"""Regenerate specific clips by tts_input block number.
|
||||
Only replaces FLAC files — run --rebuild afterwards to reassemble audio and video.
|
||||
"""
|
||||
tts_srt = out_dir / "tts_input.srt"
|
||||
job = job_dir(out_dir, args.video)
|
||||
tts_srt = job / "tts_input.srt"
|
||||
if not tts_srt.exists():
|
||||
sys.exit(f"No tts_input.srt found in {out_dir} — run a full generation first.")
|
||||
sys.exit(f"No tts_input.srt found in {job} — run a full generation first."
|
||||
+ _legacy_hint(out_dir, job))
|
||||
|
||||
blocks = load_tts_input_srt(tts_srt)
|
||||
total = len(blocks)
|
||||
regen_set = set(args.regen)
|
||||
clips_dir = out_dir / "clips"
|
||||
clips_dir.mkdir(exist_ok=True)
|
||||
clips_dir = job
|
||||
clips_dir.mkdir(parents=True, exist_ok=True)
|
||||
|
||||
bad = [n for n in regen_set if n < 1 or n > total]
|
||||
if bad:
|
||||
@@ -874,9 +980,23 @@ def _regen_clips(args, out_dir: Path):
|
||||
|
||||
def main():
|
||||
global COMFY_URL
|
||||
|
||||
# --settings is resolved before the main parser is built, because the file's
|
||||
# contents become that parser's defaults.
|
||||
pre = argparse.ArgumentParser(add_help=False)
|
||||
pre.add_argument("--settings", default=str(SETTINGS_FILE), metavar="FILE")
|
||||
pre_args, _ = pre.parse_known_args()
|
||||
settings_path = Path(pre_args.settings)
|
||||
cfg = load_settings(settings_path)
|
||||
|
||||
parser = argparse.ArgumentParser(
|
||||
description="Generate AI voiceover via ComfyUI + Higgs Audio V3, per-subtitle."
|
||||
description="Generate AI voiceover via ComfyUI + Higgs Audio V3, per-subtitle.",
|
||||
epilog=f"Defaults are read from {SETTINGS_FILE.name} next to this script; "
|
||||
f"any flag given on the command line overrides it.",
|
||||
)
|
||||
parser.add_argument("--settings", default=str(SETTINGS_FILE), metavar="FILE",
|
||||
help=f"Settings file to read defaults from "
|
||||
f"(default: {SETTINGS_FILE.name})")
|
||||
parser.add_argument("srt", nargs="?", default=None,
|
||||
help="Input SRT subtitle file (omit with --embedded-subs)")
|
||||
parser.add_argument("video", nargs="?", default=None,
|
||||
@@ -885,33 +1005,39 @@ def main():
|
||||
help="Fragment start time (default: 00:00)")
|
||||
parser.add_argument("--end", default=None, metavar="MM:SS",
|
||||
help="Fragment end time")
|
||||
parser.add_argument("--voice", default="gudowski_short",
|
||||
help="Voice prompt name or WAV path (default: gudowski_short)")
|
||||
parser.add_argument("--speed", type=float, default=1.2, metavar="FACTOR",
|
||||
help="Per-clip playback speed via atempo (default: 1.2)")
|
||||
parser.add_argument("--voice", default=cfg["voice"],
|
||||
help=f"Voice prompt name or WAV path (default: {cfg['voice']})")
|
||||
parser.add_argument("--speed", type=float, default=cfg["speed"], metavar="FACTOR",
|
||||
help=f"Per-clip playback speed via atempo (default: {cfg['speed']})")
|
||||
parser.add_argument("--seed", type=int, default=0,
|
||||
help="Base generation seed (default: 0 = random each run)")
|
||||
parser.add_argument("--orig-volume", type=float, default=0.60, metavar="0-1",
|
||||
help="Original audio volume in mix (default: 0.60)")
|
||||
parser.add_argument("--tts-volume", type=float, default=1.8, metavar="0-2",
|
||||
help="Voiceover volume in mix (default: 1.8)")
|
||||
parser.add_argument("--output-dir", default="output",
|
||||
help="Output directory (default: output/)")
|
||||
parser.add_argument("--comfy-url", default=COMFY_URL,
|
||||
help=f"ComfyUI URL (default: {COMFY_URL})")
|
||||
parser.add_argument("--orig-volume", type=float, default=cfg["orig_volume"], metavar="0-1",
|
||||
help=f"Original audio volume in mix (default: {cfg['orig_volume']})")
|
||||
parser.add_argument("--tts-volume", type=float, default=cfg["tts_volume"], metavar="0-2",
|
||||
help=f"Voiceover volume in mix (default: {cfg['tts_volume']})")
|
||||
parser.add_argument("--output-dir", default=cfg["output_dir"],
|
||||
help=f"Output directory (default: {cfg['output_dir']}/)")
|
||||
parser.add_argument("--comfy-url", default=cfg["comfy_url"],
|
||||
help=f"ComfyUI URL (default: {cfg['comfy_url']})")
|
||||
parser.add_argument("--skip-tts", metavar="AUDIO_FILE",
|
||||
help="Skip TTS and use this pre-assembled audio file")
|
||||
parser.add_argument("--temperature", type=float, default=0.9, metavar="T",
|
||||
help="Sampling temperature (default: 0.9; lower = more deterministic)")
|
||||
parser.add_argument("--strip-leading-ms", type=float, default=0.0, metavar="MS",
|
||||
help="Strip N ms from the start of each TTS clip to remove "
|
||||
"leading breath/noise artifact (default: 0, try 80–150)")
|
||||
parser.add_argument("--trailing-fade-ms", type=float, default=80.0, metavar="MS",
|
||||
help="Fade out the last N ms of each TTS clip to suppress "
|
||||
"tail pops/noise (default: 80, set 0 to disable)")
|
||||
parser.add_argument("--max-tokens", type=int, default=4096, metavar="N",
|
||||
help="Max audio tokens the TTS model may generate per clip "
|
||||
"(default: 4096; raise if long clips are cut off mid-word)")
|
||||
parser.add_argument("--temperature", type=float, default=cfg["temperature"], metavar="T",
|
||||
help=f"Sampling temperature (default: {cfg['temperature']}; "
|
||||
f"lower = more deterministic)")
|
||||
parser.add_argument("--strip-leading-ms", type=float, default=cfg["strip_leading_ms"],
|
||||
metavar="MS",
|
||||
help=f"Strip N ms from the start of each TTS clip to remove "
|
||||
f"leading breath/noise artifact "
|
||||
f"(default: {cfg['strip_leading_ms']}, try 80–150)")
|
||||
parser.add_argument("--trailing-fade-ms", type=float, default=cfg["trailing_fade_ms"],
|
||||
metavar="MS",
|
||||
help=f"Fade out the last N ms of each TTS clip to suppress "
|
||||
f"tail pops/noise (default: {cfg['trailing_fade_ms']}, "
|
||||
f"set 0 to disable)")
|
||||
parser.add_argument("--max-tokens", type=int, default=cfg["max_tokens"], metavar="N",
|
||||
help=f"Max audio tokens the TTS model may generate per clip "
|
||||
f"(default: {cfg['max_tokens']}; raise if long clips are "
|
||||
f"cut off mid-word)")
|
||||
parser.add_argument("--resume-from", type=int, default=0, metavar="N",
|
||||
help="Resume a crashed run: skip blocks 1..(N-1) by loading "
|
||||
"their existing clip files, generate from block N onwards.")
|
||||
@@ -933,6 +1059,11 @@ def main():
|
||||
|
||||
COMFY_URL = args.comfy_url
|
||||
|
||||
if settings_path.is_file():
|
||||
print(f" Settings: {settings_path}")
|
||||
else:
|
||||
print(f" Settings: {settings_path.name} not found — using built-in defaults")
|
||||
|
||||
# ── Resolve positional ambiguity for modes that don't need an SRT file ────
|
||||
# When the user writes `script.py video.mkv --embedded-subs` (or --rebuild /
|
||||
# --regen without an SRT), argparse puts the lone positional into args.srt.
|
||||
@@ -947,10 +1078,8 @@ def main():
|
||||
parser.error("Provide an SRT subtitle file, or use --embedded-subs to pick one "
|
||||
"from the video.")
|
||||
|
||||
out_dir = Path(args.output_dir)
|
||||
out_dir = Path(args.output_dir)
|
||||
out_dir.mkdir(parents=True, exist_ok=True)
|
||||
clips_dir = out_dir / "clips"
|
||||
clips_dir.mkdir(exist_ok=True)
|
||||
|
||||
# ── Rebuild mode: reassemble + remux from existing clips, no TTS ─────────
|
||||
if args.rebuild:
|
||||
@@ -962,6 +1091,12 @@ def main():
|
||||
_regen_clips(args, out_dir)
|
||||
return
|
||||
|
||||
# Everything belonging to this video lives together, so successive jobs
|
||||
# cannot overwrite each other's clips or tts_input.srt.
|
||||
clips_dir = job_dir(out_dir, args.video)
|
||||
clips_dir.mkdir(parents=True, exist_ok=True)
|
||||
print(f" Job folder: {clips_dir}")
|
||||
|
||||
start_sec = _parse_mmss(args.start)
|
||||
end_sec = _parse_mmss(args.end) if args.end else None
|
||||
|
||||
@@ -991,7 +1126,7 @@ def main():
|
||||
if chosen['codec_name'].lower() not in _TEXT_SUB_CODECS:
|
||||
sys.exit(f"Stream #{chosen['index']} ({chosen['codec_name']}) is image-based "
|
||||
"and cannot be converted to text for TTS.")
|
||||
embedded_srt = out_dir / "embedded_sub.srt"
|
||||
embedded_srt = clips_dir / "embedded_sub.srt"
|
||||
extract_subtitle_track(args.video, chosen['index'], str(embedded_srt))
|
||||
srt_path = str(embedded_srt)
|
||||
else:
|
||||
@@ -1005,7 +1140,7 @@ def main():
|
||||
blocks = split_dialogue_blocks(blocks)
|
||||
print(f" {len(blocks)} entries after dialogue splitting.")
|
||||
|
||||
srt_preview = out_dir / "tts_input.srt"
|
||||
srt_preview = clips_dir / "tts_input.srt"
|
||||
srt_preview.write_text(blocks_to_srt(blocks), encoding="utf-8")
|
||||
print(f" Saved to {srt_preview} for inspection.")
|
||||
|
||||
@@ -1035,7 +1170,7 @@ def main():
|
||||
trailing_fade_ms=args.trailing_fade_ms,
|
||||
max_new_tokens=args.max_tokens)
|
||||
|
||||
assembled_tts = out_dir / "tts_assembled.flac"
|
||||
assembled_tts = clips_dir / "tts_assembled.flac"
|
||||
assemble_timed_audio(clips, str(assembled_tts))
|
||||
print(f" Assembled: {assembled_tts} ({assembled_tts.stat().st_size // 1024} KB)")
|
||||
print(f" Clips saved to {clips_dir}")
|
||||
@@ -1044,7 +1179,7 @@ def main():
|
||||
print(f"\n[3/4] Extracting video fragment…")
|
||||
real_end = _parse_mmss(args.end) if args.end else None
|
||||
if real_end is not None:
|
||||
video_fragment = out_dir / "video_fragment.mkv"
|
||||
video_fragment = clips_dir / "video_fragment.mkv"
|
||||
extract_video_fragment(args.video, start_sec, real_end, str(video_fragment))
|
||||
else:
|
||||
video_fragment = Path(args.video)
|
||||
|
||||
@@ -29,7 +29,7 @@ from urllib.request import urlopen, Request
|
||||
COMFY_URL = "http://127.0.0.1:8189"
|
||||
COMFY_OUTPUT_DIR = Path("/home/lukas/comfy-tts-app/ComfyUI/output")
|
||||
COMFY_INPUT_DIR = Path("/home/lukas/comfy-tts-app/ComfyUI/input")
|
||||
VOICE_PROMPTS_DIR = Path("/home/lukas/Programy/lektor_ai/gudowski")
|
||||
VOICE_PROMPTS_DIR = Path("/home/lukas/Programy/lektor_ai/voices")
|
||||
|
||||
# ---------------------------------------------------------------------------
|
||||
# SRT helpers
|
||||
+29
-9
@@ -24,7 +24,17 @@ Requirements
|
||||
- Model: higgs-audio-v3-tts-4b (bosonai/higgs-audio-v3)
|
||||
- ffmpeg and ffprobe available on PATH
|
||||
- Voice reference: a WAV file + matching .txt transcript in the voice directory
|
||||
(default: gudowski/gudowski.wav + gudowski/gudowski.txt)
|
||||
(default: voices/gudowski_short.wav + voices/gudowski_short.txt)
|
||||
|
||||
Settings
|
||||
--------
|
||||
Defaults live in settings.ini next to the script:
|
||||
voice, speed, temperature, max_tokens, strip_leading_ms, trailing_fade_ms,
|
||||
orig_volume, tts_volume, output_dir, comfy_url
|
||||
|
||||
Precedence: command-line flag > settings.ini > built-in default.
|
||||
Use --settings FILE to select a different profile.
|
||||
See docs/usage.md for the full description.
|
||||
|
||||
|
||||
Basic Usage
|
||||
@@ -54,7 +64,7 @@ Both external SRT files and embedded subtitle tracks go through the same cleanin
|
||||
- Missing terminal punctuation added (… or . depending on context)
|
||||
- Dialogue lines (prefixed with -) are split into separate clips with proportional timing
|
||||
|
||||
With --embedded-subs the extracted subtitles are saved to output/embedded_sub.srt
|
||||
With --embedded-subs the extracted subtitles are saved to the job folder as embedded_sub.srt
|
||||
so you can inspect them before or after the run.
|
||||
|
||||
The final MKV preserves all subtitle tracks from the source video — text-based
|
||||
@@ -64,17 +74,27 @@ stream copy with no conversion.
|
||||
|
||||
Output Files
|
||||
------------
|
||||
Everything belonging to one video is kept together in a job folder named after
|
||||
the video file. Only the finished video is written to the top level.
|
||||
|
||||
output/
|
||||
tts_input.srt — processed subtitle text sent to TTS (inspect to verify)
|
||||
embedded_sub.srt — extracted embedded subtitle (only when --embedded-subs)
|
||||
clips/clip_NNNN.flac — individual TTS clips, one per subtitle block (lossless)
|
||||
tts_assembled.flac — all clips assembled with correct timing (lossless)
|
||||
video_fragment.mkv — extracted video segment (only when --end is used)
|
||||
clips/
|
||||
<video name>/ — job folder for one video:
|
||||
clip_NNNN.flac — individual TTS clips, one per block (lossless)
|
||||
tts_input.srt — processed subtitle text sent to TTS (inspect to verify)
|
||||
tts_assembled.flac — all clips assembled with correct timing (lossless)
|
||||
embedded_sub.srt — extracted embedded subtitle (only with --embedded-subs)
|
||||
video_fragment.mkv — extracted video segment (only when --end is used)
|
||||
{stem}_lektor.mkv — final video with two audio tracks + all original subtitles:
|
||||
Track 0 — Original (source audio, stream-copied as-is)
|
||||
Track 1 — Polish lektor (TTS + original mixed,
|
||||
AAC 192k stereo)
|
||||
|
||||
Each run prints the job folder it uses. tts_input.srt and the clips are a
|
||||
matched pair (block N is clip_NNNN.flac), which is why they live together —
|
||||
previously each new job overwrote the last one's, so --rebuild on an older
|
||||
film silently paired its clips with the newest film's text.
|
||||
|
||||
|
||||
All Options
|
||||
-----------
|
||||
@@ -139,7 +159,7 @@ Regenerate specific clips by block number (same numbering as tts_input.srt):
|
||||
|
||||
python3 lektor_ai_comfy.py input/film.mkv --regen 12 22 41
|
||||
|
||||
Only replaces the selected clips/clip_NNNN.flac files — does not reassemble
|
||||
Only replaces the selected clip_NNNN.flac files in the job folder — does not reassemble
|
||||
or rebuild the video. Run --rebuild afterwards to apply the changes.
|
||||
Useful for fixing hallucinations, mumbling, or cut-off clips.
|
||||
|
||||
@@ -158,7 +178,7 @@ Reassemble audio from all existing clips and rebuild the final video
|
||||
Tips
|
||||
----
|
||||
|
||||
- Check output/tts_input.srt before or after a run to see the exact text sent to TTS.
|
||||
- Check output/clips/<video name>/tts_input.srt to see the exact text sent to TTS.
|
||||
Block numbers in that file match the clip_NNNN.flac filenames.
|
||||
|
||||
- If a clip sounds wrong (hallucination, wrong language, cut off), note its block
|
||||
|
||||
@@ -0,0 +1,56 @@
|
||||
# lektor_ai_comfy — default settings
|
||||
#
|
||||
# These are the values used when a flag is not given on the command line.
|
||||
# Precedence: command-line flag > this file > built-in defaults
|
||||
#
|
||||
# Delete a line (or the whole file) to fall back to the built-in default.
|
||||
# A malformed value is reported and ignored, never fatal.
|
||||
#
|
||||
# Not configurable here, because they belong to a single run rather than to
|
||||
# your setup: --start --end --seed --regen --rebuild --resume-from
|
||||
# --embedded-subs --skip-tts
|
||||
|
||||
|
||||
[tts]
|
||||
# Voice reference. Either a name in voices/ (expects NAME.wav + NAME.txt),
|
||||
# or a full path to a .wav file.
|
||||
voice = gudowski_short
|
||||
|
||||
# Per-clip speed-up applied with atempo, before assembly.
|
||||
# Raise it if speech regularly overruns its subtitle window; 1.0 = no change.
|
||||
speed = 1.2
|
||||
|
||||
# Sampling temperature. 1.0 makes the model occasionally recite the reference
|
||||
# transcript instead of the subtitle; 0.9 largely prevents that. Lower is more
|
||||
# deterministic but flatter.
|
||||
temperature = 0.7
|
||||
|
||||
# Ceiling on audio tokens per clip. Raise only if long clips are cut off
|
||||
# mid-word.
|
||||
max_tokens = 4096
|
||||
|
||||
# Trim this many ms from the start of every clip, to remove an inhale artifact.
|
||||
# Try 80-150 if you hear one. Too high clips the first consonant.
|
||||
strip_leading_ms = 0
|
||||
|
||||
# Fade out the last N ms of every clip to suppress tail pops. 0 disables.
|
||||
trailing_fade_ms = 80
|
||||
|
||||
|
||||
[mix]
|
||||
# Levels inside the Polish lektor track only. Track 0 is always the untouched
|
||||
# original, so mixing aggressively here costs you nothing.
|
||||
# Dialogue-heavy sources: orig 0.5 / tts 2.0. Quiet sources: orig 1.0 / tts 1.4.
|
||||
orig_volume = 0.70
|
||||
tts_volume = 1.4
|
||||
|
||||
|
||||
[paths]
|
||||
# Where clips, assembled audio and the finished video are written.
|
||||
# Relative paths are resolved from wherever you run the script.
|
||||
output_dir = output
|
||||
|
||||
|
||||
[comfy]
|
||||
# Address of the ComfyUI backend running Higgs Audio V3.
|
||||
comfy_url = http://127.0.0.1:8189
|
||||
Reference in New Issue
Block a user