Give a wedged ComfyUI ten restarts per clip instead of three

Three was not enough in practice: a backend that dies during model load can
burn all three before it ever comes up healthy, and the clip is then skipped —
or, when the backend is simply gone, the run stalls waiting for it.

Both caps move together. COMFY_AUTOSTART_MAX is what actually gates a relaunch,
so raising CLIP_CRASH_LIMIT alone would still have stopped at three restarts.
Counting is unchanged: both are per clip, not per run.

The cost of a truly dead backend goes up accordingly — up to ten rounds of a
90s cooldown plus the 180s health timeout before a clip is given up on.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01NkWDx9dVB7xBpqntvjvm6A
This commit is contained in:
2026-08-31 21:15:36 +02:00
co-authored by Claude Opus 5
parent 39452a4281
commit e541ef7ee9
+6 -6
View File
@@ -1542,7 +1542,7 @@ BACKEND_RETRY_SECONDS = 3.0 # how often to look for ComfyUI while it is down
# Launching the backend is guarded: a ComfyUI that dies on startup would
# otherwise spawn a new terminal window every few seconds.
COMFY_AUTOSTART_MAX = 3 # restarts allowed while one clip is in flight
COMFY_AUTOSTART_MAX = 10 # restarts allowed while one clip is in flight
COMFY_AUTOSTART_COOLDOWN = 90.0 # seconds between attempts
# A clip that kills the backend this many times is skipped rather than retried
@@ -1550,7 +1550,7 @@ COMFY_AUTOSTART_COOLDOWN = 90.0 # seconds between attempts
# tripped a CUDA device-side assert inside the audio tokenizer, which cannot be
# caught in Python — and without a cap an overnight queue stops there and does
# nothing until morning.
CLIP_CRASH_LIMIT = 3
CLIP_CRASH_LIMIT = 10
# How long a restarted backend gets to answer before we conclude it is not the
# clip's fault but the backend's absence.
@@ -1752,7 +1752,7 @@ def trim_comfy_log(path: Path = None, max_lines: int = COMFY_LOG_MAX_LINES) -> i
def wait_for_backend(timeout: float = 300.0) -> bool:
"""Block until the backend answers again. After a restart the port is not
listening for a while, and submitting into it would look like another
crash and burn one of the clip's three attempts."""
crash and burn one of the clip's attempts."""
deadline = time.monotonic() + timeout
while time.monotonic() < deadline:
try:
@@ -1813,7 +1813,7 @@ def run_workflow(workflow: dict, poll_interval: float = 0.25,
prompt_id, offline = None, False
start, interval, next_dot = time.monotonic(), poll_interval, 1.0
# Restarts are counted per clip, not per run. A crash in block 4000 of an
# overnight queue deserves the same three attempts the first one got.
# overnight queue deserves the same number of attempts the first one got.
# One death is enough to hand control back: the caller retries with a
# different seed, which is the only thing that changes the outcome.
crashes = 0
@@ -2002,7 +2002,7 @@ def generate_clips(blocks: list, voice_filename: str, voice_ref_text: str,
# A crash is seed-dependent, not purely text-dependent: the block
# that killed ComfyUI three times in a 2073-clip run generated
# perfectly on its own with a different seed. So each attempt gets a
# fresh seed — three tries at the identical computation would just
# fresh seed — repeated tries at the identical computation would just
# reproduce the same abort.
seed = base_seed + idx + attempt * 100_003
try:
@@ -2015,7 +2015,7 @@ def generate_clips(blocks: list, voice_filename: str, voice_ref_text: str,
break
except BackendCrashed as exc:
attempt += 1
# The backend died three times on this clip. That means one of
# The backend died on this clip. That means one of
# two very different things, and the difference decides whether
# skipping is right: either this clip kills ComfyUI — some
# inputs abort it from inside CUDA — or ComfyUI is simply not