Give a wedged ComfyUI ten restarts per clip instead of three
Three was not enough in practice: a backend that dies during model load can burn all three before it ever comes up healthy, and the clip is then skipped — or, when the backend is simply gone, the run stalls waiting for it. Both caps move together. COMFY_AUTOSTART_MAX is what actually gates a relaunch, so raising CLIP_CRASH_LIMIT alone would still have stopped at three restarts. Counting is unchanged: both are per clip, not per run. The cost of a truly dead backend goes up accordingly — up to ten rounds of a 90s cooldown plus the 180s health timeout before a clip is given up on. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01NkWDx9dVB7xBpqntvjvm6A
This commit is contained in:
+6
-6
@@ -1542,7 +1542,7 @@ BACKEND_RETRY_SECONDS = 3.0 # how often to look for ComfyUI while it is down
|
||||
|
||||
# Launching the backend is guarded: a ComfyUI that dies on startup would
|
||||
# otherwise spawn a new terminal window every few seconds.
|
||||
COMFY_AUTOSTART_MAX = 3 # restarts allowed while one clip is in flight
|
||||
COMFY_AUTOSTART_MAX = 10 # restarts allowed while one clip is in flight
|
||||
COMFY_AUTOSTART_COOLDOWN = 90.0 # seconds between attempts
|
||||
|
||||
# A clip that kills the backend this many times is skipped rather than retried
|
||||
@@ -1550,7 +1550,7 @@ COMFY_AUTOSTART_COOLDOWN = 90.0 # seconds between attempts
|
||||
# tripped a CUDA device-side assert inside the audio tokenizer, which cannot be
|
||||
# caught in Python — and without a cap an overnight queue stops there and does
|
||||
# nothing until morning.
|
||||
CLIP_CRASH_LIMIT = 3
|
||||
CLIP_CRASH_LIMIT = 10
|
||||
|
||||
# How long a restarted backend gets to answer before we conclude it is not the
|
||||
# clip's fault but the backend's absence.
|
||||
@@ -1752,7 +1752,7 @@ def trim_comfy_log(path: Path = None, max_lines: int = COMFY_LOG_MAX_LINES) -> i
|
||||
def wait_for_backend(timeout: float = 300.0) -> bool:
|
||||
"""Block until the backend answers again. After a restart the port is not
|
||||
listening for a while, and submitting into it would look like another
|
||||
crash and burn one of the clip's three attempts."""
|
||||
crash and burn one of the clip's attempts."""
|
||||
deadline = time.monotonic() + timeout
|
||||
while time.monotonic() < deadline:
|
||||
try:
|
||||
@@ -1813,7 +1813,7 @@ def run_workflow(workflow: dict, poll_interval: float = 0.25,
|
||||
prompt_id, offline = None, False
|
||||
start, interval, next_dot = time.monotonic(), poll_interval, 1.0
|
||||
# Restarts are counted per clip, not per run. A crash in block 4000 of an
|
||||
# overnight queue deserves the same three attempts the first one got.
|
||||
# overnight queue deserves the same number of attempts the first one got.
|
||||
# One death is enough to hand control back: the caller retries with a
|
||||
# different seed, which is the only thing that changes the outcome.
|
||||
crashes = 0
|
||||
@@ -2002,7 +2002,7 @@ def generate_clips(blocks: list, voice_filename: str, voice_ref_text: str,
|
||||
# A crash is seed-dependent, not purely text-dependent: the block
|
||||
# that killed ComfyUI three times in a 2073-clip run generated
|
||||
# perfectly on its own with a different seed. So each attempt gets a
|
||||
# fresh seed — three tries at the identical computation would just
|
||||
# fresh seed — repeated tries at the identical computation would just
|
||||
# reproduce the same abort.
|
||||
seed = base_seed + idx + attempt * 100_003
|
||||
try:
|
||||
@@ -2015,7 +2015,7 @@ def generate_clips(blocks: list, voice_filename: str, voice_ref_text: str,
|
||||
break
|
||||
except BackendCrashed as exc:
|
||||
attempt += 1
|
||||
# The backend died three times on this clip. That means one of
|
||||
# The backend died on this clip. That means one of
|
||||
# two very different things, and the difference decides whether
|
||||
# skipping is right: either this clip kills ComfyUI — some
|
||||
# inputs abort it from inside CUDA — or ComfyUI is simply not
|
||||
|
||||
Reference in New Issue
Block a user