sync-v2 step 7: observability + graceful shutdown (§7)
- Heartbeat JSON now carries {owned, sockets_open, tiers:{hot,warm,cold}, ts} per
§2 so peers and operators can see each worker's load and socket footprint.
- run.py runs the sync coordinator directly (not via a placeholder arq worker) so
SIGINT/SIGTERM can stop the claimer, cancel runners, and release every lease
explicitly within a 10s drain budget — peers adopt the accounts immediately
instead of waiting out the lease TTL. Removed the sync:control-queue hack and
the supervisor indirection (start/stop_supervisor, sync_on_startup/shutdown).
- Acceptance: graceful-shutdown test proves a peer adopts released accounts within
seconds under a 30s TTL (i.e. via release, not expiry).
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
This commit is contained in:
@@ -1,8 +1,9 @@
|
||||
"""Worker job functions and the sync-worker lifecycle (refactor §1, §6).
|
||||
"""Worker job callables and queue function-sets (refactor §1, §6).
|
||||
|
||||
The process is launched via ``workers.run`` (``WORKER_ROLE`` decides which arq
|
||||
workers run). This module only defines the job callables and the two queue
|
||||
function-sets; the runner wires them onto the ``interactive`` / ``bulk`` queues.
|
||||
The sync role runs the coordinator directly (see ``workers.run``), not via arq.
|
||||
"""
|
||||
|
||||
from __future__ import annotations
|
||||
@@ -46,19 +47,3 @@ INTERACTIVE_FUNCTIONS = [
|
||||
|
||||
# Bulk work (§6): long timeout, low concurrency, isolated from interactive jobs.
|
||||
BULK_FUNCTIONS = [sync_bootstrap_account]
|
||||
|
||||
|
||||
async def sync_on_startup(ctx: dict[str, Any]) -> None:
|
||||
"""Sync-worker startup: launch the account-sync supervisor."""
|
||||
from app.sync.supervisor import start_supervisor
|
||||
|
||||
await start_supervisor(ctx)
|
||||
logger.info("sync worker: supervisor started")
|
||||
|
||||
|
||||
async def sync_on_shutdown(ctx: dict[str, Any]) -> None:
|
||||
"""Sync-worker shutdown: stop the supervisor (process cleanup is in the runner)."""
|
||||
from app.sync.supervisor import stop_supervisor
|
||||
|
||||
await stop_supervisor(ctx)
|
||||
logger.info("sync worker: supervisor stopped")
|
||||
|
||||
Reference in New Issue
Block a user