sync-v2 step 7: observability + graceful shutdown (§7)
- Heartbeat JSON now carries {owned, sockets_open, tiers:{hot,warm,cold}, ts} per
§2 so peers and operators can see each worker's load and socket footprint.
- run.py runs the sync coordinator directly (not via a placeholder arq worker) so
SIGINT/SIGTERM can stop the claimer, cancel runners, and release every lease
explicitly within a 10s drain budget — peers adopt the accounts immediately
instead of waiting out the lease TTL. Removed the sync:control-queue hack and
the supervisor indirection (start/stop_supervisor, sync_on_startup/shutdown).
- Acceptance: graceful-shutdown test proves a peer adopts released accounts within
seconds under a 30s TTL (i.e. via release, not expiry).
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
This commit is contained in:
@@ -202,7 +202,21 @@ class SyncCoordinator:
|
||||
await self._release(account_id)
|
||||
|
||||
async def _heartbeat_once(self) -> None:
|
||||
payload = json.dumps({"owned": len(self._runners), "ts": time.time()})
|
||||
"""Publish the worker's load for peer rebalancing + observability (§2)."""
|
||||
tiers = {"hot": 0, "warm": 0, "cold": 0}
|
||||
sockets_open = 0
|
||||
for runner in self._runners.values():
|
||||
tiers[runner.tier] = tiers.get(runner.tier, 0) + 1
|
||||
if runner.tier == "hot":
|
||||
sockets_open += 1 # one persistent IDLE connection per hot account
|
||||
payload = json.dumps(
|
||||
{
|
||||
"owned": len(self._runners),
|
||||
"sockets_open": sockets_open,
|
||||
"tiers": tiers,
|
||||
"ts": time.time(),
|
||||
}
|
||||
)
|
||||
await self._redis.set(leases.worker_key(self.worker_id), payload, ex=self.heartbeat_ttl)
|
||||
|
||||
async def _alive_worker_count(self) -> int:
|
||||
|
||||
@@ -1,29 +0,0 @@
|
||||
"""Live-sync entrypoint (refactor §2, step 3).
|
||||
|
||||
The sync worker's ``on_startup`` starts a :class:`SyncCoordinator`, which claims
|
||||
accounts from the ``sync:live`` registry via Redis leases and runs one
|
||||
:class:`~app.sync.runner.AccountRunner` per owned account. ``on_shutdown`` stops
|
||||
it, releasing every lease so peers can take the accounts over immediately.
|
||||
"""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import logging
|
||||
from typing import Any
|
||||
|
||||
from app.sync.coordinator import SyncCoordinator
|
||||
|
||||
logger = logging.getLogger(__name__)
|
||||
|
||||
|
||||
async def start_supervisor(ctx: dict[str, Any]) -> None:
|
||||
coordinator = SyncCoordinator()
|
||||
await coordinator.start()
|
||||
ctx["coordinator"] = coordinator
|
||||
|
||||
|
||||
async def stop_supervisor(ctx: dict[str, Any]) -> None:
|
||||
coordinator: SyncCoordinator | None = ctx.get("coordinator")
|
||||
if coordinator is not None:
|
||||
await coordinator.stop()
|
||||
ctx["coordinator"] = None
|
||||
Reference in New Issue
Block a user