sync-v2 step 7: observability + graceful shutdown (§7)

- Heartbeat JSON now carries {owned, sockets_open, tiers:{hot,warm,cold}, ts} per
  §2 so peers and operators can see each worker's load and socket footprint.
- run.py runs the sync coordinator directly (not via a placeholder arq worker) so
  SIGINT/SIGTERM can stop the claimer, cancel runners, and release every lease
  explicitly within a 10s drain budget — peers adopt the accounts immediately
  instead of waiting out the lease TTL. Removed the sync:control-queue hack and
  the supervisor indirection (start/stop_supervisor, sync_on_startup/shutdown).
- Acceptance: graceful-shutdown test proves a peer adopts released accounts within
  seconds under a 30s TTL (i.e. via release, not expiry).

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
This commit is contained in:
2026-07-19 07:18:19 +02:00
co-authored by Claude Opus 4.8
parent 9dbc5ed998
commit 6d83f7cde8
5 changed files with 80 additions and 93 deletions
+15 -1
View File
@@ -202,7 +202,21 @@ class SyncCoordinator:
await self._release(account_id)
async def _heartbeat_once(self) -> None:
payload = json.dumps({"owned": len(self._runners), "ts": time.time()})
"""Publish the worker's load for peer rebalancing + observability (§2)."""
tiers = {"hot": 0, "warm": 0, "cold": 0}
sockets_open = 0
for runner in self._runners.values():
tiers[runner.tier] = tiers.get(runner.tier, 0) + 1
if runner.tier == "hot":
sockets_open += 1 # one persistent IDLE connection per hot account
payload = json.dumps(
{
"owned": len(self._runners),
"sockets_open": sockets_open,
"tiers": tiers,
"ts": time.time(),
}
)
await self._redis.set(leases.worker_key(self.worker_id), payload, ex=self.heartbeat_ttl)
async def _alive_worker_count(self) -> int:
-29
View File
@@ -1,29 +0,0 @@
"""Live-sync entrypoint (refactor §2, step 3).
The sync worker's ``on_startup`` starts a :class:`SyncCoordinator`, which claims
accounts from the ``sync:live`` registry via Redis leases and runs one
:class:`~app.sync.runner.AccountRunner` per owned account. ``on_shutdown`` stops
it, releasing every lease so peers can take the accounts over immediately.
"""
from __future__ import annotations
import logging
from typing import Any
from app.sync.coordinator import SyncCoordinator
logger = logging.getLogger(__name__)
async def start_supervisor(ctx: dict[str, Any]) -> None:
coordinator = SyncCoordinator()
await coordinator.start()
ctx["coordinator"] = coordinator
async def stop_supervisor(ctx: dict[str, Any]) -> None:
coordinator: SyncCoordinator | None = ctx.get("coordinator")
if coordinator is not None:
await coordinator.stop()
ctx["coordinator"] = None