sync-v2 step 7: observability + graceful shutdown (§7)

- Heartbeat JSON now carries {owned, sockets_open, tiers:{hot,warm,cold}, ts} per
  §2 so peers and operators can see each worker's load and socket footprint.
- run.py runs the sync coordinator directly (not via a placeholder arq worker) so
  SIGINT/SIGTERM can stop the claimer, cancel runners, and release every lease
  explicitly within a 10s drain budget — peers adopt the accounts immediately
  instead of waiting out the lease TTL. Removed the sync:control-queue hack and
  the supervisor indirection (start/stop_supervisor, sync_on_startup/shutdown).
- Acceptance: graceful-shutdown test proves a peer adopts released accounts within
  seconds under a 30s TTL (i.e. via release, not expiry).

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
This commit is contained in:
2026-07-19 07:18:19 +02:00
co-authored by Claude Opus 4.8
parent 9dbc5ed998
commit 6d83f7cde8
5 changed files with 80 additions and 93 deletions
+29
View File
@@ -297,6 +297,35 @@ async def test_event_driven_teardown(connections, make_accounts) -> None: # typ
await c.stop()
# --- acceptance 6: graceful shutdown -----------------------------------------
async def test_acceptance_6_graceful_shutdown(connections, make_accounts) -> None: # type: ignore[no-untyped-def]
"""A graceful stop releases every lease so a peer adopts the accounts fast.
Leases are long-lived, so adoption within a couple of seconds proves the
accounts were handed over by explicit release — not by waiting out the TTL.
"""
ids = set(await make_accounts(6))
await _seed(ids) # hot => persistent IDLE
c1 = _fast(SyncCoordinator("w1"))
c2 = _fast(SyncCoordinator("w2"))
c1.lease_ttl = c2.lease_ttl = 30 # adoption must come from release, not expiry
await c1.start()
await _until(lambda: c1.owned == ids) # sole owner
await c2.start() # nothing to claim yet — all leased by w1
try:
await c1.stop() # graceful: stops claimer, cancels runners, releases leases
assert c1.owned == set()
assert await get_redis().exists(leases.worker_key("w1")) == 0 # heartbeat gone
# w2 adopts every account well within the 30s TTL, one connection each.
await _until(lambda: c2.owned == ids, timeout_s=4.0)
await _until(lambda: all(connections[a] == 1 for a in ids))
finally:
await c2.stop()
await _until(lambda: all(connections[a] == 0 for a in ids))
# --- §5 activity tiers -------------------------------------------------------