webmail-client/sync-engine-v2-refactor.md

Sync Engine v2 — Refactor Instructions (replaces SPEC §6.3)

This document is a self-contained repair/rebuild task for the implementing agent. It replaces the current single-supervisor sync design with a sharded, lease-owned, tiered sync engine targeting 1000+ accounts. Read it fully, then execute the migration steps in §9 in order.

0. Ground rules

• This is a workers-layer refactor only. Frozen and untouchable: the MailProvider interface (SPEC §5.1), the database schema (no migration is needed or allowed by this task), the REST/WS API surface, invariant I-1 (API never opens IMAP connections), and the idempotency discipline (UNIQUE (mailbox_id, uid) upserts).
• Existing tests must stay green after every migration step. Each step ends with its own acceptance checks (§10 maps them).
• All new tunables go through config.py (pydantic-settings) — no literals in code. New env vars are listed in §8.
• Redis is already the queue and event bus; it now also carries leases, the live-account set, tier state, and worker heartbeats. No new infrastructure.

1. Process model: two roles

The single worker process splits into two entrypoints (same codebase, WORKER_ROLE env):

• role=jobs — pure arq consumer for the discrete jobs (send, flags, move, drafts, backfill). Stateless, horizontally scalable; run N instances freely.
• role=sync — runs the lease claimer + per-account runners (long-lived loops). Scalable via the lease protocol in §2; run 1..N instances.

Default dev/prod-small topology: 2× sync + 2× jobs. docker-compose gets two services (worker-sync, worker-jobs) sharing the image, each with ulimits: { nofile: 65536 }.

2. Lease protocol (account ownership)

Exactly one sync worker owns each live account at any moment. Ownership is a Redis lease:

• Registry: sync:live — a Redis SET of account ids with state='live'. Maintained by the API on every account state change AND reconciled against Postgres by each sync worker every 300 s (belt and braces; reconcile only adds/removes ids, never touches leases).
• Lease key: sync:lease:{account_id} = {worker_id}, TTL LEASE_TTL (60 s). worker_id = {hostname}:{pid}:{random8} generated at startup.
• Claim: SET sync:lease:{id} {worker_id} NX EX {LEASE_TTL} — success starts the account runner (§4).
• Claimer loop: every 10 s + random jitter 0–5 s, while owned < MAX_ACCOUNTS_PER_WORKER: iterate sync:live (SSCAN), attempt claims on ids with no lease. Jitter prevents thundering-herd on mass restart.
• Renewal: every LEASE_RENEW (20 s), for each owned account run this Lua atomically (never a plain EXPIRE):

  if redis.call('GET', KEYS[1]) == ARGV[1]
  then return redis.call('EXPIRE', KEYS[1], ARGV[2]) else return 0 end
  

• Fencing (hard rule): if renewal returns 0 for an account (lease lost or taken), the worker cancels that account's runner tasks immediately, before any further provider or DB operation for it. In-flight operations are tolerable because all writes are idempotent upserts.
• Release: on graceful stop or account disable, cancel the runner, then delete the lease with the owner-checked Lua (GET==me → DEL).
• Worker registry & rebalance: each sync worker heartbeats sync:workers:{worker_id} = JSON {owned, sockets_open, tiers:{hot,warm,cold}, ts} (TTL 30 s, refresh 10 s). Every 60 s: target = |sync:live| / |alive workers|; if owned > target*1.2 + 5, release the excess, choosing least-recently-active accounts first — released accounts are claimed by under-loaded workers within one claimer cycle.

Failover story (emergent, no extra code): worker crash → leases expire ≤ LEASE_TTL → survivors claim orphans on the next claimer tick. Total re-ownership within ~90 s.

3. Event-driven account lifecycle (kills startup-only discovery + zombie loops)

Sync workers subscribe to the event bus:

• account.state_changed → live (new account finished backfill, or re-enabled): API adds the id to sync:live; some worker claims it within one claimer cycle. No restart, ever.
• account.state_changed → disabled|deleted: API removes the id from sync:live; the owning worker (it can check GET sync:lease:{id}==me cheaply, or simply receives the event and checks ownership) stops the runner, closes connections, releases the lease — target ≤ 30 s from the API call.
• Defense in depth: every runner poll cycle re-verifies the account row exists with state='live'; if not, self-terminate and release. This makes zombie loops structurally impossible even if an event is lost.
• New core event user.active (add to SPEC §8.3 catalog): published by the API on successful login and on every WS connect, payload {user_id}. Consumed for tier promotion (§5).

4. Per-account runner (replaces the fixed 2-task model)

One AccountRunner per owned account, whose task set depends on the current tier:

• hot: task A = INBOX IDLE loop (one persistent connection; re-issue IDLE ≤ 25 min; on wake → poll() INBOX). Task B = secondary CONDSTORE poll of Sent/Drafts/Archive/Trash/Junk every POLL_WARM s on a transient connection.
• warm: no persistent connection. Single task: transient CONDSTORE poll of INBOX + special folders every POLL_WARM s.
• cold: same single task, every POLL_COLD s.

Delta application, UIDVALIDITY handling, expunge soft-handling, and outbound flag write-then-enqueue semantics are unchanged from the current implementation.

Connection budgets:

• ≤ 2 concurrent IMAP connections per account from the sync side (IDLE + one transient); jobs may add short-lived connections but must acquire the same per-account semaphore (cap 4 total, as today).
• Per-destination-host semaphore (process-local), cap PER_HOST_CONN_CAP (default 50): acquired around every IMAP/SMTP connect, so a worker never opens hundreds of simultaneous sockets to one provider. Sharding by account makes process-local sufficient.

5. Activity tiers

Tier state: sync:tier:{account_id} = hot|warm|cold (TTL none; default warm on claim). Evaluated by the owning worker every 120 s and on events:

• → hot when: user.active received for the owning user (all their accounts promote within ≤ 5 s, IDLE established), OR the account received new mail in the last TIER_HOT_ACTIVITY_H (24 h). Session-based heat persists while WS pings continue + TIER_SESSION_GRACE_MIN (30 min) after the last one (API refreshes sess:active:{user_id} TTL; workers just read it).
• → warm when neither hot condition holds and last activity < TIER_COLD_AFTER_D (3 d).
• → cold when no user session and no new mail for ≥ TIER_COLD_AFTER_D days. Any new mail found by a cold poll immediately re-promotes to hot (mail arriving = activity).
• Optional global guard: if a worker's hot count exceeds HOT_BUDGET_PER_WORKER (default 250), the least-recently-active hot accounts without live sessions degrade to warm. Session-hot accounts are never degraded by the budget.

Expected steady state at ~3000 accounts: roughly 15–25 % hot → 450–750 persistent sockets across the pool, the rest on cheap transient polls. Correctness is identical in all tiers — only latency differs (seconds / ≤ POLL_WARM / ≤ POLL_COLD).

6. Queue split (interactive vs bulk)

Two arq queues, consumed by role=jobs workers:

Queue	Jobs	job_timeout	Concurrency per worker
interactive	send_job, push_flags_job, move_job, move_thread_to_folder_job, delete_thread_job, push_draft_to_imap_job, remove_imap_draft_job, refresh_account_job	120 s	20
bulk	sync_bootstrap_account (and future re-backfills)	3600 s	BULK_CONCURRENCY (default 2)

Every jobs worker consumes both queues (arq supports it) with the above per-queue caps — a storm of backfills can no longer delay a send. Retry/backoff and failure-revert semantics per job are unchanged.

7. Backfill memory bounds

• Global backfill parallelism is bounded by the bulk queue concurrency (above) — remove any other ad-hoc caps.
• Per-process parse semaphore PARSE_CONCURRENCY (default 4) around MIME parsing.
• Streaming threshold: a raw message larger than STREAM_PARSE_MB (default 8) is written to the blob store first and parsed from the file handle; smaller messages may parse in memory. Result: peak RAM ≈ PARSE_CONCURRENCY × STREAM_PARSE_MB per process, independent of mailbox sizes.
• sync.progress publication cadence unchanged (per batch).

8. New configuration (add to SPEC §14; all via config.py)

WORKER_ROLE=jobs|sync · MAX_ACCOUNTS_PER_WORKER=400 · LEASE_TTL=60 · LEASE_RENEW=20 · POLL_WARM=180 · POLL_COLD=900 · TIER_HOT_ACTIVITY_H=24 · TIER_SESSION_GRACE_MIN=30 · TIER_COLD_AFTER_D=3 · HOT_BUDGET_PER_WORKER=250 · PER_HOST_CONN_CAP=50 · BULK_CONCURRENCY=2 · PARSE_CONCURRENCY=4 · STREAM_PARSE_MB=8. Remove POLL_INTERVAL (superseded by POLL_WARM/POLL_COLD).

9. Migration steps (execute in order; suite green after each)

1. Roles + queues. Add WORKER_ROLE; move job registration to the two named queues with the §6 routing/timeouts; compose gets worker-sync / worker-jobs services with nofile ulimits. Supervisor behavior untouched.
2. Extract AccountRunner. Refactor the current per-account loops into a self-contained runner class (start/stop/teardown API), still launched by the old startup supervisor. Pure refactor, no behavior change.
3. Lease manager + claimer. Implement §2 (worker_id, claim, Lua renewal, fencing, release, heartbeats, rebalance). Replace the startup supervisor with the claimer. Verify single-worker behavior is equivalent, then run two sync workers against the same accounts.
4. Event-driven lifecycle. Implement §3 including the new user.active event in the API and the per-cycle state re-verification. Delete the startup-only discovery code path.
5. Tiers. Implement §5 tier storage, evaluation, promotions/demotions, and the tier-shaped task sets in AccountRunner.
6. Memory bounds. Implement §7 streaming threshold and semaphores.
7. Observability + graceful shutdown. Structured log fields (worker_id, account_id, tier), heartbeat JSON per §2, SIGTERM handler: stop claimer → cancel runners → release leases → drain (target < 10 s).

10. Acceptance criteria (all must pass; add as automated tests where marked)

1. [test] Two sync workers, 20 mock accounts: at no point do two workers hold the same lease; each account has exactly one IDLE connection (assert via mock provider connection counter).
2. [test] kill -9 one of two sync workers: every orphaned account is re-owned and syncing within 90 s; message table has zero duplicate (mailbox_id, uid) rows afterwards.
3. [test] Create an account via the API: its live loop starts within 30 s without any worker restart. Delete an account: all its connections closed and no further provider calls within 30 s (zombie regression test — mock provider records post-deletion calls, must be zero after the 30 s cutoff).
4. [test] Publish user.active: all of that user's accounts reach hot (IDLE established on mock) within 5 s. After session grace expiry with no activity, they demote to warm on the next evaluation cycle.
5. [test] Queue isolation: enqueue 10 bulk backfills + 20 interactive sends; all sends complete in < 5 s p95 while backfills run.
6. [test] SIGTERM a sync worker: leases released and adopted by the peer within 15 s; no fencing violations logged.
7. [load, manual/CI-nightly] 1000 simulated accounts (mock provider), 20 % with active sessions: steady-state persistent sockets ≤ hot count + 5 %; worker RSS stable over 1 h; oldest-sync-lag < POLL_COLD for every account.
8. Backfilling a mailbox containing a 40 MB message never raises process RSS by more than ~PARSE_CONCURRENCY × STREAM_PARSE_MB above baseline.

11. Out of scope for this task

Cross-worker (Redis-based) per-host connection accounting, pgbouncer deployment, JMAP provider, multi-region operation, per-user rate plans. Note them in docs/open-questions.md if the need is observed.