webmail-client/sync-engine-v2-refactor.md

# Sync Engine v2 — Refactor Instructions (replaces SPEC §6.3)

This document is a self-contained repair/rebuild task for the implementing agent. It replaces the current single-supervisor sync design with a sharded, lease-owned, tiered sync engine targeting **1000+ accounts**. Read it fully, then execute the migration steps in §9 **in order**.

## 0. Ground rules

- This is a **workers-layer refactor only**. Frozen and untouchable: the `MailProvider` interface (SPEC §5.1), the database schema (no migration is needed or allowed by this task), the REST/WS API surface, invariant **I-1** (API never opens IMAP connections), and the idempotency discipline (`UNIQUE (mailbox_id, uid)` upserts).
- Existing tests must stay green after every migration step. Each step ends with its own acceptance checks (§10 maps them).
- All new tunables go through `config.py` (pydantic-settings) — no literals in code. New env vars are listed in §8.
- Redis is already the queue and event bus; it now also carries leases, the live-account set, tier state, and worker heartbeats. No new infrastructure.

## 1. Process model: two roles

The single worker process splits into two entrypoints (same codebase, `WORKER_ROLE` env):

- **`role=jobs`** — pure arq consumer for the discrete jobs (send, flags, move, drafts, backfill). Stateless, horizontally scalable; run N instances freely.
- **`role=sync`** — runs the lease claimer + per-account runners (long-lived loops). Scalable via the lease protocol in §2; run 1..N instances.

Default dev/prod-small topology: 2× sync + 2× jobs. docker-compose gets two services (`worker-sync`, `worker-jobs`) sharing the image, each with `ulimits: { nofile: 65536 }`.

## 2. Lease protocol (account ownership)

Exactly one sync worker owns each live account at any moment. Ownership is a Redis lease:

- **Registry:** `sync:live` — a Redis SET of account ids with `state='live'`. Maintained by the API on every account state change AND reconciled against Postgres by each sync worker every 300 s (belt and braces; reconcile only adds/removes ids, never touches leases).
- **Lease key:** `sync:lease:{account_id}` = `{worker_id}`, TTL `LEASE_TTL` (60 s). `worker_id` = `{hostname}:{pid}:{random8}` generated at startup.
- **Claim:** `SET sync:lease:{id} {worker_id} NX EX {LEASE_TTL}` — success starts the account runner (§4).
- **Claimer loop:** every 10 s + random jitter 0–5 s, while `owned < MAX_ACCOUNTS_PER_WORKER`: iterate `sync:live` (SSCAN), attempt claims on ids with no lease. Jitter prevents thundering-herd on mass restart.
- **Renewal:** every `LEASE_RENEW` (20 s), for each owned account run this Lua atomically (never a plain EXPIRE):
  ```lua
  if redis.call('GET', KEYS[1]) == ARGV[1]
  then return redis.call('EXPIRE', KEYS[1], ARGV[2]) else return 0 end
  ```
- **Fencing (hard rule):** if renewal returns 0 for an account (lease lost or taken), the worker cancels that account's runner tasks **immediately, before any further provider or DB operation for it**. In-flight operations are tolerable because all writes are idempotent upserts.
- **Release:** on graceful stop or account disable, cancel the runner, then delete the lease with the owner-checked Lua (`GET==me → DEL`).
- **Worker registry & rebalance:** each sync worker heartbeats `sync:workers:{worker_id}` = JSON `{owned, sockets_open, tiers:{hot,warm,cold}, ts}` (TTL 30 s, refresh 10 s). Every 60 s: `target = |sync:live| / |alive workers|`; if `owned > target*1.2 + 5`, release the excess, choosing least-recently-active accounts first — released accounts are claimed by under-loaded workers within one claimer cycle.

**Failover story (emergent, no extra code):** worker crash → leases expire ≤ `LEASE_TTL` → survivors claim orphans on the next claimer tick. Total re-ownership within ~90 s.

## 3. Event-driven account lifecycle (kills startup-only discovery + zombie loops)

Sync workers subscribe to the event bus:

- `account.state_changed → live` (new account finished backfill, or re-enabled): API adds the id to `sync:live`; some worker claims it within one claimer cycle. **No restart, ever.**
- `account.state_changed → disabled|deleted`: API removes the id from `sync:live`; the owning worker (it can check `GET sync:lease:{id}==me` cheaply, or simply receives the event and checks ownership) stops the runner, closes connections, releases the lease — target ≤ 30 s from the API call.
- Defense in depth: every runner poll cycle re-verifies the account row exists with `state='live'`; if not, self-terminate and release. This makes zombie loops structurally impossible even if an event is lost.
- **New core event `user.active`** (add to SPEC §8.3 catalog): published by the API on successful login and on every WS connect, payload `{user_id}`. Consumed for tier promotion (§5).

## 4. Per-account runner (replaces the fixed 2-task model)

One `AccountRunner` per owned account, whose task set depends on the current tier:

- **hot:** task A = INBOX IDLE loop (one persistent connection; re-issue IDLE ≤ 25 min; on wake → `poll()` INBOX). Task B = secondary CONDSTORE poll of Sent/Drafts/Archive/Trash/Junk every `POLL_WARM` s on a transient connection.
- **warm:** no persistent connection. Single task: transient CONDSTORE poll of INBOX + special folders every `POLL_WARM` s.
- **cold:** same single task, every `POLL_COLD` s.

Delta application, UIDVALIDITY handling, expunge soft-handling, and outbound flag write-then-enqueue semantics are **unchanged** from the current implementation.

**Connection budgets:**
- ≤ 2 concurrent IMAP connections per account from the sync side (IDLE + one transient); jobs may add short-lived connections but must acquire the same per-account semaphore (cap 4 total, as today).
- Per-destination-host semaphore (process-local), cap `PER_HOST_CONN_CAP` (default 50): acquired around every IMAP/SMTP connect, so a worker never opens hundreds of simultaneous sockets to one provider. Sharding by account makes process-local sufficient.

## 5. Activity tiers

Tier state: `sync:tier:{account_id}` = `hot|warm|cold` (TTL none; default `warm` on claim). Evaluated by the owning worker every 120 s and on events:

- **→ hot** when: `user.active` received for the owning user (all their accounts promote within ≤ 5 s, IDLE established), OR the account received new mail in the last `TIER_HOT_ACTIVITY_H` (24 h). Session-based heat persists while WS pings continue + `TIER_SESSION_GRACE_MIN` (30 min) after the last one (API refreshes `sess:active:{user_id}` TTL; workers just read it).
- **→ warm** when neither hot condition holds and last activity < `TIER_COLD_AFTER_D` (3 d).
- **→ cold** when no user session and no new mail for ≥ `TIER_COLD_AFTER_D` days. Any new mail found by a cold poll immediately re-promotes to hot (mail arriving = activity).
- Optional global guard: if a worker's hot count exceeds `HOT_BUDGET_PER_WORKER` (default 250), the least-recently-active hot accounts without live sessions degrade to warm. Session-hot accounts are never degraded by the budget.

Expected steady state at ~3000 accounts: roughly 15–25 % hot → 450–750 persistent sockets across the pool, the rest on cheap transient polls. Correctness is identical in all tiers — only latency differs (seconds / ≤ `POLL_WARM` / ≤ `POLL_COLD`).

## 6. Queue split (interactive vs bulk)

Two arq queues, consumed by `role=jobs` workers:

| Queue | Jobs | `job_timeout` | Concurrency per worker |
|---|---|---|---|
| `interactive` | send_job, push_flags_job, move_job, move_thread_to_folder_job, delete_thread_job, push_draft_to_imap_job, remove_imap_draft_job, refresh_account_job | 120 s | 20 |
| `bulk` | sync_bootstrap_account (and future re-backfills) | 3600 s | `BULK_CONCURRENCY` (default 2) |

Every jobs worker consumes both queues (arq supports it) with the above per-queue caps — a storm of backfills can no longer delay a send. Retry/backoff and failure-revert semantics per job are unchanged.

## 7. Backfill memory bounds

- Global backfill parallelism is bounded by the `bulk` queue concurrency (above) — remove any other ad-hoc caps.
- Per-process parse semaphore `PARSE_CONCURRENCY` (default 4) around MIME parsing.
- Streaming threshold: a raw message larger than `STREAM_PARSE_MB` (default 8) is written to the blob store first and parsed from the file handle; smaller messages may parse in memory. Result: peak RAM ≈ `PARSE_CONCURRENCY × STREAM_PARSE_MB` per process, independent of mailbox sizes.
- `sync.progress` publication cadence unchanged (per batch).

## 8. New configuration (add to SPEC §14; all via `config.py`)

`WORKER_ROLE=jobs|sync` · `MAX_ACCOUNTS_PER_WORKER=400` · `LEASE_TTL=60` · `LEASE_RENEW=20` · `POLL_WARM=180` · `POLL_COLD=900` · `TIER_HOT_ACTIVITY_H=24` · `TIER_SESSION_GRACE_MIN=30` · `TIER_COLD_AFTER_D=3` · `HOT_BUDGET_PER_WORKER=250` · `PER_HOST_CONN_CAP=50` · `BULK_CONCURRENCY=2` · `PARSE_CONCURRENCY=4` · `STREAM_PARSE_MB=8`. Remove `POLL_INTERVAL` (superseded by POLL_WARM/POLL_COLD).

## 9. Migration steps (execute in order; suite green after each)

1. **Roles + queues.** Add `WORKER_ROLE`; move job registration to the two named queues with the §6 routing/timeouts; compose gets `worker-sync` / `worker-jobs` services with `nofile` ulimits. Supervisor behavior untouched.
2. **Extract `AccountRunner`.** Refactor the current per-account loops into a self-contained runner class (start/stop/teardown API), still launched by the old startup supervisor. Pure refactor, no behavior change.
3. **Lease manager + claimer.** Implement §2 (worker_id, claim, Lua renewal, fencing, release, heartbeats, rebalance). Replace the startup supervisor with the claimer. Verify single-worker behavior is equivalent, then run two sync workers against the same accounts.
4. **Event-driven lifecycle.** Implement §3 including the new `user.active` event in the API and the per-cycle state re-verification. Delete the startup-only discovery code path.
5. **Tiers.** Implement §5 tier storage, evaluation, promotions/demotions, and the tier-shaped task sets in `AccountRunner`.
6. **Memory bounds.** Implement §7 streaming threshold and semaphores.
7. **Observability + graceful shutdown.** Structured log fields (`worker_id`, `account_id`, `tier`), heartbeat JSON per §2, SIGTERM handler: stop claimer → cancel runners → release leases → drain (target < 10 s).

## 10. Acceptance criteria (all must pass; add as automated tests where marked)

1. **[test]** Two sync workers, 20 mock accounts: at no point do two workers hold the same lease; each account has exactly one IDLE connection (assert via mock provider connection counter).
2. **[test]** `kill -9` one of two sync workers: every orphaned account is re-owned and syncing within 90 s; message table has zero duplicate `(mailbox_id, uid)` rows afterwards.
3. **[test]** Create an account via the API: its live loop starts within 30 s **without any worker restart**. Delete an account: all its connections closed and no further provider calls within 30 s (zombie regression test — mock provider records post-deletion calls, must be zero after the 30 s cutoff).
4. **[test]** Publish `user.active`: all of that user's accounts reach `hot` (IDLE established on mock) within 5 s. After session grace expiry with no activity, they demote to `warm` on the next evaluation cycle.
5. **[test]** Queue isolation: enqueue 10 bulk backfills + 20 interactive sends; all sends complete in < 5 s p95 while backfills run.
6. **[test]** SIGTERM a sync worker: leases released and adopted by the peer within 15 s; no fencing violations logged.
7. **[load, manual/CI-nightly]** 1000 simulated accounts (mock provider), 20 % with active sessions: steady-state persistent sockets ≤ hot count + 5 %; worker RSS stable over 1 h; oldest-sync-lag < `POLL_COLD` for every account.
8. Backfilling a mailbox containing a 40 MB message never raises process RSS by more than ~`PARSE_CONCURRENCY × STREAM_PARSE_MB` above baseline.

## 11. Out of scope for this task

Cross-worker (Redis-based) per-host connection accounting, pgbouncer deployment, JMAP provider, multi-region operation, per-user rate plans. Note them in `docs/open-questions.md` if the need is observed.