diff --git a/sync-engine-v2-refactor.md b/sync-engine-v2-refactor.md new file mode 100644 index 0000000..c611b9e --- /dev/null +++ b/sync-engine-v2-refactor.md @@ -0,0 +1,119 @@ +# Sync Engine v2 — Refactor Instructions (replaces SPEC §6.3) + +This document is a self-contained repair/rebuild task for the implementing agent. It replaces the current single-supervisor sync design with a sharded, lease-owned, tiered sync engine targeting **1000+ accounts**. Read it fully, then execute the migration steps in §9 **in order**. + +## 0. Ground rules + +- This is a **workers-layer refactor only**. Frozen and untouchable: the `MailProvider` interface (SPEC §5.1), the database schema (no migration is needed or allowed by this task), the REST/WS API surface, invariant **I-1** (API never opens IMAP connections), and the idempotency discipline (`UNIQUE (mailbox_id, uid)` upserts). +- Existing tests must stay green after every migration step. Each step ends with its own acceptance checks (§10 maps them). +- All new tunables go through `config.py` (pydantic-settings) — no literals in code. New env vars are listed in §8. +- Redis is already the queue and event bus; it now also carries leases, the live-account set, tier state, and worker heartbeats. No new infrastructure. + +## 1. Process model: two roles + +The single worker process splits into two entrypoints (same codebase, `WORKER_ROLE` env): + +- **`role=jobs`** — pure arq consumer for the discrete jobs (send, flags, move, drafts, backfill). Stateless, horizontally scalable; run N instances freely. +- **`role=sync`** — runs the lease claimer + per-account runners (long-lived loops). Scalable via the lease protocol in §2; run 1..N instances. + +Default dev/prod-small topology: 2× sync + 2× jobs. docker-compose gets two services (`worker-sync`, `worker-jobs`) sharing the image, each with `ulimits: { nofile: 65536 }`. + +## 2. Lease protocol (account ownership) + +Exactly one sync worker owns each live account at any moment. Ownership is a Redis lease: + +- **Registry:** `sync:live` — a Redis SET of account ids with `state='live'`. Maintained by the API on every account state change AND reconciled against Postgres by each sync worker every 300 s (belt and braces; reconcile only adds/removes ids, never touches leases). +- **Lease key:** `sync:lease:{account_id}` = `{worker_id}`, TTL `LEASE_TTL` (60 s). `worker_id` = `{hostname}:{pid}:{random8}` generated at startup. +- **Claim:** `SET sync:lease:{id} {worker_id} NX EX {LEASE_TTL}` — success starts the account runner (§4). +- **Claimer loop:** every 10 s + random jitter 0–5 s, while `owned < MAX_ACCOUNTS_PER_WORKER`: iterate `sync:live` (SSCAN), attempt claims on ids with no lease. Jitter prevents thundering-herd on mass restart. +- **Renewal:** every `LEASE_RENEW` (20 s), for each owned account run this Lua atomically (never a plain EXPIRE): + ```lua + if redis.call('GET', KEYS[1]) == ARGV[1] + then return redis.call('EXPIRE', KEYS[1], ARGV[2]) else return 0 end + ``` +- **Fencing (hard rule):** if renewal returns 0 for an account (lease lost or taken), the worker cancels that account's runner tasks **immediately, before any further provider or DB operation for it**. In-flight operations are tolerable because all writes are idempotent upserts. +- **Release:** on graceful stop or account disable, cancel the runner, then delete the lease with the owner-checked Lua (`GET==me → DEL`). +- **Worker registry & rebalance:** each sync worker heartbeats `sync:workers:{worker_id}` = JSON `{owned, sockets_open, tiers:{hot,warm,cold}, ts}` (TTL 30 s, refresh 10 s). Every 60 s: `target = |sync:live| / |alive workers|`; if `owned > target*1.2 + 5`, release the excess, choosing least-recently-active accounts first — released accounts are claimed by under-loaded workers within one claimer cycle. + +**Failover story (emergent, no extra code):** worker crash → leases expire ≤ `LEASE_TTL` → survivors claim orphans on the next claimer tick. Total re-ownership within ~90 s. + +## 3. Event-driven account lifecycle (kills startup-only discovery + zombie loops) + +Sync workers subscribe to the event bus: + +- `account.state_changed → live` (new account finished backfill, or re-enabled): API adds the id to `sync:live`; some worker claims it within one claimer cycle. **No restart, ever.** +- `account.state_changed → disabled|deleted`: API removes the id from `sync:live`; the owning worker (it can check `GET sync:lease:{id}==me` cheaply, or simply receives the event and checks ownership) stops the runner, closes connections, releases the lease — target ≤ 30 s from the API call. +- Defense in depth: every runner poll cycle re-verifies the account row exists with `state='live'`; if not, self-terminate and release. This makes zombie loops structurally impossible even if an event is lost. +- **New core event `user.active`** (add to SPEC §8.3 catalog): published by the API on successful login and on every WS connect, payload `{user_id}`. Consumed for tier promotion (§5). + +## 4. Per-account runner (replaces the fixed 2-task model) + +One `AccountRunner` per owned account, whose task set depends on the current tier: + +- **hot:** task A = INBOX IDLE loop (one persistent connection; re-issue IDLE ≤ 25 min; on wake → `poll()` INBOX). Task B = secondary CONDSTORE poll of Sent/Drafts/Archive/Trash/Junk every `POLL_WARM` s on a transient connection. +- **warm:** no persistent connection. Single task: transient CONDSTORE poll of INBOX + special folders every `POLL_WARM` s. +- **cold:** same single task, every `POLL_COLD` s. + +Delta application, UIDVALIDITY handling, expunge soft-handling, and outbound flag write-then-enqueue semantics are **unchanged** from the current implementation. + +**Connection budgets:** +- ≤ 2 concurrent IMAP connections per account from the sync side (IDLE + one transient); jobs may add short-lived connections but must acquire the same per-account semaphore (cap 4 total, as today). +- Per-destination-host semaphore (process-local), cap `PER_HOST_CONN_CAP` (default 50): acquired around every IMAP/SMTP connect, so a worker never opens hundreds of simultaneous sockets to one provider. Sharding by account makes process-local sufficient. + +## 5. Activity tiers + +Tier state: `sync:tier:{account_id}` = `hot|warm|cold` (TTL none; default `warm` on claim). Evaluated by the owning worker every 120 s and on events: + +- **→ hot** when: `user.active` received for the owning user (all their accounts promote within ≤ 5 s, IDLE established), OR the account received new mail in the last `TIER_HOT_ACTIVITY_H` (24 h). Session-based heat persists while WS pings continue + `TIER_SESSION_GRACE_MIN` (30 min) after the last one (API refreshes `sess:active:{user_id}` TTL; workers just read it). +- **→ warm** when neither hot condition holds and last activity < `TIER_COLD_AFTER_D` (3 d). +- **→ cold** when no user session and no new mail for ≥ `TIER_COLD_AFTER_D` days. Any new mail found by a cold poll immediately re-promotes to hot (mail arriving = activity). +- Optional global guard: if a worker's hot count exceeds `HOT_BUDGET_PER_WORKER` (default 250), the least-recently-active hot accounts without live sessions degrade to warm. Session-hot accounts are never degraded by the budget. + +Expected steady state at ~3000 accounts: roughly 15–25 % hot → 450–750 persistent sockets across the pool, the rest on cheap transient polls. Correctness is identical in all tiers — only latency differs (seconds / ≤ `POLL_WARM` / ≤ `POLL_COLD`). + +## 6. Queue split (interactive vs bulk) + +Two arq queues, consumed by `role=jobs` workers: + +| Queue | Jobs | `job_timeout` | Concurrency per worker | +|---|---|---|---| +| `interactive` | send_job, push_flags_job, move_job, move_thread_to_folder_job, delete_thread_job, push_draft_to_imap_job, remove_imap_draft_job, refresh_account_job | 120 s | 20 | +| `bulk` | sync_bootstrap_account (and future re-backfills) | 3600 s | `BULK_CONCURRENCY` (default 2) | + +Every jobs worker consumes both queues (arq supports it) with the above per-queue caps — a storm of backfills can no longer delay a send. Retry/backoff and failure-revert semantics per job are unchanged. + +## 7. Backfill memory bounds + +- Global backfill parallelism is bounded by the `bulk` queue concurrency (above) — remove any other ad-hoc caps. +- Per-process parse semaphore `PARSE_CONCURRENCY` (default 4) around MIME parsing. +- Streaming threshold: a raw message larger than `STREAM_PARSE_MB` (default 8) is written to the blob store first and parsed from the file handle; smaller messages may parse in memory. Result: peak RAM ≈ `PARSE_CONCURRENCY × STREAM_PARSE_MB` per process, independent of mailbox sizes. +- `sync.progress` publication cadence unchanged (per batch). + +## 8. New configuration (add to SPEC §14; all via `config.py`) + +`WORKER_ROLE=jobs|sync` · `MAX_ACCOUNTS_PER_WORKER=400` · `LEASE_TTL=60` · `LEASE_RENEW=20` · `POLL_WARM=180` · `POLL_COLD=900` · `TIER_HOT_ACTIVITY_H=24` · `TIER_SESSION_GRACE_MIN=30` · `TIER_COLD_AFTER_D=3` · `HOT_BUDGET_PER_WORKER=250` · `PER_HOST_CONN_CAP=50` · `BULK_CONCURRENCY=2` · `PARSE_CONCURRENCY=4` · `STREAM_PARSE_MB=8`. Remove `POLL_INTERVAL` (superseded by POLL_WARM/POLL_COLD). + +## 9. Migration steps (execute in order; suite green after each) + +1. **Roles + queues.** Add `WORKER_ROLE`; move job registration to the two named queues with the §6 routing/timeouts; compose gets `worker-sync` / `worker-jobs` services with `nofile` ulimits. Supervisor behavior untouched. +2. **Extract `AccountRunner`.** Refactor the current per-account loops into a self-contained runner class (start/stop/teardown API), still launched by the old startup supervisor. Pure refactor, no behavior change. +3. **Lease manager + claimer.** Implement §2 (worker_id, claim, Lua renewal, fencing, release, heartbeats, rebalance). Replace the startup supervisor with the claimer. Verify single-worker behavior is equivalent, then run two sync workers against the same accounts. +4. **Event-driven lifecycle.** Implement §3 including the new `user.active` event in the API and the per-cycle state re-verification. Delete the startup-only discovery code path. +5. **Tiers.** Implement §5 tier storage, evaluation, promotions/demotions, and the tier-shaped task sets in `AccountRunner`. +6. **Memory bounds.** Implement §7 streaming threshold and semaphores. +7. **Observability + graceful shutdown.** Structured log fields (`worker_id`, `account_id`, `tier`), heartbeat JSON per §2, SIGTERM handler: stop claimer → cancel runners → release leases → drain (target < 10 s). + +## 10. Acceptance criteria (all must pass; add as automated tests where marked) + +1. **[test]** Two sync workers, 20 mock accounts: at no point do two workers hold the same lease; each account has exactly one IDLE connection (assert via mock provider connection counter). +2. **[test]** `kill -9` one of two sync workers: every orphaned account is re-owned and syncing within 90 s; message table has zero duplicate `(mailbox_id, uid)` rows afterwards. +3. **[test]** Create an account via the API: its live loop starts within 30 s **without any worker restart**. Delete an account: all its connections closed and no further provider calls within 30 s (zombie regression test — mock provider records post-deletion calls, must be zero after the 30 s cutoff). +4. **[test]** Publish `user.active`: all of that user's accounts reach `hot` (IDLE established on mock) within 5 s. After session grace expiry with no activity, they demote to `warm` on the next evaluation cycle. +5. **[test]** Queue isolation: enqueue 10 bulk backfills + 20 interactive sends; all sends complete in < 5 s p95 while backfills run. +6. **[test]** SIGTERM a sync worker: leases released and adopted by the peer within 15 s; no fencing violations logged. +7. **[load, manual/CI-nightly]** 1000 simulated accounts (mock provider), 20 % with active sessions: steady-state persistent sockets ≤ hot count + 5 %; worker RSS stable over 1 h; oldest-sync-lag < `POLL_COLD` for every account. +8. Backfilling a mailbox containing a 40 MB message never raises process RSS by more than ~`PARSE_CONCURRENCY × STREAM_PARSE_MB` above baseline. + +## 11. Out of scope for this task + +Cross-worker (Redis-based) per-host connection accounting, pgbouncer deployment, JMAP provider, multi-region operation, per-user rate plans. Note them in `docs/open-questions.md` if the need is observed. diff --git a/webmail-client-spec_v2.md b/webmail-client-spec_v2.md new file mode 100644 index 0000000..063c121 --- /dev/null +++ b/webmail-client-spec_v2.md @@ -0,0 +1,744 @@ +# Webmail Client — Implementation Specification (v1.3, agent-ready) + +This document is a complete, self-contained build specification. It is written to be handed to an AI coding agent. Read it fully before writing code. The companion document `webmail-client-concept.md` gives background; where they differ, **this spec wins**. + +--- + +## 0. Instructions to the implementing agent + +1. **Build milestone by milestone** (§12), in order. Do not start a milestone before the previous one's acceptance criteria pass. +2. **Do not add features beyond this spec.** Anything not specified here is out of scope; leave `TODO(spec)` comments instead of inventing behavior. +3. **Definition of done** for every task: code + type hints + unit tests + passing `ruff check`, `ruff format --check`, `mypy --strict` (backend) / `tsc --noEmit`, `eslint` (frontend) + updated docs in `docs/` if an interface changed. +4. **Stability over novelty:** use exactly the pinned dependencies in §3. Do not swap libraries or add dependencies without a `# DEPENDENCY:` comment explaining why, and never for something the stdlib does adequately. +5. **All interfaces in §5 are frozen contracts.** Implementations may evolve; signatures may not (without bumping the version comment at the top of the interface file). +6. **Security rules in §10 are hard requirements**, not suggestions. +7. Every module publishing or consuming events must use the event names in §8 verbatim. +8. Commit in small, working increments; every commit must leave the test suite green. + +## 1. Architecture invariants (never violate) + +- **I-1 Sync-first:** the web UI and REST API never open IMAP connections. Only sync workers talk to mail servers. The API reads exclusively from Postgres/Meilisearch. +- **I-2 Independent identity:** login uses app accounts (`localpart@our-web-client.example`) with Argon2id hashes. Mailbox credentials are payload data, encrypted at rest, decrypted only inside sync workers. +- **I-3 One index, many views:** unified inbox, per-account views, virtual mailboxes, and search results are all parameterized queries over the same index. No duplicated storage per view. +- **I-4 Threading is ours:** JWZ in our backend (§7). Never use the IMAP `THREAD` extension. +- **I-5 Direction-agnostic thread UI:** thread ordering (oldest-first/newest-first) is a data-ordering concern. One layout, one composer-anchor slot (§9.4). +- **I-6 Plugins extend, core stays closed:** new features beyond this spec must be expressible through the plugin surfaces (§11). +- **I-7 Multi-user isolation:** every query (SQL and search) carries a mandatory `user_id` filter. No exceptions, including admin tooling. +- **I-8 Mail HTML is hostile input:** sanitize with nh3 server-side at index time AND render in a sandboxed iframe. Both, always. + +## 2. Repository layout (monorepo) + +``` +webmail/ +├── backend/ +│ ├── pyproject.toml +│ ├── app/ +│ │ ├── main.py # FastAPI app factory +│ │ ├── config.py # pydantic-settings; all config via env +│ │ ├── db.py # SQLAlchemy async engine/session +│ │ ├── events.py # EventBus (Redis pub/sub + arq enqueue) +│ │ ├── identity/ # §6.1 auth, sessions, handles, TOTP +│ │ ├── accounts/ # §6.2 wizard, autodiscovery, credential vault +│ │ ├── providers/ +│ │ │ ├── base.py # MailProvider interface (§5.1) +│ │ │ └── imap_smtp/ # first implementation +│ │ ├── sync/ # §6.3 backfill, IDLE loops, reconciliation +│ │ ├── indexer/ # §6.4 MIME parsing → DB rows + blobs +│ │ ├── threading_/ # §7 JWZ (trailing _ avoids stdlib clash) +│ │ ├── search/ +│ │ │ ├── base.py # SearchBackend interface (§5.2) +│ │ │ ├── queryparser.py # operator grammar (§8.2) +│ │ │ ├── pg.py # Postgres FTS backend (M0–M1) +│ │ │ └── meili.py # Meilisearch backend (M2) +│ │ ├── outbox/ # §6.5 drafts, MIME assembly, send queue, undo +│ │ ├── api/ # routers, one file per §9.1 group +│ │ ├── ws.py # WebSocket hub (§9.2) +│ │ └── plugins/ # §11 loader + plugin packages +│ ├── workers/worker.py # arq worker entrypoint +│ ├── migrations/ # alembic +│ └── tests/ +├── frontend/ +│ ├── package.json +│ └── src/ +│ ├── api/ # generated client from OpenAPI + WS layer +│ ├── routes/ # §9.3 +│ ├── components/ +│ │ ├── thread/ThreadView.tsx # §9.4 direction-agnostic +│ │ ├── thread/Composer.tsx +│ │ ├── list/MessageList.tsx # virtualized +│ │ └── slots/ # plugin UI slots (§11) +│ └── lib/ +├── docker/ +│ ├── docker-compose.yml # app, worker, postgres:16, redis:7, meilisearch:v1.8, dovecot (test) +│ └── docker-compose.ci.yml # + greenmail for integration tests +└── docs/ +``` + +## 3. Pinned stack + +**Backend (Python 3.11):** fastapi ~0.115, uvicorn ~0.30, sqlalchemy ~2.0 (async, asyncpg), alembic ~1.13, pydantic ~2.8, pydantic-settings, arq ~0.26, redis ~5.0, aioimaplib ~1.1, imapclient ~3.0 (utility/fallback), aiosmtplib ~3.0, argon2-cffi ~23.1, pyotp ~2.9, dnspython ~2.6, httpx ~0.27, nh3 ~0.2, cryptography ~43.0, mail-parser ~4.0 (helpers only; primary parsing via stdlib `email.message.EmailMessage` with `policy.default`). + +**Frontend:** TypeScript ~5.5, React ~18.3, Vite ~5, TanStack Query ~5, TanStack Virtual ~3, Radix UI, Tailwind ~3.4, zod (WS payload validation). API client generated from OpenAPI (`openapi-typescript`). + +**Infra:** PostgreSQL 16 (with `citext` extension), Redis 7, Meilisearch v1.8 (M2+), MinIO or plain filesystem for the blob store (config-switchable; filesystem default). + +## 4. Database schema (Postgres 16, authoritative DDL) + +All PKs are `uuid DEFAULT gen_random_uuid()`. All tables carrying user data include `user_id` with an index. Timestamps are `timestamptz`. + +```sql +CREATE EXTENSION IF NOT EXISTS citext; + +CREATE TABLE users ( + id uuid PRIMARY KEY DEFAULT gen_random_uuid(), + handle citext UNIQUE NOT NULL, -- full form: localpart@ + password_hash text NOT NULL, -- argon2id + totp_secret bytea, -- encrypted; NULL = 2FA off + recovery_email citext, + settings jsonb NOT NULL DEFAULT '{}', -- incl. {"thread_order":"asc"|"desc"} + created_at timestamptz NOT NULL DEFAULT now() +); +-- Handle rules (enforce in code, mail-grade from day one): +-- localpart: 3–30 chars, [a-z0-9._-], no leading/trailing/double dot, +-- case-insensitive, reserved list (postmaster, abuse, admin, root, noreply, ...). + +CREATE TABLE accounts ( + id uuid PRIMARY KEY DEFAULT gen_random_uuid(), + user_id uuid NOT NULL REFERENCES users(id) ON DELETE CASCADE, + email citext NOT NULL, + display_name text NOT NULL DEFAULT '', + color text NOT NULL DEFAULT '#888888', + imap_config jsonb NOT NULL, -- {host,port,tls:"ssl"|"starttls",auth:"password"|"oauth2"} + smtp_config jsonb NOT NULL, -- same shape + secret_ct bytea NOT NULL, -- envelope-encrypted password or OAuth token bundle (§10.2) + discovery_src text NOT NULL, -- preset|autoconfig|ispdb|srv|probe|manual + state text NOT NULL DEFAULT 'backfilling', -- backfilling|live|error|disabled + state_detail text NOT NULL DEFAULT '', + last_sync_at timestamptz, + created_at timestamptz NOT NULL DEFAULT now(), + UNIQUE (user_id, email) +); + +CREATE TABLE mailboxes ( + id uuid PRIMARY KEY DEFAULT gen_random_uuid(), + account_id uuid NOT NULL REFERENCES accounts(id) ON DELETE CASCADE, + user_id uuid NOT NULL REFERENCES users(id) ON DELETE CASCADE, + path text NOT NULL, -- IMAP mailbox name, UTF-7 decoded + special_use text, -- inbox|sent|drafts|trash|junk|archive|NULL + uidvalidity bigint NOT NULL DEFAULT 0, + highestmodseq bigint, -- NULL if no CONDSTORE + last_seen_uid bigint NOT NULL DEFAULT 0, + UNIQUE (account_id, path) +); + +CREATE TABLE threads ( + id uuid PRIMARY KEY DEFAULT gen_random_uuid(), + user_id uuid NOT NULL REFERENCES users(id) ON DELETE CASCADE, + subject_norm text NOT NULL, -- normalized subject (§7.3) + snippet text NOT NULL DEFAULT '', + participants jsonb NOT NULL DEFAULT '[]', -- [{name,email}], deduped, ordered by recency + account_ids uuid[] NOT NULL DEFAULT '{}', + message_count int NOT NULL DEFAULT 0, + unread_count int NOT NULL DEFAULT 0, + has_attachments boolean NOT NULL DEFAULT false, + last_message_at timestamptz NOT NULL +); +CREATE INDEX ON threads (user_id, last_message_at DESC); + +CREATE TABLE messages ( + id uuid PRIMARY KEY DEFAULT gen_random_uuid(), + user_id uuid NOT NULL REFERENCES users(id) ON DELETE CASCADE, + account_id uuid NOT NULL REFERENCES accounts(id) ON DELETE CASCADE, + mailbox_id uuid NOT NULL REFERENCES mailboxes(id) ON DELETE CASCADE, + uid bigint NOT NULL, -- IMAP UID within mailbox + thread_id uuid REFERENCES threads(id), + message_id_hdr text, -- Message-ID, angle brackets stripped + in_reply_to text, + references_hdr text[] NOT NULL DEFAULT '{}', + subject text NOT NULL DEFAULT '', + from_addr jsonb NOT NULL DEFAULT '[]', + to_addr jsonb NOT NULL DEFAULT '[]', + cc_addr jsonb NOT NULL DEFAULT '[]', + date_hdr timestamptz, + internal_date timestamptz NOT NULL, + flags text[] NOT NULL DEFAULT '{}', -- \Seen,\Flagged,\Answered,\Draft,\Deleted + size_bytes int NOT NULL DEFAULT 0, + has_attachments boolean NOT NULL DEFAULT false, + snippet text NOT NULL DEFAULT '', -- first ~160 chars of text body + blob_sha256 text NOT NULL, -- raw RFC822 in blob store (§6.4) + body_indexed tsvector, -- Postgres FTS (M0–M1 SearchBackend) + UNIQUE (mailbox_id, uid) +); +CREATE INDEX ON messages (user_id, thread_id); +CREATE INDEX ON messages (account_id, message_id_hdr); +CREATE INDEX ON messages USING gin (body_indexed); + +CREATE TABLE body_parts ( + id uuid PRIMARY KEY DEFAULT gen_random_uuid(), + message_id uuid NOT NULL REFERENCES messages(id) ON DELETE CASCADE, + kind text NOT NULL, -- text|html_sanitized + content text NOT NULL +); + +CREATE TABLE attachments ( + id uuid PRIMARY KEY DEFAULT gen_random_uuid(), + message_id uuid NOT NULL REFERENCES messages(id) ON DELETE CASCADE, + user_id uuid NOT NULL, + filename text NOT NULL, + mime_type text NOT NULL, + size_bytes int NOT NULL, + content_id text, -- for inline cid: images + blob_sha256 text NOT NULL +); + +CREATE TABLE drafts ( + id uuid PRIMARY KEY DEFAULT gen_random_uuid(), + user_id uuid NOT NULL REFERENCES users(id) ON DELETE CASCADE, + account_id uuid NOT NULL REFERENCES accounts(id) ON DELETE CASCADE, + thread_id uuid REFERENCES threads(id), -- NULL = new mail + reply_to_message_id uuid REFERENCES messages(id), + kind text NOT NULL DEFAULT 'reply', -- reply|reply_all|forward|new + payload jsonb NOT NULL, -- {to,cc,bcc,subject,html,text,attachment_ids} + updated_at timestamptz NOT NULL DEFAULT now() +); + +CREATE TABLE outbox ( + id uuid PRIMARY KEY DEFAULT gen_random_uuid(), + user_id uuid NOT NULL, + draft_id uuid NOT NULL, + state text NOT NULL DEFAULT 'undo_window', -- undo_window|sending|sent|failed|cancelled + send_after timestamptz NOT NULL, -- now() + undo window + error text +); + +CREATE TABLE saved_searches ( + id uuid PRIMARY KEY DEFAULT gen_random_uuid(), + user_id uuid NOT NULL REFERENCES users(id) ON DELETE CASCADE, + name text NOT NULL, + query text NOT NULL, -- raw operator-syntax string (§8.2) + pinned boolean NOT NULL DEFAULT true, + UNIQUE (user_id, name) +); + +-- Core recipient autocomplete (Gmail-style, derived from mail history). +-- The Contacts plugin (Appendix B) ENRICHES suggestions; this table works without it. +CREATE TABLE known_recipients ( + user_id uuid NOT NULL REFERENCES users(id) ON DELETE CASCADE, + email citext NOT NULL, + name text NOT NULL DEFAULT '', -- last seen display name + times_used int NOT NULL DEFAULT 0, -- weighted: sent-to counts 3, received-from counts 1 + last_used_at timestamptz NOT NULL, + PRIMARY KEY (user_id, email) +); +CREATE INDEX ON known_recipients (user_id, email text_pattern_ops); +``` + +**Blob store:** raw RFC822 messages and attachment payloads are stored content-addressed at `${BLOB_ROOT}/{user_id}/{sha256[:2]}/{sha256}` (filesystem) or the equivalent object key (MinIO). Postgres holds metadata only. Deleting a user removes their blob subtree. + +**Sessions** live in Redis (`sess:{token}` → JSON `{user_id, created, ua}`, TTL 30 days sliding), not Postgres. + +## 5. Frozen core interfaces (Python) + +### 5.1 `providers/base.py` — MailProvider + +```python +# INTERFACE VERSION: 1 +from dataclasses import dataclass +from typing import AsyncIterator, Protocol +from datetime import datetime + +@dataclass(frozen=True) +class RemoteMailbox: + path: str + special_use: str | None # "inbox"|"sent"|"drafts"|"trash"|"junk"|"archive"|None + uidvalidity: int + +@dataclass(frozen=True) +class RemoteMessage: + uid: int + raw: bytes # full RFC822 + flags: frozenset[str] + internal_date: datetime + +@dataclass(frozen=True) +class FlagChange: + uid: int + flags: frozenset[str] + +@dataclass(frozen=True) +class MailboxDelta: + new: list[RemoteMessage] + flag_changes: list[FlagChange] + expunged_uids: list[int] + new_uidvalidity: int | None # non-None => UIDVALIDITY changed, caller must full-resync + new_highestmodseq: int | None + +class MailProvider(Protocol): + async def verify(self) -> None: ... + # Raises ProviderAuthError / ProviderConnectError with a human-readable .detail + + async def list_mailboxes(self) -> list[RemoteMailbox]: ... + + def backfill(self, mailbox: str, *, newest_first: bool = True, + batch_size: int = 200) -> AsyncIterator[list[RemoteMessage]]: ... + # Yields batches until the mailbox is exhausted. Headers+body in one pass. + + async def poll(self, mailbox: str, *, last_seen_uid: int, + highestmodseq: int | None) -> MailboxDelta: ... + # One reconciliation pass. Uses QRESYNC/CONDSTORE when highestmodseq is not None, + # otherwise UID SEARCH + FETCH FLAGS diff. + + async def idle(self, mailbox: str) -> None: ... + # Blocks until the server signals change or 25 min elapse (re-issue IDLE). May raise + # ProviderNotSupported => sync loop falls back to poll() every POLL_INTERVAL (default 120 s). + + async def push_flags(self, mailbox: str, uid: int, + add: frozenset[str], remove: frozenset[str]) -> None: ... + async def move(self, uid: int, src: str, dst: str) -> int: ... # returns new UID + async def append(self, mailbox: str, raw: bytes, + flags: frozenset[str]) -> int: ... # Sent/Drafts upload + async def send(self, raw: bytes, envelope_from: str, + envelope_to: list[str]) -> None: ... # SMTP submission +``` + +`ImapSmtpProvider` is the only M0–M2 implementation. A future `NativeProvider` (hosted mailboxes) or `JmapProvider` implements the same protocol — nothing above this interface may import IMAP libraries. + +### 5.2 `search/base.py` — SearchBackend + +```python +# INTERFACE VERSION: 1 +@dataclass(frozen=True) +class SearchQuery: + text: str | None # free-text remainder after operator extraction + filters: dict[str, object] # parsed operators, keys per §8.2 + user_id: UUID # ALWAYS set (invariant I-7) + +@dataclass(frozen=True) +class SearchHit: + message_id: UUID + thread_id: UUID + score: float + highlights: dict[str, str] # field -> HTML-escaped snippet with + +class SearchBackend(Protocol): + async def index(self, doc: SearchDoc) -> None: ... + async def delete(self, message_id: UUID) -> None: ... + async def delete_user(self, user_id: UUID) -> None: ... + async def query(self, q: SearchQuery, *, limit: int, cursor: str | None + ) -> tuple[list[SearchHit], str | None]: ... +``` + +`SearchDoc` fields: `message_id, thread_id, user_id, account_id, mailbox_special_use, from_ (name+email), to, cc, subject, body_text, attachment_names[], has_attachments, is_unread, is_starred, date (unix), size_bytes`. Both backends (pg.py, meili.py) index the same doc. Meilisearch: one index `messages`, filterable = every non-body field incl. `user_id`, searchable = `subject, body_text, from_, to, cc, attachment_names`, sortable = `date`; ranking rules default + `date:desc` tiebreak. + +### 5.3 `events.py` — EventBus + +```python +# INTERFACE VERSION: 1 +class EventBus(Protocol): + async def publish(self, event: str, payload: dict) -> None: + # Fan-out to Redis pub/sub channel "evt:{event}" AND enqueue arq job + # "on_{event}" for durable consumers (indexer, plugins). + def subscribe(self, event: str, handler: Handler) -> None: ... +``` + +Payloads always include `user_id`. Event names are listed in §8.3 and are the only ones allowed. + +## 6. Backend module specifications + +### 6.1 Identity (`app/identity/`) +- Registration: validate localpart per §4 rules against reserved list in `identity/reserved.py`; store full handle `localpart@{APP_DOMAIN}` (env `APP_DOMAIN`). +- Login: constant-time verify (argon2id, `time_cost=3, memory_cost=64MiB, parallelism=4`); if `totp_secret` set, require valid TOTP (±1 window). Rate limit: 10 attempts / 15 min per handle AND per IP (Redis counters); lockout returns 429 with `Retry-After`. +- Session cookie: `__Host-session`, httpOnly, Secure, SameSite=Lax. CSRF: double-submit token header `X-CSRF` for all mutating requests. +- TOTP setup: `POST /auth/totp/setup` returns provisioning URI + QR payload; enable requires one valid code; secret stored envelope-encrypted. +- Password reset: only if `recovery_email` verified; single-use token, 30 min TTL, Redis-backed. + +### 6.2 Accounts & autodiscovery (`app/accounts/`) +`discover(email) -> list[CandidateConfig]` runs the chain, **returning on first success** but recording every step's outcome in the response (`steps: [{method, status, detail}]`) for wizard UX: + +1. **Presets** — `accounts/presets.yaml` (config, not code): match by domain; ships with gmail.com, googlemail.com, outlook.com/hotmail/live, office365, yahoo, icloud/me.com, gmx, wp.pl, onet.pl, o2.pl, interia.pl. Each entry: imap/smtp host+port+tls, auth kind, oauth provider id if applicable. +2. **Autoconfig** — GET `https://autoconfig.{domain}/mail/config-v1.1.xml?emailaddress={email}` then `https://{domain}/.well-known/autoconfig/mail/config-v1.1.xml`; parse Thunderbird XML (incomingServer type=imap, outgoingServer type=smtp; `%EMAILADDRESS%`/`%EMAILLOCALPART%` substitution). +3. **ISPDB** — GET `https://autoconfig.thunderbird.net/v1.1/{domain}`; same XML format. +4. **DNS SRV (RFC 6186)** — `_imaps._tcp.{domain}` (SSL 993), `_imap._tcp` (STARTTLS 143), `_submissions._tcp` (465), `_submission._tcp` (587); lowest priority wins. +5. **Probe** — try `{imap,mail}.{domain}` × ports 993(ssl)/143(starttls) and `{smtp,mail}.{domain}` × 465(ssl)/587(starttls); TCP connect + TLS handshake + IMAP/SMTP greeting, 5 s timeout each, all concurrently. + +All HTTP/DNS lookups: 5 s timeout, no redirects off-domain, HTTPS only. `test(config, credentials)`: full IMAP LOGIN + `LIST`, and SMTP EHLO+AUTH (no message sent); errors mapped to precise user-facing strings (`auth_failed`, `tls_failed`, `connect_timeout`, `unknown_host`). On save: encrypt credentials (§10.2), create account row (`state=backfilling`), enqueue `sync.bootstrap_account`. + +### 6.3 Sync engine v2 (`app/sync/`) — sharded, lease-owned, tiered +(Authoritative detail: `sync-engine-v2-refactor.md`; this section summarizes the binding rules.) + +- **Two worker roles** (`WORKER_ROLE`): `jobs` = stateless arq consumer, N instances; `sync` = lease claimer + per-account runners, 1..N instances. Compose services `worker-jobs` / `worker-sync`, `nofile` ulimit 65536. +- **Lease ownership:** exactly one sync worker owns each live account. Redis SET `sync:live` (API-maintained, DB-reconciled every 300 s); lease `sync:lease:{account_id}` claimed `SET NX EX LEASE_TTL`, renewed every `LEASE_RENEW` via owner-checked Lua; **fencing** — on renewal failure, cancel that account's runner before any further provider/DB op. Worker heartbeats in `sync:workers:{id}`; rebalance releases excess above `target*1.2+5`. Crash failover = lease expiry + reclaim (≤ ~90 s), no coordinator. +- **Event-driven lifecycle:** `account.state_changed` adds/removes ids from `sync:live` at runtime — accounts go live ≤ 30 s after creation and stop ≤ 30 s after deletion with no restarts; runners re-verify `state='live'` each cycle (zombie-proof). New core event **`user.active`** (API publishes on login/WS connect) drives tier promotion. +- **Bootstrap (unchanged from v1, now a `bulk` job):** `list_mailboxes()` → upsert `mailboxes` (SPECIAL-USE mapping with name heuristics — Sent/Wysłane, Trash/Kosz, etc. in `sync/folder_names.py`); backfill newest-first per mailbox (INBOX first, then sent, then rest); `sync.progress` published per batch; on completion set `state='live'` → the API adds the id to `sync:live` and a sync worker claims it. +- **AccountRunner per tier:** *hot* = persistent INBOX IDLE + transient CONDSTORE poll of special folders every `POLL_WARM`; *warm* = transient poll of INBOX+specials every `POLL_WARM`; *cold* = every `POLL_COLD`. Promotion to hot on `user.active` (≤ 5 s) or new mail within `TIER_HOT_ACTIVITY_H`; demotion by inactivity; `HOT_BUDGET_PER_WORKER` degrades least-active non-session accounts. Delta application, UIDVALIDITY wipe-and-rebackfill, expunge soft-handling, and write-then-enqueue outbound flags are unchanged from v1. +- **Connection budgets:** ≤ 2 sync-side connections per account (4 total incl. jobs, shared semaphore); process-local per-destination-host semaphore cap `PER_HOST_CONN_CAP`. +- **Queues:** `interactive` (send/flags/move/draft jobs, timeout 120 s, concurrency 20) and `bulk` (backfills, timeout 3600 s, concurrency `BULK_CONCURRENCY`) — bulk work can never delay a send. +- **Memory bounds:** parse semaphore `PARSE_CONCURRENCY`; raw messages > `STREAM_PARSE_MB` stream to blob and parse from file — peak RAM ≈ `PARSE_CONCURRENCY × STREAM_PARSE_MB` per process regardless of mailbox size. +- **Shutdown:** SIGTERM → stop claimer → cancel runners → release leases → drain in < 10 s. + +### 6.4 Indexer (`app/indexer/`) +Input: `RemoteMessage`. Steps: store raw blob (sha256, dedupe) → parse with `email` stdlib (`policy.default`) → extract: headers (decode RFC2047), `references_hdr` = parsed `References` + `In-Reply-To` message-ids in order; best text body and best HTML body (multipart/alternative preference: html for display, text for indexing; if only HTML, derive text via nh3 strip); sanitize HTML with nh3 (allowlist: structural tags, inline styles filtered, `cid:` images rewritten to `/api/messages/{id}/cid/{content_id}`, remote `src` rewritten to blocked-placeholder attributes for FR-23) → write `messages`, `body_parts` (kinds `text` and `html_sanitized`), `attachments` → hand to Threader (§7) → `bus.publish("message.new", ...)` (search indexing + thread denormalization are consumers). For non-junk received messages, upsert the `From` address into `known_recipients` (weight 1). + +### 6.5 Outbox (`app/outbox/`) +- Drafts: autosave endpoint upserts `drafts` (client debounces 2 s); a background job mirrors drafts to the account's Drafts folder via `append()` at most every 60 s per draft (replace previous UID). +- Send: `POST /send {draft_id}` → build MIME (html + auto-generated text alternative via html-to-text; `In-Reply-To`/`References` from `reply_to_message_id`; attachments from blob store) → create `outbox` row with `send_after = now()+UNDO_SECONDS` (env, default 10) → schedule arq job at `send_after`. Undo endpoint flips state to `cancelled` if still in window. The job: SMTP `send()`, then `append()` to Sent (skip for Gmail preset — Gmail self-appends), mark `sent`, publish `message.sent`, and upsert every To/Cc/Bcc address into `known_recipients` (weight 3). Failure → `failed` + WS notification, draft preserved. + +## 7. Threading specification (JWZ, `app/threading_/`) + +### 7.1 Normalization +- `message_id_hdr`, `in_reply_to`, each `References` entry: strip angle brackets and whitespace, lowercase the domain part only. Discard obviously invalid ids (no `@`). +- Missing/duplicate `Message-ID`: synthesize `sha256(raw)[:32]@synthetic.local` and mark it non-authoritative (never used to join other messages to it). + +### 7.2 Incremental algorithm (per new message) +1. Candidate set = all normalized ids in `references_hdr` + `in_reply_to` + own `message_id_hdr`. +2. `SELECT DISTINCT thread_id FROM messages WHERE user_id=$1 AND message_id_hdr = ANY($2) AND thread_id IS NOT NULL`. +3. 0 threads → subject fallback (§7.3); still none → create new thread. +4. 1 thread → attach. +5. ≥2 threads → **merge**: keep the thread with the oldest `last_message_at` as canonical, repoint messages of the others, delete emptied thread rows, publish `thread.updated` for all involved. +6. Recompute denormalized thread fields (count, unread, participants, snippet = newest message's snippet, has_attachments, account_ids, last_message_at) in one UPDATE. + +Threads are cross-account by design: the candidate lookup is scoped by `user_id`, not `account_id` (invariant I-3). Sent-folder messages join threads exactly like received ones (FR-15). + +### 7.3 Subject fallback +Only when reference joining fails: normalize subject (strip repeated `Re:/Fw:/Fwd:/Odp:/PD:` prefixes case-insensitively, collapse whitespace); join to a thread with the same `subject_norm` that has activity within 7 days AND shares ≥1 participant email; never join two messages that both lack references AND have empty subjects. + +### 7.4 Test vectors (must pass in `tests/threading/`) +Linear chain; chain arriving out of order; reply missing `In-Reply-To` but carrying `References`; thread merge on late-arriving parent; broken client (subject-only) joins; distinct-topic same-subject ("hi") does NOT join; cross-account merge (message in account A references message in account B); duplicate Message-ID; synthetic-ID isolation. + +## 8. Search specification + +### 8.1 Pipeline +Consumer of `message.new` / `message.changed` / `message.deleted` → builds `SearchDoc` → active `SearchBackend` (env `SEARCH_BACKEND=pg|meili`). Backfill indexes inline with sync (per batch), never as a separate pass the user waits on (FR-30). + +### 8.2 Query grammar (`search/queryparser.py`, engine-independent) +Tokenization: whitespace-split, respecting `"quoted phrases"`. A token matching `key:value` (or `-key:value`) with a known key becomes a filter; everything else joins free text. `-` prefix negates a term or filter. `OR` (uppercase) groups adjacent terms/filters into a disjunction; default combinator is AND. + +| Operator | Filter semantics | +|---|---| +| `from:` `to:` `cc:` | substring match on name or email of that field | +| `subject:` | match constrained to subject | +| `account:` | account by `display_name` or email substring | +| `in:` | `inbox\|sent\|drafts\|trash\|junk\|archive` (special_use) or folder path | +| `is:` | `unread\|read\|starred\|answered` | +| `has:attachment` | boolean | +| `filename:` | attachment name substring | +| `before:` `after:` | `YYYY-MM-DD` | +| `newer_than:` `older_than:` | `Nd\|Nw\|Nm\|Ny` | +| `larger:` `smaller:` | `NK\|NM` bytes | + +Parser output = `SearchQuery` (§5.2). Unknown `key:` tokens are treated as free text (forward compatibility). Unit tests must cover every operator, negation, quoting, `OR`, and garbage input (parser never raises). + +### 8.3 Event catalog (complete) +`message.new`, `message.changed`, `message.deleted`, `message.sent`, `flags.changed`, `thread.updated`, `sync.progress`, `account.state_changed`, `draft.saved`, `outbox.state_changed`, `user.active` (published by the API on login and WS connect; consumed by sync workers for tier promotion — not forwarded to browsers). Payloads: `{user_id, ...ids, minimal-diff fields}` — documented in `docs/events.md` (agent: generate it from code). + +Plugins may additionally publish events namespaced `plugin.{name}.*` (e.g. `plugin.calendar.event_updated`); core code never subscribes to plugin events, but the WS hub forwards them to the owning user like any other event. + +## 9. API, realtime, frontend + +### 9.1 REST API (prefix `/api`, session-cookie auth, JSON; errors: `{error:{code,message}}`) + +**Auth:** `POST /auth/register` · `POST /auth/login` · `POST /auth/logout` · `GET /auth/me` · `GET /auth/sessions` · `DELETE /auth/sessions/{id}` · `POST /auth/totp/setup` · `POST /auth/totp/enable` · `POST /auth/password/reset-request` · `POST /auth/password/reset` + +**Accounts:** `POST /accounts/discover {email}` → `{candidates:[{imap,smtp,source}], steps:[...]}` · `POST /accounts/test {config, credentials}` → `{ok}|{error:{code,detail}}` · `POST /accounts` · `GET /accounts` (incl. `state`, sync progress) · `PATCH /accounts/{id}` · `DELETE /accounts/{id}` (cascades rows, blobs, search docs) + +**Mail:** `GET /threads?view=unified|account:{id}|mailbox:{id}|search&q=&cursor=&limit=50` → thread summaries (denormalized fields only; cursor = `last_message_at,id` keyset) · `GET /threads/{id}?sort=asc|desc` → ordered full messages (bodies of the N newest expanded per client hint `expand=`; older = envelope only) · `POST /threads/{id}/flags {add,remove}` (whole thread) · `POST /messages/{id}/flags` · `POST /messages/{id}/move {mailbox_id}` · `GET /messages/{id}/html` (sanitized, for iframe `srcdoc`) · `GET /messages/{id}/cid/{content_id}` · `GET /attachments/{id}` (Content-Disposition attachment; images/PDF also inline for preview) · `POST /messages/{id}/allow-remote-images` (per-sender allowlist in user settings) + +**Compose:** `POST /drafts` · `PUT /drafts/{id}` · `DELETE /drafts/{id}` · `POST /send {draft_id}` → `{outbox_id, undo_until}` · `POST /send/{outbox_id}/undo` · `GET /recipients/suggest?q=` → top 10 from `known_recipients` (prefix match on email or name, ranked by `times_used` decayed by recency); the composer merges these with any plugin capability providers (§11.3) + +**Search:** `GET /search?q=&cursor=` → `{threads:[...], hits_by_thread, highlights}` (grouped by thread, FR-28; `&group=message` for flat) · `GET/POST/DELETE /saved-searches` + +**Settings:** `GET/PATCH /settings` (incl. `thread_order: "asc"|"desc"`, `tz` IANA timezone — default from browser at registration, undo window, per-sender image allowlist) + +### 9.2 WebSocket `/api/ws` +Auth via session cookie. Server → client frames: `{event, payload}` for every §8.3 event belonging to the user (that's the whole contract — the UI refetches via TanStack Query invalidation keyed on ids in the payload). Client → server: `{type:"ping"}` only. Zod-validate all frames. + +### 9.3 Frontend routes +`/login`, `/register`, `/mail` (unified inbox), `/mail/a/:accountId`, `/mail/a/:accountId/f/:mailboxId`, `/mail/t/:threadId` (thread pane; on desktop rendered beside the list), `/mail/search?q=`, `/settings/{profile|security|accounts|appearance}`; `/settings/accounts/new` = wizard. + +**Account wizard flow:** step 1 email (+ password field revealed unless preset says oauth) → calls `discover` while typing pauses; step 2 shows resolved config with source badge ("found via ISPDB") and a Test button (auto-run); failure expands the manual form pre-filled with the best guess and the precise error; step 3 = created, live backfill progress bar (WS `sync.progress`) with "start reading now" (list fills as backfill proceeds). + +### 9.4 ThreadView contract (invariant I-5) +``` +ThreadView(props: { messages: Message[] /* ASC always */, order: 'asc'|'desc', + composerState, ... }) +``` +- Rendering order = `order==='asc' ? messages : [...messages].reverse()`. No other branching on `order` is allowed in layout code. +- **Composer anchor** = a slot component rendered adjacent to the newest message: after it when `asc` (bottom of thread), directly after it when `desc` (top of thread, above older messages). Same component, one placement rule: "composer is always attached to the newest message." +- Initial scroll: to the newest message (bottom for asc, top for desc). Collapse: all read messages except the newest are collapsed; "Expand all" toggles. Unread messages are never collapsed. +- Keyboard: `j`/`k` move in *visual* order; `o`/`Enter` expand; `r` reply, `a` reply-all, `f` forward (open composer at anchor); `e` archive; `/` focus search; `gi` go to unified inbox; `esc` close composer (draft persists). Toolbar has the per-thread order toggle; it updates local state only, and long-presses/menu offers "make default" (PATCH /settings). +- Message HTML renders in `