webmail-client/webmail-client-spec_v2.md

Webmail Client — Implementation Specification (v1.3, agent-ready)

This document is a complete, self-contained build specification. It is written to be handed to an AI coding agent. Read it fully before writing code. The companion document webmail-client-concept.md gives background; where they differ, this spec wins.

---

0. Instructions to the implementing agent

1. Build milestone by milestone (§12), in order. Do not start a milestone before the previous one's acceptance criteria pass.
2. Do not add features beyond this spec. Anything not specified here is out of scope; leave TODO(spec) comments instead of inventing behavior.
3. Definition of done for every task: code + type hints + unit tests + passing ruff check, ruff format --check, mypy --strict (backend) / tsc --noEmit, eslint (frontend) + updated docs in docs/ if an interface changed.
4. Stability over novelty: use exactly the pinned dependencies in §3. Do not swap libraries or add dependencies without a # DEPENDENCY: comment explaining why, and never for something the stdlib does adequately.
5. All interfaces in §5 are frozen contracts. Implementations may evolve; signatures may not (without bumping the version comment at the top of the interface file).
6. Security rules in §10 are hard requirements, not suggestions.
7. Every module publishing or consuming events must use the event names in §8 verbatim.
8. Commit in small, working increments; every commit must leave the test suite green.

1. Architecture invariants (never violate)

• I-1 Sync-first: the web UI and REST API never open IMAP connections. Only sync workers talk to mail servers. The API reads exclusively from Postgres/Meilisearch.
• I-2 Independent identity: login uses app accounts (localpart@our-web-client.example) with Argon2id hashes. Mailbox credentials are payload data, encrypted at rest, decrypted only inside sync workers.
• I-3 One index, many views: unified inbox, per-account views, virtual mailboxes, and search results are all parameterized queries over the same index. No duplicated storage per view.
• I-4 Threading is ours: JWZ in our backend (§7). Never use the IMAP THREAD extension.
• I-5 Direction-agnostic thread UI: thread ordering (oldest-first/newest-first) is a data-ordering concern. One layout, one composer-anchor slot (§9.4).
• I-6 Plugins extend, core stays closed: new features beyond this spec must be expressible through the plugin surfaces (§11).
• I-7 Multi-user isolation: every query (SQL and search) carries a mandatory user_id filter. No exceptions, including admin tooling.
• I-8 Mail HTML is hostile input: sanitize with nh3 server-side at index time AND render in a sandboxed iframe. Both, always.

2. Repository layout (monorepo)

webmail/
├── backend/
│   ├── pyproject.toml
│   ├── app/
│   │   ├── main.py                  # FastAPI app factory
│   │   ├── config.py                # pydantic-settings; all config via env
│   │   ├── db.py                    # SQLAlchemy async engine/session
│   │   ├── events.py                # EventBus (Redis pub/sub + arq enqueue)
│   │   ├── identity/                # §6.1 auth, sessions, handles, TOTP
│   │   ├── accounts/                # §6.2 wizard, autodiscovery, credential vault
│   │   ├── providers/
│   │   │   ├── base.py              # MailProvider interface (§5.1)
│   │   │   └── imap_smtp/           # first implementation
│   │   ├── sync/                    # §6.3 backfill, IDLE loops, reconciliation
│   │   ├── indexer/                 # §6.4 MIME parsing → DB rows + blobs
│   │   ├── threading_/              # §7 JWZ (trailing _ avoids stdlib clash)
│   │   ├── search/
│   │   │   ├── base.py              # SearchBackend interface (§5.2)
│   │   │   ├── queryparser.py       # operator grammar (§8.2)
│   │   │   ├── pg.py                # Postgres FTS backend (M0–M1)
│   │   │   └── meili.py             # Meilisearch backend (M2)
│   │   ├── outbox/                  # §6.5 drafts, MIME assembly, send queue, undo
│   │   ├── api/                     # routers, one file per §9.1 group
│   │   ├── ws.py                    # WebSocket hub (§9.2)
│   │   └── plugins/                 # §11 loader + plugin packages
│   ├── workers/worker.py            # arq worker entrypoint
│   ├── migrations/                  # alembic
│   └── tests/
├── frontend/
│   ├── package.json
│   └── src/
│       ├── api/                     # generated client from OpenAPI + WS layer
│       ├── routes/                  # §9.3
│       ├── components/
│       │   ├── thread/ThreadView.tsx    # §9.4 direction-agnostic
│       │   ├── thread/Composer.tsx
│       │   ├── list/MessageList.tsx     # virtualized
│       │   └── slots/               # plugin UI slots (§11)
│       └── lib/
├── docker/
│   ├── docker-compose.yml           # app, worker, postgres:16, redis:7, meilisearch:v1.8, dovecot (test)
│   └── docker-compose.ci.yml        # + greenmail for integration tests
└── docs/

3. Pinned stack

Backend (Python 3.11): fastapi ~0.115, uvicorn ~0.30, sqlalchemy ~2.0 (async, asyncpg), alembic ~1.13, pydantic ~2.8, pydantic-settings, arq ~0.26, redis ~5.0, aioimaplib ~1.1, imapclient ~3.0 (utility/fallback), aiosmtplib ~3.0, argon2-cffi ~23.1, pyotp ~2.9, dnspython ~2.6, httpx ~0.27, nh3 ~0.2, cryptography ~43.0, mail-parser ~4.0 (helpers only; primary parsing via stdlib email.message.EmailMessage with policy.default).

Frontend: TypeScript ~5.5, React ~18.3, Vite ~5, TanStack Query ~5, TanStack Virtual ~3, Radix UI, Tailwind ~3.4, zod (WS payload validation). API client generated from OpenAPI (openapi-typescript).

Infra: PostgreSQL 16 (with citext extension), Redis 7, Meilisearch v1.8 (M2+), MinIO or plain filesystem for the blob store (config-switchable; filesystem default).

4. Database schema (Postgres 16, authoritative DDL)

All PKs are uuid DEFAULT gen_random_uuid(). All tables carrying user data include user_id with an index. Timestamps are timestamptz.

CREATE EXTENSION IF NOT EXISTS citext;

CREATE TABLE users (
  id            uuid PRIMARY KEY DEFAULT gen_random_uuid(),
  handle        citext UNIQUE NOT NULL,        -- full form: localpart@<APP_DOMAIN>
  password_hash text   NOT NULL,               -- argon2id
  totp_secret   bytea,                         -- encrypted; NULL = 2FA off
  recovery_email citext,
  settings      jsonb  NOT NULL DEFAULT '{}',  -- incl. {"thread_order":"asc"|"desc"}
  created_at    timestamptz NOT NULL DEFAULT now()
);
-- Handle rules (enforce in code, mail-grade from day one):
--   localpart: 3–30 chars, [a-z0-9._-], no leading/trailing/double dot,
--   case-insensitive, reserved list (postmaster, abuse, admin, root, noreply, ...).

CREATE TABLE accounts (
  id             uuid PRIMARY KEY DEFAULT gen_random_uuid(),
  user_id        uuid NOT NULL REFERENCES users(id) ON DELETE CASCADE,
  email          citext NOT NULL,
  display_name   text NOT NULL DEFAULT '',
  color          text NOT NULL DEFAULT '#888888',
  imap_config    jsonb NOT NULL,   -- {host,port,tls:"ssl"|"starttls",auth:"password"|"oauth2"}
  smtp_config    jsonb NOT NULL,   -- same shape
  secret_ct      bytea NOT NULL,   -- envelope-encrypted password or OAuth token bundle (§10.2)
  discovery_src  text  NOT NULL,   -- preset|autoconfig|ispdb|srv|probe|manual
  state          text  NOT NULL DEFAULT 'backfilling',  -- backfilling|live|error|disabled
  state_detail   text  NOT NULL DEFAULT '',
  last_sync_at   timestamptz,
  created_at     timestamptz NOT NULL DEFAULT now(),
  UNIQUE (user_id, email)
);

CREATE TABLE mailboxes (
  id             uuid PRIMARY KEY DEFAULT gen_random_uuid(),
  account_id     uuid NOT NULL REFERENCES accounts(id) ON DELETE CASCADE,
  user_id        uuid NOT NULL REFERENCES users(id) ON DELETE CASCADE,
  path           text NOT NULL,                 -- IMAP mailbox name, UTF-7 decoded
  special_use    text,                          -- inbox|sent|drafts|trash|junk|archive|NULL
  uidvalidity    bigint NOT NULL DEFAULT 0,
  highestmodseq  bigint,                        -- NULL if no CONDSTORE
  last_seen_uid  bigint NOT NULL DEFAULT 0,
  UNIQUE (account_id, path)
);

CREATE TABLE threads (
  id              uuid PRIMARY KEY DEFAULT gen_random_uuid(),
  user_id         uuid NOT NULL REFERENCES users(id) ON DELETE CASCADE,
  subject_norm    text NOT NULL,                -- normalized subject (§7.3)
  snippet         text NOT NULL DEFAULT '',
  participants    jsonb NOT NULL DEFAULT '[]',  -- [{name,email}], deduped, ordered by recency
  account_ids     uuid[] NOT NULL DEFAULT '{}',
  message_count   int NOT NULL DEFAULT 0,
  unread_count    int NOT NULL DEFAULT 0,
  has_attachments boolean NOT NULL DEFAULT false,
  last_message_at timestamptz NOT NULL
);
CREATE INDEX ON threads (user_id, last_message_at DESC);

CREATE TABLE messages (
  id              uuid PRIMARY KEY DEFAULT gen_random_uuid(),
  user_id         uuid NOT NULL REFERENCES users(id) ON DELETE CASCADE,
  account_id      uuid NOT NULL REFERENCES accounts(id) ON DELETE CASCADE,
  mailbox_id      uuid NOT NULL REFERENCES mailboxes(id) ON DELETE CASCADE,
  uid             bigint NOT NULL,              -- IMAP UID within mailbox
  thread_id       uuid REFERENCES threads(id),
  message_id_hdr  text,                         -- Message-ID, angle brackets stripped
  in_reply_to     text,
  references_hdr  text[] NOT NULL DEFAULT '{}',
  subject         text NOT NULL DEFAULT '',
  from_addr       jsonb NOT NULL DEFAULT '[]',
  to_addr         jsonb NOT NULL DEFAULT '[]',
  cc_addr         jsonb NOT NULL DEFAULT '[]',
  date_hdr        timestamptz,
  internal_date   timestamptz NOT NULL,
  flags           text[] NOT NULL DEFAULT '{}', -- \Seen,\Flagged,\Answered,\Draft,\Deleted
  size_bytes      int NOT NULL DEFAULT 0,
  has_attachments boolean NOT NULL DEFAULT false,
  snippet         text NOT NULL DEFAULT '',     -- first ~160 chars of text body
  blob_sha256     text NOT NULL,                -- raw RFC822 in blob store (§6.4)
  body_indexed    tsvector,                     -- Postgres FTS (M0–M1 SearchBackend)
  UNIQUE (mailbox_id, uid)
);
CREATE INDEX ON messages (user_id, thread_id);
CREATE INDEX ON messages (account_id, message_id_hdr);
CREATE INDEX ON messages USING gin (body_indexed);

CREATE TABLE body_parts (
  id          uuid PRIMARY KEY DEFAULT gen_random_uuid(),
  message_id  uuid NOT NULL REFERENCES messages(id) ON DELETE CASCADE,
  kind        text NOT NULL,        -- text|html_sanitized
  content     text NOT NULL
);

CREATE TABLE attachments (
  id          uuid PRIMARY KEY DEFAULT gen_random_uuid(),
  message_id  uuid NOT NULL REFERENCES messages(id) ON DELETE CASCADE,
  user_id     uuid NOT NULL,
  filename    text NOT NULL,
  mime_type   text NOT NULL,
  size_bytes  int  NOT NULL,
  content_id  text,                 -- for inline cid: images
  blob_sha256 text NOT NULL
);

CREATE TABLE drafts (
  id          uuid PRIMARY KEY DEFAULT gen_random_uuid(),
  user_id     uuid NOT NULL REFERENCES users(id) ON DELETE CASCADE,
  account_id  uuid NOT NULL REFERENCES accounts(id) ON DELETE CASCADE,
  thread_id   uuid REFERENCES threads(id),      -- NULL = new mail
  reply_to_message_id uuid REFERENCES messages(id),
  kind        text NOT NULL DEFAULT 'reply',    -- reply|reply_all|forward|new
  payload     jsonb NOT NULL,       -- {to,cc,bcc,subject,html,text,attachment_ids}
  updated_at  timestamptz NOT NULL DEFAULT now()
);

CREATE TABLE outbox (
  id          uuid PRIMARY KEY DEFAULT gen_random_uuid(),
  user_id     uuid NOT NULL,
  draft_id    uuid NOT NULL,
  state       text NOT NULL DEFAULT 'undo_window',  -- undo_window|sending|sent|failed|cancelled
  send_after  timestamptz NOT NULL,                 -- now() + undo window
  error       text
);

CREATE TABLE saved_searches (
  id       uuid PRIMARY KEY DEFAULT gen_random_uuid(),
  user_id  uuid NOT NULL REFERENCES users(id) ON DELETE CASCADE,
  name     text NOT NULL,
  query    text NOT NULL,           -- raw operator-syntax string (§8.2)
  pinned   boolean NOT NULL DEFAULT true,
  UNIQUE (user_id, name)
);

-- Core recipient autocomplete (Gmail-style, derived from mail history).
-- The Contacts plugin (Appendix B) ENRICHES suggestions; this table works without it.
CREATE TABLE known_recipients (
  user_id      uuid NOT NULL REFERENCES users(id) ON DELETE CASCADE,
  email        citext NOT NULL,
  name         text NOT NULL DEFAULT '',      -- last seen display name
  times_used   int NOT NULL DEFAULT 0,        -- weighted: sent-to counts 3, received-from counts 1
  last_used_at timestamptz NOT NULL,
  PRIMARY KEY (user_id, email)
);
CREATE INDEX ON known_recipients (user_id, email text_pattern_ops);

Blob store: raw RFC822 messages and attachment payloads are stored content-addressed at ${BLOB_ROOT}/{user_id}/{sha256[:2]}/{sha256} (filesystem) or the equivalent object key (MinIO). Postgres holds metadata only. Deleting a user removes their blob subtree.

Sessions live in Redis (sess:{token} → JSON {user_id, created, ua}, TTL 30 days sliding), not Postgres.

5. Frozen core interfaces (Python)

5.1 providers/base.py — MailProvider

# INTERFACE VERSION: 1
from dataclasses import dataclass
from typing import AsyncIterator, Protocol
from datetime import datetime

@dataclass(frozen=True)
class RemoteMailbox:
    path: str
    special_use: str | None      # "inbox"|"sent"|"drafts"|"trash"|"junk"|"archive"|None
    uidvalidity: int

@dataclass(frozen=True)
class RemoteMessage:
    uid: int
    raw: bytes                   # full RFC822
    flags: frozenset[str]
    internal_date: datetime

@dataclass(frozen=True)
class FlagChange:
    uid: int
    flags: frozenset[str]

@dataclass(frozen=True)
class MailboxDelta:
    new: list[RemoteMessage]
    flag_changes: list[FlagChange]
    expunged_uids: list[int]
    new_uidvalidity: int | None  # non-None => UIDVALIDITY changed, caller must full-resync
    new_highestmodseq: int | None

class MailProvider(Protocol):
    async def verify(self) -> None: ...
        # Raises ProviderAuthError / ProviderConnectError with a human-readable .detail

    async def list_mailboxes(self) -> list[RemoteMailbox]: ...

    def backfill(self, mailbox: str, *, newest_first: bool = True,
                 batch_size: int = 200) -> AsyncIterator[list[RemoteMessage]]: ...
        # Yields batches until the mailbox is exhausted. Headers+body in one pass.

    async def poll(self, mailbox: str, *, last_seen_uid: int,
                   highestmodseq: int | None) -> MailboxDelta: ...
        # One reconciliation pass. Uses QRESYNC/CONDSTORE when highestmodseq is not None,
        # otherwise UID SEARCH + FETCH FLAGS diff.

    async def idle(self, mailbox: str) -> None: ...
        # Blocks until the server signals change or 25 min elapse (re-issue IDLE). May raise
        # ProviderNotSupported => sync loop falls back to poll() every POLL_INTERVAL (default 120 s).

    async def push_flags(self, mailbox: str, uid: int,
                         add: frozenset[str], remove: frozenset[str]) -> None: ...
    async def move(self, uid: int, src: str, dst: str) -> int: ...   # returns new UID
    async def append(self, mailbox: str, raw: bytes,
                     flags: frozenset[str]) -> int: ...               # Sent/Drafts upload
    async def send(self, raw: bytes, envelope_from: str,
                   envelope_to: list[str]) -> None: ...               # SMTP submission

ImapSmtpProvider is the only M0–M2 implementation. A future NativeProvider (hosted mailboxes) or JmapProvider implements the same protocol — nothing above this interface may import IMAP libraries.

5.2 search/base.py — SearchBackend

# INTERFACE VERSION: 1
@dataclass(frozen=True)
class SearchQuery:
    text: str | None                      # free-text remainder after operator extraction
    filters: dict[str, object]            # parsed operators, keys per §8.2
    user_id: UUID                         # ALWAYS set (invariant I-7)

@dataclass(frozen=True)
class SearchHit:
    message_id: UUID
    thread_id: UUID
    score: float
    highlights: dict[str, str]            # field -> HTML-escaped snippet with <mark>

class SearchBackend(Protocol):
    async def index(self, doc: SearchDoc) -> None: ...
    async def delete(self, message_id: UUID) -> None: ...
    async def delete_user(self, user_id: UUID) -> None: ...
    async def query(self, q: SearchQuery, *, limit: int, cursor: str | None
                    ) -> tuple[list[SearchHit], str | None]: ...

SearchDoc fields: message_id, thread_id, user_id, account_id, mailbox_special_use, from_ (name+email), to, cc, subject, body_text, attachment_names[], has_attachments, is_unread, is_starred, date (unix), size_bytes. Both backends (pg.py, meili.py) index the same doc. Meilisearch: one index messages, filterable = every non-body field incl. user_id, searchable = subject, body_text, from_, to, cc, attachment_names, sortable = date; ranking rules default + date:desc tiebreak.

5.3 events.py — EventBus

# INTERFACE VERSION: 1
class EventBus(Protocol):
    async def publish(self, event: str, payload: dict) -> None:
        # Fan-out to Redis pub/sub channel "evt:{event}" AND enqueue arq job
        # "on_{event}" for durable consumers (indexer, plugins).
    def subscribe(self, event: str, handler: Handler) -> None: ...

Payloads always include user_id. Event names are listed in §8.3 and are the only ones allowed.

6. Backend module specifications

6.1 Identity (app/identity/)

• Registration: validate localpart per §4 rules against reserved list in identity/reserved.py; store full handle localpart@{APP_DOMAIN} (env APP_DOMAIN).
• Login: constant-time verify (argon2id, time_cost=3, memory_cost=64MiB, parallelism=4); if totp_secret set, require valid TOTP (±1 window). Rate limit: 10 attempts / 15 min per handle AND per IP (Redis counters); lockout returns 429 with Retry-After.
• Session cookie: __Host-session, httpOnly, Secure, SameSite=Lax. CSRF: double-submit token header X-CSRF for all mutating requests.
• TOTP setup: POST /auth/totp/setup returns provisioning URI + QR payload; enable requires one valid code; secret stored envelope-encrypted.
• Password reset: only if recovery_email verified; single-use token, 30 min TTL, Redis-backed.

6.2 Accounts & autodiscovery (app/accounts/)

discover(email) -> list[CandidateConfig] runs the chain, returning on first success but recording every step's outcome in the response (steps: [{method, status, detail}]) for wizard UX:

1. Presets — accounts/presets.yaml (config, not code): match by domain; ships with gmail.com, googlemail.com, outlook.com/hotmail/live, office365, yahoo, icloud/me.com, gmx, wp.pl, onet.pl, o2.pl, interia.pl. Each entry: imap/smtp host+port+tls, auth kind, oauth provider id if applicable.
2. Autoconfig — GET https://autoconfig.{domain}/mail/config-v1.1.xml?emailaddress={email} then https://{domain}/.well-known/autoconfig/mail/config-v1.1.xml; parse Thunderbird XML (incomingServer type=imap, outgoingServer type=smtp; %EMAILADDRESS%/%EMAILLOCALPART% substitution).
3. ISPDB — GET https://autoconfig.thunderbird.net/v1.1/{domain}; same XML format.
4. DNS SRV (RFC 6186) — _imaps._tcp.{domain} (SSL 993), _imap._tcp (STARTTLS 143), _submissions._tcp (465), _submission._tcp (587); lowest priority wins.
5. Probe — try {imap,mail}.{domain} × ports 993(ssl)/143(starttls) and {smtp,mail}.{domain} × 465(ssl)/587(starttls); TCP connect + TLS handshake + IMAP/SMTP greeting, 5 s timeout each, all concurrently.

All HTTP/DNS lookups: 5 s timeout, no redirects off-domain, HTTPS only. test(config, credentials): full IMAP LOGIN + LIST, and SMTP EHLO+AUTH (no message sent); errors mapped to precise user-facing strings (auth_failed, tls_failed, connect_timeout, unknown_host). On save: encrypt credentials (§10.2), create account row (state=backfilling), enqueue sync.bootstrap_account.

6.3 Sync engine v2 (app/sync/) — sharded, lease-owned, tiered

(Authoritative detail: sync-engine-v2-refactor.md; this section summarizes the binding rules.)

• Two worker roles (WORKER_ROLE): jobs = stateless arq consumer, N instances; sync = lease claimer + per-account runners, 1..N instances. Compose services worker-jobs / worker-sync, nofile ulimit 65536.
• Lease ownership: exactly one sync worker owns each live account. Redis SET sync:live (API-maintained, DB-reconciled every 300 s); lease sync:lease:{account_id} claimed SET NX EX LEASE_TTL, renewed every LEASE_RENEW via owner-checked Lua; fencing — on renewal failure, cancel that account's runner before any further provider/DB op. Worker heartbeats in sync:workers:{id}; rebalance releases excess above target*1.2+5. Crash failover = lease expiry + reclaim (≤ ~90 s), no coordinator.
• Event-driven lifecycle: account.state_changed adds/removes ids from sync:live at runtime — accounts go live ≤ 30 s after creation and stop ≤ 30 s after deletion with no restarts; runners re-verify state='live' each cycle (zombie-proof). New core event user.active (API publishes on login/WS connect) drives tier promotion.
• Bootstrap (unchanged from v1, now a bulk job): list_mailboxes() → upsert mailboxes (SPECIAL-USE mapping with name heuristics — Sent/Wysłane, Trash/Kosz, etc. in sync/folder_names.py); backfill newest-first per mailbox (INBOX first, then sent, then rest); sync.progress published per batch; on completion set state='live' → the API adds the id to sync:live and a sync worker claims it.
• AccountRunner per tier: hot = persistent INBOX IDLE + transient CONDSTORE poll of special folders every POLL_WARM; warm = transient poll of INBOX+specials every POLL_WARM; cold = every POLL_COLD. Promotion to hot on user.active (≤ 5 s) or new mail within TIER_HOT_ACTIVITY_H; demotion by inactivity; HOT_BUDGET_PER_WORKER degrades least-active non-session accounts. Delta application, UIDVALIDITY wipe-and-rebackfill, expunge soft-handling, and write-then-enqueue outbound flags are unchanged from v1.
• Connection budgets: ≤ 2 sync-side connections per account (4 total incl. jobs, shared semaphore); process-local per-destination-host semaphore cap PER_HOST_CONN_CAP.
• Queues: interactive (send/flags/move/draft jobs, timeout 120 s, concurrency 20) and bulk (backfills, timeout 3600 s, concurrency BULK_CONCURRENCY) — bulk work can never delay a send.
• Memory bounds: parse semaphore PARSE_CONCURRENCY; raw messages > STREAM_PARSE_MB stream to blob and parse from file — peak RAM ≈ PARSE_CONCURRENCY × STREAM_PARSE_MB per process regardless of mailbox size.
• Shutdown: SIGTERM → stop claimer → cancel runners → release leases → drain in < 10 s.

6.4 Indexer (app/indexer/)

Input: RemoteMessage. Steps: store raw blob (sha256, dedupe) → parse with email stdlib (policy.default) → extract: headers (decode RFC2047), references_hdr = parsed References + In-Reply-To message-ids in order; best text body and best HTML body (multipart/alternative preference: html for display, text for indexing; if only HTML, derive text via nh3 strip); sanitize HTML with nh3 (allowlist: structural tags, inline styles filtered, cid: images rewritten to /api/messages/{id}/cid/{content_id}, remote src rewritten to blocked-placeholder attributes for FR-23) → write messages, body_parts (kinds text and html_sanitized), attachments → hand to Threader (§7) → bus.publish("message.new", ...) (search indexing + thread denormalization are consumers). For non-junk received messages, upsert the From address into known_recipients (weight 1).

6.5 Outbox (app/outbox/)

• Drafts: autosave endpoint upserts drafts (client debounces 2 s); a background job mirrors drafts to the account's Drafts folder via append() at most every 60 s per draft (replace previous UID).
• Send: POST /send {draft_id} → build MIME (html + auto-generated text alternative via html-to-text; In-Reply-To/References from reply_to_message_id; attachments from blob store) → create outbox row with send_after = now()+UNDO_SECONDS (env, default 10) → schedule arq job at send_after. Undo endpoint flips state to cancelled if still in window. The job: SMTP send(), then append() to Sent (skip for Gmail preset — Gmail self-appends), mark sent, publish message.sent, and upsert every To/Cc/Bcc address into known_recipients (weight 3). Failure → failed + WS notification, draft preserved.

7. Threading specification (JWZ, app/threading_/)

7.1 Normalization

• message_id_hdr, in_reply_to, each References entry: strip angle brackets and whitespace, lowercase the domain part only. Discard obviously invalid ids (no @).
• Missing/duplicate Message-ID: synthesize sha256(raw)[:32]@synthetic.local and mark it non-authoritative (never used to join other messages to it).

7.2 Incremental algorithm (per new message)

1. Candidate set = all normalized ids in references_hdr + in_reply_to + own message_id_hdr.
2. SELECT DISTINCT thread_id FROM messages WHERE user_id=$1 AND message_id_hdr = ANY($2) AND thread_id IS NOT NULL.
3. 0 threads → subject fallback (§7.3); still none → create new thread.
4. 1 thread → attach.
5. ≥2 threads → merge: keep the thread with the oldest last_message_at as canonical, repoint messages of the others, delete emptied thread rows, publish thread.updated for all involved.
6. Recompute denormalized thread fields (count, unread, participants, snippet = newest message's snippet, has_attachments, account_ids, last_message_at) in one UPDATE.

Threads are cross-account by design: the candidate lookup is scoped by user_id, not account_id (invariant I-3). Sent-folder messages join threads exactly like received ones (FR-15).

7.3 Subject fallback

Only when reference joining fails: normalize subject (strip repeated Re:/Fw:/Fwd:/Odp:/PD: prefixes case-insensitively, collapse whitespace); join to a thread with the same subject_norm that has activity within 7 days AND shares ≥1 participant email; never join two messages that both lack references AND have empty subjects.

7.4 Test vectors (must pass in tests/threading/)

Linear chain; chain arriving out of order; reply missing In-Reply-To but carrying References; thread merge on late-arriving parent; broken client (subject-only) joins; distinct-topic same-subject ("hi") does NOT join; cross-account merge (message in account A references message in account B); duplicate Message-ID; synthetic-ID isolation.

8. Search specification

8.1 Pipeline

Consumer of message.new / message.changed / message.deleted → builds SearchDoc → active SearchBackend (env SEARCH_BACKEND=pg|meili). Backfill indexes inline with sync (per batch), never as a separate pass the user waits on (FR-30).

8.2 Query grammar (search/queryparser.py, engine-independent)

Tokenization: whitespace-split, respecting "quoted phrases". A token matching key:value (or -key:value) with a known key becomes a filter; everything else joins free text. - prefix negates a term or filter. OR (uppercase) groups adjacent terms/filters into a disjunction; default combinator is AND.

Operator	Filter semantics
from: to: cc:	substring match on name or email of that field
subject:	match constrained to subject
account:	account by display_name or email substring
in:	inbox|sent|drafts|trash|junk|archive (special_use) or folder path
is:	unread|read|starred|answered
has:attachment	boolean
filename:	attachment name substring
before: after:	YYYY-MM-DD
newer_than: older_than:	Nd|Nw|Nm|Ny
larger: smaller:	NK|NM bytes

Parser output = SearchQuery (§5.2). Unknown key: tokens are treated as free text (forward compatibility). Unit tests must cover every operator, negation, quoting, OR, and garbage input (parser never raises).

8.3 Event catalog (complete)

message.new, message.changed, message.deleted, message.sent, flags.changed, thread.updated, sync.progress, account.state_changed, draft.saved, outbox.state_changed, user.active (published by the API on login and WS connect; consumed by sync workers for tier promotion — not forwarded to browsers). Payloads: {user_id, ...ids, minimal-diff fields} — documented in docs/events.md (agent: generate it from code).

Plugins may additionally publish events namespaced plugin.{name}.* (e.g. plugin.calendar.event_updated); core code never subscribes to plugin events, but the WS hub forwards them to the owning user like any other event.

9. API, realtime, frontend

9.1 REST API (prefix /api, session-cookie auth, JSON; errors: {error:{code,message}})

Auth: POST /auth/register · POST /auth/login · POST /auth/logout · GET /auth/me · GET /auth/sessions · DELETE /auth/sessions/{id} · POST /auth/totp/setup · POST /auth/totp/enable · POST /auth/password/reset-request · POST /auth/password/reset

Accounts: POST /accounts/discover {email} → {candidates:[{imap,smtp,source}], steps:[...]} · POST /accounts/test {config, credentials} → {ok}|{error:{code,detail}} · POST /accounts · GET /accounts (incl. state, sync progress) · PATCH /accounts/{id} · DELETE /accounts/{id} (cascades rows, blobs, search docs)

Mail: GET /threads?view=unified|account:{id}|mailbox:{id}|search&q=&cursor=&limit=50 → thread summaries (denormalized fields only; cursor = last_message_at,id keyset) · GET /threads/{id}?sort=asc|desc → ordered full messages (bodies of the N newest expanded per client hint expand=; older = envelope only) · POST /threads/{id}/flags {add,remove} (whole thread) · POST /messages/{id}/flags · POST /messages/{id}/move {mailbox_id} · GET /messages/{id}/html (sanitized, for iframe srcdoc) · GET /messages/{id}/cid/{content_id} · GET /attachments/{id} (Content-Disposition attachment; images/PDF also inline for preview) · POST /messages/{id}/allow-remote-images (per-sender allowlist in user settings)

Compose: POST /drafts · PUT /drafts/{id} · DELETE /drafts/{id} · POST /send {draft_id} → {outbox_id, undo_until} · POST /send/{outbox_id}/undo · GET /recipients/suggest?q= → top 10 from known_recipients (prefix match on email or name, ranked by times_used decayed by recency); the composer merges these with any plugin capability providers (§11.3)

Search: GET /search?q=&cursor= → {threads:[...], hits_by_thread, highlights} (grouped by thread, FR-28; &group=message for flat) · GET/POST/DELETE /saved-searches

Settings: GET/PATCH /settings (incl. thread_order: "asc"|"desc", tz IANA timezone — default from browser at registration, undo window, per-sender image allowlist)

9.2 WebSocket /api/ws

Auth via session cookie. Server → client frames: {event, payload} for every §8.3 event belonging to the user (that's the whole contract — the UI refetches via TanStack Query invalidation keyed on ids in the payload). Client → server: {type:"ping"} only. Zod-validate all frames.

9.3 Frontend routes

/login, /register, /mail (unified inbox), /mail/a/:accountId, /mail/a/:accountId/f/:mailboxId, /mail/t/:threadId (thread pane; on desktop rendered beside the list), /mail/search?q=, /settings/{profile|security|accounts|appearance}; /settings/accounts/new = wizard.

Account wizard flow: step 1 email (+ password field revealed unless preset says oauth) → calls discover while typing pauses; step 2 shows resolved config with source badge ("found via ISPDB") and a Test button (auto-run); failure expands the manual form pre-filled with the best guess and the precise error; step 3 = created, live backfill progress bar (WS sync.progress) with "start reading now" (list fills as backfill proceeds).

9.4 ThreadView contract (invariant I-5)

ThreadView(props: { messages: Message[] /* ASC always */, order: 'asc'|'desc',
                    composerState, ... })

• Rendering order = order==='asc' ? messages : [...messages].reverse(). No other branching on order is allowed in layout code.
• Composer anchor = a slot component rendered adjacent to the newest message: after it when asc (bottom of thread), directly after it when desc (top of thread, above older messages). Same component, one placement rule: "composer is always attached to the newest message."
• Initial scroll: to the newest message (bottom for asc, top for desc). Collapse: all read messages except the newest are collapsed; "Expand all" toggles. Unread messages are never collapsed.
• Keyboard: j/k move in visual order; o/Enter expand; r reply, a reply-all, f forward (open composer at anchor); e archive; / focus search; gi go to unified inbox; esc close composer (draft persists). Toolbar has the per-thread order toggle; it updates local state only, and long-presses/menu offers "make default" (PATCH /settings).
• Message HTML renders in <iframe sandbox="" srcdoc={...}> sized via ResizeObserver postMessage shim; remote images show a "load images / always from this sender" bar when the sanitizer flagged any.

10. Security requirements (hard)

1. Argon2id params per §6.1; no other password hashing anywhere.
2. Credential vault: mailbox secrets encrypted with AES-256-GCM under per-account data keys, wrapped by a master key from env MASTER_KEY (32 B, base64) — cryptography library, key id embedded for future rotation. Decryption code path exists only in workers/ and accounts/test. Secrets never appear in logs, API responses, or exceptions (custom repr).
3. CSRF double-submit on all mutating routes; __Host- cookie prefix; CSP: default-src 'self'; frame-src 'self'; img-src 'self' data: (remote mail images are proxied? No — v1 loads allowed remote images inside the sandboxed iframe only, which has its own relaxed CSP via srcdoc; document this in code).
4. nh3 sanitization at index time; re-sanitize on render if sanitizer version changed (store sanitizer_v on body_parts).
5. Rate limits: login (§6.1), POST /accounts/discover 10/min/user, POST /send 30/h/user (env-tunable).
6. All outbound autodiscovery HTTP: HTTPS only, 5 s timeout, response size cap 256 KB, no off-domain redirects (SSRF guard: resolve + reject private IP ranges).
7. Every SQL query and search query includes user_id (enforced by repository-layer helpers; raw session use in API code is a review error).
8. Uploads (attachments to drafts): 25 MB/file, 40 MB/message total, content-type sniffed server-side, stored to blob store, never executed/served with original content type unless in safe list (images/pdf/text) — others force application/octet-stream + attachment disposition.

11. Plugin API (freeze in M3 — including the heavy-plugin surfaces below)

A plugin = Python package webmail_plugin_<name> exposing:

plugin = Plugin(
    name, version,
    hooks       = {event: handler},          # subscribe to §8.3 events
    router      = APIRouter | None,          # mounted at /api/plugins/{name}
    ui_manifest = UiManifest | None,         # slots + pages, see below
    migrations  = "migrations/" | None,      # 11.1
    jobs        = [JobSpec, ...] | None,     # 11.2
)

Loader discovers plugins via entry points group webmail.plugins, forwards declared events, and serves GET /api/plugins/ui-manifest for the frontend loader. Plugins may not import from app.* internals other than app.plugins.sdk (which exposes: DB session factory scoped to plugin tables, EventBus publish under plugin.{name}.*, blob store, outbox draft/send helpers, settings accessor, HTTP client with the §10.6 SSRF guard built in).

Every plugin gets the namespaced KV store (plugin_kv(plugin, user_id, key)) for light state. Heavy plugins additionally get:

11.1 Plugin-owned tables

A plugin may ship Alembic migrations in its own version branch (alembic -n plugin_{name}). All tables MUST be prefixed plg_{name}_ and every user-data table MUST carry user_id (invariant I-7 applies to plugins). Core migrations never reference plugin tables and vice versa. Uninstalling a plugin offers --drop-data to run its downgrade chain.

11.2 Plugin background jobs

JobSpec(func, schedule: cron_str | None, singleton: bool) registers tasks on the shared arq worker: cron-scheduled (e.g. a sync loop trigger) or enqueued on demand from hooks/routes. singleton=True guarantees at most one running instance per (plugin, user) via a Redis lock. Plugin jobs run in the same worker process pool with the same retry semantics as core jobs and are visible in worker logs under plg.{name}.*.

11.3 Frontend surfaces

• Slots (lazy-loaded ESM injected into mail UI): thread.toolbar, message.actions, sidebar.section, settings.page.
• Pages (app.page): a plugin may declare full views: {route: "/calendar", nav: {icon, label, order}, bundle}. The shell adds the nav entry, and the route (including sub-routes under it) lazy-loads the bundle into the entire content area — no mail list/thread components are rendered. Page bundles receive the SDK context (API client, WS event stream filtered to plugin.{name}.* + requested core events, design-system components) and are sandboxed to their route: importing mail-component internals is a review error. Removing the plugin removes the nav entry and route cleanly.
• Capabilities: the manifest may declare named capabilities mapping to plugin endpoints, e.g. capabilities: {"recipients.suggest": "/api/plugins/contacts/suggest"}. Core UI features query all providers of a capability they support and merge results client-side (defined capabilities in v1: recipients.suggest — used by the composer To/Cc/Bcc fields and by any other plugin needing person-picking, such as calendar attendees). This keeps core backend code plugin-unaware.

Acceptance test for the API: implement filters (rules on message.new: match → move/flag) and labels entirely as plugins using only slots + KV; then implement the calendar (Appendix A) using tables + jobs + a page — all with zero core diffs.

12. Milestones with acceptance criteria

M0 — Architecture spike Login/register (no 2FA), one hardcoded-config IMAP account, backfill+IDLE for INBOX, JWZ, read-only unified list + thread view with working asc/desc toggle and correct composer-anchor placeholder, Postgres FTS free-text search. ✅ Accept: sync a real 10k-message Gmail account via app password; threads match Gmail's grouping on a 50-thread sample ≥ 90%; toggle order and the newest message + anchor swap ends correctly; search returns in < 500 ms.

M1 — MVP Full identity (2FA, sessions UI, reset), account wizard with full discovery chain, multi-account + unified inbox with account chips, flags/move/archive both directions, inline reply + reply-all/forward + SMTP send + Sent append, recipient autocomplete from known_recipients, drafts autosave, sanitized HTML + blocked remote images, operator search (full §8.2 grammar) on PG FTS, WS live updates. ✅ Accept: add gmail.com, outlook.com, and one autoconfig-only domain through the wizard typing only address+password; reply from a thread lands correctly threaded in the recipient's client; all §8.2 operators pass integration tests; docker-compose up from clean checkout works.

M2 — Hardening / v1.0 OAuth2 (Google + Microsoft, XOAUTH2 for IMAP/SMTP, server-side refresh), QRESYNC fast resync, drafts mirrored to server, undo-send, attachment upload/preview, Meilisearch backend + instant search-as-you-type UI + saved searches as virtual mailboxes, session revocation, backfill progress UX, folder-name i18n heuristics. ✅ Accept: typo query (invocie) finds "invoice" mail < 200 ms on a 100k-message corpus; kill -9 the worker mid-backfill and restart resumes without duplicates (idempotency via (mailbox_id, uid) unique key); undo-send within window results in no SMTP traffic.

M3 — Plugin platform Loader, full SDK per §11 including plugin migrations (11.1), plugin jobs (11.2), and app.page (11.3); filters + labels shipped as plugins; plugin docs. ✅ Accept: both plugins install/uninstall cleanly with zero core diffs; a third trivial plugin ("Unsubscribe header button" on message.actions) can be written from docs/plugins.md alone; a skeleton page-plugin (route + nav entry + one plugin table + one cron job) round-trips install → run → uninstall --drop-data leaving no residue.

M4 — Calendar plugin (Appendix A) Phases A1–A2: local calendars + full page views + iMIP invite handling; then A3: CalDAV client sync + ICS subscriptions. ✅ Accept: criteria in Appendix A §A.7.

M5 — Contacts plugin (Appendix B) Phases B1–B2: contact books, full page, groups, add-from-mail, recipients.suggest capability, vCard import/export; then B3: CardDAV client sync. May be built in parallel with M4 — both depend only on the M3 SDK. ✅ Accept: criteria in Appendix B §B.7.

M6 (optional) — Hosted mailboxes Inbound MTA (Stalwart) delivering to a NativeProvider; handle becomes receiving address. Specified separately when reached.

13. Testing & CI

• Unit: pytest + pytest-asyncio; threading vectors (§7.4); query parser (§8.2); autodiscovery resolver with respx-mocked HTTP + dnspython stub; MIME corpus in tests/fixtures/mime/ (agent: include ≥ 25 nasty real-world-style fixtures — encoded headers incl. RFC2047 Polish diacritics, nested multiparts, missing Message-ID, malformed dates, HTML-only, inline cid images, 8-bit subjects, winmail.dat placeholder).
• Integration: docker-compose.ci.yml with Greenmail (IMAP+SMTP) and Dovecot; full loop test: append message to Greenmail → appears in API < 10 s → reply via API → arrives in Greenmail SMTP sink with correct References.
• Frontend: vitest + React Testing Library; ThreadView ordering/anchor tests for both directions; Playwright smoke: login → add Greenmail account via wizard → read → inline reply → search.
• CI gates: ruff, mypy --strict, pytest, tsc, eslint, vitest, Playwright smoke; coverage floor 80% on threading_, search/queryparser, accounts/autodiscovery, indexer.

14. Configuration (env, complete list)

APP_DOMAIN, DATABASE_URL, REDIS_URL, MASTER_KEY, BLOB_ROOT or S3_*, SEARCH_BACKEND=pg|meili, MEILI_URL, MEILI_KEY, UNDO_SECONDS=10, SESSION_TTL_DAYS=30, OAUTH_GOOGLE_CLIENT_ID/SECRET, OAUTH_MS_CLIENT_ID/SECRET, MAX_ATTACHMENT_MB=25, LOG_LEVEL; sync engine v2 (§6.3): WORKER_ROLE=jobs|sync, MAX_ACCOUNTS_PER_WORKER=400, LEASE_TTL=60, LEASE_RENEW=20, POLL_WARM=180, POLL_COLD=900, TIER_HOT_ACTIVITY_H=24, TIER_SESSION_GRACE_MIN=30, TIER_COLD_AFTER_D=3, HOT_BUDGET_PER_WORKER=250, PER_HOST_CONN_CAP=50, BULK_CONCURRENCY=2, PARSE_CONCURRENCY=4, STREAM_PARSE_MB=8. All read once in config.py (pydantic-settings); no os.environ elsewhere.

---

Appendix A — Calendar plugin (webmail_plugin_calendar)

The calendar is the reference "heavy plugin": it lives in the same app and repo, uses the §11 SDK only, and must build with zero core diffs. It ships as a page (/calendar) beside Mail — one product, one login.

A.1 Scope by phase

• A1 — Local calendar: create/edit/delete events, multiple calendars per user, month/week/day/agenda views, recurrence, reminders (WS notification), timezone-correct display.
• A2 — Invites over mail (iMIP): detect text/calendar parts on message.new; render an invite card in the thread (message.actions + inline summary); Accept/Tentative/Decline sends an iTIP REPLY through the outbox; organizer flow sends REQUEST/CANCEL to attendees; updates (SEQUENCE bumps) modify the stored event.
• A3 — External sources: CalendarProvider interface with CalDAV client (first implementation; covers iCloud, Fastmail, Nextcloud) and read-only ICS URL subscriptions (holidays, team feeds); sync loop as a plugin job.
• A4 (deferred, spec later): Google/Microsoft API providers reusing M2 OAuth machinery; secret ICS export feed GET /api/plugins/calendar/feed/{token}.ics; CalDAV server endpoint for native phone sync — architecturally anticipated, not built now.

A.2 Dependencies

icalendar ~5.0 (parse/serialize RFC 5545), python-dateutil (RRULE expansion), caldav ~1.3 (A3), tzdata. No other additions.

A.3 Data model (plugin migrations, prefix plg_calendar_)

CREATE TABLE plg_calendar_calendars (
  id uuid PRIMARY KEY DEFAULT gen_random_uuid(),
  user_id uuid NOT NULL,
  name text NOT NULL,
  color text NOT NULL DEFAULT '#3b82f6',
  kind text NOT NULL DEFAULT 'local',     -- local|caldav|ics_subscription
  source_config jsonb NOT NULL DEFAULT '{}',  -- caldav url/collection, ics url
  secret_ct bytea,                        -- caldav credentials, §10.2 vault scheme
  sync_token text,                        -- CalDAV sync-token / ICS etag
  read_only boolean NOT NULL DEFAULT false,
  UNIQUE (user_id, name)
);

CREATE TABLE plg_calendar_events (
  id uuid PRIMARY KEY DEFAULT gen_random_uuid(),
  user_id uuid NOT NULL,
  calendar_id uuid NOT NULL REFERENCES plg_calendar_calendars(id) ON DELETE CASCADE,
  ical_uid text NOT NULL,                 -- UID from iCalendar; join key for iMIP updates
  sequence int NOT NULL DEFAULT 0,
  summary text NOT NULL DEFAULT '',
  description text NOT NULL DEFAULT '',
  location text NOT NULL DEFAULT '',
  dtstart timestamptz NOT NULL,
  dtend timestamptz,
  all_day boolean NOT NULL DEFAULT false,
  tz text NOT NULL DEFAULT 'UTC',         -- original TZID; render in user tz, store UTC
  rrule text,                             -- raw RRULE string, NULL = single
  exdates timestamptz[] NOT NULL DEFAULT '{}',
  recurrence_id timestamptz,              -- non-NULL = detached exception instance
  organizer jsonb,                        -- {name,email}
  attendees jsonb NOT NULL DEFAULT '[]',  -- [{name,email,partstat,role}]
  status text NOT NULL DEFAULT 'confirmed',   -- confirmed|tentative|cancelled
  my_partstat text,                       -- needs-action|accepted|declined|tentative
  reminders int[] NOT NULL DEFAULT '{}',  -- minutes before start
  origin_message_id uuid,                 -- link back to the invite mail thread
  raw_ical text,                          -- round-trip fidelity for iMIP/CalDAV
  UNIQUE (calendar_id, ical_uid, recurrence_id)
);
CREATE INDEX ON plg_calendar_events (user_id, dtstart);

Recurrence rule: store the RRULE string; expand instances at query time with dateutil.rrule over the requested view range only (no materialization table). Exceptions: exdates remove instances; a row with recurrence_id overrides one instance. Range queries must account for events starting before the window but recurring into it (expand from dtstart, cap iterations at 1000 per event).

A.4 CalendarProvider interface (inside the plugin, mirrors §5.1)

# PLUGIN INTERFACE VERSION: 1
class CalendarProvider(Protocol):
    async def verify(self) -> None: ...
    async def list_collections(self) -> list[RemoteCalendar]: ...
    async def pull(self, collection: str, *, sync_token: str | None
                   ) -> tuple[list[RemoteEvent], list[str], str]: ...
        # returns (changed_events_as_ical, deleted_uids, new_sync_token)
    async def push(self, collection: str, ical: str, *, uid: str,
                   etag: str | None) -> str: ...        # returns new etag
    async def delete(self, collection: str, uid: str, etag: str | None) -> None: ...

Implementations in A3: CaldavProvider, IcsSubscriptionProvider (pull-only, push/delete raise ReadOnlyCalendar). Sync loop = plugin cron job (schedule="*/5 * * * *", singleton=True per user) doing pull → upsert by (calendar_id, ical_uid, recurrence_id) → local-dirty push with etag conflict handling (conflict ⇒ server wins, local copy saved as a duplicate marked status='tentative', user notified).

A.5 iMIP flow (A2)

• Inbound: hook on message.new; if a text/calendar part exists, parse METHOD: REQUEST → upsert event (match ical_uid; apply only if SEQUENCE ≥ stored) into the user's default calendar with my_partstat='needs-action', set origin_message_id; CANCEL → status='cancelled'; REPLY → update that attendee's partstat on the organizer's copy.
• RSVP: invite card buttons call POST /api/plugins/calendar/events/{id}/rsvp {partstat} → update row → build iTIP REPLY ics → send via SDK outbox helper to the organizer (subject Accepted: {summary} etc.) → publish plugin.calendar.event_updated.
• Organizer: creating an event with attendees sends METHOD:REQUEST to each; edits bump SEQUENCE and re-send; deletion sends CANCEL. All outgoing invite mail uses the account whose address matches the organizer identity (picker if ambiguous).

A.6 UI (page /calendar, nav icon between Mail and Settings)

Routes: /calendar (month default), /calendar/{month|week|day|agenda}/:date, /calendar/event/:id, /calendar/settings (calendars CRUD, CalDAV/ICS wizard reusing the account-wizard UX pattern: URL+credentials → test → collection picker). Interactions: click-drag create, drag-move/resize (writes optimistically, reverts on API error), event popover with edit/RSVP, mini-month navigator, calendar visibility toggles (client state), t = today, arrows = prev/next period. Invite linkage both ways: invite card in thread links to /calendar/event/:id; event view links back to the mail thread when origin_message_id is set. Live updates via WS plugin.calendar.* events. Timezone: render in user setting settings.tz (add to §9.1 settings; default from browser at registration).

A.7 Acceptance criteria

1. Plugin installs/uninstalls (--drop-data) with zero diffs in app/ and frontend/src/ outside plugins//slots/ loader code.
2. A weekly-recurring event with two EXDATEs and one moved instance renders correctly in month and week views across a DST boundary (Europe/Warsaw fixtures).
3. Round-trip with Google Calendar via iMIP mail only: receive a Google invite → Accept in our UI → organizer's Google Calendar shows attendee accepted; our organizer invite → Google attendee accepts → our event shows partstat=accepted.
4. CalDAV sync against a Nextcloud/Radicale test container: create/edit/delete propagate both directions in ≤ 5 min (one cron cycle); conflict case produces the A.4 tentative-duplicate, never silent loss.
5. ICS subscription to a public holiday feed renders read-only events; editing is blocked in UI and API (403 read_only_calendar).
6. Killing the worker mid-sync and restarting causes no duplicate events (upsert key discipline).

---

Appendix B — Contacts plugin (webmail_plugin_contacts)

The second heavy plugin: same rules as the calendar — §11 SDK only, zero core diffs, own page. Division of labor with core: core owns implicit recipient history (known_recipients, §4/§9.1) so autocomplete works even without this plugin; the Contacts plugin owns explicit, user-curated contact records and enriches suggestions via the recipients.suggest capability (§11.3).

B.1 Scope by phase

• B1 — Local contacts: address books, contact CRUD with structured fields, groups, photo, search, dedicated page /contacts; "Add sender to contacts" from the thread view; suggestion capability wired into the composer.
• B2 — Portability: vCard 3.0/4.0 import (file, multi-card) and export (single contact / book), duplicate detection + merge tool.
• B3 — External sources: ContactProvider interface with CardDAV client sync (iCloud, Fastmail, Nextcloud, Radicale); per-book sync job.
• B4 (deferred, spec later): Google People / Microsoft Graph providers (reusing M2 OAuth), CardDAV server endpoint for native phone sync, book sharing.

B.2 Dependencies

vobject ~0.9 (vCard parse/serialize). Photos stored via the SDK blob store (JPEG/PNG, re-encoded server-side, ≤ 512×512). Nothing else.

B.3 Data model (plugin migrations, prefix plg_contacts_)

CREATE TABLE plg_contacts_books (
  id uuid PRIMARY KEY DEFAULT gen_random_uuid(),
  user_id uuid NOT NULL,
  name text NOT NULL,
  kind text NOT NULL DEFAULT 'local',       -- local|carddav
  source_config jsonb NOT NULL DEFAULT '{}',-- carddav url/collection
  secret_ct bytea,                          -- carddav credentials, §10.2 vault scheme
  sync_token text,
  read_only boolean NOT NULL DEFAULT false,
  UNIQUE (user_id, name)
);

CREATE TABLE plg_contacts_contacts (
  id uuid PRIMARY KEY DEFAULT gen_random_uuid(),
  user_id uuid NOT NULL,
  book_id uuid NOT NULL REFERENCES plg_contacts_books(id) ON DELETE CASCADE,
  vcard_uid text NOT NULL,                  -- UID property; join key for CardDAV
  etag text,                                -- server etag (carddav books)
  name_display text NOT NULL DEFAULT '',    -- FN
  name_parts jsonb NOT NULL DEFAULT '{}',   -- {given,family,middle,prefix,suffix} (N)
  emails jsonb NOT NULL DEFAULT '[]',       -- [{value,type,pref}]
  phones jsonb NOT NULL DEFAULT '[]',
  addresses jsonb NOT NULL DEFAULT '[]',    -- structured ADR
  org text NOT NULL DEFAULT '',
  title text NOT NULL DEFAULT '',
  birthday date,
  urls jsonb NOT NULL DEFAULT '[]',
  notes text NOT NULL DEFAULT '',
  photo_sha256 text,
  raw_vcard text,                           -- round-trip fidelity: preserve unknown props
  updated_at timestamptz NOT NULL DEFAULT now(),
  UNIQUE (book_id, vcard_uid)
);
CREATE INDEX ON plg_contacts_contacts (user_id, name_display);
CREATE INDEX ON plg_contacts_contacts USING gin (emails jsonb_path_ops);

CREATE TABLE plg_contacts_groups (
  id uuid PRIMARY KEY DEFAULT gen_random_uuid(),
  user_id uuid NOT NULL,
  name text NOT NULL,
  UNIQUE (user_id, name)
);
CREATE TABLE plg_contacts_group_members (
  group_id uuid NOT NULL REFERENCES plg_contacts_groups(id) ON DELETE CASCADE,
  contact_id uuid NOT NULL REFERENCES plg_contacts_contacts(id) ON DELETE CASCADE,
  PRIMARY KEY (group_id, contact_id)
);

vCard mapping discipline: parse known properties into columns, keep the full source in raw_vcard, and on export/push regenerate from columns merged over raw_vcard so properties we don't model (e.g. X- extensions) survive round-trips.

B.4 ContactProvider interface (inside the plugin, mirrors A.4)

# PLUGIN INTERFACE VERSION: 1
class ContactProvider(Protocol):
    async def verify(self) -> None: ...
    async def list_books(self) -> list[RemoteBook]: ...
    async def pull(self, book: str, *, sync_token: str | None
                   ) -> tuple[list[RemoteVcard], list[str], str]: ...
        # (changed vcards, deleted uids, new sync_token)
    async def push(self, book: str, vcard: str, *, uid: str,
                   etag: str | None) -> str: ...
    async def delete(self, book: str, uid: str, etag: str | None) -> None: ...

Sync loop = plugin cron job (*/5 * * * *, singleton per user), upsert by (book_id, vcard_uid), etag conflict ⇒ server wins + local copy kept as a duplicate flagged for the merge tool (never silent loss) — same policy as A.4.

B.5 API & capability (/api/plugins/contacts/)

GET /contacts?book=&group=&q=&cursor= · POST /contacts · GET/PUT/DELETE /contacts/{id} · POST /contacts/from-message {message_id} (prefill from From header) · GET/POST/DELETE /books, /groups, group membership ops · POST /import (vCard file → report {created, updated, skipped}) · GET /export?book=|contact= (.vcf) · GET /duplicates (candidate pairs: same email, or same normalized name + fuzzy) · POST /merge {keep_id, remove_id} (field union, keep wins conflicts, group memberships merged) · GET /suggest?q= — the recipients.suggest capability endpoint: matches name/email prefix and substring, returns [{name, email, source:"contact", contact_id, photo_url?}]. Composer merge rule (client-side, defined here so behavior is deterministic): contact hits rank above known_recipients hits at equal prefix quality; dedupe by email, contact entry wins and carries the photo/display name.

B.6 UI

Page /contacts (nav entry after Calendar): three-pane layout — books/groups sidebar, alphabetically grouped virtualized list with search-as-you-type, detail pane with edit-in-place form; import/export and merge review under /contacts/tools; CardDAV book wizard reusing the account-wizard UX pattern (URL + credentials → test → book picker). Mail integration via message.actions slot: "Add to Contacts" (hidden when the sender already matches a contact; then it becomes "View contact" linking to /contacts/{id}). Group compose: selecting a group in the composer expands to its members' preferred emails.

B.7 Acceptance criteria

1. Zero core diffs; install/uninstall --drop-data round-trip clean.
2. Import a 500-card vCard file mixing v3.0/v4.0, quoted-printable encoding, and Polish diacritics in names (Grzegorz Brzęczyszczykiewicz) — all names, emails, and photos correct; re-export → re-import is lossless including X- properties (raw_vcard merge discipline).
3. Composer typing a 3-char prefix returns merged suggestions in < 100 ms p95 (contacts ranked above history, deduped by email, photos shown for contacts).
4. CardDAV sync against Radicale: create/edit/delete propagate both directions within one cron cycle; concurrent edit conflict produces a flagged duplicate, never silent loss; worker kill mid-sync ⇒ no duplicate contacts on restart.
5. "Add sender to contacts" from a thread creates a prefilled contact; the same sender's messages immediately render the contact's display name, and the action flips to "View contact".
6. Merge tool: merging two contacts unions emails/phones/groups, keeps the chosen primary's conflicting fields, and repoints nothing else in the system (calendar attendees and mail remain keyed by email, by design).

---

End of specification. Ambiguities: prefer the simplest behavior consistent with the invariants in §1, leave a TODO(spec) comment, and list open questions in docs/open-questions.md rather than guessing silently.