lukas/webmail-client
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

Gmail (app-password) onboarding: auth_strategy routing + credential-helper panel

Browse Source 
Implements SPEC v1.5 §6.2.1–6.2.3 so Gmail (and Yahoo/iCloud/Outlook-personal)
onboard via a provider app password on the existing plain-password sync path —
no OAuth, no new sync code. Gmail was oauth2-only with OAuth unimplemented, so it
was previously un-addable.

Backend:
- presets.yaml: each preset declares an ordered `auth_strategy`; Gmail becomes
  `auth: password` + `[app_password, oauth2]`. App-password providers ship a
  data-driven `help:` block (deep_link, app_name_hint, i18n step keys, docs_url).
- discovery/types: CandidateConfig carries `auth_strategy` + `HelpBlock`; presets
  load them and autoconfig candidates derive a default from the detected auth.
- schemas/api: CandidateOut exposes auth_strategy + help; DiscoverResponse adds
  `google_oauth_enabled`.
- config: GOOGLE_OAUTH_ENABLED flag (default false) gates the OAuth fallback.

Frontend:
- ProviderCredentialPanel (wizard step 1.5): deep-link popup (user gesture),
  rendered steps, single paste field that strips whitespace, instant verify
  (auto POST /accounts/test at 16 chars → Connected ✓ / mapped error), and the
  auth_failed recovery branch with a gated "Sign in with Google instead" button.
- AccountWizard routes app-password providers to the panel, then a details step.
- i18n keys (en) for the panel + provider steps.

Honors the §6.2.2 prohibitions: no page automation, no real-password capture,
no scraping — deep-link + guide + paste + test only. Adds the v1.5 spec.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01U72FZiF1W1pb8XxyWBTREA
This commit is contained in:
lukas
2026-08-04 19:59:52 +02:00
co-authored by Claude Opus 4.8
parent 0f686bbd5c
commit e5a87b37b1
11 changed files with 1380 additions and 84 deletions
Download Patch File Download Diff File Expand all files Collapse all files
webmail-client-spec_v1.5.md
+833
View File
@@ -0,0 +1,833 @@
# Webmail Client — Implementation Specification (v1.5, agent-ready)
This document is a complete, self-contained build specification. It is written to be handed to an AI coding agent. Read it fully before writing code. The companion document `webmail-client-concept.md` gives background; where they differ, **this spec wins**.
---
## 0. Instructions to the implementing agent
1. **Build milestone by milestone** (§12), in order. Do not start a milestone before the previous one's acceptance criteria pass.
2. **Do not add features beyond this spec.** Anything not specified here is out of scope; leave `TODO(spec)` comments instead of inventing behavior.
3. **Definition of done** for every task: code + type hints + unit tests + passing `ruff check`, `ruff format --check`, `mypy --strict` (backend) / `tsc --noEmit`, `eslint` (frontend) + updated docs in `docs/` if an interface changed.
4. **Stability over novelty:** use exactly the pinned dependencies in §3. Do not swap libraries or add dependencies without a `# DEPENDENCY:` comment explaining why, and never for something the stdlib does adequately.
5. **All interfaces in §5 are frozen contracts.** Implementations may evolve; signatures may not (without bumping the version comment at the top of the interface file).
6. **Security rules in §10 are hard requirements**, not suggestions.
7. Every module publishing or consuming events must use the event names in §8 verbatim.
8. Commit in small, working increments; every commit must leave the test suite green.
## 1. Architecture invariants (never violate)
- **I-1 Sync-first:** the web UI and REST API never open IMAP connections. Only sync workers talk to mail servers. The API reads exclusively from Postgres/Meilisearch.
- **I-2 Independent identity:** login uses app accounts (`localpart@our-web-client.example`) with Argon2id hashes. Mailbox credentials are payload data, encrypted at rest, decrypted only inside sync workers.
- **I-3 One index, many views:** unified inbox, per-account views, virtual mailboxes, and search results are all parameterized queries over the same index. No duplicated storage per view.
- **I-4 Threading is ours:** JWZ in our backend (§7). Never use the IMAP `THREAD` extension.
- **I-5 Direction-agnostic thread UI:** thread ordering (oldest-first/newest-first) is a data-ordering concern. One layout, one composer-anchor slot (§9.4).
- **I-6 Plugins extend, core stays closed:** new features beyond this spec must be expressible through the plugin surfaces (§11).
- **I-7 Multi-user isolation:** every query (SQL and search) carries a mandatory `user_id` filter. No exceptions, including admin tooling.
- **I-8 Mail HTML is hostile input:** sanitize with nh3 server-side at index time AND render in a sandboxed iframe. Both, always.
## 2. Repository layout (monorepo)
```
webmail/
├── backend/
│ ├── pyproject.toml
│ ├── app/
│ │ ├── main.py # FastAPI app factory
│ │ ├── config.py # pydantic-settings; all config via env
│ │ ├── db.py # SQLAlchemy async engine/session
│ │ ├── events.py # EventBus (Redis pub/sub + arq enqueue)
│ │ ├── identity/ # §6.1 auth, sessions, handles, TOTP
│ │ ├── accounts/ # §6.2 wizard, autodiscovery, credential vault
│ │ ├── providers/
│ │ │ ├── base.py # MailProvider interface (§5.1)
│ │ │ └── imap_smtp/ # first implementation
│ │ ├── sync/ # §6.3 backfill, IDLE loops, reconciliation
│ │ ├── indexer/ # §6.4 MIME parsing → DB rows + blobs
│ │ ├── threading_/ # §7 JWZ (trailing _ avoids stdlib clash)
│ │ ├── search/
│ │ │ ├── base.py # SearchBackend interface (§5.2)
│ │ │ ├── queryparser.py # operator grammar (§8.2)
│ │ │ ├── pg.py # Postgres FTS backend (M0–M1)
│ │ │ └── meili.py # Meilisearch backend (M2)
│ │ ├── outbox/ # §6.5 drafts, MIME assembly, send queue, undo
│ │ ├── api/ # routers, one file per §9.1 group
│ │ ├── ws.py # WebSocket hub (§9.2)
│ │ └── plugins/ # §11 loader + plugin packages
│ ├── workers/worker.py # arq worker entrypoint
│ ├── migrations/ # alembic
│ └── tests/
├── frontend/
│ ├── package.json
│ └── src/
│ ├── api/ # generated client from OpenAPI + WS layer
│ ├── routes/ # §9.3
│ ├── components/
│ │ ├── thread/ThreadView.tsx # §9.4 direction-agnostic
│ │ ├── thread/Composer.tsx
│ │ ├── list/MessageList.tsx # virtualized
│ │ └── slots/ # plugin UI slots (§11)
│ └── lib/
├── docker/
│ ├── docker-compose.yml # app, worker, postgres:16, redis:7, meilisearch:v1.8, dovecot (test)
│ └── docker-compose.ci.yml # + greenmail for integration tests
└── docs/
```
## 3. Pinned stack
**Backend (Python 3.11):** fastapi ~0.115, uvicorn ~0.30, sqlalchemy ~2.0 (async, asyncpg), alembic ~1.13, pydantic ~2.8, pydantic-settings, arq ~0.26, redis ~5.0, aioimaplib ~1.1, imapclient ~3.0 (utility/fallback), aiosmtplib ~3.0, argon2-cffi ~23.1, pyotp ~2.9, dnspython ~2.6, httpx ~0.27, nh3 ~0.2, cryptography ~43.0, mail-parser ~4.0 (helpers only; primary parsing via stdlib `email.message.EmailMessage` with `policy.default`).
**Frontend:** TypeScript ~5.5, React ~18.3, Vite ~5, TanStack Query ~5, TanStack Virtual ~3, Radix UI, Tailwind ~3.4, zod (WS payload validation). API client generated from OpenAPI (`openapi-typescript`).
**Infra:** PostgreSQL 16 (with `citext` extension), Redis 7, Meilisearch v1.8 (M2+), MinIO or plain filesystem for the blob store (config-switchable; filesystem default).
## 4. Database schema (Postgres 16, authoritative DDL)
All PKs are `uuid DEFAULT gen_random_uuid()`. All tables carrying user data include `user_id` with an index. Timestamps are `timestamptz`.
```sql
CREATE EXTENSION IF NOT EXISTS citext;
CREATE TABLE users (
id uuid PRIMARY KEY DEFAULT gen_random_uuid(),
handle citext UNIQUE NOT NULL, -- full form: localpart@<APP_DOMAIN>
password_hash text NOT NULL, -- argon2id
totp_secret bytea, -- encrypted; NULL = 2FA off
recovery_email citext,
settings jsonb NOT NULL DEFAULT '{}', -- incl. {"thread_order":"asc"|"desc"}
created_at timestamptz NOT NULL DEFAULT now()
);
-- Handle rules (enforce in code, mail-grade from day one):
-- localpart: 3–30 chars, [a-z0-9._-], no leading/trailing/double dot,
-- case-insensitive, reserved list (postmaster, abuse, admin, root, noreply, ...).
CREATE TABLE accounts (
id uuid PRIMARY KEY DEFAULT gen_random_uuid(),
user_id uuid NOT NULL REFERENCES users(id) ON DELETE CASCADE,
email citext NOT NULL,
display_name text NOT NULL DEFAULT '',
color text NOT NULL DEFAULT '#888888',
imap_config jsonb NOT NULL, -- {host,port,tls:"ssl"|"starttls",auth:"password"|"oauth2"}
smtp_config jsonb NOT NULL, -- same shape
secret_ct bytea NOT NULL, -- envelope-encrypted password or OAuth token bundle (§10.2)
discovery_src text NOT NULL, -- preset|autoconfig|ispdb|srv|probe|manual
state text NOT NULL DEFAULT 'backfilling', -- backfilling|live|error|disabled
state_detail text NOT NULL DEFAULT '',
last_sync_at timestamptz,
created_at timestamptz NOT NULL DEFAULT now(),
UNIQUE (user_id, email)
);
CREATE TABLE mailboxes (
id uuid PRIMARY KEY DEFAULT gen_random_uuid(),
account_id uuid NOT NULL REFERENCES accounts(id) ON DELETE CASCADE,
user_id uuid NOT NULL REFERENCES users(id) ON DELETE CASCADE,
path text NOT NULL, -- IMAP mailbox name, UTF-7 decoded
special_use text, -- inbox|sent|drafts|trash|junk|archive|NULL
uidvalidity bigint NOT NULL DEFAULT 0,
highestmodseq bigint, -- NULL if no CONDSTORE
last_seen_uid bigint NOT NULL DEFAULT 0,
UNIQUE (account_id, path)
);
CREATE TABLE threads (
id uuid PRIMARY KEY DEFAULT gen_random_uuid(),
user_id uuid NOT NULL REFERENCES users(id) ON DELETE CASCADE,
subject_norm text NOT NULL, -- normalized subject (§7.3)
snippet text NOT NULL DEFAULT '',
participants jsonb NOT NULL DEFAULT '[]', -- [{name,email}], deduped, ordered by recency
account_ids uuid[] NOT NULL DEFAULT '{}',
message_count int NOT NULL DEFAULT 0,
unread_count int NOT NULL DEFAULT 0,
has_attachments boolean NOT NULL DEFAULT false,
last_message_at timestamptz NOT NULL
);
CREATE INDEX ON threads (user_id, last_message_at DESC);
CREATE TABLE messages (
id uuid PRIMARY KEY DEFAULT gen_random_uuid(),
user_id uuid NOT NULL REFERENCES users(id) ON DELETE CASCADE,
account_id uuid NOT NULL REFERENCES accounts(id) ON DELETE CASCADE,
mailbox_id uuid NOT NULL REFERENCES mailboxes(id) ON DELETE CASCADE,
uid bigint NOT NULL, -- IMAP UID within mailbox
thread_id uuid REFERENCES threads(id),
message_id_hdr text, -- Message-ID, angle brackets stripped
in_reply_to text,
references_hdr text[] NOT NULL DEFAULT '{}',
subject text NOT NULL DEFAULT '',
from_addr jsonb NOT NULL DEFAULT '[]',
to_addr jsonb NOT NULL DEFAULT '[]',
cc_addr jsonb NOT NULL DEFAULT '[]',
date_hdr timestamptz,
internal_date timestamptz NOT NULL,
flags text[] NOT NULL DEFAULT '{}', -- \Seen,\Flagged,\Answered,\Draft,\Deleted
size_bytes int NOT NULL DEFAULT 0,
has_attachments boolean NOT NULL DEFAULT false,
snippet text NOT NULL DEFAULT '', -- first ~160 chars of text body
blob_sha256 text NOT NULL, -- raw RFC822 in blob store (§6.4)
body_indexed tsvector, -- Postgres FTS (M0–M1 SearchBackend)
UNIQUE (mailbox_id, uid)
);
CREATE INDEX ON messages (user_id, thread_id);
CREATE INDEX ON messages (account_id, message_id_hdr);
CREATE INDEX ON messages USING gin (body_indexed);
CREATE TABLE body_parts (
id uuid PRIMARY KEY DEFAULT gen_random_uuid(),
message_id uuid NOT NULL REFERENCES messages(id) ON DELETE CASCADE,
kind text NOT NULL, -- text|html_sanitized
content text NOT NULL
);
CREATE TABLE attachments (
id uuid PRIMARY KEY DEFAULT gen_random_uuid(),
message_id uuid NOT NULL REFERENCES messages(id) ON DELETE CASCADE,
user_id uuid NOT NULL,
filename text NOT NULL,
mime_type text NOT NULL,
size_bytes int NOT NULL,
content_id text, -- for inline cid: images
blob_sha256 text NOT NULL
);
CREATE TABLE drafts (
id uuid PRIMARY KEY DEFAULT gen_random_uuid(),
user_id uuid NOT NULL REFERENCES users(id) ON DELETE CASCADE,
account_id uuid NOT NULL REFERENCES accounts(id) ON DELETE CASCADE,
thread_id uuid REFERENCES threads(id), -- NULL = new mail
reply_to_message_id uuid REFERENCES messages(id),
kind text NOT NULL DEFAULT 'reply', -- reply|reply_all|forward|new
payload jsonb NOT NULL, -- {to,cc,bcc,subject,html,text,attachment_ids}
updated_at timestamptz NOT NULL DEFAULT now()
);
CREATE TABLE outbox (
id uuid PRIMARY KEY DEFAULT gen_random_uuid(),
user_id uuid NOT NULL,
draft_id uuid NOT NULL,
state text NOT NULL DEFAULT 'undo_window', -- undo_window|sending|sent|failed|cancelled
send_after timestamptz NOT NULL, -- now() + undo window
error text
);
CREATE TABLE saved_searches (
id uuid PRIMARY KEY DEFAULT gen_random_uuid(),
user_id uuid NOT NULL REFERENCES users(id) ON DELETE CASCADE,
name text NOT NULL,
query text NOT NULL, -- raw operator-syntax string (§8.2)
pinned boolean NOT NULL DEFAULT true,
UNIQUE (user_id, name)
);
-- Core recipient autocomplete (Gmail-style, derived from mail history).
-- The Contacts plugin (Appendix B) ENRICHES suggestions; this table works without it.
CREATE TABLE known_recipients (
user_id uuid NOT NULL REFERENCES users(id) ON DELETE CASCADE,
email citext NOT NULL,
name text NOT NULL DEFAULT '', -- last seen display name
times_used int NOT NULL DEFAULT 0, -- weighted: sent-to counts 3, received-from counts 1
last_used_at timestamptz NOT NULL,
PRIMARY KEY (user_id, email)
);
CREATE INDEX ON known_recipients (user_id, email text_pattern_ops);
```
**Blob store:** raw RFC822 messages and attachment payloads are stored content-addressed at `${BLOB_ROOT}/{user_id}/{sha256[:2]}/{sha256}` (filesystem) or the equivalent object key (MinIO). Postgres holds metadata only. Deleting a user removes their blob subtree.
**Sessions** live in Redis (`sess:{token}` → JSON `{user_id, created, ua}`, TTL 30 days sliding), not Postgres.
## 5. Frozen core interfaces (Python)
### 5.1 `providers/base.py` — MailProvider
```python
# INTERFACE VERSION: 1
from dataclasses import dataclass
from typing import AsyncIterator, Protocol
from datetime import datetime
@dataclass(frozen=True)
class RemoteMailbox:
path: str
special_use: str | None # "inbox"|"sent"|"drafts"|"trash"|"junk"|"archive"|None
uidvalidity: int
@dataclass(frozen=True)
class RemoteMessage:
uid: int
raw: bytes # full RFC822
flags: frozenset[str]
internal_date: datetime
@dataclass(frozen=True)
class FlagChange:
uid: int
flags: frozenset[str]
@dataclass(frozen=True)
class MailboxDelta:
new: list[RemoteMessage]
flag_changes: list[FlagChange]
expunged_uids: list[int]
new_uidvalidity: int | None # non-None => UIDVALIDITY changed, caller must full-resync
new_highestmodseq: int | None
class MailProvider(Protocol):
async def verify(self) -> None: ...
# Raises ProviderAuthError / ProviderConnectError with a human-readable .detail
async def list_mailboxes(self) -> list[RemoteMailbox]: ...
def backfill(self, mailbox: str, *, newest_first: bool = True,
batch_size: int = 200) -> AsyncIterator[list[RemoteMessage]]: ...
# Yields batches until the mailbox is exhausted. Headers+body in one pass.
async def poll(self, mailbox: str, *, last_seen_uid: int,
highestmodseq: int | None) -> MailboxDelta: ...
# One reconciliation pass. Uses QRESYNC/CONDSTORE when highestmodseq is not None,
# otherwise UID SEARCH + FETCH FLAGS diff.
async def idle(self, mailbox: str) -> None: ...
# Blocks until the server signals change or 25 min elapse (re-issue IDLE). May raise
# ProviderNotSupported => sync loop falls back to poll() every POLL_INTERVAL (default 120 s).
async def push_flags(self, mailbox: str, uid: int,
add: frozenset[str], remove: frozenset[str]) -> None: ...
async def move(self, uid: int, src: str, dst: str) -> int: ... # returns new UID
async def append(self, mailbox: str, raw: bytes,
flags: frozenset[str]) -> int: ... # Sent/Drafts upload
async def send(self, raw: bytes, envelope_from: str,
envelope_to: list[str]) -> None: ... # SMTP submission
```
`ImapSmtpProvider` is the only M0–M2 implementation. A future `NativeProvider` (hosted mailboxes) or `JmapProvider` implements the same protocol — nothing above this interface may import IMAP libraries.
### 5.2 `search/base.py` — SearchBackend
```python
# INTERFACE VERSION: 1
@dataclass(frozen=True)
class SearchQuery:
text: str | None # free-text remainder after operator extraction
filters: dict[str, object] # parsed operators, keys per §8.2
user_id: UUID # ALWAYS set (invariant I-7)
@dataclass(frozen=True)
class SearchHit:
message_id: UUID
thread_id: UUID
score: float
highlights: dict[str, str] # field -> HTML-escaped snippet with <mark>
class SearchBackend(Protocol):
async def index(self, doc: SearchDoc) -> None: ...
async def delete(self, message_id: UUID) -> None: ...
async def delete_user(self, user_id: UUID) -> None: ...
async def query(self, q: SearchQuery, *, limit: int, cursor: str | None
) -> tuple[list[SearchHit], str | None]: ...
```
`SearchDoc` fields: `message_id, thread_id, user_id, account_id, mailbox_special_use, from_ (name+email), to, cc, subject, body_text, attachment_names[], has_attachments, is_unread, is_starred, date (unix), size_bytes`. Both backends (pg.py, meili.py) index the same doc. Meilisearch: one index `messages`, filterable = every non-body field incl. `user_id`, searchable = `subject, body_text, from_, to, cc, attachment_names`, sortable = `date`; ranking rules default + `date:desc` tiebreak.
### 5.3 `events.py` — EventBus
```python
# INTERFACE VERSION: 1
class EventBus(Protocol):
async def publish(self, event: str, payload: dict) -> None:
# Fan-out to Redis pub/sub channel "evt:{event}" AND enqueue arq job
# "on_{event}" for durable consumers (indexer, plugins).
def subscribe(self, event: str, handler: Handler) -> None: ...
```
Payloads always include `user_id`. Event names are listed in §8.3 and are the only ones allowed.
## 6. Backend module specifications
### 6.1 Identity (`app/identity/`)
- Registration: validate localpart per §4 rules against reserved list in `identity/reserved.py`; store full handle `localpart@{APP_DOMAIN}` (env `APP_DOMAIN`).
- Login: constant-time verify (argon2id, `time_cost=3, memory_cost=64MiB, parallelism=4`); if `totp_secret` set, require valid TOTP (±1 window). Rate limit: 10 attempts / 15 min per handle AND per IP (Redis counters); lockout returns 429 with `Retry-After`.
- Session cookie: `__Host-session`, httpOnly, Secure, SameSite=Lax. CSRF: double-submit token header `X-CSRF` for all mutating requests.
- TOTP setup: `POST /auth/totp/setup` returns provisioning URI + QR payload; enable requires one valid code; secret stored envelope-encrypted.
- Password reset: only if `recovery_email` verified; single-use token, 30 min TTL, Redis-backed.
### 6.2 Accounts & autodiscovery (`app/accounts/`)
`discover(email) -> list[CandidateConfig]` runs the chain, **returning on first success** but recording every step's outcome in the response (`steps: [{method, status, detail}]`) for wizard UX:
1. **Presets** — `accounts/presets.yaml` (config, not code): match by domain; ships with gmail.com, googlemail.com, outlook.com/hotmail/live, office365, yahoo, icloud/me.com, gmx, wp.pl, onet.pl, o2.pl, interia.pl. Each entry: imap/smtp host+port+tls, auth kind, oauth provider id if applicable.
2. **Autoconfig** — GET `https://autoconfig.{domain}/mail/config-v1.1.xml?emailaddress={email}` then `https://{domain}/.well-known/autoconfig/mail/config-v1.1.xml`; parse Thunderbird XML (incomingServer type=imap, outgoingServer type=smtp; `%EMAILADDRESS%`/`%EMAILLOCALPART%` substitution).
3. **ISPDB** — GET `https://autoconfig.thunderbird.net/v1.1/{domain}`; same XML format.
4. **DNS SRV (RFC 6186)** — `_imaps._tcp.{domain}` (SSL 993), `_imap._tcp` (STARTTLS 143), `_submissions._tcp` (465), `_submission._tcp` (587); lowest priority wins.
5. **Probe** — try `{imap,mail}.{domain}` × ports 993(ssl)/143(starttls) and `{smtp,mail}.{domain}` × 465(ssl)/587(starttls); TCP connect + TLS handshake + IMAP/SMTP greeting, 5 s timeout each, all concurrently.
All HTTP/DNS lookups: 5 s timeout, no redirects off-domain, HTTPS only. `test(config, credentials)`: full IMAP LOGIN + `LIST`, and SMTP EHLO+AUTH (no message sent); errors mapped to precise user-facing strings (`auth_failed`, `tls_failed`, `connect_timeout`, `unknown_host`). On save: encrypt credentials (§10.2), create account row (`state=backfilling`), enqueue `sync.bootstrap_account`.
#### 6.2.1 Provider onboarding strategy (`auth_strategy`)
Each preset in `presets.yaml` declares an ordered `auth_strategy` list telling the wizard which credential flows to offer and in what order. This is the routing that keeps Gmail (and similar) onboarding smooth without any server-side automation of the provider's pages. Values:
- `app_password` — provider supports a per-user application-specific password used as an ordinary IMAP/SMTP password (Gmail, Yahoo, iCloud, outlook.com personal). The wizard shows the provider-specific helper panel (§6.2.2) rather than the generic password box.
- `oauth2` — XOAUTH2 flow (M2). For Google this is the **fallback** offered when `app_password` fails or is unavailable (brand-new accounts, Advanced Protection, admin-disabled). Gmail's `auth_strategy` is therefore `[app_password, oauth2]`.
- `password` — plain IMAP password (self-hosted/ISP accounts): the generic password box.
Rationale recorded so the agent doesn't "optimize" it away: full-IMAP OAuth for Google requires restricted-scope verification + an annual CASA assessment, which this deployment declines by default; app passwords are a first-class Google-provided mechanism that needs none of it and reuses the existing password code path (zero new sync code). OAuth remains implemented for M2 as the fallback and for deployments that do complete verification. `SEARCH_BACKEND`-style toggle `GOOGLE_OAUTH_ENABLED` (default false) hides the OAuth option entirely when an operator hasn't configured a Cloud project.
#### 6.2.2 Provider credential-helper panel (frontend, wizard step 1.5)
When the resolved preset's first `auth_strategy` is `app_password`, the wizard renders a `ProviderCredentialPanel` keyed by the preset's `help_id` (e.g. `gmail`, `yahoo`, `icloud`) instead of the generic password field. Content is data-driven from `presets.yaml` (`help: {deep_link, app_name_hint, steps[], docs_url}`) so adding a provider needs no code. For Gmail specifically:
- **Deep-link button** — "Open Google App Passwords" opens `https://myaccount.google.com/apppasswords` in a popup/new tab. Must fire from the user's click (a direct gesture) so popup blockers allow it. The panel does not and cannot prefill or read anything on Google's page.
- **Inline steps** (rendered from `help.steps`): enable 2-Step Verification if needed → name the password (suggest `app_name_hint`, e.g. "1.email") → copy the 16 characters.
- **Single paste field** — on paste/change, client strips all whitespace (Google displays four groups of four) before use.
- **Instant verify** — on a complete-looking value (16 chars), auto-fire `POST /accounts/test` with the resolved IMAP/SMTP config; show `Connected ✓` or the precise mapped error without the user pressing anything else.
- **Failure recovery (the two real dead-ends):** if `test` returns `auth_failed`, show a branch explaining the likely causes — 2FA not fully enabled, **or a brand-new Google account for which Google withholds app passwords until it ages/gains trust**, or a Workspace account with the feature admin-disabled — each with the relevant link. When the preset's `auth_strategy` also contains `oauth2` **and** `GOOGLE_OAUTH_ENABLED`, render a prominent **"Sign in with Google instead"** button that switches the wizard to the OAuth flow (which is not gated by account age). This single fallback converts the most common onboarding failure into a one-click recovery.
**Hard prohibitions (security + trust — do not implement under any framing):** no headless/automated browser driving of Google (or any provider) sign-in or app-password pages; never request or accept the user's actual Google account password; never scrape, screen-read, or otherwise extract the generated app password from the provider's page — the user copies and pastes it themselves. The panel's ceiling is deep-link + guide + paste + test. Anything beyond that gets user accounts flagged, breaks on provider markup changes, and is out of scope permanently, not just for now.
#### 6.2.3 presets.yaml additions
Each preset gains `auth_strategy: [...]` and, where the first strategy is `app_password`, a `help:` block: `{help_id, deep_link, app_name_hint, steps: [i18n keys], docs_url}`. `steps` are i18n message keys (Polish + English at minimum, per §languages) so the copy is translatable, not hardcoded. Gmail, Yahoo, iCloud, and outlook.com personal ship with populated `help` blocks; corporate/M365 and self-hosted presets use `oauth2` or `password` and no panel.
### 6.3 Sync engine v2 (`app/sync/`) — sharded, lease-owned, tiered
(Authoritative detail: `sync-engine-v2-refactor.md`; this section summarizes the binding rules.)
- **Two worker roles** (`WORKER_ROLE`): `jobs` = stateless arq consumer, N instances; `sync` = lease claimer + per-account runners, 1..N instances. Compose services `worker-jobs` / `worker-sync`, `nofile` ulimit 65536.
- **Lease ownership:** exactly one sync worker owns each live account. Redis SET `sync:live` (API-maintained, DB-reconciled every 300 s); lease `sync:lease:{account_id}` claimed `SET NX EX LEASE_TTL`, renewed every `LEASE_RENEW` via owner-checked Lua; **fencing** — on renewal failure, cancel that account's runner before any further provider/DB op. Worker heartbeats in `sync:workers:{id}`; rebalance releases excess above `target*1.2+5`. Crash failover = lease expiry + reclaim (≤ ~90 s), no coordinator.
- **Event-driven lifecycle:** `account.state_changed` adds/removes ids from `sync:live` at runtime — accounts go live ≤ 30 s after creation and stop ≤ 30 s after deletion with no restarts; runners re-verify `state='live'` each cycle (zombie-proof). New core event **`user.active`** (API publishes on login/WS connect) drives tier promotion.
- **Bootstrap (unchanged from v1, now a `bulk` job):** `list_mailboxes()` → upsert `mailboxes` (SPECIAL-USE mapping with name heuristics — Sent/Wysłane, Trash/Kosz, etc. in `sync/folder_names.py`); backfill newest-first per mailbox (INBOX first, then sent, then rest); `sync.progress` published per batch; on completion set `state='live'` → the API adds the id to `sync:live` and a sync worker claims it.
- **AccountRunner per tier:** *hot* = persistent INBOX IDLE + transient CONDSTORE poll of special folders every `POLL_WARM`; *warm* = transient poll of INBOX+specials every `POLL_WARM`; *cold* = every `POLL_COLD`. Promotion to hot on `user.active` (≤ 5 s) or new mail within `TIER_HOT_ACTIVITY_H`; demotion by inactivity; `HOT_BUDGET_PER_WORKER` degrades least-active non-session accounts. Delta application, UIDVALIDITY wipe-and-rebackfill, expunge soft-handling, and write-then-enqueue outbound flags are unchanged from v1.
- **Connection budgets:** ≤ 2 sync-side connections per account (4 total incl. jobs, shared semaphore); process-local per-destination-host semaphore cap `PER_HOST_CONN_CAP`.
- **Queues:** `interactive` (send/flags/move/draft jobs, timeout 120 s, concurrency 20) and `bulk` (backfills, timeout 3600 s, concurrency `BULK_CONCURRENCY`) — bulk work can never delay a send.
- **Memory bounds:** parse semaphore `PARSE_CONCURRENCY`; raw messages > `STREAM_PARSE_MB` stream to blob and parse from file — peak RAM ≈ `PARSE_CONCURRENCY × STREAM_PARSE_MB` per process regardless of mailbox size.
- **Shutdown:** SIGTERM → stop claimer → cancel runners → release leases → drain in < 10 s.
### 6.4 Indexer (`app/indexer/`)
Input: `RemoteMessage`. Steps: store raw blob (sha256, dedupe) → parse with `email` stdlib (`policy.default`) → extract: headers (decode RFC2047), `references_hdr` = parsed `References` + `In-Reply-To` message-ids in order; best text body and best HTML body (multipart/alternative preference: html for display, text for indexing; if only HTML, derive text via nh3 strip); sanitize HTML with nh3 (allowlist: structural tags, inline styles filtered, `cid:` images rewritten to `/api/messages/{id}/cid/{content_id}`, remote `src` rewritten to blocked-placeholder attributes for FR-23) → write `messages`, `body_parts` (kinds `text` and `html_sanitized`), `attachments` → hand to Threader (§7) → `bus.publish("message.new", ...)` (search indexing + thread denormalization are consumers). For non-junk received messages, upsert the `From` address into `known_recipients` (weight 1).
### 6.5 Outbox (`app/outbox/`)
- Drafts: autosave endpoint upserts `drafts` (client debounces 2 s); a background job mirrors drafts to the account's Drafts folder via `append()` at most every 60 s per draft (replace previous UID).
- Send: `POST /send {draft_id}` → build MIME (html + auto-generated text alternative via html-to-text; `In-Reply-To`/`References` from `reply_to_message_id`; attachments from blob store) → create `outbox` row with `send_after = now()+UNDO_SECONDS` (env, default 10) → schedule arq job at `send_after`. Undo endpoint flips state to `cancelled` if still in window. The job: SMTP `send()`, then `append()` to Sent (skip for Gmail preset — Gmail self-appends), mark `sent`, publish `message.sent`, and upsert every To/Cc/Bcc address into `known_recipients` (weight 3). Failure → `failed` + WS notification, draft preserved.
## 7. Threading specification (JWZ, `app/threading_/`)
### 7.1 Normalization
- `message_id_hdr`, `in_reply_to`, each `References` entry: strip angle brackets and whitespace, lowercase the domain part only. Discard obviously invalid ids (no `@`).
- Missing/duplicate `Message-ID`: synthesize `sha256(raw)[:32]@synthetic.local` and mark it non-authoritative (never used to join other messages to it).
### 7.2 Incremental algorithm (per new message)
1. Candidate set = all normalized ids in `references_hdr` + `in_reply_to` + own `message_id_hdr`.
2. `SELECT DISTINCT thread_id FROM messages WHERE user_id=$1 AND message_id_hdr = ANY($2) AND thread_id IS NOT NULL`.
3. 0 threads → subject fallback (§7.3); still none → create new thread.
4. 1 thread → attach.
5. ≥2 threads → **merge**: keep the thread with the oldest `last_message_at` as canonical, repoint messages of the others, delete emptied thread rows, publish `thread.updated` for all involved.
6. Recompute denormalized thread fields (count, unread, participants, snippet = newest message's snippet, has_attachments, account_ids, last_message_at) in one UPDATE.
Threads are cross-account by design: the candidate lookup is scoped by `user_id`, not `account_id` (invariant I-3). Sent-folder messages join threads exactly like received ones (FR-15).
### 7.3 Subject fallback
Only when reference joining fails: normalize subject (strip repeated `Re:/Fw:/Fwd:/Odp:/PD:` prefixes case-insensitively, collapse whitespace); join to a thread with the same `subject_norm` that has activity within 7 days AND shares ≥1 participant email; never join two messages that both lack references AND have empty subjects.
### 7.4 Test vectors (must pass in `tests/threading/`)
Linear chain; chain arriving out of order; reply missing `In-Reply-To` but carrying `References`; thread merge on late-arriving parent; broken client (subject-only) joins; distinct-topic same-subject ("hi") does NOT join; cross-account merge (message in account A references message in account B); duplicate Message-ID; synthetic-ID isolation.
## 8. Search specification
### 8.1 Pipeline
Consumer of `message.new` / `message.changed` / `message.deleted` → builds `SearchDoc` → active `SearchBackend` (env `SEARCH_BACKEND=pg|meili`). Backfill indexes inline with sync (per batch), never as a separate pass the user waits on (FR-30).
### 8.2 Query grammar (`search/queryparser.py`, engine-independent)
Tokenization: whitespace-split, respecting `"quoted phrases"`. A token matching `key:value` (or `-key:value`) with a known key becomes a filter; everything else joins free text. `-` prefix negates a term or filter. `OR` (uppercase) groups adjacent terms/filters into a disjunction; default combinator is AND.
| Operator | Filter semantics |
|---|---|
| `from:` `to:` `cc:` | substring match on name or email of that field |
| `subject:` | match constrained to subject |
| `account:` | account by `display_name` or email substring |
| `in:` | `inbox\|sent\|drafts\|trash\|junk\|archive` (special_use) or folder path |
| `is:` | `unread\|read\|starred\|answered` |
| `has:attachment` | boolean |
| `filename:` | attachment name substring |
| `before:` `after:` | `YYYY-MM-DD` |
| `newer_than:` `older_than:` | `Nd\|Nw\|Nm\|Ny` |
| `larger:` `smaller:` | `NK\|NM` bytes |
Parser output = `SearchQuery` (§5.2). Unknown `key:` tokens are treated as free text (forward compatibility). Unit tests must cover every operator, negation, quoting, `OR`, and garbage input (parser never raises).
### 8.3 Event catalog (complete)
`message.new`, `message.changed`, `message.deleted`, `message.sent`, `flags.changed`, `thread.updated`, `sync.progress`, `account.state_changed`, `draft.saved`, `outbox.state_changed`, `user.active` (published by the API on login and WS connect; consumed by sync workers for tier promotion — not forwarded to browsers). Payloads: `{user_id, ...ids, minimal-diff fields}` — documented in `docs/events.md` (agent: generate it from code).
Plugins may additionally publish events namespaced `plugin.{name}.*` (e.g. `plugin.calendar.event_updated`); core code never subscribes to plugin events, but the WS hub forwards them to the owning user like any other event.
## 9. API, realtime, frontend
### 9.1 REST API (prefix `/api`, session-cookie auth, JSON; errors: `{error:{code,message}}`)
**Auth:** `POST /auth/register` · `POST /auth/login` · `POST /auth/logout` · `GET /auth/me` · `GET /auth/sessions` · `DELETE /auth/sessions/{id}` · `POST /auth/totp/setup` · `POST /auth/totp/enable` · `POST /auth/password/reset-request` · `POST /auth/password/reset`
**Accounts:** `POST /accounts/discover {email}` → `{candidates:[{imap,smtp,source}], steps:[...]}` · `POST /accounts/test {config, credentials}` → `{ok}|{error:{code,detail}}` · `POST /accounts` · `GET /accounts` (incl. `state`, sync progress) · `PATCH /accounts/{id}` · `DELETE /accounts/{id}` (cascades rows, blobs, search docs)
**Mail:** `GET /threads?view=unified|account:{id}|mailbox:{id}|search&q=&cursor=&limit=50` → thread summaries (denormalized fields only; cursor = `last_message_at,id` keyset) · `GET /threads/{id}?sort=asc|desc` → ordered full messages (bodies of the N newest expanded per client hint `expand=`; older = envelope only) · `POST /threads/{id}/flags {add,remove}` (whole thread) · `POST /messages/{id}/flags` · `POST /messages/{id}/move {mailbox_id}` · `GET /messages/{id}/html` (sanitized, for iframe `srcdoc`) · `GET /messages/{id}/cid/{content_id}` · `GET /attachments/{id}` (Content-Disposition attachment; images/PDF also inline for preview) · `POST /messages/{id}/allow-remote-images` (per-sender allowlist in user settings)
**Compose:** `POST /drafts` · `PUT /drafts/{id}` · `DELETE /drafts/{id}` · `POST /send {draft_id}` → `{outbox_id, undo_until}` · `POST /send/{outbox_id}/undo` · `GET /recipients/suggest?q=` → top 10 from `known_recipients` (prefix match on email or name, ranked by `times_used` decayed by recency); the composer merges these with any plugin capability providers (§11.3)
**Search:** `GET /search?q=&cursor=` → `{threads:[...], hits_by_thread, highlights}` (grouped by thread, FR-28; `&group=message` for flat) · `GET/POST/DELETE /saved-searches`
**Settings:** `GET/PATCH /settings` (incl. `thread_order: "asc"|"desc"`, `tz` IANA timezone — default from browser at registration, undo window, per-sender image allowlist)
### 9.2 WebSocket `/api/ws`
Auth via session cookie. Server → client frames: `{event, payload}` for every §8.3 event belonging to the user (that's the whole contract — the UI refetches via TanStack Query invalidation keyed on ids in the payload). Client → server: `{type:"ping"}` only. Zod-validate all frames.
### 9.3 Frontend routes
`/login`, `/register`, `/mail` (unified inbox), `/mail/a/:accountId`, `/mail/a/:accountId/f/:mailboxId`, `/mail/t/:threadId` (thread pane; on desktop rendered beside the list), `/mail/search?q=`, `/settings/{profile|security|accounts|appearance}`; `/settings/accounts/new` = wizard.
**Account wizard flow:** step 1 email → calls `discover` while typing pauses; step 1.5 credential entry, whose form depends on the resolved preset's `auth_strategy` (§6.2.1): the `ProviderCredentialPanel` (§6.2.2) for `app_password` providers such as Gmail, the "Sign in with Google/Microsoft" button for `oauth2`, or the generic password box for `password`; step 2 shows resolved config with source badge ("found via ISPDB") and a Test button (auto-run, or auto-fired by the panel on paste); failure expands the manual form pre-filled with the best guess and the precise error, and — for providers whose `auth_strategy` includes a fallback — offers the alternate flow (e.g. Gmail app-password failure → "Sign in with Google instead"); step 3 = created, live backfill progress bar (WS `sync.progress`) with "start reading now" (list fills as backfill proceeds).
### 9.4 ThreadView contract (invariant I-5)
```
ThreadView(props: { messages: Message[] /* ASC always */, order: 'asc'|'desc',
composerState, ... })
```
- Rendering order = `order==='asc' ? messages : [...messages].reverse()`. No other branching on `order` is allowed in layout code.
- **Composer anchor** = a slot component rendered adjacent to the newest message: after it when `asc` (bottom of thread), directly after it when `desc` (top of thread, above older messages). Same component, one placement rule: "composer is always attached to the newest message."
- Initial scroll: to the newest message (bottom for asc, top for desc). Collapse: all read messages except the newest are collapsed; "Expand all" toggles. Unread messages are never collapsed.
- Keyboard: `j`/`k` move in *visual* order; `o`/`Enter` expand; `r` reply, `a` reply-all, `f` forward (open composer at anchor); `e` archive; `/` focus search; `gi` go to unified inbox; `esc` close composer (draft persists). Toolbar has the per-thread order toggle; it updates local state only, and long-presses/menu offers "make default" (PATCH /settings).
- Message HTML renders in `<iframe sandbox="" srcdoc={...}>` sized via ResizeObserver postMessage shim; remote images show a "load images / always from this sender" bar when the sanitizer flagged any.
## 10. Security requirements (hard)
1. Argon2id params per §6.1; no other password hashing anywhere.
2. **Credential vault:** mailbox secrets encrypted with AES-256-GCM under per-account data keys, wrapped by a master key from env `MASTER_KEY` (32 B, base64) — `cryptography` library, key id embedded for future rotation. Decryption code path exists only in `workers/` and `accounts/test`. Secrets never appear in logs, API responses, or exceptions (custom repr).
3. CSRF double-submit on all mutating routes; `__Host-` cookie prefix; CSP: `default-src 'self'; frame-src 'self'; img-src 'self' data:` (remote mail images are proxied? **No** — v1 loads allowed remote images inside the sandboxed iframe only, which has its own relaxed CSP via srcdoc; document this in code).
4. nh3 sanitization at index time; re-sanitize on render if sanitizer version changed (store `sanitizer_v` on body_parts).
5. Rate limits: login (§6.1), `POST /accounts/discover` 10/min/user, `POST /send` 30/h/user (env-tunable).
6. All outbound autodiscovery HTTP: HTTPS only, 5 s timeout, response size cap 256 KB, no off-domain redirects (SSRF guard: resolve + reject private IP ranges).
7. Every SQL query and search query includes `user_id` (enforced by repository-layer helpers; raw session use in API code is a review error).
8. Uploads (attachments to drafts): 25 MB/file, 40 MB/message total, content-type sniffed server-side, stored to blob store, never executed/served with original content type unless in safe list (images/pdf/text) — others force `application/octet-stream` + attachment disposition.
## 11. Plugin API (freeze in M3 — including the heavy-plugin surfaces below)
A plugin = Python package `webmail_plugin_<name>` exposing:
```python
plugin = Plugin(
name, version,
hooks = {event: handler}, # subscribe to §8.3 events
router = APIRouter | None, # mounted at /api/plugins/{name}
ui_manifest = UiManifest | None, # slots + pages, see below
migrations = "migrations/" | None, # 11.1
jobs = [JobSpec, ...] | None, # 11.2
)
```
Loader discovers plugins via entry points group `webmail.plugins`, forwards declared events, and serves `GET /api/plugins/ui-manifest` for the frontend loader. Plugins may not import from `app.*` internals other than `app.plugins.sdk` (which exposes: DB session factory scoped to plugin tables, EventBus publish under `plugin.{name}.*`, blob store, outbox draft/send helpers, settings accessor, HTTP client with the §10.6 SSRF guard built in).
Every plugin gets the namespaced KV store (`plugin_kv(plugin, user_id, key)`) for light state. Heavy plugins additionally get:
### 11.1 Plugin-owned tables
A plugin may ship Alembic migrations in its own version branch (`alembic -n plugin_{name}`). All tables MUST be prefixed `plg_{name}_` and every user-data table MUST carry `user_id` (invariant I-7 applies to plugins). Core migrations never reference plugin tables and vice versa. Uninstalling a plugin offers `--drop-data` to run its downgrade chain.
### 11.2 Plugin background jobs
`JobSpec(func, schedule: cron_str | None, singleton: bool)` registers tasks on the shared arq worker: cron-scheduled (e.g. a sync loop trigger) or enqueued on demand from hooks/routes. `singleton=True` guarantees at most one running instance per (plugin, user) via a Redis lock. Plugin jobs run in the same worker process pool with the same retry semantics as core jobs and are visible in worker logs under `plg.{name}.*`.
### 11.3 Frontend surfaces
- **Slots** (lazy-loaded ESM injected into mail UI): `thread.toolbar`, `message.actions`, `message.banner` (a strip rendered above a message body — used for spam warnings, calendar invite cards, and similar per-message notices), `sidebar.section`, `settings.page`.
- **Pages** (`app.page`): a plugin may declare full views: `{route: "/calendar", nav: {icon, label, order}, bundle}`. The shell adds the nav entry, and the route (including sub-routes under it) lazy-loads the bundle into the **entire content area** — no mail list/thread components are rendered. Page bundles receive the SDK context (API client, WS event stream filtered to `plugin.{name}.*` + requested core events, design-system components) and are sandboxed to their route: importing mail-component internals is a review error. Removing the plugin removes the nav entry and route cleanly.
- **Capabilities**: the manifest may declare named capabilities mapping to plugin endpoints, e.g. `capabilities: {"recipients.suggest": "/api/plugins/contacts/suggest"}`. Core UI features query all providers of a capability they support and merge results client-side (defined capabilities in v1: `recipients.suggest` — used by the composer To/Cc/Bcc fields and by any other plugin needing person-picking, such as calendar attendees). This keeps core backend code plugin-unaware.
Acceptance test for the API: implement **filters** (rules on `message.new`: match → move/flag) and **labels** entirely as plugins using only slots + KV; then implement the **calendar** (Appendix A) using tables + jobs + a page — all with zero core diffs.
## 12. Milestones with acceptance criteria
**M0 — Architecture spike**
Login/register (no 2FA), one hardcoded-config IMAP account, backfill+IDLE for INBOX, JWZ, read-only unified list + thread view with working asc/desc toggle and correct composer-anchor placeholder, Postgres FTS free-text search.
✅ Accept: sync a real 10k-message Gmail account via app password; threads match Gmail's grouping on a 50-thread sample ≥ 90%; toggle order and the newest message + anchor swap ends correctly; search returns in < 500 ms.
**M1 — MVP**
Full identity (2FA, sessions UI, reset), account wizard with full discovery chain, multi-account + unified inbox with account chips, flags/move/archive both directions, inline reply + reply-all/forward + SMTP send + Sent append, recipient autocomplete from `known_recipients`, drafts autosave, sanitized HTML + blocked remote images, operator search (full §8.2 grammar) on PG FTS, WS live updates.
✅ Accept: add gmail.com (via the app-password helper panel: deep-link → paste → auto-test → Connected ✓), outlook.com, and one autoconfig-only domain through the wizard; the Gmail panel's `auth_failed` branch offers the OAuth fallback when `GOOGLE_OAUTH_ENABLED`; reply from a thread lands correctly threaded in the recipient's client; all §8.2 operators pass integration tests; `docker-compose up` from clean checkout works.
**M2 — Hardening / v1.0**
OAuth2 (Google + Microsoft, XOAUTH2 for IMAP/SMTP, server-side refresh), QRESYNC fast resync, drafts mirrored to server, undo-send, attachment upload/preview, Meilisearch backend + instant search-as-you-type UI + saved searches as virtual mailboxes, session revocation, backfill progress UX, folder-name i18n heuristics.
✅ Accept: typo query (`invocie`) finds "invoice" mail < 200 ms on a 100k-message corpus; kill -9 the worker mid-backfill and restart resumes without duplicates (idempotency via `(mailbox_id, uid)` unique key); undo-send within window results in no SMTP traffic.
**M3 — Plugin platform**
Loader, full SDK per §11 **including** plugin migrations (11.1), plugin jobs (11.2), and `app.page` (11.3); filters + labels shipped as plugins; plugin docs.
✅ Accept: both plugins install/uninstall cleanly with zero core diffs; a third trivial plugin ("Unsubscribe header button" on `message.actions`) can be written from `docs/plugins.md` alone; a skeleton page-plugin (route + nav entry + one plugin table + one cron job) round-trips install → run → uninstall `--drop-data` leaving no residue.
**M4 — Calendar plugin (Appendix A)**
Phases A1–A2: local calendars + full page views + iMIP invite handling; then A3: CalDAV client sync + ICS subscriptions.
✅ Accept: criteria in Appendix A §A.7.
**M5 — Contacts plugin (Appendix B)**
Phases B1–B2: contact books, full page, groups, add-from-mail, `recipients.suggest` capability, vCard import/export; then B3: CardDAV client sync. May be built in parallel with M4 — both depend only on the M3 SDK.
✅ Accept: criteria in Appendix B §B.7.
**M6 (optional) — Hosted mailboxes**
Inbound MTA (Stalwart) delivering to a `NativeProvider`; handle becomes receiving address. Specified separately when reached.
**M7 — Anti-spam plugin (Appendix C)**
May be built any time after M3 (depends only on the plugin SDK; parallelizable with M4/M5). Phases C1–C2 client-side; C3 activates at the M6 edge.
✅ Accept: criteria in Appendix C §C.8.
## 13. Testing & CI
- **Unit:** pytest + pytest-asyncio; threading vectors (§7.4); query parser (§8.2); autodiscovery resolver with `respx`-mocked HTTP + `dnspython` stub; MIME corpus in `tests/fixtures/mime/` (agent: include ≥ 25 nasty real-world-style fixtures — encoded headers incl. RFC2047 Polish diacritics, nested multiparts, missing Message-ID, malformed dates, HTML-only, inline cid images, 8-bit subjects, winmail.dat placeholder).
- **Integration:** docker-compose.ci.yml with Greenmail (IMAP+SMTP) and Dovecot; full loop test: append message to Greenmail → appears in API < 10 s → reply via API → arrives in Greenmail SMTP sink with correct References.
- **Frontend:** vitest + React Testing Library; ThreadView ordering/anchor tests for both directions; Playwright smoke: login → add Greenmail account via wizard → read → inline reply → search.
- **CI gates:** ruff, mypy --strict, pytest, tsc, eslint, vitest, Playwright smoke; coverage floor 80% on `threading_`, `search/queryparser`, `accounts/autodiscovery`, `indexer`.
## 14. Configuration (env, complete list)
`APP_DOMAIN`, `DATABASE_URL`, `REDIS_URL`, `MASTER_KEY`, `BLOB_ROOT` or `S3_*`, `SEARCH_BACKEND=pg|meili`, `MEILI_URL`, `MEILI_KEY`, `UNDO_SECONDS=10`, `SESSION_TTL_DAYS=30`, `OAUTH_GOOGLE_CLIENT_ID/SECRET`, `OAUTH_MS_CLIENT_ID/SECRET`, `GOOGLE_OAUTH_ENABLED=false` (hides the Gmail OAuth option until a Cloud project is configured; app-password onboarding is the default Gmail path — §6.2.1), `MAX_ATTACHMENT_MB=25`, `LOG_LEVEL`; sync engine v2 (§6.3): `WORKER_ROLE=jobs|sync`, `MAX_ACCOUNTS_PER_WORKER=400`, `LEASE_TTL=60`, `LEASE_RENEW=20`, `POLL_WARM=180`, `POLL_COLD=900`, `TIER_HOT_ACTIVITY_H=24`, `TIER_SESSION_GRACE_MIN=30`, `TIER_COLD_AFTER_D=3`, `HOT_BUDGET_PER_WORKER=250`, `PER_HOST_CONN_CAP=50`, `BULK_CONCURRENCY=2`, `PARSE_CONCURRENCY=4`, `STREAM_PARSE_MB=8`. All read once in `config.py` (pydantic-settings); no `os.environ` elsewhere.
---
## Appendix A — Calendar plugin (`webmail_plugin_calendar`)
The calendar is the reference "heavy plugin": it lives in the same app and repo, uses the §11 SDK only, and must build with **zero core diffs**. It ships as a page (`/calendar`) beside Mail — one product, one login.
### A.1 Scope by phase
- **A1 — Local calendar:** create/edit/delete events, multiple calendars per user, month/week/day/agenda views, recurrence, reminders (WS notification), timezone-correct display.
- **A2 — Invites over mail (iMIP):** detect `text/calendar` parts on `message.new`; render an invite card in the thread (`message.actions` + inline summary); Accept/Tentative/Decline sends an iTIP REPLY through the outbox; organizer flow sends REQUEST/CANCEL to attendees; updates (SEQUENCE bumps) modify the stored event.
- **A3 — External sources:** `CalendarProvider` interface with **CalDAV client** (first implementation; covers iCloud, Fastmail, Nextcloud) and read-only **ICS URL subscriptions** (holidays, team feeds); sync loop as a plugin job.
- **A4 (deferred, spec later):** Google/Microsoft API providers reusing M2 OAuth machinery; secret **ICS export feed** `GET /api/plugins/calendar/feed/{token}.ics`; CalDAV **server** endpoint for native phone sync — architecturally anticipated, not built now.
### A.2 Dependencies
`icalendar ~5.0` (parse/serialize RFC 5545), `python-dateutil` (RRULE expansion), `caldav ~1.3` (A3), `tzdata`. No other additions.
### A.3 Data model (plugin migrations, prefix `plg_calendar_`)
```sql
CREATE TABLE plg_calendar_calendars (
id uuid PRIMARY KEY DEFAULT gen_random_uuid(),
user_id uuid NOT NULL,
name text NOT NULL,
color text NOT NULL DEFAULT '#3b82f6',
kind text NOT NULL DEFAULT 'local', -- local|caldav|ics_subscription
source_config jsonb NOT NULL DEFAULT '{}', -- caldav url/collection, ics url
secret_ct bytea, -- caldav credentials, §10.2 vault scheme
sync_token text, -- CalDAV sync-token / ICS etag
read_only boolean NOT NULL DEFAULT false,
UNIQUE (user_id, name)
);
CREATE TABLE plg_calendar_events (
id uuid PRIMARY KEY DEFAULT gen_random_uuid(),
user_id uuid NOT NULL,
calendar_id uuid NOT NULL REFERENCES plg_calendar_calendars(id) ON DELETE CASCADE,
ical_uid text NOT NULL, -- UID from iCalendar; join key for iMIP updates
sequence int NOT NULL DEFAULT 0,
summary text NOT NULL DEFAULT '',
description text NOT NULL DEFAULT '',
location text NOT NULL DEFAULT '',
dtstart timestamptz NOT NULL,
dtend timestamptz,
all_day boolean NOT NULL DEFAULT false,
tz text NOT NULL DEFAULT 'UTC', -- original TZID; render in user tz, store UTC
rrule text, -- raw RRULE string, NULL = single
exdates timestamptz[] NOT NULL DEFAULT '{}',
recurrence_id timestamptz, -- non-NULL = detached exception instance
organizer jsonb, -- {name,email}
attendees jsonb NOT NULL DEFAULT '[]', -- [{name,email,partstat,role}]
status text NOT NULL DEFAULT 'confirmed', -- confirmed|tentative|cancelled
my_partstat text, -- needs-action|accepted|declined|tentative
reminders int[] NOT NULL DEFAULT '{}', -- minutes before start
origin_message_id uuid, -- link back to the invite mail thread
raw_ical text, -- round-trip fidelity for iMIP/CalDAV
UNIQUE (calendar_id, ical_uid, recurrence_id)
);
CREATE INDEX ON plg_calendar_events (user_id, dtstart);
```
**Recurrence rule:** store the RRULE string; expand instances at query time with `dateutil.rrule` over the requested view range only (no materialization table). Exceptions: `exdates` remove instances; a row with `recurrence_id` overrides one instance. Range queries must account for events starting before the window but recurring into it (expand from `dtstart`, cap iterations at 1000 per event).
### A.4 CalendarProvider interface (inside the plugin, mirrors §5.1)
```python
# PLUGIN INTERFACE VERSION: 1
class CalendarProvider(Protocol):
async def verify(self) -> None: ...
async def list_collections(self) -> list[RemoteCalendar]: ...
async def pull(self, collection: str, *, sync_token: str | None
) -> tuple[list[RemoteEvent], list[str], str]: ...
# returns (changed_events_as_ical, deleted_uids, new_sync_token)
async def push(self, collection: str, ical: str, *, uid: str,
etag: str | None) -> str: ... # returns new etag
async def delete(self, collection: str, uid: str, etag: str | None) -> None: ...
```
Implementations in A3: `CaldavProvider`, `IcsSubscriptionProvider` (pull-only, `push`/`delete` raise `ReadOnlyCalendar`). Sync loop = plugin cron job (`schedule="*/5 * * * *"`, `singleton=True` per user) doing pull → upsert by `(calendar_id, ical_uid, recurrence_id)` → local-dirty push with etag conflict handling (conflict ⇒ server wins, local copy saved as a duplicate marked `status='tentative'`, user notified).
### A.5 iMIP flow (A2)
- **Inbound:** hook on `message.new`; if a `text/calendar` part exists, parse METHOD:
`REQUEST` → upsert event (match `ical_uid`; apply only if `SEQUENCE` ≥ stored) into the user's default calendar with `my_partstat='needs-action'`, set `origin_message_id`; `CANCEL` → `status='cancelled'`; `REPLY` → update that attendee's `partstat` on the organizer's copy.
- **RSVP:** invite card buttons call `POST /api/plugins/calendar/events/{id}/rsvp {partstat}` → update row → build iTIP REPLY ics → send via SDK outbox helper to the organizer (subject `Accepted: {summary}` etc.) → publish `plugin.calendar.event_updated`.
- **Organizer:** creating an event with attendees sends METHOD:REQUEST to each; edits bump `SEQUENCE` and re-send; deletion sends CANCEL. All outgoing invite mail uses the account whose address matches the organizer identity (picker if ambiguous).
### A.6 UI (page `/calendar`, nav icon between Mail and Settings)
Routes: `/calendar` (month default), `/calendar/{month|week|day|agenda}/:date`, `/calendar/event/:id`, `/calendar/settings` (calendars CRUD, CalDAV/ICS wizard reusing the account-wizard UX pattern: URL+credentials → test → collection picker). Interactions: click-drag create, drag-move/resize (writes optimistically, reverts on API error), event popover with edit/RSVP, mini-month navigator, calendar visibility toggles (client state), `t` = today, arrows = prev/next period. Invite linkage both ways: invite card in thread links to `/calendar/event/:id`; event view links back to the mail thread when `origin_message_id` is set. Live updates via WS `plugin.calendar.*` events. Timezone: render in user setting `settings.tz` (add to §9.1 settings; default from browser at registration).
### A.7 Acceptance criteria
1. Plugin installs/uninstalls (`--drop-data`) with zero diffs in `app/` and `frontend/src/` outside `plugins/`/`slots/` loader code.
2. A weekly-recurring event with two EXDATEs and one moved instance renders correctly in month and week views across a DST boundary (Europe/Warsaw fixtures).
3. Round-trip with Google Calendar **via iMIP mail only**: receive a Google invite → Accept in our UI → organizer's Google Calendar shows attendee accepted; our organizer invite → Google attendee accepts → our event shows `partstat=accepted`.
4. CalDAV sync against a Nextcloud/Radicale test container: create/edit/delete propagate both directions in ≤ 5 min (one cron cycle); conflict case produces the A.4 tentative-duplicate, never silent loss.
5. ICS subscription to a public holiday feed renders read-only events; editing is blocked in UI and API (403 `read_only_calendar`).
6. Killing the worker mid-sync and restarting causes no duplicate events (upsert key discipline).
---
## Appendix B — Contacts plugin (`webmail_plugin_contacts`)
The second heavy plugin: same rules as the calendar — §11 SDK only, zero core diffs, own page. Division of labor with core: core owns *implicit* recipient history (`known_recipients`, §4/§9.1) so autocomplete works even without this plugin; the Contacts plugin owns *explicit*, user-curated contact records and enriches suggestions via the `recipients.suggest` capability (§11.3).
### B.1 Scope by phase
- **B1 — Local contacts:** address books, contact CRUD with structured fields, groups, photo, search, dedicated page `/contacts`; "Add sender to contacts" from the thread view; suggestion capability wired into the composer.
- **B2 — Portability:** vCard 3.0/4.0 import (file, multi-card) and export (single contact / book), duplicate detection + merge tool.
- **B3 — External sources:** `ContactProvider` interface with **CardDAV client** sync (iCloud, Fastmail, Nextcloud, Radicale); per-book sync job.
- **B4 (deferred, spec later):** Google People / Microsoft Graph providers (reusing M2 OAuth), CardDAV **server** endpoint for native phone sync, book sharing.
### B.2 Dependencies
`vobject ~0.9` (vCard parse/serialize). Photos stored via the SDK blob store (JPEG/PNG, re-encoded server-side, ≤ 512×512). Nothing else.
### B.3 Data model (plugin migrations, prefix `plg_contacts_`)
```sql
CREATE TABLE plg_contacts_books (
id uuid PRIMARY KEY DEFAULT gen_random_uuid(),
user_id uuid NOT NULL,
name text NOT NULL,
kind text NOT NULL DEFAULT 'local', -- local|carddav
source_config jsonb NOT NULL DEFAULT '{}',-- carddav url/collection
secret_ct bytea, -- carddav credentials, §10.2 vault scheme
sync_token text,
read_only boolean NOT NULL DEFAULT false,
UNIQUE (user_id, name)
);
CREATE TABLE plg_contacts_contacts (
id uuid PRIMARY KEY DEFAULT gen_random_uuid(),
user_id uuid NOT NULL,
book_id uuid NOT NULL REFERENCES plg_contacts_books(id) ON DELETE CASCADE,
vcard_uid text NOT NULL, -- UID property; join key for CardDAV
etag text, -- server etag (carddav books)
name_display text NOT NULL DEFAULT '', -- FN
name_parts jsonb NOT NULL DEFAULT '{}', -- {given,family,middle,prefix,suffix} (N)
emails jsonb NOT NULL DEFAULT '[]', -- [{value,type,pref}]
phones jsonb NOT NULL DEFAULT '[]',
addresses jsonb NOT NULL DEFAULT '[]', -- structured ADR
org text NOT NULL DEFAULT '',
title text NOT NULL DEFAULT '',
birthday date,
urls jsonb NOT NULL DEFAULT '[]',
notes text NOT NULL DEFAULT '',
photo_sha256 text,
raw_vcard text, -- round-trip fidelity: preserve unknown props
updated_at timestamptz NOT NULL DEFAULT now(),
UNIQUE (book_id, vcard_uid)
);
CREATE INDEX ON plg_contacts_contacts (user_id, name_display);
CREATE INDEX ON plg_contacts_contacts USING gin (emails jsonb_path_ops);
CREATE TABLE plg_contacts_groups (
id uuid PRIMARY KEY DEFAULT gen_random_uuid(),
user_id uuid NOT NULL,
name text NOT NULL,
UNIQUE (user_id, name)
);
CREATE TABLE plg_contacts_group_members (
group_id uuid NOT NULL REFERENCES plg_contacts_groups(id) ON DELETE CASCADE,
contact_id uuid NOT NULL REFERENCES plg_contacts_contacts(id) ON DELETE CASCADE,
PRIMARY KEY (group_id, contact_id)
);
```
vCard mapping discipline: parse known properties into columns, keep the full source in `raw_vcard`, and on export/push regenerate from columns **merged over** `raw_vcard` so properties we don't model (e.g. `X-` extensions) survive round-trips.
### B.4 ContactProvider interface (inside the plugin, mirrors A.4)
```python
# PLUGIN INTERFACE VERSION: 1
class ContactProvider(Protocol):
async def verify(self) -> None: ...
async def list_books(self) -> list[RemoteBook]: ...
async def pull(self, book: str, *, sync_token: str | None
) -> tuple[list[RemoteVcard], list[str], str]: ...
# (changed vcards, deleted uids, new sync_token)
async def push(self, book: str, vcard: str, *, uid: str,
etag: str | None) -> str: ...
async def delete(self, book: str, uid: str, etag: str | None) -> None: ...
```
Sync loop = plugin cron job (`*/5 * * * *`, singleton per user), upsert by `(book_id, vcard_uid)`, etag conflict ⇒ server wins + local copy kept as a duplicate flagged for the merge tool (never silent loss) — same policy as A.4.
### B.5 API & capability (`/api/plugins/contacts/`)
`GET /contacts?book=&group=&q=&cursor=` · `POST /contacts` · `GET/PUT/DELETE /contacts/{id}` · `POST /contacts/from-message {message_id}` (prefill from From header) · `GET/POST/DELETE /books`, `/groups`, group membership ops · `POST /import` (vCard file → report `{created, updated, skipped}`) · `GET /export?book=|contact=` (`.vcf`) · `GET /duplicates` (candidate pairs: same email, or same normalized name + fuzzy) · `POST /merge {keep_id, remove_id}` (field union, keep wins conflicts, group memberships merged) · **`GET /suggest?q=`** — the `recipients.suggest` capability endpoint: matches name/email prefix and substring, returns `[{name, email, source:"contact", contact_id, photo_url?}]`. Composer merge rule (client-side, defined here so behavior is deterministic): contact hits rank above `known_recipients` hits at equal prefix quality; dedupe by email, contact entry wins and carries the photo/display name.
### B.6 UI
Page `/contacts` (nav entry after Calendar): three-pane layout — books/groups sidebar, alphabetically grouped virtualized list with search-as-you-type, detail pane with edit-in-place form; import/export and merge review under `/contacts/tools`; CardDAV book wizard reusing the account-wizard UX pattern (URL + credentials → test → book picker). Mail integration via `message.actions` slot: "Add to Contacts" (hidden when the sender already matches a contact; then it becomes "View contact" linking to `/contacts/{id}`). Group compose: selecting a group in the composer expands to its members' preferred emails.
### B.7 Acceptance criteria
1. Zero core diffs; install/uninstall `--drop-data` round-trip clean.
2. Import a 500-card vCard file mixing v3.0/v4.0, quoted-printable encoding, and Polish diacritics in names (`Grzegorz Brzęczyszczykiewicz`) — all names, emails, and photos correct; re-export → re-import is lossless including `X-` properties (raw_vcard merge discipline).
3. Composer typing a 3-char prefix returns merged suggestions in < 100 ms p95 (contacts ranked above history, deduped by email, photos shown for contacts).
4. CardDAV sync against Radicale: create/edit/delete propagate both directions within one cron cycle; concurrent edit conflict produces a flagged duplicate, never silent loss; worker kill mid-sync ⇒ no duplicate contacts on restart.
5. "Add sender to contacts" from a thread creates a prefilled contact; the same sender's messages immediately render the contact's display name, and the action flips to "View contact".
6. Merge tool: merging two contacts unions emails/phones/groups, keeps the chosen primary's conflicting fields, and repoints nothing else in the system (calendar attendees and mail remain keyed by email, by design).
---
## Appendix C — Anti-spam plugin (`webmail_plugin_antispam`)
Third heavy plugin; §11 SDK only, zero core diffs. Architectural context: for **synced** accounts the upstream provider has usually already filtered spam, so this plugin is a *second-pass* filter (weak-provider accounts, unified experience, per-user trainable Bayes). It becomes the *first-pass* filter only for M6 hosted mailboxes, where the same engine also runs at the SMTP edge (§C.7).
### C.1 Scope by phase
- **C1 — Classification & actions:** score every newly synced inbox message; move high-scoring mail to Junk (through the existing move machinery, so the verdict propagates to the IMAP server); banner low-confidence suspects; per-user allow/deny lists; per-account enable toggle.
- **C2 — Learning loop:** explicit "Spam"/"Not spam" actions train per-user Bayes; passive learning from user moves in/out of Junk and (sampled, capped) from provider-classified Junk; stats page.
- **C3 (M6 era):** the same Rspamd configuration/expertise deployed at the mail edge (VPS), where full IP/SPF/DKIM/DMARC/greylisting checks are actually meaningful; hosted-mailbox delivery is filtered *before* it reaches the index.
### C.2 Engine decision
**Rspamd** as a service container (mature, fast, HTTP API) with a small **dedicated Redis** container for its Bayes/fuzzy storage (isolated from the app Redis on purpose — different persistence and flush semantics). No Python spam engine is written from scratch; the plugin is glue + policy. Compose additions: `rspamd` (`rspamd/rspamd:3.x`) + `redis-rspamd` (`redis:7`, `--save 300 1`). Rspamd's network checks (RBL/SURBL/fuzzy lookups leak message-derived hashes/URLs to third parties) are **off by default**, one settings toggle to enable (privacy default matches the product's self-hosted ethos).
### C.3 Data model (plugin migrations, prefix `plg_antispam_`)
```sql
CREATE TABLE plg_antispam_settings (
user_id uuid PRIMARY KEY,
enabled_accounts uuid[] NOT NULL DEFAULT '{}', -- empty = all accounts
move_threshold real NOT NULL DEFAULT 12.0, -- score >= : move to Junk
banner_threshold real NOT NULL DEFAULT 6.0, -- score >= : warn banner
network_checks boolean NOT NULL DEFAULT false,
learn_from_provider_junk boolean NOT NULL DEFAULT true
);
CREATE TABLE plg_antispam_rules (
id uuid PRIMARY KEY DEFAULT gen_random_uuid(),
user_id uuid NOT NULL,
kind text NOT NULL, -- allow|deny
pattern citext NOT NULL, -- email or @domain
UNIQUE (user_id, kind, pattern)
);
CREATE TABLE plg_antispam_verdicts (
message_id uuid PRIMARY KEY,
user_id uuid NOT NULL,
score real NOT NULL,
symbols jsonb NOT NULL DEFAULT '[]', -- rspamd symbol list for explainability
action text NOT NULL, -- none|banner|moved|allow_listed|deny_listed
learned text -- spam|ham|NULL
);
```
### C.4 Classification flow
Hook on `message.new`; process only messages landing in `special_use='inbox'` mailboxes of enabled accounts (never Sent/Drafts/Junk/Trash; never re-classify on flag changes). Order: (1) deny list → move to Junk, action `deny_listed`; (2) allow list → action `allow_listed`, stop; (3) POST raw blob to Rspamd `/checkv2` with `Deliver-To`-style per-user context header (per-user Bayes statistics via Rspamd user settings); (4) apply thresholds → move via the existing move job (verdict thus syncs to the server's Junk folder) or record `banner`; (5) write verdict row, publish `plugin.antispam.classified`. Rspamd unavailability = fail-open (deliver normally, log, retry queue for learning only — spam filtering must never delay or lose mail).
### C.5 Learning
- Explicit: `message.actions` buttons "Spam" / "Not spam" → `POST /api/plugins/antispam/learn {message_id, verdict}` → Rspamd `/learnspam|/learnham` with the per-user context → move message accordingly → update `verdicts.learned`.
- Passive: hook on `flags.changed`/move events — user moves message out of Junk ⇒ learn ham; into Junk ⇒ learn spam (dedup via `verdicts.learned`).
- Provider bootstrap (per account, if `learn_from_provider_junk`): during/after backfill, sample up to 500 provider-Junk messages as spam corpus and 500 aged inbox messages as ham, rate-limited background job. Gives per-user Bayes a warm start.
### C.6 API & UI
`GET/PATCH /api/plugins/antispam/settings` · `GET/POST/DELETE /rules` · `POST /learn` · `GET /stats` (verdict counts, top symbols, Bayes confidence). UI: `settings.page` (thresholds with plain-language labels, per-account toggles, lists editor, privacy toggle, stats); `message.banner` for the suspect zone ("Looks like spam — [Move to Junk] [Not spam] [Always trust sender]"); `message.actions` Spam/Not-spam. Junk folder view is core (SPECIAL-USE) and needs no changes.
### C.7 M6 edge integration (assumption, specified fully with M6)
At the hosted-mail phase the VPS edge (Stalwart's built-in filter or Postfix+Rspamd) performs first-pass filtering with the checks that only make sense at SMTP time (connecting IP, SPF/DKIM/DMARC verification, greylisting, rate limits); accepted mail is tagged with its edge score in a header, and this plugin trusts that header for hosted accounts instead of re-scanning — one user-visible spam system, two enforcement points.
### C.8 Acceptance criteria
1. Zero core diffs; install/uninstall `--drop-data` clean; with the plugin absent, mail flow is byte-identical to v1.4 core.
2. Rspamd container stopped ⇒ new mail still indexes and renders normally within normal latency (fail-open proven by test).
3. Verdict propagation: a moved-to-Junk verdict appears in the provider's Junk folder via IMAP within one sync cycle; "Not spam" moves it back and the same message is not re-classified.
4. Learning: after 20 spam + 20 ham teachings on a fixture corpus, a held-out obvious-spam fixture scores above `move_threshold` for that user and a ham fixture below `banner_threshold` (per-user isolation: a second user's scores are unaffected).
5. Deny-listed sender is moved without engine involvement; allow-listed sender bypasses even a high engine score.
6. Privacy: with `network_checks=false` (default), a network-capture test during classification shows zero outbound connections from Rspamd.
---
*End of specification. Ambiguities: prefer the simplest behavior consistent with the invariants in §1, leave a `TODO(spec)` comment, and list open questions in `docs/open-questions.md` rather than guessing silently.*