lukas/webmail-client
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity
Files
08d428de8a87879b636bc8c3efbed3c7fa9dd736
webmail-client/webmail-client-spec.md
T

740 lines
56 KiB
Markdown
Raw Normal View History

Initial commit: webmail client (M0/M1)
2026-07-17 23:20:34 +02:00
# Webmail Client — Implementation Specification (v1.2, agent-ready)
This document is a complete, self-contained build specification. It is written to be handed to an AI coding agent. Read it fully before writing code. The companion document `webmail-client-concept.md` gives background; where they differ, **this spec wins**.
---
## 0. Instructions to the implementing agent
1. **Build milestone by milestone** (§12), in order. Do not start a milestone before the previous one's acceptance criteria pass.
2. **Do not add features beyond this spec.** Anything not specified here is out of scope; leave `TODO(spec)` comments instead of inventing behavior.
3. **Definition of done** for every task: code + type hints + unit tests + passing `ruff check`, `ruff format --check`, `mypy --strict` (backend) / `tsc --noEmit`, `eslint` (frontend) + updated docs in `docs/` if an interface changed.
4. **Stability over novelty:** use exactly the pinned dependencies in §3. Do not swap libraries or add dependencies without a `# DEPENDENCY:` comment explaining why, and never for something the stdlib does adequately.
5. **All interfaces in §5 are frozen contracts.** Implementations may evolve; signatures may not (without bumping the version comment at the top of the interface file).
6. **Security rules in §10 are hard requirements**, not suggestions.
7. Every module publishing or consuming events must use the event names in §8 verbatim.
8. Commit in small, working increments; every commit must leave the test suite green.
## 1. Architecture invariants (never violate)
- **I-1 Sync-first:** the web UI and REST API never open IMAP connections. Only sync workers talk to mail servers. The API reads exclusively from Postgres/Meilisearch.
- **I-2 Independent identity:** login uses app accounts (`localpart@our-web-client.example`) with Argon2id hashes. Mailbox credentials are payload data, encrypted at rest, decrypted only inside sync workers.
- **I-3 One index, many views:** unified inbox, per-account views, virtual mailboxes, and search results are all parameterized queries over the same index. No duplicated storage per view.
- **I-4 Threading is ours:** JWZ in our backend (§7). Never use the IMAP `THREAD` extension.
- **I-5 Direction-agnostic thread UI:** thread ordering (oldest-first/newest-first) is a data-ordering concern. One layout, one composer-anchor slot (§9.4).
- **I-6 Plugins extend, core stays closed:** new features beyond this spec must be expressible through the plugin surfaces (§11).
- **I-7 Multi-user isolation:** every query (SQL and search) carries a mandatory `user_id` filter. No exceptions, including admin tooling.
- **I-8 Mail HTML is hostile input:** sanitize with nh3 server-side at index time AND render in a sandboxed iframe. Both, always.
## 2. Repository layout (monorepo)
```
webmail/
├── backend/
│ ├── pyproject.toml
│ ├── app/
│ │ ├── main.py # FastAPI app factory
│ │ ├── config.py # pydantic-settings; all config via env
│ │ ├── db.py # SQLAlchemy async engine/session
│ │ ├── events.py # EventBus (Redis pub/sub + arq enqueue)
│ │ ├── identity/ # §6.1 auth, sessions, handles, TOTP
│ │ ├── accounts/ # §6.2 wizard, autodiscovery, credential vault
│ │ ├── providers/
│ │ │ ├── base.py # MailProvider interface (§5.1)
│ │ │ └── imap_smtp/ # first implementation
│ │ ├── sync/ # §6.3 backfill, IDLE loops, reconciliation
│ │ ├── indexer/ # §6.4 MIME parsing → DB rows + blobs
│ │ ├── threading_/ # §7 JWZ (trailing _ avoids stdlib clash)
│ │ ├── search/
│ │ │ ├── base.py # SearchBackend interface (§5.2)
│ │ │ ├── queryparser.py # operator grammar (§8.2)
│ │ │ ├── pg.py # Postgres FTS backend (M0–M1)
│ │ │ └── meili.py # Meilisearch backend (M2)
│ │ ├── outbox/ # §6.5 drafts, MIME assembly, send queue, undo
│ │ ├── api/ # routers, one file per §9.1 group
│ │ ├── ws.py # WebSocket hub (§9.2)
│ │ └── plugins/ # §11 loader + plugin packages
│ ├── workers/worker.py # arq worker entrypoint
│ ├── migrations/ # alembic
│ └── tests/
├── frontend/
│ ├── package.json
│ └── src/
│ ├── api/ # generated client from OpenAPI + WS layer
│ ├── routes/ # §9.3
│ ├── components/
│ │ ├── thread/ThreadView.tsx # §9.4 direction-agnostic
│ │ ├── thread/Composer.tsx
│ │ ├── list/MessageList.tsx # virtualized
│ │ └── slots/ # plugin UI slots (§11)
│ └── lib/
├── docker/
│ ├── docker-compose.yml # app, worker, postgres:16, redis:7, meilisearch:v1.8, dovecot (test)
│ └── docker-compose.ci.yml # + greenmail for integration tests
└── docs/
```
## 3. Pinned stack
**Backend (Python 3.11):** fastapi ~0.115, uvicorn ~0.30, sqlalchemy ~2.0 (async, asyncpg), alembic ~1.13, pydantic ~2.8, pydantic-settings, arq ~0.26, redis ~5.0, aioimaplib ~1.1, imapclient ~3.0 (utility/fallback), aiosmtplib ~3.0, argon2-cffi ~23.1, pyotp ~2.9, dnspython ~2.6, httpx ~0.27, nh3 ~0.2, cryptography ~43.0, mail-parser ~4.0 (helpers only; primary parsing via stdlib `email.message.EmailMessage` with `policy.default`).
**Frontend:** TypeScript ~5.5, React ~18.3, Vite ~5, TanStack Query ~5, TanStack Virtual ~3, Radix UI, Tailwind ~3.4, zod (WS payload validation). API client generated from OpenAPI (`openapi-typescript`).
**Infra:** PostgreSQL 16 (with `citext` extension), Redis 7, Meilisearch v1.8 (M2+), MinIO or plain filesystem for the blob store (config-switchable; filesystem default).
## 4. Database schema (Postgres 16, authoritative DDL)
All PKs are `uuid DEFAULT gen_random_uuid()`. All tables carrying user data include `user_id` with an index. Timestamps are `timestamptz`.
```sql
CREATE EXTENSION IF NOT EXISTS citext;
CREATE TABLE users (
id uuid PRIMARY KEY DEFAULT gen_random_uuid(),
handle citext UNIQUE NOT NULL, -- full form: localpart@<APP_DOMAIN>
password_hash text NOT NULL, -- argon2id
totp_secret bytea, -- encrypted; NULL = 2FA off
recovery_email citext,
settings jsonb NOT NULL DEFAULT '{}', -- incl. {"thread_order":"asc"|"desc"}
created_at timestamptz NOT NULL DEFAULT now()
);
-- Handle rules (enforce in code, mail-grade from day one):
-- localpart: 3–30 chars, [a-z0-9._-], no leading/trailing/double dot,
-- case-insensitive, reserved list (postmaster, abuse, admin, root, noreply, ...).
CREATE TABLE accounts (
id uuid PRIMARY KEY DEFAULT gen_random_uuid(),
user_id uuid NOT NULL REFERENCES users(id) ON DELETE CASCADE,
email citext NOT NULL,
display_name text NOT NULL DEFAULT '',
color text NOT NULL DEFAULT '#888888',
imap_config jsonb NOT NULL, -- {host,port,tls:"ssl"|"starttls",auth:"password"|"oauth2"}
smtp_config jsonb NOT NULL, -- same shape
secret_ct bytea NOT NULL, -- envelope-encrypted password or OAuth token bundle (§10.2)
discovery_src text NOT NULL, -- preset|autoconfig|ispdb|srv|probe|manual
state text NOT NULL DEFAULT 'backfilling', -- backfilling|live|error|disabled
state_detail text NOT NULL DEFAULT '',
last_sync_at timestamptz,
created_at timestamptz NOT NULL DEFAULT now(),
UNIQUE (user_id, email)
);
CREATE TABLE mailboxes (
id uuid PRIMARY KEY DEFAULT gen_random_uuid(),
account_id uuid NOT NULL REFERENCES accounts(id) ON DELETE CASCADE,
user_id uuid NOT NULL REFERENCES users(id) ON DELETE CASCADE,
path text NOT NULL, -- IMAP mailbox name, UTF-7 decoded
special_use text, -- inbox|sent|drafts|trash|junk|archive|NULL
uidvalidity bigint NOT NULL DEFAULT 0,
highestmodseq bigint, -- NULL if no CONDSTORE
last_seen_uid bigint NOT NULL DEFAULT 0,
UNIQUE (account_id, path)
);
CREATE TABLE threads (
id uuid PRIMARY KEY DEFAULT gen_random_uuid(),
user_id uuid NOT NULL REFERENCES users(id) ON DELETE CASCADE,
subject_norm text NOT NULL, -- normalized subject (§7.3)
snippet text NOT NULL DEFAULT '',
participants jsonb NOT NULL DEFAULT '[]', -- [{name,email}], deduped, ordered by recency
account_ids uuid[] NOT NULL DEFAULT '{}',
message_count int NOT NULL DEFAULT 0,
unread_count int NOT NULL DEFAULT 0,
has_attachments boolean NOT NULL DEFAULT false,
last_message_at timestamptz NOT NULL
);
CREATE INDEX ON threads (user_id, last_message_at DESC);
CREATE TABLE messages (
id uuid PRIMARY KEY DEFAULT gen_random_uuid(),
user_id uuid NOT NULL REFERENCES users(id) ON DELETE CASCADE,
account_id uuid NOT NULL REFERENCES accounts(id) ON DELETE CASCADE,
mailbox_id uuid NOT NULL REFERENCES mailboxes(id) ON DELETE CASCADE,
uid bigint NOT NULL, -- IMAP UID within mailbox
thread_id uuid REFERENCES threads(id),
message_id_hdr text, -- Message-ID, angle brackets stripped
in_reply_to text,
references_hdr text[] NOT NULL DEFAULT '{}',
subject text NOT NULL DEFAULT '',
from_addr jsonb NOT NULL DEFAULT '[]',
to_addr jsonb NOT NULL DEFAULT '[]',
cc_addr jsonb NOT NULL DEFAULT '[]',
date_hdr timestamptz,
internal_date timestamptz NOT NULL,
flags text[] NOT NULL DEFAULT '{}', -- \Seen,\Flagged,\Answered,\Draft,\Deleted
size_bytes int NOT NULL DEFAULT 0,
has_attachments boolean NOT NULL DEFAULT false,
snippet text NOT NULL DEFAULT '', -- first ~160 chars of text body
blob_sha256 text NOT NULL, -- raw RFC822 in blob store (§6.4)
body_indexed tsvector, -- Postgres FTS (M0–M1 SearchBackend)
UNIQUE (mailbox_id, uid)
);
CREATE INDEX ON messages (user_id, thread_id);
CREATE INDEX ON messages (account_id, message_id_hdr);
CREATE INDEX ON messages USING gin (body_indexed);
CREATE TABLE body_parts (
id uuid PRIMARY KEY DEFAULT gen_random_uuid(),
message_id uuid NOT NULL REFERENCES messages(id) ON DELETE CASCADE,
kind text NOT NULL, -- text|html_sanitized
content text NOT NULL
);
CREATE TABLE attachments (
id uuid PRIMARY KEY DEFAULT gen_random_uuid(),
message_id uuid NOT NULL REFERENCES messages(id) ON DELETE CASCADE,
user_id uuid NOT NULL,
filename text NOT NULL,
mime_type text NOT NULL,
size_bytes int NOT NULL,
content_id text, -- for inline cid: images
blob_sha256 text NOT NULL
);
CREATE TABLE drafts (
id uuid PRIMARY KEY DEFAULT gen_random_uuid(),
user_id uuid NOT NULL REFERENCES users(id) ON DELETE CASCADE,
account_id uuid NOT NULL REFERENCES accounts(id) ON DELETE CASCADE,
thread_id uuid REFERENCES threads(id), -- NULL = new mail
reply_to_message_id uuid REFERENCES messages(id),
kind text NOT NULL DEFAULT 'reply', -- reply|reply_all|forward|new
payload jsonb NOT NULL, -- {to,cc,bcc,subject,html,text,attachment_ids}
updated_at timestamptz NOT NULL DEFAULT now()
);
CREATE TABLE outbox (
id uuid PRIMARY KEY DEFAULT gen_random_uuid(),
user_id uuid NOT NULL,
draft_id uuid NOT NULL,
state text NOT NULL DEFAULT 'undo_window', -- undo_window|sending|sent|failed|cancelled
send_after timestamptz NOT NULL, -- now() + undo window
error text
);
CREATE TABLE saved_searches (
id uuid PRIMARY KEY DEFAULT gen_random_uuid(),
user_id uuid NOT NULL REFERENCES users(id) ON DELETE CASCADE,
name text NOT NULL,
query text NOT NULL, -- raw operator-syntax string (§8.2)
pinned boolean NOT NULL DEFAULT true,
UNIQUE (user_id, name)
);
-- Core recipient autocomplete (Gmail-style, derived from mail history).
-- The Contacts plugin (Appendix B) ENRICHES suggestions; this table works without it.
CREATE TABLE known_recipients (
user_id uuid NOT NULL REFERENCES users(id) ON DELETE CASCADE,
email citext NOT NULL,
name text NOT NULL DEFAULT '', -- last seen display name
times_used int NOT NULL DEFAULT 0, -- weighted: sent-to counts 3, received-from counts 1
last_used_at timestamptz NOT NULL,
PRIMARY KEY (user_id, email)
);
CREATE INDEX ON known_recipients (user_id, email text_pattern_ops);
```
**Blob store:** raw RFC822 messages and attachment payloads are stored content-addressed at `${BLOB_ROOT}/{user_id}/{sha256[:2]}/{sha256}` (filesystem) or the equivalent object key (MinIO). Postgres holds metadata only. Deleting a user removes their blob subtree.
**Sessions** live in Redis (`sess:{token}` → JSON `{user_id, created, ua}`, TTL 30 days sliding), not Postgres.
## 5. Frozen core interfaces (Python)
### 5.1 `providers/base.py` — MailProvider
```python
# INTERFACE VERSION: 1
from dataclasses import dataclass
from typing import AsyncIterator, Protocol
from datetime import datetime
@dataclass(frozen=True)
class RemoteMailbox:
path: str
special_use: str | None # "inbox"|"sent"|"drafts"|"trash"|"junk"|"archive"|None
uidvalidity: int
@dataclass(frozen=True)
class RemoteMessage:
uid: int
raw: bytes # full RFC822
flags: frozenset[str]
internal_date: datetime
@dataclass(frozen=True)
class FlagChange:
uid: int
flags: frozenset[str]
@dataclass(frozen=True)
class MailboxDelta:
new: list[RemoteMessage]
flag_changes: list[FlagChange]
expunged_uids: list[int]
new_uidvalidity: int | None # non-None => UIDVALIDITY changed, caller must full-resync
new_highestmodseq: int | None
class MailProvider(Protocol):
async def verify(self) -> None: ...
# Raises ProviderAuthError / ProviderConnectError with a human-readable .detail
async def list_mailboxes(self) -> list[RemoteMailbox]: ...
def backfill(self, mailbox: str, *, newest_first: bool = True,
batch_size: int = 200) -> AsyncIterator[list[RemoteMessage]]: ...
# Yields batches until the mailbox is exhausted. Headers+body in one pass.
async def poll(self, mailbox: str, *, last_seen_uid: int,
highestmodseq: int | None) -> MailboxDelta: ...
# One reconciliation pass. Uses QRESYNC/CONDSTORE when highestmodseq is not None,
# otherwise UID SEARCH + FETCH FLAGS diff.
async def idle(self, mailbox: str) -> None: ...
# Blocks until the server signals change or 25 min elapse (re-issue IDLE). May raise
# ProviderNotSupported => sync loop falls back to poll() every POLL_INTERVAL (default 120 s).
async def push_flags(self, mailbox: str, uid: int,
add: frozenset[str], remove: frozenset[str]) -> None: ...
async def move(self, uid: int, src: str, dst: str) -> int: ... # returns new UID
async def append(self, mailbox: str, raw: bytes,
flags: frozenset[str]) -> int: ... # Sent/Drafts upload
async def send(self, raw: bytes, envelope_from: str,
envelope_to: list[str]) -> None: ... # SMTP submission
```
`ImapSmtpProvider` is the only M0–M2 implementation. A future `NativeProvider` (hosted mailboxes) or `JmapProvider` implements the same protocol — nothing above this interface may import IMAP libraries.
### 5.2 `search/base.py` — SearchBackend
```python
# INTERFACE VERSION: 1
@dataclass(frozen=True)
class SearchQuery:
text: str | None # free-text remainder after operator extraction
filters: dict[str, object] # parsed operators, keys per §8.2
user_id: UUID # ALWAYS set (invariant I-7)
@dataclass(frozen=True)
class SearchHit:
message_id: UUID
thread_id: UUID
score: float
highlights: dict[str, str] # field -> HTML-escaped snippet with <mark>
class SearchBackend(Protocol):
async def index(self, doc: SearchDoc) -> None: ...
async def delete(self, message_id: UUID) -> None: ...
async def delete_user(self, user_id: UUID) -> None: ...
async def query(self, q: SearchQuery, *, limit: int, cursor: str | None
) -> tuple[list[SearchHit], str | None]: ...
```
`SearchDoc` fields: `message_id, thread_id, user_id, account_id, mailbox_special_use, from_ (name+email), to, cc, subject, body_text, attachment_names[], has_attachments, is_unread, is_starred, date (unix), size_bytes`. Both backends (pg.py, meili.py) index the same doc. Meilisearch: one index `messages`, filterable = every non-body field incl. `user_id`, searchable = `subject, body_text, from_, to, cc, attachment_names`, sortable = `date`; ranking rules default + `date:desc` tiebreak.
### 5.3 `events.py` — EventBus
```python
# INTERFACE VERSION: 1
class EventBus(Protocol):
async def publish(self, event: str, payload: dict) -> None:
# Fan-out to Redis pub/sub channel "evt:{event}" AND enqueue arq job
# "on_{event}" for durable consumers (indexer, plugins).
def subscribe(self, event: str, handler: Handler) -> None: ...
```
Payloads always include `user_id`. Event names are listed in §8.3 and are the only ones allowed.
## 6. Backend module specifications
### 6.1 Identity (`app/identity/`)
- Registration: validate localpart per §4 rules against reserved list in `identity/reserved.py`; store full handle `localpart@{APP_DOMAIN}` (env `APP_DOMAIN`).
- Login: constant-time verify (argon2id, `time_cost=3, memory_cost=64MiB, parallelism=4`); if `totp_secret` set, require valid TOTP (±1 window). Rate limit: 10 attempts / 15 min per handle AND per IP (Redis counters); lockout returns 429 with `Retry-After`.
- Session cookie: `__Host-session`, httpOnly, Secure, SameSite=Lax. CSRF: double-submit token header `X-CSRF` for all mutating requests.
- TOTP setup: `POST /auth/totp/setup` returns provisioning URI + QR payload; enable requires one valid code; secret stored envelope-encrypted.
- Password reset: only if `recovery_email` verified; single-use token, 30 min TTL, Redis-backed.
### 6.2 Accounts & autodiscovery (`app/accounts/`)
`discover(email) -> list[CandidateConfig]` runs the chain, **returning on first success** but recording every step's outcome in the response (`steps: [{method, status, detail}]`) for wizard UX:
1. **Presets** — `accounts/presets.yaml` (config, not code): match by domain; ships with gmail.com, googlemail.com, outlook.com/hotmail/live, office365, yahoo, icloud/me.com, gmx, wp.pl, onet.pl, o2.pl, interia.pl. Each entry: imap/smtp host+port+tls, auth kind, oauth provider id if applicable.
2. **Autoconfig** — GET `https://autoconfig.{domain}/mail/config-v1.1.xml?emailaddress={email}` then `https://{domain}/.well-known/autoconfig/mail/config-v1.1.xml`; parse Thunderbird XML (incomingServer type=imap, outgoingServer type=smtp; `%EMAILADDRESS%`/`%EMAILLOCALPART%` substitution).
3. **ISPDB** — GET `https://autoconfig.thunderbird.net/v1.1/{domain}`; same XML format.
4. **DNS SRV (RFC 6186)** — `_imaps._tcp.{domain}` (SSL 993), `_imap._tcp` (STARTTLS 143), `_submissions._tcp` (465), `_submission._tcp` (587); lowest priority wins.
5. **Probe** — try `{imap,mail}.{domain}` × ports 993(ssl)/143(starttls) and `{smtp,mail}.{domain}` × 465(ssl)/587(starttls); TCP connect + TLS handshake + IMAP/SMTP greeting, 5 s timeout each, all concurrently.
All HTTP/DNS lookups: 5 s timeout, no redirects off-domain, HTTPS only. `test(config, credentials)`: full IMAP LOGIN + `LIST`, and SMTP EHLO+AUTH (no message sent); errors mapped to precise user-facing strings (`auth_failed`, `tls_failed`, `connect_timeout`, `unknown_host`). On save: encrypt credentials (§10.2), create account row (`state=backfilling`), enqueue `sync.bootstrap_account`.
### 6.3 Sync engine (`app/sync/`)
Per live account, the worker maintains one **supervisor task**:
- Bootstrap: `list_mailboxes()` → upsert `mailboxes` rows (map SPECIAL-USE; fallback name heuristics: Sent/Wysłane, Trash/Kosz, etc. in `sync/folder_names.py`); enqueue backfill jobs newest-first per mailbox (INBOX first, then sent, then rest).
- Backfill job: consume `backfill()` batches → indexer; update `last_seen_uid`; publish `sync.progress` every batch (`{account_id, mailbox, done, total}`); on completion of all mailboxes set `state=live`.
- Live loop per mailbox of interest (inbox, sent, drafts + any opened recently): `idle()` → on wake, `poll()` → apply `MailboxDelta` (new → indexer; flag_changes → update rows + `flags.changed`; expunged → soft-handling: if the same `message_id_hdr` exists elsewhere in the account keep thread intact, else delete row + search doc). `new_uidvalidity` ⇒ wipe mailbox rows and re-backfill it.
- Outbound flag changes (user actions) are written to DB first, then enqueued as `sync.push_flags` jobs; on provider failure retry ×5 exponential, then revert DB and emit `flags.changed` with `reverted=true`.
- Concurrency limits: ≤ 4 IMAP connections per account (1 IDLE inbox + up to 3 job connections), global worker cap via arq `max_jobs`.
### 6.4 Indexer (`app/indexer/`)
Input: `RemoteMessage`. Steps: store raw blob (sha256, dedupe) → parse with `email` stdlib (`policy.default`) → extract: headers (decode RFC2047), `references_hdr` = parsed `References` + `In-Reply-To` message-ids in order; best text body and best HTML body (multipart/alternative preference: html for display, text for indexing; if only HTML, derive text via nh3 strip); sanitize HTML with nh3 (allowlist: structural tags, inline styles filtered, `cid:` images rewritten to `/api/messages/{id}/cid/{content_id}`, remote `src` rewritten to blocked-placeholder attributes for FR-23) → write `messages`, `body_parts` (kinds `text` and `html_sanitized`), `attachments` → hand to Threader (§7) → `bus.publish("message.new", ...)` (search indexing + thread denormalization are consumers). For non-junk received messages, upsert the `From` address into `known_recipients` (weight 1).
### 6.5 Outbox (`app/outbox/`)
- Drafts: autosave endpoint upserts `drafts` (client debounces 2 s); a background job mirrors drafts to the account's Drafts folder via `append()` at most every 60 s per draft (replace previous UID).
- Send: `POST /send {draft_id}` → build MIME (html + auto-generated text alternative via html-to-text; `In-Reply-To`/`References` from `reply_to_message_id`; attachments from blob store) → create `outbox` row with `send_after = now()+UNDO_SECONDS` (env, default 10) → schedule arq job at `send_after`. Undo endpoint flips state to `cancelled` if still in window. The job: SMTP `send()`, then `append()` to Sent (skip for Gmail preset — Gmail self-appends), mark `sent`, publish `message.sent`, and upsert every To/Cc/Bcc address into `known_recipients` (weight 3). Failure → `failed` + WS notification, draft preserved.
## 7. Threading specification (JWZ, `app/threading_/`)
### 7.1 Normalization
- `message_id_hdr`, `in_reply_to`, each `References` entry: strip angle brackets and whitespace, lowercase the domain part only. Discard obviously invalid ids (no `@`).
- Missing/duplicate `Message-ID`: synthesize `sha256(raw)[:32]@synthetic.local` and mark it non-authoritative (never used to join other messages to it).
### 7.2 Incremental algorithm (per new message)
1. Candidate set = all normalized ids in `references_hdr` + `in_reply_to` + own `message_id_hdr`.
2. `SELECT DISTINCT thread_id FROM messages WHERE user_id=$1 AND message_id_hdr = ANY($2) AND thread_id IS NOT NULL`.
3. 0 threads → subject fallback (§7.3); still none → create new thread.
4. 1 thread → attach.
5. ≥2 threads → **merge**: keep the thread with the oldest `last_message_at` as canonical, repoint messages of the others, delete emptied thread rows, publish `thread.updated` for all involved.
6. Recompute denormalized thread fields (count, unread, participants, snippet = newest message's snippet, has_attachments, account_ids, last_message_at) in one UPDATE.
Threads are cross-account by design: the candidate lookup is scoped by `user_id`, not `account_id` (invariant I-3). Sent-folder messages join threads exactly like received ones (FR-15).
### 7.3 Subject fallback
Only when reference joining fails: normalize subject (strip repeated `Re:/Fw:/Fwd:/Odp:/PD:` prefixes case-insensitively, collapse whitespace); join to a thread with the same `subject_norm` that has activity within 7 days AND shares ≥1 participant email; never join two messages that both lack references AND have empty subjects.
### 7.4 Test vectors (must pass in `tests/threading/`)
Linear chain; chain arriving out of order; reply missing `In-Reply-To` but carrying `References`; thread merge on late-arriving parent; broken client (subject-only) joins; distinct-topic same-subject ("hi") does NOT join; cross-account merge (message in account A references message in account B); duplicate Message-ID; synthetic-ID isolation.
## 8. Search specification
### 8.1 Pipeline
Consumer of `message.new` / `message.changed` / `message.deleted` → builds `SearchDoc` → active `SearchBackend` (env `SEARCH_BACKEND=pg|meili`). Backfill indexes inline with sync (per batch), never as a separate pass the user waits on (FR-30).
### 8.2 Query grammar (`search/queryparser.py`, engine-independent)
Tokenization: whitespace-split, respecting `"quoted phrases"`. A token matching `key:value` (or `-key:value`) with a known key becomes a filter; everything else joins free text. `-` prefix negates a term or filter. `OR` (uppercase) groups adjacent terms/filters into a disjunction; default combinator is AND.
| Operator | Filter semantics |
|---|---|
| `from:` `to:` `cc:` | substring match on name or email of that field |
| `subject:` | match constrained to subject |
| `account:` | account by `display_name` or email substring |
| `in:` | `inbox\|sent\|drafts\|trash\|junk\|archive` (special_use) or folder path |
| `is:` | `unread\|read\|starred\|answered` |
| `has:attachment` | boolean |
| `filename:` | attachment name substring |
| `before:` `after:` | `YYYY-MM-DD` |
| `newer_than:` `older_than:` | `Nd\|Nw\|Nm\|Ny` |
| `larger:` `smaller:` | `NK\|NM` bytes |
Parser output = `SearchQuery` (§5.2). Unknown `key:` tokens are treated as free text (forward compatibility). Unit tests must cover every operator, negation, quoting, `OR`, and garbage input (parser never raises).
### 8.3 Event catalog (complete)
`message.new`, `message.changed`, `message.deleted`, `message.sent`, `flags.changed`, `thread.updated`, `sync.progress`, `account.state_changed`, `draft.saved`, `outbox.state_changed`. Payloads: `{user_id, ...ids, minimal-diff fields}` — documented in `docs/events.md` (agent: generate it from code).
Plugins may additionally publish events namespaced `plugin.{name}.*` (e.g. `plugin.calendar.event_updated`); core code never subscribes to plugin events, but the WS hub forwards them to the owning user like any other event.
## 9. API, realtime, frontend
### 9.1 REST API (prefix `/api`, session-cookie auth, JSON; errors: `{error:{code,message}}`)
**Auth:** `POST /auth/register` · `POST /auth/login` · `POST /auth/logout` · `GET /auth/me` · `GET /auth/sessions` · `DELETE /auth/sessions/{id}` · `POST /auth/totp/setup` · `POST /auth/totp/enable` · `POST /auth/password/reset-request` · `POST /auth/password/reset`
**Accounts:** `POST /accounts/discover {email}` → `{candidates:[{imap,smtp,source}], steps:[...]}` · `POST /accounts/test {config, credentials}` → `{ok}|{error:{code,detail}}` · `POST /accounts` · `GET /accounts` (incl. `state`, sync progress) · `PATCH /accounts/{id}` · `DELETE /accounts/{id}` (cascades rows, blobs, search docs)
**Mail:** `GET /threads?view=unified|account:{id}|mailbox:{id}|search&q=&cursor=&limit=50` → thread summaries (denormalized fields only; cursor = `last_message_at,id` keyset) · `GET /threads/{id}?sort=asc|desc` → ordered full messages (bodies of the N newest expanded per client hint `expand=`; older = envelope only) · `POST /threads/{id}/flags {add,remove}` (whole thread) · `POST /messages/{id}/flags` · `POST /messages/{id}/move {mailbox_id}` · `GET /messages/{id}/html` (sanitized, for iframe `srcdoc`) · `GET /messages/{id}/cid/{content_id}` · `GET /attachments/{id}` (Content-Disposition attachment; images/PDF also inline for preview) · `POST /messages/{id}/allow-remote-images` (per-sender allowlist in user settings)
**Compose:** `POST /drafts` · `PUT /drafts/{id}` · `DELETE /drafts/{id}` · `POST /send {draft_id}` → `{outbox_id, undo_until}` · `POST /send/{outbox_id}/undo` · `GET /recipients/suggest?q=` → top 10 from `known_recipients` (prefix match on email or name, ranked by `times_used` decayed by recency); the composer merges these with any plugin capability providers (§11.3)
**Search:** `GET /search?q=&cursor=` → `{threads:[...], hits_by_thread, highlights}` (grouped by thread, FR-28; `&group=message` for flat) · `GET/POST/DELETE /saved-searches`
**Settings:** `GET/PATCH /settings` (incl. `thread_order: "asc"|"desc"`, `tz` IANA timezone — default from browser at registration, undo window, per-sender image allowlist)
### 9.2 WebSocket `/api/ws`
Auth via session cookie. Server → client frames: `{event, payload}` for every §8.3 event belonging to the user (that's the whole contract — the UI refetches via TanStack Query invalidation keyed on ids in the payload). Client → server: `{type:"ping"}` only. Zod-validate all frames.
### 9.3 Frontend routes
`/login`, `/register`, `/mail` (unified inbox), `/mail/a/:accountId`, `/mail/a/:accountId/f/:mailboxId`, `/mail/t/:threadId` (thread pane; on desktop rendered beside the list), `/mail/search?q=`, `/settings/{profile|security|accounts|appearance}`; `/settings/accounts/new` = wizard.
**Account wizard flow:** step 1 email (+ password field revealed unless preset says oauth) → calls `discover` while typing pauses; step 2 shows resolved config with source badge ("found via ISPDB") and a Test button (auto-run); failure expands the manual form pre-filled with the best guess and the precise error; step 3 = created, live backfill progress bar (WS `sync.progress`) with "start reading now" (list fills as backfill proceeds).
### 9.4 ThreadView contract (invariant I-5)
```
ThreadView(props: { messages: Message[] /* ASC always */, order: 'asc'|'desc',
composerState, ... })
```
- Rendering order = `order==='asc' ? messages : [...messages].reverse()`. No other branching on `order` is allowed in layout code.
- **Composer anchor** = a slot component rendered adjacent to the newest message: after it when `asc` (bottom of thread), directly after it when `desc` (top of thread, above older messages). Same component, one placement rule: "composer is always attached to the newest message."
- Initial scroll: to the newest message (bottom for asc, top for desc). Collapse: all read messages except the newest are collapsed; "Expand all" toggles. Unread messages are never collapsed.
- Keyboard: `j`/`k` move in *visual* order; `o`/`Enter` expand; `r` reply, `a` reply-all, `f` forward (open composer at anchor); `e` archive; `/` focus search; `gi` go to unified inbox; `esc` close composer (draft persists). Toolbar has the per-thread order toggle; it updates local state only, and long-presses/menu offers "make default" (PATCH /settings).
- Message HTML renders in `<iframe sandbox="" srcdoc={...}>` sized via ResizeObserver postMessage shim; remote images show a "load images / always from this sender" bar when the sanitizer flagged any.
## 10. Security requirements (hard)
1. Argon2id params per §6.1; no other password hashing anywhere.
2. **Credential vault:** mailbox secrets encrypted with AES-256-GCM under per-account data keys, wrapped by a master key from env `MASTER_KEY` (32 B, base64) — `cryptography` library, key id embedded for future rotation. Decryption code path exists only in `workers/` and `accounts/test`. Secrets never appear in logs, API responses, or exceptions (custom repr).
3. CSRF double-submit on all mutating routes; `__Host-` cookie prefix; CSP: `default-src 'self'; frame-src 'self'; img-src 'self' data:` (remote mail images are proxied? **No** — v1 loads allowed remote images inside the sandboxed iframe only, which has its own relaxed CSP via srcdoc; document this in code).
4. nh3 sanitization at index time; re-sanitize on render if sanitizer version changed (store `sanitizer_v` on body_parts).
5. Rate limits: login (§6.1), `POST /accounts/discover` 10/min/user, `POST /send` 30/h/user (env-tunable).
6. All outbound autodiscovery HTTP: HTTPS only, 5 s timeout, response size cap 256 KB, no off-domain redirects (SSRF guard: resolve + reject private IP ranges).
7. Every SQL query and search query includes `user_id` (enforced by repository-layer helpers; raw session use in API code is a review error).
8. Uploads (attachments to drafts): 25 MB/file, 40 MB/message total, content-type sniffed server-side, stored to blob store, never executed/served with original content type unless in safe list (images/pdf/text) — others force `application/octet-stream` + attachment disposition.
## 11. Plugin API (freeze in M3 — including the heavy-plugin surfaces below)
A plugin = Python package `webmail_plugin_<name>` exposing:
```python
plugin = Plugin(
name, version,
hooks = {event: handler}, # subscribe to §8.3 events
router = APIRouter | None, # mounted at /api/plugins/{name}
ui_manifest = UiManifest | None, # slots + pages, see below
migrations = "migrations/" | None, # 11.1
jobs = [JobSpec, ...] | None, # 11.2
)
```
Loader discovers plugins via entry points group `webmail.plugins`, forwards declared events, and serves `GET /api/plugins/ui-manifest` for the frontend loader. Plugins may not import from `app.*` internals other than `app.plugins.sdk` (which exposes: DB session factory scoped to plugin tables, EventBus publish under `plugin.{name}.*`, blob store, outbox draft/send helpers, settings accessor, HTTP client with the §10.6 SSRF guard built in).
Every plugin gets the namespaced KV store (`plugin_kv(plugin, user_id, key)`) for light state. Heavy plugins additionally get:
### 11.1 Plugin-owned tables
A plugin may ship Alembic migrations in its own version branch (`alembic -n plugin_{name}`). All tables MUST be prefixed `plg_{name}_` and every user-data table MUST carry `user_id` (invariant I-7 applies to plugins). Core migrations never reference plugin tables and vice versa. Uninstalling a plugin offers `--drop-data` to run its downgrade chain.
### 11.2 Plugin background jobs
`JobSpec(func, schedule: cron_str | None, singleton: bool)` registers tasks on the shared arq worker: cron-scheduled (e.g. a sync loop trigger) or enqueued on demand from hooks/routes. `singleton=True` guarantees at most one running instance per (plugin, user) via a Redis lock. Plugin jobs run in the same worker process pool with the same retry semantics as core jobs and are visible in worker logs under `plg.{name}.*`.
### 11.3 Frontend surfaces
- **Slots** (lazy-loaded ESM injected into mail UI): `thread.toolbar`, `message.actions`, `sidebar.section`, `settings.page`.
- **Pages** (`app.page`): a plugin may declare full views: `{route: "/calendar", nav: {icon, label, order}, bundle}`. The shell adds the nav entry, and the route (including sub-routes under it) lazy-loads the bundle into the **entire content area** — no mail list/thread components are rendered. Page bundles receive the SDK context (API client, WS event stream filtered to `plugin.{name}.*` + requested core events, design-system components) and are sandboxed to their route: importing mail-component internals is a review error. Removing the plugin removes the nav entry and route cleanly.
- **Capabilities**: the manifest may declare named capabilities mapping to plugin endpoints, e.g. `capabilities: {"recipients.suggest": "/api/plugins/contacts/suggest"}`. Core UI features query all providers of a capability they support and merge results client-side (defined capabilities in v1: `recipients.suggest` — used by the composer To/Cc/Bcc fields and by any other plugin needing person-picking, such as calendar attendees). This keeps core backend code plugin-unaware.
Acceptance test for the API: implement **filters** (rules on `message.new`: match → move/flag) and **labels** entirely as plugins using only slots + KV; then implement the **calendar** (Appendix A) using tables + jobs + a page — all with zero core diffs.
## 12. Milestones with acceptance criteria
**M0 — Architecture spike**
Login/register (no 2FA), one hardcoded-config IMAP account, backfill+IDLE for INBOX, JWZ, read-only unified list + thread view with working asc/desc toggle and correct composer-anchor placeholder, Postgres FTS free-text search.
✅ Accept: sync a real 10k-message Gmail account via app password; threads match Gmail's grouping on a 50-thread sample ≥ 90%; toggle order and the newest message + anchor swap ends correctly; search returns in < 500 ms.
**M1 — MVP**
Full identity (2FA, sessions UI, reset), account wizard with full discovery chain, multi-account + unified inbox with account chips, flags/move/archive both directions, inline reply + reply-all/forward + SMTP send + Sent append, recipient autocomplete from `known_recipients`, drafts autosave, sanitized HTML + blocked remote images, operator search (full §8.2 grammar) on PG FTS, WS live updates.
✅ Accept: add gmail.com, outlook.com, and one autoconfig-only domain through the wizard typing only address+password; reply from a thread lands correctly threaded in the recipient's client; all §8.2 operators pass integration tests; `docker-compose up` from clean checkout works.
**M2 — Hardening / v1.0**
OAuth2 (Google + Microsoft, XOAUTH2 for IMAP/SMTP, server-side refresh), QRESYNC fast resync, drafts mirrored to server, undo-send, attachment upload/preview, Meilisearch backend + instant search-as-you-type UI + saved searches as virtual mailboxes, session revocation, backfill progress UX, folder-name i18n heuristics.
✅ Accept: typo query (`invocie`) finds "invoice" mail < 200 ms on a 100k-message corpus; kill -9 the worker mid-backfill and restart resumes without duplicates (idempotency via `(mailbox_id, uid)` unique key); undo-send within window results in no SMTP traffic.
**M3 — Plugin platform**
Loader, full SDK per §11 **including** plugin migrations (11.1), plugin jobs (11.2), and `app.page` (11.3); filters + labels shipped as plugins; plugin docs.
✅ Accept: both plugins install/uninstall cleanly with zero core diffs; a third trivial plugin ("Unsubscribe header button" on `message.actions`) can be written from `docs/plugins.md` alone; a skeleton page-plugin (route + nav entry + one plugin table + one cron job) round-trips install → run → uninstall `--drop-data` leaving no residue.
**M4 — Calendar plugin (Appendix A)**
Phases A1–A2: local calendars + full page views + iMIP invite handling; then A3: CalDAV client sync + ICS subscriptions.
✅ Accept: criteria in Appendix A §A.7.
**M5 — Contacts plugin (Appendix B)**
Phases B1–B2: contact books, full page, groups, add-from-mail, `recipients.suggest` capability, vCard import/export; then B3: CardDAV client sync. May be built in parallel with M4 — both depend only on the M3 SDK.
✅ Accept: criteria in Appendix B §B.7.
**M6 (optional) — Hosted mailboxes**
Inbound MTA (Stalwart) delivering to a `NativeProvider`; handle becomes receiving address. Specified separately when reached.
## 13. Testing & CI
- **Unit:** pytest + pytest-asyncio; threading vectors (§7.4); query parser (§8.2); autodiscovery resolver with `respx`-mocked HTTP + `dnspython` stub; MIME corpus in `tests/fixtures/mime/` (agent: include ≥ 25 nasty real-world-style fixtures — encoded headers incl. RFC2047 Polish diacritics, nested multiparts, missing Message-ID, malformed dates, HTML-only, inline cid images, 8-bit subjects, winmail.dat placeholder).
- **Integration:** docker-compose.ci.yml with Greenmail (IMAP+SMTP) and Dovecot; full loop test: append message to Greenmail → appears in API < 10 s → reply via API → arrives in Greenmail SMTP sink with correct References.
- **Frontend:** vitest + React Testing Library; ThreadView ordering/anchor tests for both directions; Playwright smoke: login → add Greenmail account via wizard → read → inline reply → search.
- **CI gates:** ruff, mypy --strict, pytest, tsc, eslint, vitest, Playwright smoke; coverage floor 80% on `threading_`, `search/queryparser`, `accounts/autodiscovery`, `indexer`.
## 14. Configuration (env, complete list)
`APP_DOMAIN`, `DATABASE_URL`, `REDIS_URL`, `MASTER_KEY`, `BLOB_ROOT` or `S3_*`, `SEARCH_BACKEND=pg|meili`, `MEILI_URL`, `MEILI_KEY`, `UNDO_SECONDS=10`, `POLL_INTERVAL=120`, `SESSION_TTL_DAYS=30`, `OAUTH_GOOGLE_CLIENT_ID/SECRET`, `OAUTH_MS_CLIENT_ID/SECRET`, `MAX_ATTACHMENT_MB=25`, `LOG_LEVEL`. All read once in `config.py` (pydantic-settings); no `os.environ` elsewhere.
---
## Appendix A — Calendar plugin (`webmail_plugin_calendar`)
The calendar is the reference "heavy plugin": it lives in the same app and repo, uses the §11 SDK only, and must build with **zero core diffs**. It ships as a page (`/calendar`) beside Mail — one product, one login.
### A.1 Scope by phase
- **A1 — Local calendar:** create/edit/delete events, multiple calendars per user, month/week/day/agenda views, recurrence, reminders (WS notification), timezone-correct display.
- **A2 — Invites over mail (iMIP):** detect `text/calendar` parts on `message.new`; render an invite card in the thread (`message.actions` + inline summary); Accept/Tentative/Decline sends an iTIP REPLY through the outbox; organizer flow sends REQUEST/CANCEL to attendees; updates (SEQUENCE bumps) modify the stored event.
- **A3 — External sources:** `CalendarProvider` interface with **CalDAV client** (first implementation; covers iCloud, Fastmail, Nextcloud) and read-only **ICS URL subscriptions** (holidays, team feeds); sync loop as a plugin job.
- **A4 (deferred, spec later):** Google/Microsoft API providers reusing M2 OAuth machinery; secret **ICS export feed** `GET /api/plugins/calendar/feed/{token}.ics`; CalDAV **server** endpoint for native phone sync — architecturally anticipated, not built now.
### A.2 Dependencies
`icalendar ~5.0` (parse/serialize RFC 5545), `python-dateutil` (RRULE expansion), `caldav ~1.3` (A3), `tzdata`. No other additions.
### A.3 Data model (plugin migrations, prefix `plg_calendar_`)
```sql
CREATE TABLE plg_calendar_calendars (
id uuid PRIMARY KEY DEFAULT gen_random_uuid(),
user_id uuid NOT NULL,
name text NOT NULL,
color text NOT NULL DEFAULT '#3b82f6',
kind text NOT NULL DEFAULT 'local', -- local|caldav|ics_subscription
source_config jsonb NOT NULL DEFAULT '{}', -- caldav url/collection, ics url
secret_ct bytea, -- caldav credentials, §10.2 vault scheme
sync_token text, -- CalDAV sync-token / ICS etag
read_only boolean NOT NULL DEFAULT false,
UNIQUE (user_id, name)
);
CREATE TABLE plg_calendar_events (
id uuid PRIMARY KEY DEFAULT gen_random_uuid(),
user_id uuid NOT NULL,
calendar_id uuid NOT NULL REFERENCES plg_calendar_calendars(id) ON DELETE CASCADE,
ical_uid text NOT NULL, -- UID from iCalendar; join key for iMIP updates
sequence int NOT NULL DEFAULT 0,
summary text NOT NULL DEFAULT '',
description text NOT NULL DEFAULT '',
location text NOT NULL DEFAULT '',
dtstart timestamptz NOT NULL,
dtend timestamptz,
all_day boolean NOT NULL DEFAULT false,
tz text NOT NULL DEFAULT 'UTC', -- original TZID; render in user tz, store UTC
rrule text, -- raw RRULE string, NULL = single
exdates timestamptz[] NOT NULL DEFAULT '{}',
recurrence_id timestamptz, -- non-NULL = detached exception instance
organizer jsonb, -- {name,email}
attendees jsonb NOT NULL DEFAULT '[]', -- [{name,email,partstat,role}]
status text NOT NULL DEFAULT 'confirmed', -- confirmed|tentative|cancelled
my_partstat text, -- needs-action|accepted|declined|tentative
reminders int[] NOT NULL DEFAULT '{}', -- minutes before start
origin_message_id uuid, -- link back to the invite mail thread
raw_ical text, -- round-trip fidelity for iMIP/CalDAV
UNIQUE (calendar_id, ical_uid, recurrence_id)
);
CREATE INDEX ON plg_calendar_events (user_id, dtstart);
```
**Recurrence rule:** store the RRULE string; expand instances at query time with `dateutil.rrule` over the requested view range only (no materialization table). Exceptions: `exdates` remove instances; a row with `recurrence_id` overrides one instance. Range queries must account for events starting before the window but recurring into it (expand from `dtstart`, cap iterations at 1000 per event).
### A.4 CalendarProvider interface (inside the plugin, mirrors §5.1)
```python
# PLUGIN INTERFACE VERSION: 1
class CalendarProvider(Protocol):
async def verify(self) -> None: ...
async def list_collections(self) -> list[RemoteCalendar]: ...
async def pull(self, collection: str, *, sync_token: str | None
) -> tuple[list[RemoteEvent], list[str], str]: ...
# returns (changed_events_as_ical, deleted_uids, new_sync_token)
async def push(self, collection: str, ical: str, *, uid: str,
etag: str | None) -> str: ... # returns new etag
async def delete(self, collection: str, uid: str, etag: str | None) -> None: ...
```
Implementations in A3: `CaldavProvider`, `IcsSubscriptionProvider` (pull-only, `push`/`delete` raise `ReadOnlyCalendar`). Sync loop = plugin cron job (`schedule="*/5 * * * *"`, `singleton=True` per user) doing pull → upsert by `(calendar_id, ical_uid, recurrence_id)` → local-dirty push with etag conflict handling (conflict ⇒ server wins, local copy saved as a duplicate marked `status='tentative'`, user notified).
### A.5 iMIP flow (A2)
- **Inbound:** hook on `message.new`; if a `text/calendar` part exists, parse METHOD:
`REQUEST` → upsert event (match `ical_uid`; apply only if `SEQUENCE` ≥ stored) into the user's default calendar with `my_partstat='needs-action'`, set `origin_message_id`; `CANCEL` → `status='cancelled'`; `REPLY` → update that attendee's `partstat` on the organizer's copy.
- **RSVP:** invite card buttons call `POST /api/plugins/calendar/events/{id}/rsvp {partstat}` → update row → build iTIP REPLY ics → send via SDK outbox helper to the organizer (subject `Accepted: {summary}` etc.) → publish `plugin.calendar.event_updated`.
- **Organizer:** creating an event with attendees sends METHOD:REQUEST to each; edits bump `SEQUENCE` and re-send; deletion sends CANCEL. All outgoing invite mail uses the account whose address matches the organizer identity (picker if ambiguous).
### A.6 UI (page `/calendar`, nav icon between Mail and Settings)
Routes: `/calendar` (month default), `/calendar/{month|week|day|agenda}/:date`, `/calendar/event/:id`, `/calendar/settings` (calendars CRUD, CalDAV/ICS wizard reusing the account-wizard UX pattern: URL+credentials → test → collection picker). Interactions: click-drag create, drag-move/resize (writes optimistically, reverts on API error), event popover with edit/RSVP, mini-month navigator, calendar visibility toggles (client state), `t` = today, arrows = prev/next period. Invite linkage both ways: invite card in thread links to `/calendar/event/:id`; event view links back to the mail thread when `origin_message_id` is set. Live updates via WS `plugin.calendar.*` events. Timezone: render in user setting `settings.tz` (add to §9.1 settings; default from browser at registration).
### A.7 Acceptance criteria
1. Plugin installs/uninstalls (`--drop-data`) with zero diffs in `app/` and `frontend/src/` outside `plugins/`/`slots/` loader code.
2. A weekly-recurring event with two EXDATEs and one moved instance renders correctly in month and week views across a DST boundary (Europe/Warsaw fixtures).
3. Round-trip with Google Calendar **via iMIP mail only**: receive a Google invite → Accept in our UI → organizer's Google Calendar shows attendee accepted; our organizer invite → Google attendee accepts → our event shows `partstat=accepted`.
4. CalDAV sync against a Nextcloud/Radicale test container: create/edit/delete propagate both directions in ≤ 5 min (one cron cycle); conflict case produces the A.4 tentative-duplicate, never silent loss.
5. ICS subscription to a public holiday feed renders read-only events; editing is blocked in UI and API (403 `read_only_calendar`).
6. Killing the worker mid-sync and restarting causes no duplicate events (upsert key discipline).
---
## Appendix B — Contacts plugin (`webmail_plugin_contacts`)
The second heavy plugin: same rules as the calendar — §11 SDK only, zero core diffs, own page. Division of labor with core: core owns *implicit* recipient history (`known_recipients`, §4/§9.1) so autocomplete works even without this plugin; the Contacts plugin owns *explicit*, user-curated contact records and enriches suggestions via the `recipients.suggest` capability (§11.3).
### B.1 Scope by phase
- **B1 — Local contacts:** address books, contact CRUD with structured fields, groups, photo, search, dedicated page `/contacts`; "Add sender to contacts" from the thread view; suggestion capability wired into the composer.
- **B2 — Portability:** vCard 3.0/4.0 import (file, multi-card) and export (single contact / book), duplicate detection + merge tool.
- **B3 — External sources:** `ContactProvider` interface with **CardDAV client** sync (iCloud, Fastmail, Nextcloud, Radicale); per-book sync job.
- **B4 (deferred, spec later):** Google People / Microsoft Graph providers (reusing M2 OAuth), CardDAV **server** endpoint for native phone sync, book sharing.
### B.2 Dependencies
`vobject ~0.9` (vCard parse/serialize). Photos stored via the SDK blob store (JPEG/PNG, re-encoded server-side, ≤ 512×512). Nothing else.
### B.3 Data model (plugin migrations, prefix `plg_contacts_`)
```sql
CREATE TABLE plg_contacts_books (
id uuid PRIMARY KEY DEFAULT gen_random_uuid(),
user_id uuid NOT NULL,
name text NOT NULL,
kind text NOT NULL DEFAULT 'local', -- local|carddav
source_config jsonb NOT NULL DEFAULT '{}',-- carddav url/collection
secret_ct bytea, -- carddav credentials, §10.2 vault scheme
sync_token text,
read_only boolean NOT NULL DEFAULT false,
UNIQUE (user_id, name)
);
CREATE TABLE plg_contacts_contacts (
id uuid PRIMARY KEY DEFAULT gen_random_uuid(),
user_id uuid NOT NULL,
book_id uuid NOT NULL REFERENCES plg_contacts_books(id) ON DELETE CASCADE,
vcard_uid text NOT NULL, -- UID property; join key for CardDAV
etag text, -- server etag (carddav books)
name_display text NOT NULL DEFAULT '', -- FN
name_parts jsonb NOT NULL DEFAULT '{}', -- {given,family,middle,prefix,suffix} (N)
emails jsonb NOT NULL DEFAULT '[]', -- [{value,type,pref}]
phones jsonb NOT NULL DEFAULT '[]',
addresses jsonb NOT NULL DEFAULT '[]', -- structured ADR
org text NOT NULL DEFAULT '',
title text NOT NULL DEFAULT '',
birthday date,
urls jsonb NOT NULL DEFAULT '[]',
notes text NOT NULL DEFAULT '',
photo_sha256 text,
raw_vcard text, -- round-trip fidelity: preserve unknown props
updated_at timestamptz NOT NULL DEFAULT now(),
UNIQUE (book_id, vcard_uid)
);
CREATE INDEX ON plg_contacts_contacts (user_id, name_display);
CREATE INDEX ON plg_contacts_contacts USING gin (emails jsonb_path_ops);
CREATE TABLE plg_contacts_groups (
id uuid PRIMARY KEY DEFAULT gen_random_uuid(),
user_id uuid NOT NULL,
name text NOT NULL,
UNIQUE (user_id, name)
);
CREATE TABLE plg_contacts_group_members (
group_id uuid NOT NULL REFERENCES plg_contacts_groups(id) ON DELETE CASCADE,
contact_id uuid NOT NULL REFERENCES plg_contacts_contacts(id) ON DELETE CASCADE,
PRIMARY KEY (group_id, contact_id)
);
```
vCard mapping discipline: parse known properties into columns, keep the full source in `raw_vcard`, and on export/push regenerate from columns **merged over** `raw_vcard` so properties we don't model (e.g. `X-` extensions) survive round-trips.
### B.4 ContactProvider interface (inside the plugin, mirrors A.4)
```python
# PLUGIN INTERFACE VERSION: 1
class ContactProvider(Protocol):
async def verify(self) -> None: ...
async def list_books(self) -> list[RemoteBook]: ...
async def pull(self, book: str, *, sync_token: str | None
) -> tuple[list[RemoteVcard], list[str], str]: ...
# (changed vcards, deleted uids, new sync_token)
async def push(self, book: str, vcard: str, *, uid: str,
etag: str | None) -> str: ...
async def delete(self, book: str, uid: str, etag: str | None) -> None: ...
```
Sync loop = plugin cron job (`*/5 * * * *`, singleton per user), upsert by `(book_id, vcard_uid)`, etag conflict ⇒ server wins + local copy kept as a duplicate flagged for the merge tool (never silent loss) — same policy as A.4.
### B.5 API & capability (`/api/plugins/contacts/`)
`GET /contacts?book=&group=&q=&cursor=` · `POST /contacts` · `GET/PUT/DELETE /contacts/{id}` · `POST /contacts/from-message {message_id}` (prefill from From header) · `GET/POST/DELETE /books`, `/groups`, group membership ops · `POST /import` (vCard file → report `{created, updated, skipped}`) · `GET /export?book=|contact=` (`.vcf`) · `GET /duplicates` (candidate pairs: same email, or same normalized name + fuzzy) · `POST /merge {keep_id, remove_id}` (field union, keep wins conflicts, group memberships merged) · **`GET /suggest?q=`** — the `recipients.suggest` capability endpoint: matches name/email prefix and substring, returns `[{name, email, source:"contact", contact_id, photo_url?}]`. Composer merge rule (client-side, defined here so behavior is deterministic): contact hits rank above `known_recipients` hits at equal prefix quality; dedupe by email, contact entry wins and carries the photo/display name.
### B.6 UI
Page `/contacts` (nav entry after Calendar): three-pane layout — books/groups sidebar, alphabetically grouped virtualized list with search-as-you-type, detail pane with edit-in-place form; import/export and merge review under `/contacts/tools`; CardDAV book wizard reusing the account-wizard UX pattern (URL + credentials → test → book picker). Mail integration via `message.actions` slot: "Add to Contacts" (hidden when the sender already matches a contact; then it becomes "View contact" linking to `/contacts/{id}`). Group compose: selecting a group in the composer expands to its members' preferred emails.
### B.7 Acceptance criteria
1. Zero core diffs; install/uninstall `--drop-data` round-trip clean.
2. Import a 500-card vCard file mixing v3.0/v4.0, quoted-printable encoding, and Polish diacritics in names (`Grzegorz Brzęczyszczykiewicz`) — all names, emails, and photos correct; re-export → re-import is lossless including `X-` properties (raw_vcard merge discipline).
3. Composer typing a 3-char prefix returns merged suggestions in < 100 ms p95 (contacts ranked above history, deduped by email, photos shown for contacts).
4. CardDAV sync against Radicale: create/edit/delete propagate both directions within one cron cycle; concurrent edit conflict produces a flagged duplicate, never silent loss; worker kill mid-sync ⇒ no duplicate contacts on restart.
5. "Add sender to contacts" from a thread creates a prefilled contact; the same sender's messages immediately render the contact's display name, and the action flips to "View contact".
6. Merge tool: merging two contacts unions emails/phones/groups, keeps the chosen primary's conflicting fields, and repoints nothing else in the system (calendar attendees and mail remain keyed by email, by design).
---
*End of specification. Ambiguities: prefer the simplest behavior consistent with the invariants in §1, leave a `TODO(spec)` comment, and list open questions in `docs/open-questions.md` rather than guessing silently.*
Reference in New Issue Copy Permalink