webmail-client/webmail-client-spec_v1.5.md

# Webmail Client — Implementation Specification (v1.5, agent-ready)

This document is a complete, self-contained build specification. It is written to be handed to an AI coding agent. Read it fully before writing code. The companion document `webmail-client-concept.md` gives background; where they differ, **this spec wins**.

---

## 0. Instructions to the implementing agent

1. **Build milestone by milestone** (§12), in order. Do not start a milestone before the previous one's acceptance criteria pass.
2. **Do not add features beyond this spec.** Anything not specified here is out of scope; leave `TODO(spec)` comments instead of inventing behavior.
3. **Definition of done** for every task: code + type hints + unit tests + passing `ruff check`, `ruff format --check`, `mypy --strict` (backend) / `tsc --noEmit`, `eslint` (frontend) + updated docs in `docs/` if an interface changed.
4. **Stability over novelty:** use exactly the pinned dependencies in §3. Do not swap libraries or add dependencies without a `# DEPENDENCY:` comment explaining why, and never for something the stdlib does adequately.
5. **All interfaces in §5 are frozen contracts.** Implementations may evolve; signatures may not (without bumping the version comment at the top of the interface file).
6. **Security rules in §10 are hard requirements**, not suggestions.
7. Every module publishing or consuming events must use the event names in §8 verbatim.
8. Commit in small, working increments; every commit must leave the test suite green.

## 1. Architecture invariants (never violate)

- **I-1 Sync-first:** the web UI and REST API never open IMAP connections. Only sync workers talk to mail servers. The API reads exclusively from Postgres/Meilisearch.
- **I-2 Independent identity:** login uses app accounts (`localpart@our-web-client.example`) with Argon2id hashes. Mailbox credentials are payload data, encrypted at rest, decrypted only inside sync workers.
- **I-3 One index, many views:** unified inbox, per-account views, virtual mailboxes, and search results are all parameterized queries over the same index. No duplicated storage per view.
- **I-4 Threading is ours:** JWZ in our backend (§7). Never use the IMAP `THREAD` extension.
- **I-5 Direction-agnostic thread UI:** thread ordering (oldest-first/newest-first) is a data-ordering concern. One layout, one composer-anchor slot (§9.4).
- **I-6 Plugins extend, core stays closed:** new features beyond this spec must be expressible through the plugin surfaces (§11).
- **I-7 Multi-user isolation:** every query (SQL and search) carries a mandatory `user_id` filter. No exceptions, including admin tooling.
- **I-8 Mail HTML is hostile input:** sanitize with nh3 server-side at index time AND render in a sandboxed iframe. Both, always.

## 2. Repository layout (monorepo)

```
webmail/
├── backend/
│   ├── pyproject.toml
│   ├── app/
│   │   ├── main.py                  # FastAPI app factory
│   │   ├── config.py                # pydantic-settings; all config via env
│   │   ├── db.py                    # SQLAlchemy async engine/session
│   │   ├── events.py                # EventBus (Redis pub/sub + arq enqueue)
│   │   ├── identity/                # §6.1 auth, sessions, handles, TOTP
│   │   ├── accounts/                # §6.2 wizard, autodiscovery, credential vault
│   │   ├── providers/
│   │   │   ├── base.py              # MailProvider interface (§5.1)
│   │   │   └── imap_smtp/           # first implementation
│   │   ├── sync/                    # §6.3 backfill, IDLE loops, reconciliation
│   │   ├── indexer/                 # §6.4 MIME parsing → DB rows + blobs
│   │   ├── threading_/              # §7 JWZ (trailing _ avoids stdlib clash)
│   │   ├── search/
│   │   │   ├── base.py              # SearchBackend interface (§5.2)
│   │   │   ├── queryparser.py       # operator grammar (§8.2)
│   │   │   ├── pg.py                # Postgres FTS backend (M0–M1)
│   │   │   └── meili.py             # Meilisearch backend (M2)
│   │   ├── outbox/                  # §6.5 drafts, MIME assembly, send queue, undo
│   │   ├── api/                     # routers, one file per §9.1 group
│   │   ├── ws.py                    # WebSocket hub (§9.2)
│   │   └── plugins/                 # §11 loader + plugin packages
│   ├── workers/worker.py            # arq worker entrypoint
│   ├── migrations/                  # alembic
│   └── tests/
├── frontend/
│   ├── package.json
│   └── src/
│       ├── api/                     # generated client from OpenAPI + WS layer
│       ├── routes/                  # §9.3
│       ├── components/
│       │   ├── thread/ThreadView.tsx    # §9.4 direction-agnostic
│       │   ├── thread/Composer.tsx
│       │   ├── list/MessageList.tsx     # virtualized
│       │   └── slots/               # plugin UI slots (§11)
│       └── lib/
├── docker/
│   ├── docker-compose.yml           # app, worker, postgres:16, redis:7, meilisearch:v1.8, dovecot (test)
│   └── docker-compose.ci.yml        # + greenmail for integration tests
└── docs/
```

## 3. Pinned stack

**Backend (Python 3.11):** fastapi ~0.115, uvicorn ~0.30, sqlalchemy ~2.0 (async, asyncpg), alembic ~1.13, pydantic ~2.8, pydantic-settings, arq ~0.26, redis ~5.0, aioimaplib ~1.1, imapclient ~3.0 (utility/fallback), aiosmtplib ~3.0, argon2-cffi ~23.1, pyotp ~2.9, dnspython ~2.6, httpx ~0.27, nh3 ~0.2, cryptography ~43.0, mail-parser ~4.0 (helpers only; primary parsing via stdlib `email.message.EmailMessage` with `policy.default`).

**Frontend:** TypeScript ~5.5, React ~18.3, Vite ~5, TanStack Query ~5, TanStack Virtual ~3, Radix UI, Tailwind ~3.4, zod (WS payload validation). API client generated from OpenAPI (`openapi-typescript`).

**Infra:** PostgreSQL 16 (with `citext` extension), Redis 7, Meilisearch v1.8 (M2+), MinIO or plain filesystem for the blob store (config-switchable; filesystem default).

## 4. Database schema (Postgres 16, authoritative DDL)

All PKs are `uuid DEFAULT gen_random_uuid()`. All tables carrying user data include `user_id` with an index. Timestamps are `timestamptz`.

```sql
CREATE EXTENSION IF NOT EXISTS citext;

CREATE TABLE users (
  id            uuid PRIMARY KEY DEFAULT gen_random_uuid(),
  handle        citext UNIQUE NOT NULL,        -- full form: localpart@<APP_DOMAIN>
  password_hash text   NOT NULL,               -- argon2id
  totp_secret   bytea,                         -- encrypted; NULL = 2FA off
  recovery_email citext,
  settings      jsonb  NOT NULL DEFAULT '{}',  -- incl. {"thread_order":"asc"|"desc"}
  created_at    timestamptz NOT NULL DEFAULT now()
);
-- Handle rules (enforce in code, mail-grade from day one):
--   localpart: 3–30 chars, [a-z0-9._-], no leading/trailing/double dot,
--   case-insensitive, reserved list (postmaster, abuse, admin, root, noreply, ...).

CREATE TABLE accounts (
  id             uuid PRIMARY KEY DEFAULT gen_random_uuid(),
  user_id        uuid NOT NULL REFERENCES users(id) ON DELETE CASCADE,
  email          citext NOT NULL,
  display_name   text NOT NULL DEFAULT '',
  color          text NOT NULL DEFAULT '#888888',
  imap_config    jsonb NOT NULL,   -- {host,port,tls:"ssl"|"starttls",auth:"password"|"oauth2"}
  smtp_config    jsonb NOT NULL,   -- same shape
  secret_ct      bytea NOT NULL,   -- envelope-encrypted password or OAuth token bundle (§10.2)
  discovery_src  text  NOT NULL,   -- preset|autoconfig|ispdb|srv|probe|manual
  state          text  NOT NULL DEFAULT 'backfilling',  -- backfilling|live|error|disabled
  state_detail   text  NOT NULL DEFAULT '',
  last_sync_at   timestamptz,
  created_at     timestamptz NOT NULL DEFAULT now(),
  UNIQUE (user_id, email)
);

CREATE TABLE mailboxes (
  id             uuid PRIMARY KEY DEFAULT gen_random_uuid(),
  account_id     uuid NOT NULL REFERENCES accounts(id) ON DELETE CASCADE,
  user_id        uuid NOT NULL REFERENCES users(id) ON DELETE CASCADE,
  path           text NOT NULL,                 -- IMAP mailbox name, UTF-7 decoded
  special_use    text,                          -- inbox|sent|drafts|trash|junk|archive|NULL
  uidvalidity    bigint NOT NULL DEFAULT 0,
  highestmodseq  bigint,                        -- NULL if no CONDSTORE
  last_seen_uid  bigint NOT NULL DEFAULT 0,
  UNIQUE (account_id, path)
);

CREATE TABLE threads (
  id              uuid PRIMARY KEY DEFAULT gen_random_uuid(),
  user_id         uuid NOT NULL REFERENCES users(id) ON DELETE CASCADE,
  subject_norm    text NOT NULL,                -- normalized subject (§7.3)
  snippet         text NOT NULL DEFAULT '',
  participants    jsonb NOT NULL DEFAULT '[]',  -- [{name,email}], deduped, ordered by recency
  account_ids     uuid[] NOT NULL DEFAULT '{}',
  message_count   int NOT NULL DEFAULT 0,
  unread_count    int NOT NULL DEFAULT 0,
  has_attachments boolean NOT NULL DEFAULT false,
  last_message_at timestamptz NOT NULL
);
CREATE INDEX ON threads (user_id, last_message_at DESC);

CREATE TABLE messages (
  id              uuid PRIMARY KEY DEFAULT gen_random_uuid(),
  user_id         uuid NOT NULL REFERENCES users(id) ON DELETE CASCADE,
  account_id      uuid NOT NULL REFERENCES accounts(id) ON DELETE CASCADE,
  mailbox_id      uuid NOT NULL REFERENCES mailboxes(id) ON DELETE CASCADE,
  uid             bigint NOT NULL,              -- IMAP UID within mailbox
  thread_id       uuid REFERENCES threads(id),
  message_id_hdr  text,                         -- Message-ID, angle brackets stripped
  in_reply_to     text,
  references_hdr  text[] NOT NULL DEFAULT '{}',
  subject         text NOT NULL DEFAULT '',
  from_addr       jsonb NOT NULL DEFAULT '[]',
  to_addr         jsonb NOT NULL DEFAULT '[]',
  cc_addr         jsonb NOT NULL DEFAULT '[]',
  date_hdr        timestamptz,
  internal_date   timestamptz NOT NULL,
  flags           text[] NOT NULL DEFAULT '{}', -- \Seen,\Flagged,\Answered,\Draft,\Deleted
  size_bytes      int NOT NULL DEFAULT 0,
  has_attachments boolean NOT NULL DEFAULT false,
  snippet         text NOT NULL DEFAULT '',     -- first ~160 chars of text body
  blob_sha256     text NOT NULL,                -- raw RFC822 in blob store (§6.4)
  body_indexed    tsvector,                     -- Postgres FTS (M0–M1 SearchBackend)
  UNIQUE (mailbox_id, uid)
);
CREATE INDEX ON messages (user_id, thread_id);
CREATE INDEX ON messages (account_id, message_id_hdr);
CREATE INDEX ON messages USING gin (body_indexed);

CREATE TABLE body_parts (
  id          uuid PRIMARY KEY DEFAULT gen_random_uuid(),
  message_id  uuid NOT NULL REFERENCES messages(id) ON DELETE CASCADE,
  kind        text NOT NULL,        -- text|html_sanitized
  content     text NOT NULL
);

CREATE TABLE attachments (
  id          uuid PRIMARY KEY DEFAULT gen_random_uuid(),
  message_id  uuid NOT NULL REFERENCES messages(id) ON DELETE CASCADE,
  user_id     uuid NOT NULL,
  filename    text NOT NULL,
  mime_type   text NOT NULL,
  size_bytes  int  NOT NULL,
  content_id  text,                 -- for inline cid: images
  blob_sha256 text NOT NULL
);

CREATE TABLE drafts (
  id          uuid PRIMARY KEY DEFAULT gen_random_uuid(),
  user_id     uuid NOT NULL REFERENCES users(id) ON DELETE CASCADE,
  account_id  uuid NOT NULL REFERENCES accounts(id) ON DELETE CASCADE,
  thread_id   uuid REFERENCES threads(id),      -- NULL = new mail
  reply_to_message_id uuid REFERENCES messages(id),
  kind        text NOT NULL DEFAULT 'reply',    -- reply|reply_all|forward|new
  payload     jsonb NOT NULL,       -- {to,cc,bcc,subject,html,text,attachment_ids}
  updated_at  timestamptz NOT NULL DEFAULT now()
);

CREATE TABLE outbox (
  id          uuid PRIMARY KEY DEFAULT gen_random_uuid(),
  user_id     uuid NOT NULL,
  draft_id    uuid NOT NULL,
  state       text NOT NULL DEFAULT 'undo_window',  -- undo_window|sending|sent|failed|cancelled
  send_after  timestamptz NOT NULL,                 -- now() + undo window
  error       text
);

CREATE TABLE saved_searches (
  id       uuid PRIMARY KEY DEFAULT gen_random_uuid(),
  user_id  uuid NOT NULL REFERENCES users(id) ON DELETE CASCADE,
  name     text NOT NULL,
  query    text NOT NULL,           -- raw operator-syntax string (§8.2)
  pinned   boolean NOT NULL DEFAULT true,
  UNIQUE (user_id, name)
);

-- Core recipient autocomplete (Gmail-style, derived from mail history).
-- The Contacts plugin (Appendix B) ENRICHES suggestions; this table works without it.
CREATE TABLE known_recipients (
  user_id      uuid NOT NULL REFERENCES users(id) ON DELETE CASCADE,
  email        citext NOT NULL,
  name         text NOT NULL DEFAULT '',      -- last seen display name
  times_used   int NOT NULL DEFAULT 0,        -- weighted: sent-to counts 3, received-from counts 1
  last_used_at timestamptz NOT NULL,
  PRIMARY KEY (user_id, email)
);
CREATE INDEX ON known_recipients (user_id, email text_pattern_ops);
```

**Blob store:** raw RFC822 messages and attachment payloads are stored content-addressed at `${BLOB_ROOT}/{user_id}/{sha256[:2]}/{sha256}` (filesystem) or the equivalent object key (MinIO). Postgres holds metadata only. Deleting a user removes their blob subtree.

**Sessions** live in Redis (`sess:{token}` → JSON `{user_id, created, ua}`, TTL 30 days sliding), not Postgres.

## 5. Frozen core interfaces (Python)

### 5.1 `providers/base.py` — MailProvider

```python
# INTERFACE VERSION: 1
from dataclasses import dataclass
from typing import AsyncIterator, Protocol
from datetime import datetime

@dataclass(frozen=True)
class RemoteMailbox:
    path: str
    special_use: str | None      # "inbox"|"sent"|"drafts"|"trash"|"junk"|"archive"|None
    uidvalidity: int

@dataclass(frozen=True)
class RemoteMessage:
    uid: int
    raw: bytes                   # full RFC822
    flags: frozenset[str]
    internal_date: datetime

@dataclass(frozen=True)
class FlagChange:
    uid: int
    flags: frozenset[str]

@dataclass(frozen=True)
class MailboxDelta:
    new: list[RemoteMessage]
    flag_changes: list[FlagChange]
    expunged_uids: list[int]
    new_uidvalidity: int | None  # non-None => UIDVALIDITY changed, caller must full-resync
    new_highestmodseq: int | None

class MailProvider(Protocol):
    async def verify(self) -> None: ...
        # Raises ProviderAuthError / ProviderConnectError with a human-readable .detail

    async def list_mailboxes(self) -> list[RemoteMailbox]: ...

    def backfill(self, mailbox: str, *, newest_first: bool = True,
                 batch_size: int = 200) -> AsyncIterator[list[RemoteMessage]]: ...
        # Yields batches until the mailbox is exhausted. Headers+body in one pass.

    async def poll(self, mailbox: str, *, last_seen_uid: int,
                   highestmodseq: int | None) -> MailboxDelta: ...
        # One reconciliation pass. Uses QRESYNC/CONDSTORE when highestmodseq is not None,
        # otherwise UID SEARCH + FETCH FLAGS diff.

    async def idle(self, mailbox: str) -> None: ...
        # Blocks until the server signals change or 25 min elapse (re-issue IDLE). May raise
        # ProviderNotSupported => sync loop falls back to poll() every POLL_INTERVAL (default 120 s).

    async def push_flags(self, mailbox: str, uid: int,
                         add: frozenset[str], remove: frozenset[str]) -> None: ...
    async def move(self, uid: int, src: str, dst: str) -> int: ...   # returns new UID
    async def append(self, mailbox: str, raw: bytes,
                     flags: frozenset[str]) -> int: ...               # Sent/Drafts upload
    async def send(self, raw: bytes, envelope_from: str,
                   envelope_to: list[str]) -> None: ...               # SMTP submission
```

`ImapSmtpProvider` is the only M0–M2 implementation. A future `NativeProvider` (hosted mailboxes) or `JmapProvider` implements the same protocol — nothing above this interface may import IMAP libraries.

### 5.2 `search/base.py` — SearchBackend

```python
# INTERFACE VERSION: 1
@dataclass(frozen=True)
class SearchQuery:
    text: str | None                      # free-text remainder after operator extraction
    filters: dict[str, object]            # parsed operators, keys per §8.2
    user_id: UUID                         # ALWAYS set (invariant I-7)

@dataclass(frozen=True)
class SearchHit:
    message_id: UUID
    thread_id: UUID
    score: float
    highlights: dict[str, str]            # field -> HTML-escaped snippet with <mark>

class SearchBackend(Protocol):
    async def index(self, doc: SearchDoc) -> None: ...
    async def delete(self, message_id: UUID) -> None: ...
    async def delete_user(self, user_id: UUID) -> None: ...
    async def query(self, q: SearchQuery, *, limit: int, cursor: str | None
                    ) -> tuple[list[SearchHit], str | None]: ...
```

`SearchDoc` fields: `message_id, thread_id, user_id, account_id, mailbox_special_use, from_ (name+email), to, cc, subject, body_text, attachment_names[], has_attachments, is_unread, is_starred, date (unix), size_bytes`. Both backends (pg.py, meili.py) index the same doc. Meilisearch: one index `messages`, filterable = every non-body field incl. `user_id`, searchable = `subject, body_text, from_, to, cc, attachment_names`, sortable = `date`; ranking rules default + `date:desc` tiebreak.

### 5.3 `events.py` — EventBus

```python
# INTERFACE VERSION: 1
class EventBus(Protocol):
    async def publish(self, event: str, payload: dict) -> None:
        # Fan-out to Redis pub/sub channel "evt:{event}" AND enqueue arq job
        # "on_{event}" for durable consumers (indexer, plugins).
    def subscribe(self, event: str, handler: Handler) -> None: ...
```

Payloads always include `user_id`. Event names are listed in §8.3 and are the only ones allowed.

## 6. Backend module specifications

### 6.1 Identity (`app/identity/`)
- Registration: validate localpart per §4 rules against reserved list in `identity/reserved.py`; store full handle `localpart@{APP_DOMAIN}` (env `APP_DOMAIN`).
- Login: constant-time verify (argon2id, `time_cost=3, memory_cost=64MiB, parallelism=4`); if `totp_secret` set, require valid TOTP (±1 window). Rate limit: 10 attempts / 15 min per handle AND per IP (Redis counters); lockout returns 429 with `Retry-After`.
- Session cookie: `__Host-session`, httpOnly, Secure, SameSite=Lax. CSRF: double-submit token header `X-CSRF` for all mutating requests.
- TOTP setup: `POST /auth/totp/setup` returns provisioning URI + QR payload; enable requires one valid code; secret stored envelope-encrypted.
- Password reset: only if `recovery_email` verified; single-use token, 30 min TTL, Redis-backed.

### 6.2 Accounts & autodiscovery (`app/accounts/`)
`discover(email) -> list[CandidateConfig]` runs the chain, **returning on first success** but recording every step's outcome in the response (`steps: [{method, status, detail}]`) for wizard UX:

1. **Presets** — `accounts/presets.yaml` (config, not code): match by domain; ships with gmail.com, googlemail.com, outlook.com/hotmail/live, office365, yahoo, icloud/me.com, gmx, wp.pl, onet.pl, o2.pl, interia.pl. Each entry: imap/smtp host+port+tls, auth kind, oauth provider id if applicable.
2. **Autoconfig** — GET `https://autoconfig.{domain}/mail/config-v1.1.xml?emailaddress={email}` then `https://{domain}/.well-known/autoconfig/mail/config-v1.1.xml`; parse Thunderbird XML (incomingServer type=imap, outgoingServer type=smtp; `%EMAILADDRESS%`/`%EMAILLOCALPART%` substitution).
3. **ISPDB** — GET `https://autoconfig.thunderbird.net/v1.1/{domain}`; same XML format.
4. **DNS SRV (RFC 6186)** — `_imaps._tcp.{domain}` (SSL 993), `_imap._tcp` (STARTTLS 143), `_submissions._tcp` (465), `_submission._tcp` (587); lowest priority wins.
5. **Probe** — try `{imap,mail}.{domain}` × ports 993(ssl)/143(starttls) and `{smtp,mail}.{domain}` × 465(ssl)/587(starttls); TCP connect + TLS handshake + IMAP/SMTP greeting, 5 s timeout each, all concurrently.

All HTTP/DNS lookups: 5 s timeout, no redirects off-domain, HTTPS only. `test(config, credentials)`: full IMAP LOGIN + `LIST`, and SMTP EHLO+AUTH (no message sent); errors mapped to precise user-facing strings (`auth_failed`, `tls_failed`, `connect_timeout`, `unknown_host`). On save: encrypt credentials (§10.2), create account row (`state=backfilling`), enqueue `sync.bootstrap_account`.

#### 6.2.1 Provider onboarding strategy (`auth_strategy`)
Each preset in `presets.yaml` declares an ordered `auth_strategy` list telling the wizard which credential flows to offer and in what order. This is the routing that keeps Gmail (and similar) onboarding smooth without any server-side automation of the provider's pages. Values:
- `app_password` — provider supports a per-user application-specific password used as an ordinary IMAP/SMTP password (Gmail, Yahoo, iCloud, outlook.com personal). The wizard shows the provider-specific helper panel (§6.2.2) rather than the generic password box.
- `oauth2` — XOAUTH2 flow (M2). For Google this is the **fallback** offered when `app_password` fails or is unavailable (brand-new accounts, Advanced Protection, admin-disabled). Gmail's `auth_strategy` is therefore `[app_password, oauth2]`.
- `password` — plain IMAP password (self-hosted/ISP accounts): the generic password box.

Rationale recorded so the agent doesn't "optimize" it away: full-IMAP OAuth for Google requires restricted-scope verification + an annual CASA assessment, which this deployment declines by default; app passwords are a first-class Google-provided mechanism that needs none of it and reuses the existing password code path (zero new sync code). OAuth remains implemented for M2 as the fallback and for deployments that do complete verification. `SEARCH_BACKEND`-style toggle `GOOGLE_OAUTH_ENABLED` (default false) hides the OAuth option entirely when an operator hasn't configured a Cloud project.

#### 6.2.2 Provider credential-helper panel (frontend, wizard step 1.5)
When the resolved preset's first `auth_strategy` is `app_password`, the wizard renders a `ProviderCredentialPanel` keyed by the preset's `help_id` (e.g. `gmail`, `yahoo`, `icloud`) instead of the generic password field. Content is data-driven from `presets.yaml` (`help: {deep_link, app_name_hint, steps[], docs_url}`) so adding a provider needs no code. For Gmail specifically:

- **Deep-link button** — "Open Google App Passwords" opens `https://myaccount.google.com/apppasswords` in a popup/new tab. Must fire from the user's click (a direct gesture) so popup blockers allow it. The panel does not and cannot prefill or read anything on Google's page.
- **Inline steps** (rendered from `help.steps`): enable 2-Step Verification if needed → name the password (suggest `app_name_hint`, e.g. "1.email") → copy the 16 characters.
- **Single paste field** — on paste/change, client strips all whitespace (Google displays four groups of four) before use.
- **Instant verify** — on a complete-looking value (16 chars), auto-fire `POST /accounts/test` with the resolved IMAP/SMTP config; show `Connected ✓` or the precise mapped error without the user pressing anything else.
- **Failure recovery (the two real dead-ends):** if `test` returns `auth_failed`, show a branch explaining the likely causes — 2FA not fully enabled, **or a brand-new Google account for which Google withholds app passwords until it ages/gains trust**, or a Workspace account with the feature admin-disabled — each with the relevant link. When the preset's `auth_strategy` also contains `oauth2` **and** `GOOGLE_OAUTH_ENABLED`, render a prominent **"Sign in with Google instead"** button that switches the wizard to the OAuth flow (which is not gated by account age). This single fallback converts the most common onboarding failure into a one-click recovery.

**Hard prohibitions (security + trust — do not implement under any framing):** no headless/automated browser driving of Google (or any provider) sign-in or app-password pages; never request or accept the user's actual Google account password; never scrape, screen-read, or otherwise extract the generated app password from the provider's page — the user copies and pastes it themselves. The panel's ceiling is deep-link + guide + paste + test. Anything beyond that gets user accounts flagged, breaks on provider markup changes, and is out of scope permanently, not just for now.

#### 6.2.3 presets.yaml additions
Each preset gains `auth_strategy: [...]` and, where the first strategy is `app_password`, a `help:` block: `{help_id, deep_link, app_name_hint, steps: [i18n keys], docs_url}`. `steps` are i18n message keys (Polish + English at minimum, per §languages) so the copy is translatable, not hardcoded. Gmail, Yahoo, iCloud, and outlook.com personal ship with populated `help` blocks; corporate/M365 and self-hosted presets use `oauth2` or `password` and no panel.

### 6.3 Sync engine v2 (`app/sync/`) — sharded, lease-owned, tiered
(Authoritative detail: `sync-engine-v2-refactor.md`; this section summarizes the binding rules.)

- **Two worker roles** (`WORKER_ROLE`): `jobs` = stateless arq consumer, N instances; `sync` = lease claimer + per-account runners, 1..N instances. Compose services `worker-jobs` / `worker-sync`, `nofile` ulimit 65536.
- **Lease ownership:** exactly one sync worker owns each live account. Redis SET `sync:live` (API-maintained, DB-reconciled every 300 s); lease `sync:lease:{account_id}` claimed `SET NX EX LEASE_TTL`, renewed every `LEASE_RENEW` via owner-checked Lua; **fencing** — on renewal failure, cancel that account's runner before any further provider/DB op. Worker heartbeats in `sync:workers:{id}`; rebalance releases excess above `target*1.2+5`. Crash failover = lease expiry + reclaim (≤ ~90 s), no coordinator.
- **Event-driven lifecycle:** `account.state_changed` adds/removes ids from `sync:live` at runtime — accounts go live ≤ 30 s after creation and stop ≤ 30 s after deletion with no restarts; runners re-verify `state='live'` each cycle (zombie-proof). New core event **`user.active`** (API publishes on login/WS connect) drives tier promotion.
- **Bootstrap (unchanged from v1, now a `bulk` job):** `list_mailboxes()` → upsert `mailboxes` (SPECIAL-USE mapping with name heuristics — Sent/Wysłane, Trash/Kosz, etc. in `sync/folder_names.py`); backfill newest-first per mailbox (INBOX first, then sent, then rest); `sync.progress` published per batch; on completion set `state='live'` → the API adds the id to `sync:live` and a sync worker claims it.
- **AccountRunner per tier:** *hot* = persistent INBOX IDLE + transient CONDSTORE poll of special folders every `POLL_WARM`; *warm* = transient poll of INBOX+specials every `POLL_WARM`; *cold* = every `POLL_COLD`. Promotion to hot on `user.active` (≤ 5 s) or new mail within `TIER_HOT_ACTIVITY_H`; demotion by inactivity; `HOT_BUDGET_PER_WORKER` degrades least-active non-session accounts. Delta application, UIDVALIDITY wipe-and-rebackfill, expunge soft-handling, and write-then-enqueue outbound flags are unchanged from v1.
- **Connection budgets:** ≤ 2 sync-side connections per account (4 total incl. jobs, shared semaphore); process-local per-destination-host semaphore cap `PER_HOST_CONN_CAP`.
- **Queues:** `interactive` (send/flags/move/draft jobs, timeout 120 s, concurrency 20) and `bulk` (backfills, timeout 3600 s, concurrency `BULK_CONCURRENCY`) — bulk work can never delay a send.
- **Memory bounds:** parse semaphore `PARSE_CONCURRENCY`; raw messages > `STREAM_PARSE_MB` stream to blob and parse from file — peak RAM ≈ `PARSE_CONCURRENCY × STREAM_PARSE_MB` per process regardless of mailbox size.
- **Shutdown:** SIGTERM → stop claimer → cancel runners → release leases → drain in < 10 s.

### 6.4 Indexer (`app/indexer/`)
Input: `RemoteMessage`. Steps: store raw blob (sha256, dedupe) → parse with `email` stdlib (`policy.default`) → extract: headers (decode RFC2047), `references_hdr` = parsed `References` + `In-Reply-To` message-ids in order; best text body and best HTML body (multipart/alternative preference: html for display, text for indexing; if only HTML, derive text via nh3 strip); sanitize HTML with nh3 (allowlist: structural tags, inline styles filtered, `cid:` images rewritten to `/api/messages/{id}/cid/{content_id}`, remote `src` rewritten to blocked-placeholder attributes for FR-23) → write `messages`, `body_parts` (kinds `text` and `html_sanitized`), `attachments` → hand to Threader (§7) → `bus.publish("message.new", ...)` (search indexing + thread denormalization are consumers). For non-junk received messages, upsert the `From` address into `known_recipients` (weight 1).

### 6.5 Outbox (`app/outbox/`)
- Drafts: autosave endpoint upserts `drafts` (client debounces 2 s); a background job mirrors drafts to the account's Drafts folder via `append()` at most every 60 s per draft (replace previous UID).
- Send: `POST /send {draft_id}` → build MIME (html + auto-generated text alternative via html-to-text; `In-Reply-To`/`References` from `reply_to_message_id`; attachments from blob store) → create `outbox` row with `send_after = now()+UNDO_SECONDS` (env, default 10) → schedule arq job at `send_after`. Undo endpoint flips state to `cancelled` if still in window. The job: SMTP `send()`, then `append()` to Sent (skip for Gmail preset — Gmail self-appends), mark `sent`, publish `message.sent`, and upsert every To/Cc/Bcc address into `known_recipients` (weight 3). Failure → `failed` + WS notification, draft preserved.

## 7. Threading specification (JWZ, `app/threading_/`)

### 7.1 Normalization
- `message_id_hdr`, `in_reply_to`, each `References` entry: strip angle brackets and whitespace, lowercase the domain part only. Discard obviously invalid ids (no `@`).
- Missing/duplicate `Message-ID`: synthesize `sha256(raw)[:32]@synthetic.local` and mark it non-authoritative (never used to join other messages to it).

### 7.2 Incremental algorithm (per new message)
1. Candidate set = all normalized ids in `references_hdr` + `in_reply_to` + own `message_id_hdr`.
2. `SELECT DISTINCT thread_id FROM messages WHERE user_id=$1 AND message_id_hdr = ANY($2) AND thread_id IS NOT NULL`.
3. 0 threads → subject fallback (§7.3); still none → create new thread.
4. 1 thread → attach.
5. ≥2 threads → **merge**: keep the thread with the oldest `last_message_at` as canonical, repoint messages of the others, delete emptied thread rows, publish `thread.updated` for all involved.
6. Recompute denormalized thread fields (count, unread, participants, snippet = newest message's snippet, has_attachments, account_ids, last_message_at) in one UPDATE.

Threads are cross-account by design: the candidate lookup is scoped by `user_id`, not `account_id` (invariant I-3). Sent-folder messages join threads exactly like received ones (FR-15).

### 7.3 Subject fallback
Only when reference joining fails: normalize subject (strip repeated `Re:/Fw:/Fwd:/Odp:/PD:` prefixes case-insensitively, collapse whitespace); join to a thread with the same `subject_norm` that has activity within 7 days AND shares ≥1 participant email; never join two messages that both lack references AND have empty subjects.

### 7.4 Test vectors (must pass in `tests/threading/`)
Linear chain; chain arriving out of order; reply missing `In-Reply-To` but carrying `References`; thread merge on late-arriving parent; broken client (subject-only) joins; distinct-topic same-subject ("hi") does NOT join; cross-account merge (message in account A references message in account B); duplicate Message-ID; synthetic-ID isolation.

## 8. Search specification

### 8.1 Pipeline
Consumer of `message.new` / `message.changed` / `message.deleted` → builds `SearchDoc` → active `SearchBackend` (env `SEARCH_BACKEND=pg|meili`). Backfill indexes inline with sync (per batch), never as a separate pass the user waits on (FR-30).

### 8.2 Query grammar (`search/queryparser.py`, engine-independent)
Tokenization: whitespace-split, respecting `"quoted phrases"`. A token matching `key:value` (or `-key:value`) with a known key becomes a filter; everything else joins free text. `-` prefix negates a term or filter. `OR` (uppercase) groups adjacent terms/filters into a disjunction; default combinator is AND.

| Operator | Filter semantics |
|---|---|
| `from:` `to:` `cc:` | substring match on name or email of that field |
| `subject:` | match constrained to subject |
| `account:` | account by `display_name` or email substring |
| `in:` | `inbox\|sent\|drafts\|trash\|junk\|archive` (special_use) or folder path |
| `is:` | `unread\|read\|starred\|answered` |
| `has:attachment` | boolean |
| `filename:` | attachment name substring |
| `before:` `after:` | `YYYY-MM-DD` |
| `newer_than:` `older_than:` | `Nd\|Nw\|Nm\|Ny` |
| `larger:` `smaller:` | `NK\|NM` bytes |

Parser output = `SearchQuery` (§5.2). Unknown `key:` tokens are treated as free text (forward compatibility). Unit tests must cover every operator, negation, quoting, `OR`, and garbage input (parser never raises).

### 8.3 Event catalog (complete)
`message.new`, `message.changed`, `message.deleted`, `message.sent`, `flags.changed`, `thread.updated`, `sync.progress`, `account.state_changed`, `draft.saved`, `outbox.state_changed`, `user.active` (published by the API on login and WS connect; consumed by sync workers for tier promotion — not forwarded to browsers). Payloads: `{user_id, ...ids, minimal-diff fields}` — documented in `docs/events.md` (agent: generate it from code).

Plugins may additionally publish events namespaced `plugin.{name}.*` (e.g. `plugin.calendar.event_updated`); core code never subscribes to plugin events, but the WS hub forwards them to the owning user like any other event.

## 9. API, realtime, frontend

### 9.1 REST API (prefix `/api`, session-cookie auth, JSON; errors: `{error:{code,message}}`)

**Auth:** `POST /auth/register` · `POST /auth/login` · `POST /auth/logout` · `GET /auth/me` · `GET /auth/sessions` · `DELETE /auth/sessions/{id}` · `POST /auth/totp/setup` · `POST /auth/totp/enable` · `POST /auth/password/reset-request` · `POST /auth/password/reset`

**Accounts:** `POST /accounts/discover {email}` → `{candidates:[{imap,smtp,source}], steps:[...]}` · `POST /accounts/test {config, credentials}` → `{ok}|{error:{code,detail}}` · `POST /accounts` · `GET /accounts` (incl. `state`, sync progress) · `PATCH /accounts/{id}` · `DELETE /accounts/{id}` (cascades rows, blobs, search docs)

**Mail:** `GET /threads?view=unified|account:{id}|mailbox:{id}|search&q=&cursor=&limit=50` → thread summaries (denormalized fields only; cursor = `last_message_at,id` keyset) · `GET /threads/{id}?sort=asc|desc` → ordered full messages (bodies of the N newest expanded per client hint `expand=`; older = envelope only) · `POST /threads/{id}/flags {add,remove}` (whole thread) · `POST /messages/{id}/flags` · `POST /messages/{id}/move {mailbox_id}` · `GET /messages/{id}/html` (sanitized, for iframe `srcdoc`) · `GET /messages/{id}/cid/{content_id}` · `GET /attachments/{id}` (Content-Disposition attachment; images/PDF also inline for preview) · `POST /messages/{id}/allow-remote-images` (per-sender allowlist in user settings)

**Compose:** `POST /drafts` · `PUT /drafts/{id}` · `DELETE /drafts/{id}` · `POST /send {draft_id}` → `{outbox_id, undo_until}` · `POST /send/{outbox_id}/undo` · `GET /recipients/suggest?q=` → top 10 from `known_recipients` (prefix match on email or name, ranked by `times_used` decayed by recency); the composer merges these with any plugin capability providers (§11.3)

**Search:** `GET /search?q=&cursor=` → `{threads:[...], hits_by_thread, highlights}` (grouped by thread, FR-28; `&group=message` for flat) · `GET/POST/DELETE /saved-searches`

**Settings:** `GET/PATCH /settings` (incl. `thread_order: "asc"|"desc"`, `tz` IANA timezone — default from browser at registration, undo window, per-sender image allowlist)

### 9.2 WebSocket `/api/ws`
Auth via session cookie. Server → client frames: `{event, payload}` for every §8.3 event belonging to the user (that's the whole contract — the UI refetches via TanStack Query invalidation keyed on ids in the payload). Client → server: `{type:"ping"}` only. Zod-validate all frames.

### 9.3 Frontend routes
`/login`, `/register`, `/mail` (unified inbox), `/mail/a/:accountId`, `/mail/a/:accountId/f/:mailboxId`, `/mail/t/:threadId` (thread pane; on desktop rendered beside the list), `/mail/search?q=`, `/settings/{profile|security|accounts|appearance}`; `/settings/accounts/new` = wizard.

**Account wizard flow:** step 1 email → calls `discover` while typing pauses; step 1.5 credential entry, whose form depends on the resolved preset's `auth_strategy` (§6.2.1): the `ProviderCredentialPanel` (§6.2.2) for `app_password` providers such as Gmail, the "Sign in with Google/Microsoft" button for `oauth2`, or the generic password box for `password`; step 2 shows resolved config with source badge ("found via ISPDB") and a Test button (auto-run, or auto-fired by the panel on paste); failure expands the manual form pre-filled with the best guess and the precise error, and — for providers whose `auth_strategy` includes a fallback — offers the alternate flow (e.g. Gmail app-password failure → "Sign in with Google instead"); step 3 = created, live backfill progress bar (WS `sync.progress`) with "start reading now" (list fills as backfill proceeds).

### 9.4 ThreadView contract (invariant I-5)
```
ThreadView(props: { messages: Message[] /* ASC always */, order: 'asc'|'desc',
                    composerState, ... })
```
- Rendering order = `order==='asc' ? messages : [...messages].reverse()`. No other branching on `order` is allowed in layout code.
- **Composer anchor** = a slot component rendered adjacent to the newest message: after it when `asc` (bottom of thread), directly after it when `desc` (top of thread, above older messages). Same component, one placement rule: "composer is always attached to the newest message."
- Initial scroll: to the newest message (bottom for asc, top for desc). Collapse: all read messages except the newest are collapsed; "Expand all" toggles. Unread messages are never collapsed.
- Keyboard: `j`/`k` move in *visual* order; `o`/`Enter` expand; `r` reply, `a` reply-all, `f` forward (open composer at anchor); `e` archive; `/` focus search; `gi` go to unified inbox; `esc` close composer (draft persists). Toolbar has the per-thread order toggle; it updates local state only, and long-presses/menu offers "make default" (PATCH /settings).
- Message HTML renders in `<iframe sandbox="" srcdoc={...}>` sized via ResizeObserver postMessage shim; remote images show a "load images / always from this sender" bar when the sanitizer flagged any.

## 10. Security requirements (hard)

1. Argon2id params per §6.1; no other password hashing anywhere.
2. **Credential vault:** mailbox secrets encrypted with AES-256-GCM under per-account data keys, wrapped by a master key from env `MASTER_KEY` (32 B, base64) — `cryptography` library, key id embedded for future rotation. Decryption code path exists only in `workers/` and `accounts/test`. Secrets never appear in logs, API responses, or exceptions (custom repr).
3. CSRF double-submit on all mutating routes; `__Host-` cookie prefix; CSP: `default-src 'self'; frame-src 'self'; img-src 'self' data:` (remote mail images are proxied? **No** — v1 loads allowed remote images inside the sandboxed iframe only, which has its own relaxed CSP via srcdoc; document this in code).
4. nh3 sanitization at index time; re-sanitize on render if sanitizer version changed (store `sanitizer_v` on body_parts).
5. Rate limits: login (§6.1), `POST /accounts/discover` 10/min/user, `POST /send` 30/h/user (env-tunable).
6. All outbound autodiscovery HTTP: HTTPS only, 5 s timeout, response size cap 256 KB, no off-domain redirects (SSRF guard: resolve + reject private IP ranges).
7. Every SQL query and search query includes `user_id` (enforced by repository-layer helpers; raw session use in API code is a review error).
8. Uploads (attachments to drafts): 25 MB/file, 40 MB/message total, content-type sniffed server-side, stored to blob store, never executed/served with original content type unless in safe list (images/pdf/text) — others force `application/octet-stream` + attachment disposition.

## 11. Plugin API (freeze in M3 — including the heavy-plugin surfaces below)

A plugin = Python package `webmail_plugin_<name>` exposing:

```python
plugin = Plugin(
    name, version,
    hooks       = {event: handler},          # subscribe to §8.3 events
    router      = APIRouter | None,          # mounted at /api/plugins/{name}
    ui_manifest = UiManifest | None,         # slots + pages, see below
    migrations  = "migrations/" | None,      # 11.1
    jobs        = [JobSpec, ...] | None,     # 11.2
)
```

Loader discovers plugins via entry points group `webmail.plugins`, forwards declared events, and serves `GET /api/plugins/ui-manifest` for the frontend loader. Plugins may not import from `app.*` internals other than `app.plugins.sdk` (which exposes: DB session factory scoped to plugin tables, EventBus publish under `plugin.{name}.*`, blob store, outbox draft/send helpers, settings accessor, HTTP client with the §10.6 SSRF guard built in).

Every plugin gets the namespaced KV store (`plugin_kv(plugin, user_id, key)`) for light state. Heavy plugins additionally get:

### 11.1 Plugin-owned tables
A plugin may ship Alembic migrations in its own version branch (`alembic -n plugin_{name}`). All tables MUST be prefixed `plg_{name}_` and every user-data table MUST carry `user_id` (invariant I-7 applies to plugins). Core migrations never reference plugin tables and vice versa. Uninstalling a plugin offers `--drop-data` to run its downgrade chain.

### 11.2 Plugin background jobs
`JobSpec(func, schedule: cron_str | None, singleton: bool)` registers tasks on the shared arq worker: cron-scheduled (e.g. a sync loop trigger) or enqueued on demand from hooks/routes. `singleton=True` guarantees at most one running instance per (plugin, user) via a Redis lock. Plugin jobs run in the same worker process pool with the same retry semantics as core jobs and are visible in worker logs under `plg.{name}.*`.

### 11.3 Frontend surfaces
- **Slots** (lazy-loaded ESM injected into mail UI): `thread.toolbar`, `message.actions`, `message.banner` (a strip rendered above a message body — used for spam warnings, calendar invite cards, and similar per-message notices), `sidebar.section`, `settings.page`.
- **Pages** (`app.page`): a plugin may declare full views: `{route: "/calendar", nav: {icon, label, order}, bundle}`. The shell adds the nav entry, and the route (including sub-routes under it) lazy-loads the bundle into the **entire content area** — no mail list/thread components are rendered. Page bundles receive the SDK context (API client, WS event stream filtered to `plugin.{name}.*` + requested core events, design-system components) and are sandboxed to their route: importing mail-component internals is a review error. Removing the plugin removes the nav entry and route cleanly.
- **Capabilities**: the manifest may declare named capabilities mapping to plugin endpoints, e.g. `capabilities: {"recipients.suggest": "/api/plugins/contacts/suggest"}`. Core UI features query all providers of a capability they support and merge results client-side (defined capabilities in v1: `recipients.suggest` — used by the composer To/Cc/Bcc fields and by any other plugin needing person-picking, such as calendar attendees). This keeps core backend code plugin-unaware.

Acceptance test for the API: implement **filters** (rules on `message.new`: match → move/flag) and **labels** entirely as plugins using only slots + KV; then implement the **calendar** (Appendix A) using tables + jobs + a page — all with zero core diffs.

## 12. Milestones with acceptance criteria

**M0 — Architecture spike**
Login/register (no 2FA), one hardcoded-config IMAP account, backfill+IDLE for INBOX, JWZ, read-only unified list + thread view with working asc/desc toggle and correct composer-anchor placeholder, Postgres FTS free-text search.
✅ Accept: sync a real 10k-message Gmail account via app password; threads match Gmail's grouping on a 50-thread sample ≥ 90%; toggle order and the newest message + anchor swap ends correctly; search returns in < 500 ms.

**M1 — MVP**
Full identity (2FA, sessions UI, reset), account wizard with full discovery chain, multi-account + unified inbox with account chips, flags/move/archive both directions, inline reply + reply-all/forward + SMTP send + Sent append, recipient autocomplete from `known_recipients`, drafts autosave, sanitized HTML + blocked remote images, operator search (full §8.2 grammar) on PG FTS, WS live updates.
✅ Accept: add gmail.com (via the app-password helper panel: deep-link → paste → auto-test → Connected ✓), outlook.com, and one autoconfig-only domain through the wizard; the Gmail panel's `auth_failed` branch offers the OAuth fallback when `GOOGLE_OAUTH_ENABLED`; reply from a thread lands correctly threaded in the recipient's client; all §8.2 operators pass integration tests; `docker-compose up` from clean checkout works.

**M2 — Hardening / v1.0**
OAuth2 (Google + Microsoft, XOAUTH2 for IMAP/SMTP, server-side refresh), QRESYNC fast resync, drafts mirrored to server, undo-send, attachment upload/preview, Meilisearch backend + instant search-as-you-type UI + saved searches as virtual mailboxes, session revocation, backfill progress UX, folder-name i18n heuristics.
✅ Accept: typo query (`invocie`) finds "invoice" mail < 200 ms on a 100k-message corpus; kill -9 the worker mid-backfill and restart resumes without duplicates (idempotency via `(mailbox_id, uid)` unique key); undo-send within window results in no SMTP traffic.

**M3 — Plugin platform**
Loader, full SDK per §11 **including** plugin migrations (11.1), plugin jobs (11.2), and `app.page` (11.3); filters + labels shipped as plugins; plugin docs.
✅ Accept: both plugins install/uninstall cleanly with zero core diffs; a third trivial plugin ("Unsubscribe header button" on `message.actions`) can be written from `docs/plugins.md` alone; a skeleton page-plugin (route + nav entry + one plugin table + one cron job) round-trips install → run → uninstall `--drop-data` leaving no residue.

**M4 — Calendar plugin (Appendix A)**
Phases A1–A2: local calendars + full page views + iMIP invite handling; then A3: CalDAV client sync + ICS subscriptions.
✅ Accept: criteria in Appendix A §A.7.

**M5 — Contacts plugin (Appendix B)**
Phases B1–B2: contact books, full page, groups, add-from-mail, `recipients.suggest` capability, vCard import/export; then B3: CardDAV client sync. May be built in parallel with M4 — both depend only on the M3 SDK.
✅ Accept: criteria in Appendix B §B.7.

**M6 (optional) — Hosted mailboxes (Appendix D)**
A Stalwart mail server beside the app: every user gets a mailbox at their handle (`localpart@APP_DOMAIN`) with the same password, added to their webmail as a permanent (non-removable) account; webmail stops keeping its own full copy of that mail. Phases D0 (hosting decision) → D1 (attachment de-duplication, useful on its own) → D2 (Stalwart spike) → D3 (native account) → D4 (on-demand bodies) → D5 (security) → D6 (production).
✅ Accept: criteria in Appendix D §D.9. **Status: planned — awaiting the D0 decisions (§D.10).**

**M7 — Anti-spam plugin (Appendix C)**
May be built any time after M3 (depends only on the plugin SDK; parallelizable with M4/M5). Phases C1–C2 client-side; C3 activates at the M6 edge.
✅ Accept: criteria in Appendix C §C.8.

## 13. Testing & CI

- **Unit:** pytest + pytest-asyncio; threading vectors (§7.4); query parser (§8.2); autodiscovery resolver with `respx`-mocked HTTP + `dnspython` stub; MIME corpus in `tests/fixtures/mime/` (agent: include ≥ 25 nasty real-world-style fixtures — encoded headers incl. RFC2047 Polish diacritics, nested multiparts, missing Message-ID, malformed dates, HTML-only, inline cid images, 8-bit subjects, winmail.dat placeholder).
- **Integration:** docker-compose.ci.yml with Greenmail (IMAP+SMTP) and Dovecot; full loop test: append message to Greenmail → appears in API < 10 s → reply via API → arrives in Greenmail SMTP sink with correct References.
- **Frontend:** vitest + React Testing Library; ThreadView ordering/anchor tests for both directions; Playwright smoke: login → add Greenmail account via wizard → read → inline reply → search.
- **CI gates:** ruff, mypy --strict, pytest, tsc, eslint, vitest, Playwright smoke; coverage floor 80% on `threading_`, `search/queryparser`, `accounts/autodiscovery`, `indexer`.

## 14. Configuration (env, complete list)

`APP_DOMAIN`, `DATABASE_URL`, `REDIS_URL`, `MASTER_KEY`, `BLOB_ROOT` or `S3_*`, `SEARCH_BACKEND=pg|meili`, `MEILI_URL`, `MEILI_KEY`, `UNDO_SECONDS=10`, `SESSION_TTL_DAYS=30`, `OAUTH_GOOGLE_CLIENT_ID/SECRET`, `OAUTH_MS_CLIENT_ID/SECRET`, `GOOGLE_OAUTH_ENABLED=false` (hides the Gmail OAuth option until a Cloud project is configured; app-password onboarding is the default Gmail path — §6.2.1), `MAX_ATTACHMENT_MB=25`, `LOG_LEVEL`; sync engine v2 (§6.3): `WORKER_ROLE=jobs|sync`, `MAX_ACCOUNTS_PER_WORKER=400`, `LEASE_TTL=60`, `LEASE_RENEW=20`, `POLL_WARM=180`, `POLL_COLD=900`, `TIER_HOT_ACTIVITY_H=24`, `TIER_SESSION_GRACE_MIN=30`, `TIER_COLD_AFTER_D=3`, `HOT_BUDGET_PER_WORKER=250`, `PER_HOST_CONN_CAP=50`, `BULK_CONCURRENCY=2`, `PARSE_CONCURRENCY=4`, `STREAM_PARSE_MB=8`. All read once in `config.py` (pydantic-settings); no `os.environ` elsewhere.

---

## Appendix A — Calendar plugin (`webmail_plugin_calendar`)

The calendar is the reference "heavy plugin": it lives in the same app and repo, uses the §11 SDK only, and must build with **zero core diffs**. It ships as a page (`/calendar`) beside Mail — one product, one login.

### A.1 Scope by phase
- **A1 — Local calendar:** create/edit/delete events, multiple calendars per user, month/week/day/agenda views, recurrence, reminders (WS notification), timezone-correct display.
- **A2 — Invites over mail (iMIP):** detect `text/calendar` parts on `message.new`; render an invite card in the thread (`message.actions` + inline summary); Accept/Tentative/Decline sends an iTIP REPLY through the outbox; organizer flow sends REQUEST/CANCEL to attendees; updates (SEQUENCE bumps) modify the stored event.
- **A3 — External sources:** `CalendarProvider` interface with **CalDAV client** (first implementation; covers iCloud, Fastmail, Nextcloud) and read-only **ICS URL subscriptions** (holidays, team feeds); sync loop as a plugin job.
- **A4 (deferred, spec later):** Google/Microsoft API providers reusing M2 OAuth machinery; secret **ICS export feed** `GET /api/plugins/calendar/feed/{token}.ics`; CalDAV **server** endpoint for native phone sync — architecturally anticipated, not built now.

### A.2 Dependencies
`icalendar ~5.0` (parse/serialize RFC 5545), `python-dateutil` (RRULE expansion), `caldav ~1.3` (A3), `tzdata`. No other additions.

### A.3 Data model (plugin migrations, prefix `plg_calendar_`)
```sql
CREATE TABLE plg_calendar_calendars (
  id uuid PRIMARY KEY DEFAULT gen_random_uuid(),
  user_id uuid NOT NULL,
  name text NOT NULL,
  color text NOT NULL DEFAULT '#3b82f6',
  kind text NOT NULL DEFAULT 'local',     -- local|caldav|ics_subscription
  source_config jsonb NOT NULL DEFAULT '{}',  -- caldav url/collection, ics url
  secret_ct bytea,                        -- caldav credentials, §10.2 vault scheme
  sync_token text,                        -- CalDAV sync-token / ICS etag
  read_only boolean NOT NULL DEFAULT false,
  UNIQUE (user_id, name)
);

CREATE TABLE plg_calendar_events (
  id uuid PRIMARY KEY DEFAULT gen_random_uuid(),
  user_id uuid NOT NULL,
  calendar_id uuid NOT NULL REFERENCES plg_calendar_calendars(id) ON DELETE CASCADE,
  ical_uid text NOT NULL,                 -- UID from iCalendar; join key for iMIP updates
  sequence int NOT NULL DEFAULT 0,
  summary text NOT NULL DEFAULT '',
  description text NOT NULL DEFAULT '',
  location text NOT NULL DEFAULT '',
  dtstart timestamptz NOT NULL,
  dtend timestamptz,
  all_day boolean NOT NULL DEFAULT false,
  tz text NOT NULL DEFAULT 'UTC',         -- original TZID; render in user tz, store UTC
  rrule text,                             -- raw RRULE string, NULL = single
  exdates timestamptz[] NOT NULL DEFAULT '{}',
  recurrence_id timestamptz,              -- non-NULL = detached exception instance
  organizer jsonb,                        -- {name,email}
  attendees jsonb NOT NULL DEFAULT '[]',  -- [{name,email,partstat,role}]
  status text NOT NULL DEFAULT 'confirmed',   -- confirmed|tentative|cancelled
  my_partstat text,                       -- needs-action|accepted|declined|tentative
  reminders int[] NOT NULL DEFAULT '{}',  -- minutes before start
  origin_message_id uuid,                 -- link back to the invite mail thread
  raw_ical text,                          -- round-trip fidelity for iMIP/CalDAV
  UNIQUE (calendar_id, ical_uid, recurrence_id)
);
CREATE INDEX ON plg_calendar_events (user_id, dtstart);
```

**Recurrence rule:** store the RRULE string; expand instances at query time with `dateutil.rrule` over the requested view range only (no materialization table). Exceptions: `exdates` remove instances; a row with `recurrence_id` overrides one instance. Range queries must account for events starting before the window but recurring into it (expand from `dtstart`, cap iterations at 1000 per event).

### A.4 CalendarProvider interface (inside the plugin, mirrors §5.1)
```python
# PLUGIN INTERFACE VERSION: 1
class CalendarProvider(Protocol):
    async def verify(self) -> None: ...
    async def list_collections(self) -> list[RemoteCalendar]: ...
    async def pull(self, collection: str, *, sync_token: str | None
                   ) -> tuple[list[RemoteEvent], list[str], str]: ...
        # returns (changed_events_as_ical, deleted_uids, new_sync_token)
    async def push(self, collection: str, ical: str, *, uid: str,
                   etag: str | None) -> str: ...        # returns new etag
    async def delete(self, collection: str, uid: str, etag: str | None) -> None: ...
```
Implementations in A3: `CaldavProvider`, `IcsSubscriptionProvider` (pull-only, `push`/`delete` raise `ReadOnlyCalendar`). Sync loop = plugin cron job (`schedule="*/5 * * * *"`, `singleton=True` per user) doing pull → upsert by `(calendar_id, ical_uid, recurrence_id)` → local-dirty push with etag conflict handling (conflict ⇒ server wins, local copy saved as a duplicate marked `status='tentative'`, user notified).

### A.5 iMIP flow (A2)
- **Inbound:** hook on `message.new`; if a `text/calendar` part exists, parse METHOD:
  `REQUEST` → upsert event (match `ical_uid`; apply only if `SEQUENCE` ≥ stored) into the user's default calendar with `my_partstat='needs-action'`, set `origin_message_id`; `CANCEL` → `status='cancelled'`; `REPLY` → update that attendee's `partstat` on the organizer's copy.
- **RSVP:** invite card buttons call `POST /api/plugins/calendar/events/{id}/rsvp {partstat}` → update row → build iTIP REPLY ics → send via SDK outbox helper to the organizer (subject `Accepted: {summary}` etc.) → publish `plugin.calendar.event_updated`.
- **Organizer:** creating an event with attendees sends METHOD:REQUEST to each; edits bump `SEQUENCE` and re-send; deletion sends CANCEL. All outgoing invite mail uses the account whose address matches the organizer identity (picker if ambiguous).

### A.6 UI (page `/calendar`, nav icon between Mail and Settings)
Routes: `/calendar` (month default), `/calendar/{month|week|day|agenda}/:date`, `/calendar/event/:id`, `/calendar/settings` (calendars CRUD, CalDAV/ICS wizard reusing the account-wizard UX pattern: URL+credentials → test → collection picker). Interactions: click-drag create, drag-move/resize (writes optimistically, reverts on API error), event popover with edit/RSVP, mini-month navigator, calendar visibility toggles (client state), `t` = today, arrows = prev/next period. Invite linkage both ways: invite card in thread links to `/calendar/event/:id`; event view links back to the mail thread when `origin_message_id` is set. Live updates via WS `plugin.calendar.*` events. Timezone: render in user setting `settings.tz` (add to §9.1 settings; default from browser at registration).

### A.7 Acceptance criteria
1. Plugin installs/uninstalls (`--drop-data`) with zero diffs in `app/` and `frontend/src/` outside `plugins/`/`slots/` loader code.
2. A weekly-recurring event with two EXDATEs and one moved instance renders correctly in month and week views across a DST boundary (Europe/Warsaw fixtures).
3. Round-trip with Google Calendar **via iMIP mail only**: receive a Google invite → Accept in our UI → organizer's Google Calendar shows attendee accepted; our organizer invite → Google attendee accepts → our event shows `partstat=accepted`.
4. CalDAV sync against a Nextcloud/Radicale test container: create/edit/delete propagate both directions in ≤ 5 min (one cron cycle); conflict case produces the A.4 tentative-duplicate, never silent loss.
5. ICS subscription to a public holiday feed renders read-only events; editing is blocked in UI and API (403 `read_only_calendar`).
6. Killing the worker mid-sync and restarting causes no duplicate events (upsert key discipline).

---

## Appendix B — Contacts plugin (`webmail_plugin_contacts`)

The second heavy plugin: same rules as the calendar — §11 SDK only, zero core diffs, own page. Division of labor with core: core owns *implicit* recipient history (`known_recipients`, §4/§9.1) so autocomplete works even without this plugin; the Contacts plugin owns *explicit*, user-curated contact records and enriches suggestions via the `recipients.suggest` capability (§11.3).

### B.1 Scope by phase
- **B1 — Local contacts:** address books, contact CRUD with structured fields, groups, photo, search, dedicated page `/contacts`; "Add sender to contacts" from the thread view; suggestion capability wired into the composer.
- **B2 — Portability:** vCard 3.0/4.0 import (file, multi-card) and export (single contact / book), duplicate detection + merge tool.
- **B3 — External sources:** `ContactProvider` interface with **CardDAV client** sync (iCloud, Fastmail, Nextcloud, Radicale); per-book sync job.
- **B4 (deferred, spec later):** Google People / Microsoft Graph providers (reusing M2 OAuth), CardDAV **server** endpoint for native phone sync, book sharing.

### B.2 Dependencies
`vobject ~0.9` (vCard parse/serialize). Photos stored via the SDK blob store (JPEG/PNG, re-encoded server-side, ≤ 512×512). Nothing else.

### B.3 Data model (plugin migrations, prefix `plg_contacts_`)
```sql
CREATE TABLE plg_contacts_books (
  id uuid PRIMARY KEY DEFAULT gen_random_uuid(),
  user_id uuid NOT NULL,
  name text NOT NULL,
  kind text NOT NULL DEFAULT 'local',       -- local|carddav
  source_config jsonb NOT NULL DEFAULT '{}',-- carddav url/collection
  secret_ct bytea,                          -- carddav credentials, §10.2 vault scheme
  sync_token text,
  read_only boolean NOT NULL DEFAULT false,
  UNIQUE (user_id, name)
);

CREATE TABLE plg_contacts_contacts (
  id uuid PRIMARY KEY DEFAULT gen_random_uuid(),
  user_id uuid NOT NULL,
  book_id uuid NOT NULL REFERENCES plg_contacts_books(id) ON DELETE CASCADE,
  vcard_uid text NOT NULL,                  -- UID property; join key for CardDAV
  etag text,                                -- server etag (carddav books)
  name_display text NOT NULL DEFAULT '',    -- FN
  name_parts jsonb NOT NULL DEFAULT '{}',   -- {given,family,middle,prefix,suffix} (N)
  emails jsonb NOT NULL DEFAULT '[]',       -- [{value,type,pref}]
  phones jsonb NOT NULL DEFAULT '[]',
  addresses jsonb NOT NULL DEFAULT '[]',    -- structured ADR
  org text NOT NULL DEFAULT '',
  title text NOT NULL DEFAULT '',
  birthday date,
  urls jsonb NOT NULL DEFAULT '[]',
  notes text NOT NULL DEFAULT '',
  photo_sha256 text,
  raw_vcard text,                           -- round-trip fidelity: preserve unknown props
  updated_at timestamptz NOT NULL DEFAULT now(),
  UNIQUE (book_id, vcard_uid)
);
CREATE INDEX ON plg_contacts_contacts (user_id, name_display);
CREATE INDEX ON plg_contacts_contacts USING gin (emails jsonb_path_ops);

CREATE TABLE plg_contacts_groups (
  id uuid PRIMARY KEY DEFAULT gen_random_uuid(),
  user_id uuid NOT NULL,
  name text NOT NULL,
  UNIQUE (user_id, name)
);
CREATE TABLE plg_contacts_group_members (
  group_id uuid NOT NULL REFERENCES plg_contacts_groups(id) ON DELETE CASCADE,
  contact_id uuid NOT NULL REFERENCES plg_contacts_contacts(id) ON DELETE CASCADE,
  PRIMARY KEY (group_id, contact_id)
);
```

vCard mapping discipline: parse known properties into columns, keep the full source in `raw_vcard`, and on export/push regenerate from columns **merged over** `raw_vcard` so properties we don't model (e.g. `X-` extensions) survive round-trips.

### B.4 ContactProvider interface (inside the plugin, mirrors A.4)
```python
# PLUGIN INTERFACE VERSION: 1
class ContactProvider(Protocol):
    async def verify(self) -> None: ...
    async def list_books(self) -> list[RemoteBook]: ...
    async def pull(self, book: str, *, sync_token: str | None
                   ) -> tuple[list[RemoteVcard], list[str], str]: ...
        # (changed vcards, deleted uids, new sync_token)
    async def push(self, book: str, vcard: str, *, uid: str,
                   etag: str | None) -> str: ...
    async def delete(self, book: str, uid: str, etag: str | None) -> None: ...
```
Sync loop = plugin cron job (`*/5 * * * *`, singleton per user), upsert by `(book_id, vcard_uid)`, etag conflict ⇒ server wins + local copy kept as a duplicate flagged for the merge tool (never silent loss) — same policy as A.4.

### B.5 API & capability (`/api/plugins/contacts/`)
`GET /contacts?book=&group=&q=&cursor=` · `POST /contacts` · `GET/PUT/DELETE /contacts/{id}` · `POST /contacts/from-message {message_id}` (prefill from From header) · `GET/POST/DELETE /books`, `/groups`, group membership ops · `POST /import` (vCard file → report `{created, updated, skipped}`) · `GET /export?book=|contact=` (`.vcf`) · `GET /duplicates` (candidate pairs: same email, or same normalized name + fuzzy) · `POST /merge {keep_id, remove_id}` (field union, keep wins conflicts, group memberships merged) · **`GET /suggest?q=`** — the `recipients.suggest` capability endpoint: matches name/email prefix and substring, returns `[{name, email, source:"contact", contact_id, photo_url?}]`. Composer merge rule (client-side, defined here so behavior is deterministic): contact hits rank above `known_recipients` hits at equal prefix quality; dedupe by email, contact entry wins and carries the photo/display name.

### B.6 UI
Page `/contacts` (nav entry after Calendar): three-pane layout — books/groups sidebar, alphabetically grouped virtualized list with search-as-you-type, detail pane with edit-in-place form; import/export and merge review under `/contacts/tools`; CardDAV book wizard reusing the account-wizard UX pattern (URL + credentials → test → book picker). Mail integration via `message.actions` slot: "Add to Contacts" (hidden when the sender already matches a contact; then it becomes "View contact" linking to `/contacts/{id}`). Group compose: selecting a group in the composer expands to its members' preferred emails.

### B.7 Acceptance criteria
1. Zero core diffs; install/uninstall `--drop-data` round-trip clean.
2. Import a 500-card vCard file mixing v3.0/v4.0, quoted-printable encoding, and Polish diacritics in names (`Grzegorz Brzęczyszczykiewicz`) — all names, emails, and photos correct; re-export → re-import is lossless including `X-` properties (raw_vcard merge discipline).
3. Composer typing a 3-char prefix returns merged suggestions in < 100 ms p95 (contacts ranked above history, deduped by email, photos shown for contacts).
4. CardDAV sync against Radicale: create/edit/delete propagate both directions within one cron cycle; concurrent edit conflict produces a flagged duplicate, never silent loss; worker kill mid-sync ⇒ no duplicate contacts on restart.
5. "Add sender to contacts" from a thread creates a prefilled contact; the same sender's messages immediately render the contact's display name, and the action flips to "View contact".
6. Merge tool: merging two contacts unions emails/phones/groups, keeps the chosen primary's conflicting fields, and repoints nothing else in the system (calendar attendees and mail remain keyed by email, by design).

---

## Appendix C — Anti-spam plugin (`webmail_plugin_antispam`)

Third heavy plugin; §11 SDK only, zero core diffs. Architectural context: for **synced** accounts the upstream provider has usually already filtered spam, so this plugin is a *second-pass* filter (weak-provider accounts, unified experience, per-user trainable Bayes). It becomes the *first-pass* filter only for M6 hosted mailboxes, where the same engine also runs at the SMTP edge (§C.7).

### C.1 Scope by phase
- **C1 — Classification & actions:** score every newly synced inbox message; move high-scoring mail to Junk (through the existing move machinery, so the verdict propagates to the IMAP server); banner low-confidence suspects; per-user allow/deny lists; per-account enable toggle.
- **C2 — Learning loop:** explicit "Spam"/"Not spam" actions train per-user Bayes; passive learning from user moves in/out of Junk and (sampled, capped) from provider-classified Junk; stats page.
- **C3 (M6 era):** the same Rspamd configuration/expertise deployed at the mail edge (VPS), where full IP/SPF/DKIM/DMARC/greylisting checks are actually meaningful; hosted-mailbox delivery is filtered *before* it reaches the index.

### C.2 Engine decision
**Rspamd** as a service container (mature, fast, HTTP API) with a small **dedicated Redis** container for its Bayes/fuzzy storage (isolated from the app Redis on purpose — different persistence and flush semantics). No Python spam engine is written from scratch; the plugin is glue + policy. Compose additions: `rspamd` (`rspamd/rspamd:3.x`) + `redis-rspamd` (`redis:7`, `--save 300 1`). Rspamd's network checks (RBL/SURBL/fuzzy lookups leak message-derived hashes/URLs to third parties) are **off by default**, one settings toggle to enable (privacy default matches the product's self-hosted ethos).

### C.3 Data model (plugin migrations, prefix `plg_antispam_`)
```sql
CREATE TABLE plg_antispam_settings (
  user_id uuid PRIMARY KEY,
  enabled_accounts uuid[] NOT NULL DEFAULT '{}',   -- empty = all accounts
  move_threshold real NOT NULL DEFAULT 12.0,       -- score >= : move to Junk
  banner_threshold real NOT NULL DEFAULT 6.0,      -- score >= : warn banner
  network_checks boolean NOT NULL DEFAULT false,
  learn_from_provider_junk boolean NOT NULL DEFAULT true
);
CREATE TABLE plg_antispam_rules (
  id uuid PRIMARY KEY DEFAULT gen_random_uuid(),
  user_id uuid NOT NULL,
  kind text NOT NULL,          -- allow|deny
  pattern citext NOT NULL,     -- email or @domain
  UNIQUE (user_id, kind, pattern)
);
CREATE TABLE plg_antispam_verdicts (
  message_id uuid PRIMARY KEY,
  user_id uuid NOT NULL,
  score real NOT NULL,
  symbols jsonb NOT NULL DEFAULT '[]',   -- rspamd symbol list for explainability
  action text NOT NULL,        -- none|banner|moved|allow_listed|deny_listed
  learned text                 -- spam|ham|NULL
);
```

### C.4 Classification flow
Hook on `message.new`; process only messages landing in `special_use='inbox'` mailboxes of enabled accounts (never Sent/Drafts/Junk/Trash; never re-classify on flag changes). Order: (1) deny list → move to Junk, action `deny_listed`; (2) allow list → action `allow_listed`, stop; (3) POST raw blob to Rspamd `/checkv2` with `Deliver-To`-style per-user context header (per-user Bayes statistics via Rspamd user settings); (4) apply thresholds → move via the existing move job (verdict thus syncs to the server's Junk folder) or record `banner`; (5) write verdict row, publish `plugin.antispam.classified`. Rspamd unavailability = fail-open (deliver normally, log, retry queue for learning only — spam filtering must never delay or lose mail).

### C.5 Learning
- Explicit: `message.actions` buttons "Spam" / "Not spam" → `POST /api/plugins/antispam/learn {message_id, verdict}` → Rspamd `/learnspam|/learnham` with the per-user context → move message accordingly → update `verdicts.learned`.
- Passive: hook on `flags.changed`/move events — user moves message out of Junk ⇒ learn ham; into Junk ⇒ learn spam (dedup via `verdicts.learned`).
- Provider bootstrap (per account, if `learn_from_provider_junk`): during/after backfill, sample up to 500 provider-Junk messages as spam corpus and 500 aged inbox messages as ham, rate-limited background job. Gives per-user Bayes a warm start.

### C.6 API & UI
`GET/PATCH /api/plugins/antispam/settings` · `GET/POST/DELETE /rules` · `POST /learn` · `GET /stats` (verdict counts, top symbols, Bayes confidence). UI: `settings.page` (thresholds with plain-language labels, per-account toggles, lists editor, privacy toggle, stats); `message.banner` for the suspect zone ("Looks like spam — [Move to Junk] [Not spam] [Always trust sender]"); `message.actions` Spam/Not-spam. Junk folder view is core (SPECIAL-USE) and needs no changes.

### C.7 M6 edge integration (assumption, specified fully with M6)
At the hosted-mail phase the VPS edge (Stalwart's built-in filter or Postfix+Rspamd) performs first-pass filtering with the checks that only make sense at SMTP time (connecting IP, SPF/DKIM/DMARC verification, greylisting, rate limits); accepted mail is tagged with its edge score in a header, and this plugin trusts that header for hosted accounts instead of re-scanning — one user-visible spam system, two enforcement points.

### C.8 Acceptance criteria
1. Zero core diffs; install/uninstall `--drop-data` clean; with the plugin absent, mail flow is byte-identical to v1.4 core.
2. Rspamd container stopped ⇒ new mail still indexes and renders normally within normal latency (fail-open proven by test).
3. Verdict propagation: a moved-to-Junk verdict appears in the provider's Junk folder via IMAP within one sync cycle; "Not spam" moves it back and the same message is not re-classified.
4. Learning: after 20 spam + 20 ham teachings on a fixture corpus, a held-out obvious-spam fixture scores above `move_threshold` for that user and a ham fixture below `banner_threshold` (per-user isolation: a second user's scores are unaffected).
5. Deny-listed sender is moved without engine involvement; allow-listed sender bypasses even a high engine score.
6. Privacy: with `network_checks=false` (default), a network-capture test during classification shows zero outbound connections from Rspamd.

## Appendix D — Hosted mailboxes (M6, Stalwart)

Every webmail user gets a real mailbox on a **Stalwart** server running beside the app. The address is the user's handle (`localpart@APP_DOMAIN`, already mail-grade per §4), and the password is the webmail password. That mailbox appears in webmail as an account the user cannot remove. Because Stalwart holds the mail, webmail stops keeping its own full copy of it (§D.6).

### D.1 Hosting (decide first — this, not code, is the hard part)
Receiving needs inbound port 25, an MX record and a fixed address; many home ISPs block port 25. Sending from a home IP is commonly spam-foldered or refused, and needs reverse DNS (PTR), SPF, DKIM and DMARC.

| Option | Inbound | Outbound | Notes |
|---|---|---|---|
| **A. All on the NAS** | MX → home IP, port 25 forwarded | direct | Only with a static IP, settable PTR, unblocked port 25. |
| **B. NAS + relay (recommended)** | Small VPS holds the MX, forwards to the NAS over WireGuard | Relay service (SMTP2GO / Postmark / SES) or the VPS | Mail stays on the NAS; reliable delivery; the VPS is the SMTP-time spam edge (§C.7). |
| **C. Stalwart on a VPS** | VPS | VPS | Mail stored off the NAS; webmail reaches it over the internet. |

**Chosen (2026-10-10): option A** — everything on the QNAP; see `docs/mail-server-network.md` for the network, DNS and port plan and the connection checks.

### D.2 Infrastructure
- `stalwart` service in both compose stacks (dev `webmail`, prod `webmail-prod` — never share the project name), image pinned to an exact version.
- **Stores:** data store = a separate `stalwart` database on the existing Postgres; blob store = filesystem under `DATA_ROOT/stalwart` (bind mount, not a named volume). `backup.sh` and `webmail.sh update` (backup → migrate → roll → health-gate) then cover it; no RocksDB to snapshot separately.
- **TLS:** Stalwart's built-in ACME with DNS-01 (nginx owns 80/443), or the existing certificates mounted read-only.
- **Spam:** SMTP-time filtering at the edge (Stalwart's filter or Rspamd) tags an edge-score header; the anti-spam plugin trusts it for hosted accounts (§C.7). The existing Rspamd keeps per-user learning.

### D.3 Same username and password — no password copying
Stalwart uses a **SQL directory** reading a read-only view over `users` (dedicated read-only role):

```sql
CREATE VIEW mail_principals AS
  SELECT handle AS name, handle AS email, password_hash AS secret,
         quota_bytes AS quota, (blocked_at IS NOT NULL) AS disabled
  FROM users;
```

- Stalwart verifies argon2id PHC hashes, which is what `identity/passwords.py` produces, so registration, password change, reset, block and quota apply **instantly** with no provisioning or sync step.
- User deletion additionally purges the mailbox through Stalwart's management API.
- Webmail never stores the user's password: the worker signs in as a **master user** (`handle%master`, secret in `.env` as `STALWART_MASTER_SECRET`), not via `accounts.secret_ct`. *Verify the master-user feature and syntax in the pinned version during D2.*
- Reserved handles (`postmaster`, `abuse`, …) stay unregistrable; route them to the administrator.

### D.4 The permanent account in webmail
- Migration: `accounts.kind text NOT NULL DEFAULT 'external'` (`external` | `native`). The native account is created in `register_user`; a one-off command creates it for users that already exist.
- `DELETE /api/accounts/{id}` on a native account → `409 {error:{code:"native_account"}}`. UI hides Remove and the server-settings fields; signature, colour and display name stay editable.
- `provider_factory`: native accounts connect to `stalwart:993` / `stalwart:465` on the Docker network with the master credentials.
- **Sync:** first reuse the IMAP provider unchanged (IDLE + QRESYNC already work). Optional later: a JMAP provider with push (Stalwart's native protocol).
- Sending goes through Stalwart; Sent copy, retries and the Scheduled folder behave as for any account.

### D.5 External mail clients and 2FA
If phones/desktop clients may use the mailbox over IMAP/SMTP, a password alone would bypass TOTP. Rule: users with 2FA enabled must use **app passwords** for IMAP/SMTP — a `user_app_passwords` table (argon2id hashes) added to the directory view, plus a Settings screen to create/revoke them. Stalwart's login rate limits / auto-ban on. If external clients are not wanted, IMAP/SMTP submission is exposed only on the Docker network and this section shrinks to nothing.

**App passwords — built (v1.3.37, ahead of D5):** table `user_app_passwords` (migration 0024: `id, user_id, name, password_hash, created_at, last_used_at`), `app/identity/app_passwords.py`, routes `GET/POST /api/auth/app-passwords`, `DELETE /api/auth/app-passwords/{id}`, Settings → Security → App passwords.
- 16 random lowercase letters (~75 bits), shown once in groups of four, copied and entered **without spaces**: the hash is of the bare letters, because the mail server compares exactly what an app sends.
- Creating one takes the account password; at most 25 per user; each is revoked on its own; a password reset revokes them all (as Google does). The webmail's own sign-in never accepts one.
- Still for D2/D5: add the hashes to the directory's secrets query (a user's secrets = `users.password_hash` unless 2FA is on, plus every `user_app_passwords.password_hash`), and set `last_used_at` from the mail server's successful sign-ins.

### D.6 Storing mail once
Today the indexer keeps every message in full (raw `.eml` blob + sanitized body in `body_parts` + search index), so external accounts are already a second copy, and with Stalwart both copies would sit on the same disk. Ideas, best first:

1. **Attachments are stored twice inside webmail today.** `indexer.py` stores each attachment as its own blob *and* it is inside the raw blob (base64), ≈2.3× its size; attachments are most of mail's bytes. Fix: keep only the raw message, extract an attachment when downloaded, keep thumbnails cached. Cuts attachment storage by about 40% (≈1.37× instead of ≈2.37× the file size) for **all** accounts, independent of Stalwart. A background pass frees existing duplicates. **Done (v1.3.34):** `app/indexer/attachments.py`, migration 0023.
2. **Native accounts: no raw copy in webmail.** Stalwart is the master copy; webmail keeps metadata, sanitized body, snippet and search index (~5–10% of the size). Full message, attachments and "show original" are fetched on demand over the local network (IMAP `BODY[part]` or JMAP blob download), with a small cache for repeat opens and thumbnails.
3. **External accounts (optional setting): age out old raw copies.** Keep raw blobs for N days or the newest X GB; refetch older ones over IMAP when needed. Readable bodies stay local; old attachments then need the external server reachable.
4. **Filesystem:** at least lz4 compression on the ZFS data share (QuTS hero). ZFS dedup would merge identical copies but costs a lot of RAM and only hits uncompressed byte-identical data — last resort.
5. **Rejected — MTA-only Stalwart** delivering straight into webmail with no IMAP mailbox: zero duplication, but phones/Thunderbird lose access unless webmail implements IMAP. Items 1 + 2 give nearly the same saving.

**Recommendation:** item 1 now, item 2 together with Stalwart, item 3 later as a setting.

### D.7 Phases
- **D0 — Decisions:** hosting option (§D.1), domain, whether external IMAP clients are allowed (§D.5).
- **D1 — Attachment de-duplication** (§D.6 item 1): independent, can ship any time.
- **D2 — Spike (dev):** Stalwart container, Postgres stores, SQL-directory view, master-user login, IMAP sync of a test user.
- **D3 — Native account:** `accounts.kind`, creation at registration + for existing users, undeletable in API and UI, provider wiring, delete/block hooks; tests against a Stalwart container (may replace greenmail).
- **D4 — On-demand bodies for native accounts** (§D.6 item 2).
- **D5 — Security:** app passwords + 2FA rule, rate limits.
- **D6 — Production:** compose, DNS (MX/SPF/DKIM/DMARC/PTR), relay or VPS per §D.1, backup, `docs/nas-install.md` runbook, release.

### D.8 Configuration additions (§14)
`STALWART_URL` (management API), `STALWART_IMAP_HOST` / `STALWART_SMTP_HOST`, `STALWART_MASTER_SECRET`, `STALWART_ADMIN_SECRET`, `HOSTED_MAILBOXES` (on/off; off = M6 inert, byte-identical to pre-M6 behaviour).

### D.9 Acceptance criteria
1. A newly registered user can receive mail from an outside address at their handle and sees it in webmail within one sync cycle; mail they send arrives at Gmail not in spam (SPF/DKIM/DMARC pass).
2. The same handle + password signs in to IMAP (when §D.5 allows it); after a webmail password change or reset, the old password fails and the new one works with no other step; a blocked user cannot sign in to IMAP.
3. The native account cannot be removed via API (409) or UI; deleting the user removes the Stalwart mailbox.
4. Webmail stores no copy of the account's password (`secret_ct` unused for native accounts).
5. After D1, an attachment-heavy fixture mailbox uses ≤ 60% of the pre-D1 blob storage, and every attachment still downloads byte-identical.
6. After D4, a native message's raw blob is absent from `BLOB_ROOT`, yet opening, downloading attachments and "show original" all work; with Stalwart stopped, list and bodies still render from local data.
7. Backup → restore on a fresh host brings back both webmail and Stalwart mail intact.

### D.10 Open questions (owner: product)
1. Hosting option A, B or C? Static IP and open port 25 available?
2. Which domain do the addresses use?
3. May external mail clients (IMAP/SMTP) use these mailboxes?
4. Ship D1 ahead of the rest?

## Appendix E — CalDAV / CardDAV server (phones, Thunderbird, macOS)

Ultimo serves the calendars and address books it already holds to standard clients — iPhone/iPad and macOS Calendar & Contacts, Thunderbird, Android via DAVx5, GNOME/Evolution — so they sync two-way with the web app. No second store: clients read and write the calendar/contacts plugins' own tables. (Chosen over Stalwart's DAV, which would make Stalwart the store and the plugins its clients, §D.)

### E.1 Shape
- **Core `app/dav/`** — the protocol: routes, Basic sign-in, discovery, principal, XML (PROPFIND / PROPPATCH / REPORT multistatus), ETag preconditions, sync tokens (RFC 6578). No knowledge of events or contacts.
- **Providers in the plugins** — `Plugin(dav=...)`: the calendar plugin serves calendar collections (iCalendar, RFC 4791), the contacts plugin address books (vCard, RFC 6352). Interface: list collections; list resources (name + ETag); get; put (If-Match / If-None-Match); delete; changes since a sync token; create / update / delete a collection; calendar time-range query. Each runs in the user's RLS session (`sdk.session`).
- **URLs** — `/.well-known/caldav`, `/.well-known/carddav` → 301 `/dav/`; principal `/dav/principals/{user_id}/`; homes `/dav/calendars/{user_id}/`, `/dav/addressbooks/{user_id}/`; a collection `…/{collection_id}/`; a resource `…/{name}.ics|.vcf`. nginx proxies `/dav/` and the two well-known paths to the API.

### E.2 Sign-in
HTTP Basic over HTTPS: the handle (full address or local part) and an **app password** (§D.5) — never the account password, so 2FA cannot be bypassed and each device can be revoked. A successful check is cached briefly (keyed by a hash of the password, invalidated when app passwords change) so a sync's many requests don't each cost an argon2id verify; failures are rate-limited per address and IP. `last_used_at` is set (at most every few minutes).

### E.3 Calendars
- A **resource** is one iCalendar object per UID: the series plus its moved/cancelled occurrences (`RECURRENCE-ID`). Name: what the client PUT it as, else derived from the UID.
- **Fidelity:** what a client PUTs is kept verbatim and served back unchanged while the event's modelled fields are still what it said; once the web app (or iMIP, or an upstream sync) changes them, the object is regenerated from the fields — SUMMARY, DESCRIPTION, LOCATION, DTSTART/DTEND with TZID and VTIMEZONE, all-day dates, RRULE, EXDATE, overrides, STATUS, TRANSP, ORGANIZER, ATTENDEEs with PARTSTAT, VALARMs from reminders, COLOR, the video link.
- **Writes** go through the same columns the web app uses, so reminders, invitations and upstream CalDAV write-back (the `dav_dirty` trigger, tombstones on synced calendars) behave identically. Subscribed (ICS) and read-only calendars are served read-only.
- **Change tracking** by triggers on the plugin tables (any write path counts) into a per-calendar change log; a calendar's sync token / CTag is its latest change number; deletions are reported by name. Old log entries are pruned; a client holding a pruned token gets `valid-sync-token` and resyncs.
- REPORTs: `calendar-query` (time-range on VEVENT), `calendar-multiget`, `sync-collection`. MKCALENDAR and PROPPATCH (name, colour) create / restyle local calendars; DELETE of a collection deletes a local calendar.

### E.4 Contacts
Address books = `plg_contacts_books`; resources = contacts as vCard (3.0 served; 3.0 and 4.0 accepted): FN, N, NICKNAME, EMAIL (type, pref), TEL, ADR, URL, ORG/department, TITLE, BDAY, NOTE, CATEGORIES ↔ groups, the VIP star as an `X-` property. Same fidelity rule (`raw_vcard` kept until the web app edits), same change tracking. REPORTs: `addressbook-query` (basic), `addressbook-multiget`, `sync-collection`.

### E.5 Phases
- **E1** core + discovery + sign-in + calendars read/write (tested with a CalDAV client library);
- **E2** vCard read/write + address books;
- **E3** deployment (nginx, docs: how to add the account on each client), then checks with real devices.

---
*End of specification. Ambiguities: prefer the simplest behavior consistent with the invariants in §1, leave a `TODO(spec)` comment, and list open questions in `docs/open-questions.md` rather than guessing silently.*