webmail-client/webmail-client-spec_v1.5.md

Webmail Client — Implementation Specification (v1.5, agent-ready)

This document is a complete, self-contained build specification. It is written to be handed to an AI coding agent. Read it fully before writing code. The companion document webmail-client-concept.md gives background; where they differ, this spec wins.

---

0. Instructions to the implementing agent

1. Build milestone by milestone (§12), in order. Do not start a milestone before the previous one's acceptance criteria pass.
2. Do not add features beyond this spec. Anything not specified here is out of scope; leave TODO(spec) comments instead of inventing behavior.
3. Definition of done for every task: code + type hints + unit tests + passing ruff check, ruff format --check, mypy --strict (backend) / tsc --noEmit, eslint (frontend) + updated docs in docs/ if an interface changed.
4. Stability over novelty: use exactly the pinned dependencies in §3. Do not swap libraries or add dependencies without a # DEPENDENCY: comment explaining why, and never for something the stdlib does adequately.
5. All interfaces in §5 are frozen contracts. Implementations may evolve; signatures may not (without bumping the version comment at the top of the interface file).
6. Security rules in §10 are hard requirements, not suggestions.
7. Every module publishing or consuming events must use the event names in §8 verbatim.
8. Commit in small, working increments; every commit must leave the test suite green.

1. Architecture invariants (never violate)

• I-1 Sync-first: the web UI and REST API never open IMAP connections. Only sync workers talk to mail servers. The API reads exclusively from Postgres/Meilisearch.
• I-2 Independent identity: login uses app accounts (localpart@our-web-client.example) with Argon2id hashes. Mailbox credentials are payload data, encrypted at rest, decrypted only inside sync workers.
• I-3 One index, many views: unified inbox, per-account views, virtual mailboxes, and search results are all parameterized queries over the same index. No duplicated storage per view.
• I-4 Threading is ours: JWZ in our backend (§7). Never use the IMAP THREAD extension.
• I-5 Direction-agnostic thread UI: thread ordering (oldest-first/newest-first) is a data-ordering concern. One layout, one composer-anchor slot (§9.4).
• I-6 Plugins extend, core stays closed: new features beyond this spec must be expressible through the plugin surfaces (§11).
• I-7 Multi-user isolation: every query (SQL and search) carries a mandatory user_id filter. No exceptions, including admin tooling.
• I-8 Mail HTML is hostile input: sanitize with nh3 server-side at index time AND render in a sandboxed iframe. Both, always.

2. Repository layout (monorepo)

webmail/
├── backend/
│   ├── pyproject.toml
│   ├── app/
│   │   ├── main.py                  # FastAPI app factory
│   │   ├── config.py                # pydantic-settings; all config via env
│   │   ├── db.py                    # SQLAlchemy async engine/session
│   │   ├── events.py                # EventBus (Redis pub/sub + arq enqueue)
│   │   ├── identity/                # §6.1 auth, sessions, handles, TOTP
│   │   ├── accounts/                # §6.2 wizard, autodiscovery, credential vault
│   │   ├── providers/
│   │   │   ├── base.py              # MailProvider interface (§5.1)
│   │   │   └── imap_smtp/           # first implementation
│   │   ├── sync/                    # §6.3 backfill, IDLE loops, reconciliation
│   │   ├── indexer/                 # §6.4 MIME parsing → DB rows + blobs
│   │   ├── threading_/              # §7 JWZ (trailing _ avoids stdlib clash)
│   │   ├── search/
│   │   │   ├── base.py              # SearchBackend interface (§5.2)
│   │   │   ├── queryparser.py       # operator grammar (§8.2)
│   │   │   ├── pg.py                # Postgres FTS backend (M0–M1)
│   │   │   └── meili.py             # Meilisearch backend (M2)
│   │   ├── outbox/                  # §6.5 drafts, MIME assembly, send queue, undo
│   │   ├── api/                     # routers, one file per §9.1 group
│   │   ├── ws.py                    # WebSocket hub (§9.2)
│   │   └── plugins/                 # §11 loader + plugin packages
│   ├── workers/worker.py            # arq worker entrypoint
│   ├── migrations/                  # alembic
│   └── tests/
├── frontend/
│   ├── package.json
│   └── src/
│       ├── api/                     # generated client from OpenAPI + WS layer
│       ├── routes/                  # §9.3
│       ├── components/
│       │   ├── thread/ThreadView.tsx    # §9.4 direction-agnostic
│       │   ├── thread/Composer.tsx
│       │   ├── list/MessageList.tsx     # virtualized
│       │   └── slots/               # plugin UI slots (§11)
│       └── lib/
├── docker/
│   ├── docker-compose.yml           # app, worker, postgres:16, redis:7, meilisearch:v1.8, dovecot (test)
│   └── docker-compose.ci.yml        # + greenmail for integration tests
└── docs/

3. Pinned stack

Backend (Python 3.11): fastapi ~0.115, uvicorn ~0.30, sqlalchemy ~2.0 (async, asyncpg), alembic ~1.13, pydantic ~2.8, pydantic-settings, arq ~0.26, redis ~5.0, aioimaplib ~1.1, imapclient ~3.0 (utility/fallback), aiosmtplib ~3.0, argon2-cffi ~23.1, pyotp ~2.9, dnspython ~2.6, httpx ~0.27, nh3 ~0.2, cryptography ~43.0, mail-parser ~4.0 (helpers only; primary parsing via stdlib email.message.EmailMessage with policy.default).

Frontend: TypeScript ~5.5, React ~18.3, Vite ~5, TanStack Query ~5, TanStack Virtual ~3, Radix UI, Tailwind ~3.4, zod (WS payload validation). API client generated from OpenAPI (openapi-typescript).

Infra: PostgreSQL 16 (with citext extension), Redis 7, Meilisearch v1.8 (M2+), MinIO or plain filesystem for the blob store (config-switchable; filesystem default).

4. Database schema (Postgres 16, authoritative DDL)

All PKs are uuid DEFAULT gen_random_uuid(). All tables carrying user data include user_id with an index. Timestamps are timestamptz.

CREATE EXTENSION IF NOT EXISTS citext;

CREATE TABLE users (
  id            uuid PRIMARY KEY DEFAULT gen_random_uuid(),
  handle        citext UNIQUE NOT NULL,        -- full form: localpart@<APP_DOMAIN>
  password_hash text   NOT NULL,               -- argon2id
  totp_secret   bytea,                         -- encrypted; NULL = 2FA off
  recovery_email citext,
  settings      jsonb  NOT NULL DEFAULT '{}',  -- incl. {"thread_order":"asc"|"desc"}
  created_at    timestamptz NOT NULL DEFAULT now()
);
-- Handle rules (enforce in code, mail-grade from day one):
--   localpart: 3–30 chars, [a-z0-9._-], no leading/trailing/double dot,
--   case-insensitive, reserved list (postmaster, abuse, admin, root, noreply, ...).

CREATE TABLE accounts (
  id             uuid PRIMARY KEY DEFAULT gen_random_uuid(),
  user_id        uuid NOT NULL REFERENCES users(id) ON DELETE CASCADE,
  email          citext NOT NULL,
  display_name   text NOT NULL DEFAULT '',
  color          text NOT NULL DEFAULT '#888888',
  imap_config    jsonb NOT NULL,   -- {host,port,tls:"ssl"|"starttls",auth:"password"|"oauth2"}
  smtp_config    jsonb NOT NULL,   -- same shape
  secret_ct      bytea NOT NULL,   -- envelope-encrypted password or OAuth token bundle (§10.2)
  discovery_src  text  NOT NULL,   -- preset|autoconfig|ispdb|srv|probe|manual
  state          text  NOT NULL DEFAULT 'backfilling',  -- backfilling|live|error|disabled
  state_detail   text  NOT NULL DEFAULT '',
  last_sync_at   timestamptz,
  created_at     timestamptz NOT NULL DEFAULT now(),
  UNIQUE (user_id, email)
);

CREATE TABLE mailboxes (
  id             uuid PRIMARY KEY DEFAULT gen_random_uuid(),
  account_id     uuid NOT NULL REFERENCES accounts(id) ON DELETE CASCADE,
  user_id        uuid NOT NULL REFERENCES users(id) ON DELETE CASCADE,
  path           text NOT NULL,                 -- IMAP mailbox name, UTF-7 decoded
  special_use    text,                          -- inbox|sent|drafts|trash|junk|archive|NULL
  uidvalidity    bigint NOT NULL DEFAULT 0,
  highestmodseq  bigint,                        -- NULL if no CONDSTORE
  last_seen_uid  bigint NOT NULL DEFAULT 0,
  UNIQUE (account_id, path)
);

CREATE TABLE threads (
  id              uuid PRIMARY KEY DEFAULT gen_random_uuid(),
  user_id         uuid NOT NULL REFERENCES users(id) ON DELETE CASCADE,
  subject_norm    text NOT NULL,                -- normalized subject (§7.3)
  snippet         text NOT NULL DEFAULT '',
  participants    jsonb NOT NULL DEFAULT '[]',  -- [{name,email}], deduped, ordered by recency
  account_ids     uuid[] NOT NULL DEFAULT '{}',
  message_count   int NOT NULL DEFAULT 0,
  unread_count    int NOT NULL DEFAULT 0,
  has_attachments boolean NOT NULL DEFAULT false,
  last_message_at timestamptz NOT NULL
);
CREATE INDEX ON threads (user_id, last_message_at DESC);

CREATE TABLE messages (
  id              uuid PRIMARY KEY DEFAULT gen_random_uuid(),
  user_id         uuid NOT NULL REFERENCES users(id) ON DELETE CASCADE,
  account_id      uuid NOT NULL REFERENCES accounts(id) ON DELETE CASCADE,
  mailbox_id      uuid NOT NULL REFERENCES mailboxes(id) ON DELETE CASCADE,
  uid             bigint NOT NULL,              -- IMAP UID within mailbox
  thread_id       uuid REFERENCES threads(id),
  message_id_hdr  text,                         -- Message-ID, angle brackets stripped
  in_reply_to     text,
  references_hdr  text[] NOT NULL DEFAULT '{}',
  subject         text NOT NULL DEFAULT '',
  from_addr       jsonb NOT NULL DEFAULT '[]',
  to_addr         jsonb NOT NULL DEFAULT '[]',
  cc_addr         jsonb NOT NULL DEFAULT '[]',
  date_hdr        timestamptz,
  internal_date   timestamptz NOT NULL,
  flags           text[] NOT NULL DEFAULT '{}', -- \Seen,\Flagged,\Answered,\Draft,\Deleted
  size_bytes      int NOT NULL DEFAULT 0,
  has_attachments boolean NOT NULL DEFAULT false,
  snippet         text NOT NULL DEFAULT '',     -- first ~160 chars of text body
  blob_sha256     text NOT NULL,                -- raw RFC822 in blob store (§6.4)
  body_indexed    tsvector,                     -- Postgres FTS (M0–M1 SearchBackend)
  UNIQUE (mailbox_id, uid)
);
CREATE INDEX ON messages (user_id, thread_id);
CREATE INDEX ON messages (account_id, message_id_hdr);
CREATE INDEX ON messages USING gin (body_indexed);

CREATE TABLE body_parts (
  id          uuid PRIMARY KEY DEFAULT gen_random_uuid(),
  message_id  uuid NOT NULL REFERENCES messages(id) ON DELETE CASCADE,
  kind        text NOT NULL,        -- text|html_sanitized
  content     text NOT NULL
);

CREATE TABLE attachments (
  id          uuid PRIMARY KEY DEFAULT gen_random_uuid(),
  message_id  uuid NOT NULL REFERENCES messages(id) ON DELETE CASCADE,
  user_id     uuid NOT NULL,
  filename    text NOT NULL,
  mime_type   text NOT NULL,
  size_bytes  int  NOT NULL,
  content_id  text,                 -- for inline cid: images
  blob_sha256 text NOT NULL
);

CREATE TABLE drafts (
  id          uuid PRIMARY KEY DEFAULT gen_random_uuid(),
  user_id     uuid NOT NULL REFERENCES users(id) ON DELETE CASCADE,
  account_id  uuid NOT NULL REFERENCES accounts(id) ON DELETE CASCADE,
  thread_id   uuid REFERENCES threads(id),      -- NULL = new mail
  reply_to_message_id uuid REFERENCES messages(id),
  kind        text NOT NULL DEFAULT 'reply',    -- reply|reply_all|forward|new
  payload     jsonb NOT NULL,       -- {to,cc,bcc,subject,html,text,attachment_ids}
  updated_at  timestamptz NOT NULL DEFAULT now()
);

CREATE TABLE outbox (
  id          uuid PRIMARY KEY DEFAULT gen_random_uuid(),
  user_id     uuid NOT NULL,
  draft_id    uuid NOT NULL,
  state       text NOT NULL DEFAULT 'undo_window',  -- undo_window|sending|sent|failed|cancelled
  send_after  timestamptz NOT NULL,                 -- now() + undo window
  error       text
);

CREATE TABLE saved_searches (
  id       uuid PRIMARY KEY DEFAULT gen_random_uuid(),
  user_id  uuid NOT NULL REFERENCES users(id) ON DELETE CASCADE,
  name     text NOT NULL,
  query    text NOT NULL,           -- raw operator-syntax string (§8.2)
  pinned   boolean NOT NULL DEFAULT true,
  UNIQUE (user_id, name)
);

-- Core recipient autocomplete (Gmail-style, derived from mail history).
-- The Contacts plugin (Appendix B) ENRICHES suggestions; this table works without it.
CREATE TABLE known_recipients (
  user_id      uuid NOT NULL REFERENCES users(id) ON DELETE CASCADE,
  email        citext NOT NULL,
  name         text NOT NULL DEFAULT '',      -- last seen display name
  times_used   int NOT NULL DEFAULT 0,        -- weighted: sent-to counts 3, received-from counts 1
  last_used_at timestamptz NOT NULL,
  PRIMARY KEY (user_id, email)
);
CREATE INDEX ON known_recipients (user_id, email text_pattern_ops);

Blob store: raw RFC822 messages and attachment payloads are stored content-addressed at ${BLOB_ROOT}/{user_id}/{sha256[:2]}/{sha256} (filesystem) or the equivalent object key (MinIO). Postgres holds metadata only. Deleting a user removes their blob subtree.

Sessions live in Redis (sess:{token} → JSON {user_id, created, ua}, TTL 30 days sliding), not Postgres.

5. Frozen core interfaces (Python)

5.1 providers/base.py — MailProvider

# INTERFACE VERSION: 1
from dataclasses import dataclass
from typing import AsyncIterator, Protocol
from datetime import datetime

@dataclass(frozen=True)
class RemoteMailbox:
    path: str
    special_use: str | None      # "inbox"|"sent"|"drafts"|"trash"|"junk"|"archive"|None
    uidvalidity: int

@dataclass(frozen=True)
class RemoteMessage:
    uid: int
    raw: bytes                   # full RFC822
    flags: frozenset[str]
    internal_date: datetime

@dataclass(frozen=True)
class FlagChange:
    uid: int
    flags: frozenset[str]

@dataclass(frozen=True)
class MailboxDelta:
    new: list[RemoteMessage]
    flag_changes: list[FlagChange]
    expunged_uids: list[int]
    new_uidvalidity: int | None  # non-None => UIDVALIDITY changed, caller must full-resync
    new_highestmodseq: int | None

class MailProvider(Protocol):
    async def verify(self) -> None: ...
        # Raises ProviderAuthError / ProviderConnectError with a human-readable .detail

    async def list_mailboxes(self) -> list[RemoteMailbox]: ...

    def backfill(self, mailbox: str, *, newest_first: bool = True,
                 batch_size: int = 200) -> AsyncIterator[list[RemoteMessage]]: ...
        # Yields batches until the mailbox is exhausted. Headers+body in one pass.

    async def poll(self, mailbox: str, *, last_seen_uid: int,
                   highestmodseq: int | None) -> MailboxDelta: ...
        # One reconciliation pass. Uses QRESYNC/CONDSTORE when highestmodseq is not None,
        # otherwise UID SEARCH + FETCH FLAGS diff.

    async def idle(self, mailbox: str) -> None: ...
        # Blocks until the server signals change or 25 min elapse (re-issue IDLE). May raise
        # ProviderNotSupported => sync loop falls back to poll() every POLL_INTERVAL (default 120 s).

    async def push_flags(self, mailbox: str, uid: int,
                         add: frozenset[str], remove: frozenset[str]) -> None: ...
    async def move(self, uid: int, src: str, dst: str) -> int: ...   # returns new UID
    async def append(self, mailbox: str, raw: bytes,
                     flags: frozenset[str]) -> int: ...               # Sent/Drafts upload
    async def send(self, raw: bytes, envelope_from: str,
                   envelope_to: list[str]) -> None: ...               # SMTP submission

ImapSmtpProvider is the only M0–M2 implementation. A future NativeProvider (hosted mailboxes) or JmapProvider implements the same protocol — nothing above this interface may import IMAP libraries.

5.2 search/base.py — SearchBackend

# INTERFACE VERSION: 1
@dataclass(frozen=True)
class SearchQuery:
    text: str | None                      # free-text remainder after operator extraction
    filters: dict[str, object]            # parsed operators, keys per §8.2
    user_id: UUID                         # ALWAYS set (invariant I-7)

@dataclass(frozen=True)
class SearchHit:
    message_id: UUID
    thread_id: UUID
    score: float
    highlights: dict[str, str]            # field -> HTML-escaped snippet with <mark>

class SearchBackend(Protocol):
    async def index(self, doc: SearchDoc) -> None: ...
    async def delete(self, message_id: UUID) -> None: ...
    async def delete_user(self, user_id: UUID) -> None: ...
    async def query(self, q: SearchQuery, *, limit: int, cursor: str | None
                    ) -> tuple[list[SearchHit], str | None]: ...

SearchDoc fields: message_id, thread_id, user_id, account_id, mailbox_special_use, from_ (name+email), to, cc, subject, body_text, attachment_names[], has_attachments, is_unread, is_starred, date (unix), size_bytes. Both backends (pg.py, meili.py) index the same doc. Meilisearch: one index messages, filterable = every non-body field incl. user_id, searchable = subject, body_text, from_, to, cc, attachment_names, sortable = date; ranking rules default + date:desc tiebreak.

5.3 events.py — EventBus

# INTERFACE VERSION: 1
class EventBus(Protocol):
    async def publish(self, event: str, payload: dict) -> None:
        # Fan-out to Redis pub/sub channel "evt:{event}" AND enqueue arq job
        # "on_{event}" for durable consumers (indexer, plugins).
    def subscribe(self, event: str, handler: Handler) -> None: ...

Payloads always include user_id. Event names are listed in §8.3 and are the only ones allowed.

6. Backend module specifications

6.1 Identity (app/identity/)

• Registration: validate localpart per §4 rules against reserved list in identity/reserved.py; store full handle localpart@{APP_DOMAIN} (env APP_DOMAIN).
• Login: constant-time verify (argon2id, time_cost=3, memory_cost=64MiB, parallelism=4); if totp_secret set, require valid TOTP (±1 window). Rate limit: 10 attempts / 15 min per handle AND per IP (Redis counters); lockout returns 429 with Retry-After.
• Session cookie: __Host-session, httpOnly, Secure, SameSite=Lax. CSRF: double-submit token header X-CSRF for all mutating requests.
• TOTP setup: POST /auth/totp/setup returns provisioning URI + QR payload; enable requires one valid code; secret stored envelope-encrypted.
• Password reset: only if recovery_email verified; single-use token, 30 min TTL, Redis-backed.

6.2 Accounts & autodiscovery (app/accounts/)

discover(email) -> list[CandidateConfig] runs the chain, returning on first success but recording every step's outcome in the response (steps: [{method, status, detail}]) for wizard UX:

1. Presets — accounts/presets.yaml (config, not code): match by domain; ships with gmail.com, googlemail.com, outlook.com/hotmail/live, office365, yahoo, icloud/me.com, gmx, wp.pl, onet.pl, o2.pl, interia.pl. Each entry: imap/smtp host+port+tls, auth kind, oauth provider id if applicable.
2. Autoconfig — GET https://autoconfig.{domain}/mail/config-v1.1.xml?emailaddress={email} then https://{domain}/.well-known/autoconfig/mail/config-v1.1.xml; parse Thunderbird XML (incomingServer type=imap, outgoingServer type=smtp; %EMAILADDRESS%/%EMAILLOCALPART% substitution).
3. ISPDB — GET https://autoconfig.thunderbird.net/v1.1/{domain}; same XML format.
4. DNS SRV (RFC 6186) — _imaps._tcp.{domain} (SSL 993), _imap._tcp (STARTTLS 143), _submissions._tcp (465), _submission._tcp (587); lowest priority wins.
5. Probe — try {imap,mail}.{domain} × ports 993(ssl)/143(starttls) and {smtp,mail}.{domain} × 465(ssl)/587(starttls); TCP connect + TLS handshake + IMAP/SMTP greeting, 5 s timeout each, all concurrently.

All HTTP/DNS lookups: 5 s timeout, no redirects off-domain, HTTPS only. test(config, credentials): full IMAP LOGIN + LIST, and SMTP EHLO+AUTH (no message sent); errors mapped to precise user-facing strings (auth_failed, tls_failed, connect_timeout, unknown_host). On save: encrypt credentials (§10.2), create account row (state=backfilling), enqueue sync.bootstrap_account.

6.2.1 Provider onboarding strategy (auth_strategy)

Each preset in presets.yaml declares an ordered auth_strategy list telling the wizard which credential flows to offer and in what order. This is the routing that keeps Gmail (and similar) onboarding smooth without any server-side automation of the provider's pages. Values:

• app_password — provider supports a per-user application-specific password used as an ordinary IMAP/SMTP password (Gmail, Yahoo, iCloud, outlook.com personal). The wizard shows the provider-specific helper panel (§6.2.2) rather than the generic password box.
• oauth2 — XOAUTH2 flow (M2). For Google this is the fallback offered when app_password fails or is unavailable (brand-new accounts, Advanced Protection, admin-disabled). Gmail's auth_strategy is therefore [app_password, oauth2].
• password — plain IMAP password (self-hosted/ISP accounts): the generic password box.

Rationale recorded so the agent doesn't "optimize" it away: full-IMAP OAuth for Google requires restricted-scope verification + an annual CASA assessment, which this deployment declines by default; app passwords are a first-class Google-provided mechanism that needs none of it and reuses the existing password code path (zero new sync code). OAuth remains implemented for M2 as the fallback and for deployments that do complete verification. SEARCH_BACKEND-style toggle GOOGLE_OAUTH_ENABLED (default false) hides the OAuth option entirely when an operator hasn't configured a Cloud project.

6.2.2 Provider credential-helper panel (frontend, wizard step 1.5)

When the resolved preset's first auth_strategy is app_password, the wizard renders a ProviderCredentialPanel keyed by the preset's help_id (e.g. gmail, yahoo, icloud) instead of the generic password field. Content is data-driven from presets.yaml (help: {deep_link, app_name_hint, steps[], docs_url}) so adding a provider needs no code. For Gmail specifically:

• Deep-link button — "Open Google App Passwords" opens https://myaccount.google.com/apppasswords in a popup/new tab. Must fire from the user's click (a direct gesture) so popup blockers allow it. The panel does not and cannot prefill or read anything on Google's page.
• Inline steps (rendered from help.steps): enable 2-Step Verification if needed → name the password (suggest app_name_hint, e.g. "1.email") → copy the 16 characters.
• Single paste field — on paste/change, client strips all whitespace (Google displays four groups of four) before use.
• Instant verify — on a complete-looking value (16 chars), auto-fire POST /accounts/test with the resolved IMAP/SMTP config; show Connected ✓ or the precise mapped error without the user pressing anything else.
• Failure recovery (the two real dead-ends): if test returns auth_failed, show a branch explaining the likely causes — 2FA not fully enabled, or a brand-new Google account for which Google withholds app passwords until it ages/gains trust, or a Workspace account with the feature admin-disabled — each with the relevant link. When the preset's auth_strategy also contains oauth2 and GOOGLE_OAUTH_ENABLED, render a prominent "Sign in with Google instead" button that switches the wizard to the OAuth flow (which is not gated by account age). This single fallback converts the most common onboarding failure into a one-click recovery.

Hard prohibitions (security + trust — do not implement under any framing): no headless/automated browser driving of Google (or any provider) sign-in or app-password pages; never request or accept the user's actual Google account password; never scrape, screen-read, or otherwise extract the generated app password from the provider's page — the user copies and pastes it themselves. The panel's ceiling is deep-link + guide + paste + test. Anything beyond that gets user accounts flagged, breaks on provider markup changes, and is out of scope permanently, not just for now.

6.2.3 presets.yaml additions

Each preset gains auth_strategy: [...] and, where the first strategy is app_password, a help: block: {help_id, deep_link, app_name_hint, steps: [i18n keys], docs_url}. steps are i18n message keys (Polish + English at minimum, per §languages) so the copy is translatable, not hardcoded. Gmail, Yahoo, iCloud, and outlook.com personal ship with populated help blocks; corporate/M365 and self-hosted presets use oauth2 or password and no panel.

6.3 Sync engine v2 (app/sync/) — sharded, lease-owned, tiered

(Authoritative detail: sync-engine-v2-refactor.md; this section summarizes the binding rules.)

• Two worker roles (WORKER_ROLE): jobs = stateless arq consumer, N instances; sync = lease claimer + per-account runners, 1..N instances. Compose services worker-jobs / worker-sync, nofile ulimit 65536.
• Lease ownership: exactly one sync worker owns each live account. Redis SET sync:live (API-maintained, DB-reconciled every 300 s); lease sync:lease:{account_id} claimed SET NX EX LEASE_TTL, renewed every LEASE_RENEW via owner-checked Lua; fencing — on renewal failure, cancel that account's runner before any further provider/DB op. Worker heartbeats in sync:workers:{id}; rebalance releases excess above target*1.2+5. Crash failover = lease expiry + reclaim (≤ ~90 s), no coordinator.
• Event-driven lifecycle: account.state_changed adds/removes ids from sync:live at runtime — accounts go live ≤ 30 s after creation and stop ≤ 30 s after deletion with no restarts; runners re-verify state='live' each cycle (zombie-proof). New core event user.active (API publishes on login/WS connect) drives tier promotion.
• Bootstrap (unchanged from v1, now a bulk job): list_mailboxes() → upsert mailboxes (SPECIAL-USE mapping with name heuristics — Sent/Wysłane, Trash/Kosz, etc. in sync/folder_names.py); backfill newest-first per mailbox (INBOX first, then sent, then rest); sync.progress published per batch; on completion set state='live' → the API adds the id to sync:live and a sync worker claims it.
• AccountRunner per tier: hot = persistent INBOX IDLE + transient CONDSTORE poll of special folders every POLL_WARM; warm = transient poll of INBOX+specials every POLL_WARM; cold = every POLL_COLD. Promotion to hot on user.active (≤ 5 s) or new mail within TIER_HOT_ACTIVITY_H; demotion by inactivity; HOT_BUDGET_PER_WORKER degrades least-active non-session accounts. Delta application, UIDVALIDITY wipe-and-rebackfill, expunge soft-handling, and write-then-enqueue outbound flags are unchanged from v1.
• Connection budgets: ≤ 2 sync-side connections per account (4 total incl. jobs, shared semaphore); process-local per-destination-host semaphore cap PER_HOST_CONN_CAP.
• Queues: interactive (send/flags/move/draft jobs, timeout 120 s, concurrency 20) and bulk (backfills, timeout 3600 s, concurrency BULK_CONCURRENCY) — bulk work can never delay a send.
• Memory bounds: parse semaphore PARSE_CONCURRENCY; raw messages > STREAM_PARSE_MB stream to blob and parse from file — peak RAM ≈ PARSE_CONCURRENCY × STREAM_PARSE_MB per process regardless of mailbox size.
• Shutdown: SIGTERM → stop claimer → cancel runners → release leases → drain in < 10 s.

6.4 Indexer (app/indexer/)

Input: RemoteMessage. Steps: store raw blob (sha256, dedupe) → parse with email stdlib (policy.default) → extract: headers (decode RFC2047), references_hdr = parsed References + In-Reply-To message-ids in order; best text body and best HTML body (multipart/alternative preference: html for display, text for indexing; if only HTML, derive text via nh3 strip); sanitize HTML with nh3 (allowlist: structural tags, inline styles filtered, cid: images rewritten to /api/messages/{id}/cid/{content_id}, remote src rewritten to blocked-placeholder attributes for FR-23) → write messages, body_parts (kinds text and html_sanitized), attachments → hand to Threader (§7) → bus.publish("message.new", ...) (search indexing + thread denormalization are consumers). For non-junk received messages, upsert the From address into known_recipients (weight 1).

6.5 Outbox (app/outbox/)

• Drafts: autosave endpoint upserts drafts (client debounces 2 s); a background job mirrors drafts to the account's Drafts folder via append() at most every 60 s per draft (replace previous UID).
• Send: POST /send {draft_id} → build MIME (html + auto-generated text alternative via html-to-text; In-Reply-To/References from reply_to_message_id; attachments from blob store) → create outbox row with send_after = now()+UNDO_SECONDS (env, default 10) → schedule arq job at send_after. Undo endpoint flips state to cancelled if still in window. The job: SMTP send(), then append() to Sent (skip for Gmail preset — Gmail self-appends), mark sent, publish message.sent, and upsert every To/Cc/Bcc address into known_recipients (weight 3). Failure → failed + WS notification, draft preserved.

7. Threading specification (JWZ, app/threading_/)

7.1 Normalization

• message_id_hdr, in_reply_to, each References entry: strip angle brackets and whitespace, lowercase the domain part only. Discard obviously invalid ids (no @).
• Missing/duplicate Message-ID: synthesize sha256(raw)[:32]@synthetic.local and mark it non-authoritative (never used to join other messages to it).

7.2 Incremental algorithm (per new message)

1. Candidate set = all normalized ids in references_hdr + in_reply_to + own message_id_hdr.
2. SELECT DISTINCT thread_id FROM messages WHERE user_id=$1 AND message_id_hdr = ANY($2) AND thread_id IS NOT NULL.
3. 0 threads → subject fallback (§7.3); still none → create new thread.
4. 1 thread → attach.
5. ≥2 threads → merge: keep the thread with the oldest last_message_at as canonical, repoint messages of the others, delete emptied thread rows, publish thread.updated for all involved.
6. Recompute denormalized thread fields (count, unread, participants, snippet = newest message's snippet, has_attachments, account_ids, last_message_at) in one UPDATE.

Threads are cross-account by design: the candidate lookup is scoped by user_id, not account_id (invariant I-3). Sent-folder messages join threads exactly like received ones (FR-15).

7.3 Subject fallback

Only when reference joining fails: normalize subject (strip repeated Re:/Fw:/Fwd:/Odp:/PD: prefixes case-insensitively, collapse whitespace); join to a thread with the same subject_norm that has activity within 7 days AND shares ≥1 participant email; never join two messages that both lack references AND have empty subjects.

7.4 Test vectors (must pass in tests/threading/)

Linear chain; chain arriving out of order; reply missing In-Reply-To but carrying References; thread merge on late-arriving parent; broken client (subject-only) joins; distinct-topic same-subject ("hi") does NOT join; cross-account merge (message in account A references message in account B); duplicate Message-ID; synthetic-ID isolation.

8. Search specification

8.1 Pipeline

Consumer of message.new / message.changed / message.deleted → builds SearchDoc → active SearchBackend (env SEARCH_BACKEND=pg|meili). Backfill indexes inline with sync (per batch), never as a separate pass the user waits on (FR-30).

8.2 Query grammar (search/queryparser.py, engine-independent)

Tokenization: whitespace-split, respecting "quoted phrases". A token matching key:value (or -key:value) with a known key becomes a filter; everything else joins free text. - prefix negates a term or filter. OR (uppercase) groups adjacent terms/filters into a disjunction; default combinator is AND.

Operator	Filter semantics
from: to: cc:	substring match on name or email of that field
subject:	match constrained to subject
account:	account by display_name or email substring
in:	inbox|sent|drafts|trash|junk|archive (special_use) or folder path
is:	unread|read|starred|answered
has:attachment	boolean
filename:	attachment name substring
before: after:	YYYY-MM-DD
newer_than: older_than:	Nd|Nw|Nm|Ny
larger: smaller:	NK|NM bytes

Parser output = SearchQuery (§5.2). Unknown key: tokens are treated as free text (forward compatibility). Unit tests must cover every operator, negation, quoting, OR, and garbage input (parser never raises).

8.3 Event catalog (complete)

message.new, message.changed, message.deleted, message.sent, flags.changed, thread.updated, sync.progress, account.state_changed, draft.saved, outbox.state_changed, user.active (published by the API on login and WS connect; consumed by sync workers for tier promotion — not forwarded to browsers). Payloads: {user_id, ...ids, minimal-diff fields} — documented in docs/events.md (agent: generate it from code).

Plugins may additionally publish events namespaced plugin.{name}.* (e.g. plugin.calendar.event_updated); core code never subscribes to plugin events, but the WS hub forwards them to the owning user like any other event.

9. API, realtime, frontend

9.1 REST API (prefix /api, session-cookie auth, JSON; errors: {error:{code,message}})

Auth: POST /auth/register · POST /auth/login · POST /auth/logout · GET /auth/me · GET /auth/sessions · DELETE /auth/sessions/{id} · POST /auth/totp/setup · POST /auth/totp/enable · POST /auth/password/reset-request · POST /auth/password/reset

Accounts: POST /accounts/discover {email} → {candidates:[{imap,smtp,source}], steps:[...]} · POST /accounts/test {config, credentials} → {ok}|{error:{code,detail}} · POST /accounts · GET /accounts (incl. state, sync progress) · PATCH /accounts/{id} · DELETE /accounts/{id} (cascades rows, blobs, search docs)

Mail: GET /threads?view=unified|account:{id}|mailbox:{id}|search&q=&cursor=&limit=50 → thread summaries (denormalized fields only; cursor = last_message_at,id keyset) · GET /threads/{id}?sort=asc|desc → ordered full messages (bodies of the N newest expanded per client hint expand=; older = envelope only) · POST /threads/{id}/flags {add,remove} (whole thread) · POST /messages/{id}/flags · POST /messages/{id}/move {mailbox_id} · GET /messages/{id}/html (sanitized, for iframe srcdoc) · GET /messages/{id}/cid/{content_id} · GET /attachments/{id} (Content-Disposition attachment; images/PDF also inline for preview) · POST /messages/{id}/allow-remote-images (per-sender allowlist in user settings)

Compose: POST /drafts · PUT /drafts/{id} · DELETE /drafts/{id} · POST /send {draft_id} → {outbox_id, undo_until} · POST /send/{outbox_id}/undo · GET /recipients/suggest?q= → top 10 from known_recipients (prefix match on email or name, ranked by times_used decayed by recency); the composer merges these with any plugin capability providers (§11.3)

Search: GET /search?q=&cursor= → {threads:[...], hits_by_thread, highlights} (grouped by thread, FR-28; &group=message for flat) · GET/POST/DELETE /saved-searches

Settings: GET/PATCH /settings (incl. thread_order: "asc"|"desc", tz IANA timezone — default from browser at registration, undo window, per-sender image allowlist)

9.2 WebSocket /api/ws

Auth via session cookie. Server → client frames: {event, payload} for every §8.3 event belonging to the user (that's the whole contract — the UI refetches via TanStack Query invalidation keyed on ids in the payload). Client → server: {type:"ping"} only. Zod-validate all frames.

9.3 Frontend routes

/login, /register, /mail (unified inbox), /mail/a/:accountId, /mail/a/:accountId/f/:mailboxId, /mail/t/:threadId (thread pane; on desktop rendered beside the list), /mail/search?q=, /settings/{profile|security|accounts|appearance}; /settings/accounts/new = wizard.

Account wizard flow: step 1 email → calls discover while typing pauses; step 1.5 credential entry, whose form depends on the resolved preset's auth_strategy (§6.2.1): the ProviderCredentialPanel (§6.2.2) for app_password providers such as Gmail, the "Sign in with Google/Microsoft" button for oauth2, or the generic password box for password; step 2 shows resolved config with source badge ("found via ISPDB") and a Test button (auto-run, or auto-fired by the panel on paste); failure expands the manual form pre-filled with the best guess and the precise error, and — for providers whose auth_strategy includes a fallback — offers the alternate flow (e.g. Gmail app-password failure → "Sign in with Google instead"); step 3 = created, live backfill progress bar (WS sync.progress) with "start reading now" (list fills as backfill proceeds).

9.4 ThreadView contract (invariant I-5)

ThreadView(props: { messages: Message[] /* ASC always */, order: 'asc'|'desc',
                    composerState, ... })

• Rendering order = order==='asc' ? messages : [...messages].reverse(). No other branching on order is allowed in layout code.
• Composer anchor = a slot component rendered adjacent to the newest message: after it when asc (bottom of thread), directly after it when desc (top of thread, above older messages). Same component, one placement rule: "composer is always attached to the newest message."
• Initial scroll: to the newest message (bottom for asc, top for desc). Collapse: all read messages except the newest are collapsed; "Expand all" toggles. Unread messages are never collapsed.
• Keyboard: j/k move in visual order; o/Enter expand; r reply, a reply-all, f forward (open composer at anchor); e archive; / focus search; gi go to unified inbox; esc close composer (draft persists). Toolbar has the per-thread order toggle; it updates local state only, and long-presses/menu offers "make default" (PATCH /settings).
• Message HTML renders in <iframe sandbox="" srcdoc={...}> sized via ResizeObserver postMessage shim; remote images show a "load images / always from this sender" bar when the sanitizer flagged any.

10. Security requirements (hard)

1. Argon2id params per §6.1; no other password hashing anywhere.
2. Credential vault: mailbox secrets encrypted with AES-256-GCM under per-account data keys, wrapped by a master key from env MASTER_KEY (32 B, base64) — cryptography library, key id embedded for future rotation. Decryption code path exists only in workers/ and accounts/test. Secrets never appear in logs, API responses, or exceptions (custom repr).
3. CSRF double-submit on all mutating routes; __Host- cookie prefix; CSP: default-src 'self'; frame-src 'self'; img-src 'self' data: (remote mail images are proxied? No — v1 loads allowed remote images inside the sandboxed iframe only, which has its own relaxed CSP via srcdoc; document this in code).
4. nh3 sanitization at index time; re-sanitize on render if sanitizer version changed (store sanitizer_v on body_parts).
5. Rate limits: login (§6.1), POST /accounts/discover 10/min/user, POST /send 30/h/user (env-tunable).
6. All outbound autodiscovery HTTP: HTTPS only, 5 s timeout, response size cap 256 KB, no off-domain redirects (SSRF guard: resolve + reject private IP ranges).
7. Every SQL query and search query includes user_id (enforced by repository-layer helpers; raw session use in API code is a review error).
8. Uploads (attachments to drafts): 25 MB/file, 40 MB/message total, content-type sniffed server-side, stored to blob store, never executed/served with original content type unless in safe list (images/pdf/text) — others force application/octet-stream + attachment disposition.

11. Plugin API (freeze in M3 — including the heavy-plugin surfaces below)

A plugin = Python package webmail_plugin_<name> exposing:

plugin = Plugin(
    name, version,
    hooks       = {event: handler},          # subscribe to §8.3 events
    router      = APIRouter | None,          # mounted at /api/plugins/{name}
    ui_manifest = UiManifest | None,         # slots + pages, see below
    migrations  = "migrations/" | None,      # 11.1
    jobs        = [JobSpec, ...] | None,     # 11.2
)

Loader discovers plugins via entry points group webmail.plugins, forwards declared events, and serves GET /api/plugins/ui-manifest for the frontend loader. Plugins may not import from app.* internals other than app.plugins.sdk (which exposes: DB session factory scoped to plugin tables, EventBus publish under plugin.{name}.*, blob store, outbox draft/send helpers, settings accessor, HTTP client with the §10.6 SSRF guard built in).

Every plugin gets the namespaced KV store (plugin_kv(plugin, user_id, key)) for light state. Heavy plugins additionally get:

11.1 Plugin-owned tables

A plugin may ship Alembic migrations in its own version branch (alembic -n plugin_{name}). All tables MUST be prefixed plg_{name}_ and every user-data table MUST carry user_id (invariant I-7 applies to plugins). Core migrations never reference plugin tables and vice versa. Uninstalling a plugin offers --drop-data to run its downgrade chain.

11.2 Plugin background jobs

JobSpec(func, schedule: cron_str | None, singleton: bool) registers tasks on the shared arq worker: cron-scheduled (e.g. a sync loop trigger) or enqueued on demand from hooks/routes. singleton=True guarantees at most one running instance per (plugin, user) via a Redis lock. Plugin jobs run in the same worker process pool with the same retry semantics as core jobs and are visible in worker logs under plg.{name}.*.

11.3 Frontend surfaces

• Slots (lazy-loaded ESM injected into mail UI): thread.toolbar, message.actions, message.banner (a strip rendered above a message body — used for spam warnings, calendar invite cards, and similar per-message notices), sidebar.section, settings.page.
• Pages (app.page): a plugin may declare full views: {route: "/calendar", nav: {icon, label, order}, bundle}. The shell adds the nav entry, and the route (including sub-routes under it) lazy-loads the bundle into the entire content area — no mail list/thread components are rendered. Page bundles receive the SDK context (API client, WS event stream filtered to plugin.{name}.* + requested core events, design-system components) and are sandboxed to their route: importing mail-component internals is a review error. Removing the plugin removes the nav entry and route cleanly.
• Capabilities: the manifest may declare named capabilities mapping to plugin endpoints, e.g. capabilities: {"recipients.suggest": "/api/plugins/contacts/suggest"}. Core UI features query all providers of a capability they support and merge results client-side (defined capabilities in v1: recipients.suggest — used by the composer To/Cc/Bcc fields and by any other plugin needing person-picking, such as calendar attendees). This keeps core backend code plugin-unaware.

Acceptance test for the API: implement filters (rules on message.new: match → move/flag) and labels entirely as plugins using only slots + KV; then implement the calendar (Appendix A) using tables + jobs + a page — all with zero core diffs.

12. Milestones with acceptance criteria

M0 — Architecture spike Login/register (no 2FA), one hardcoded-config IMAP account, backfill+IDLE for INBOX, JWZ, read-only unified list + thread view with working asc/desc toggle and correct composer-anchor placeholder, Postgres FTS free-text search. ✅ Accept: sync a real 10k-message Gmail account via app password; threads match Gmail's grouping on a 50-thread sample ≥ 90%; toggle order and the newest message + anchor swap ends correctly; search returns in < 500 ms.

M1 — MVP Full identity (2FA, sessions UI, reset), account wizard with full discovery chain, multi-account + unified inbox with account chips, flags/move/archive both directions, inline reply + reply-all/forward + SMTP send + Sent append, recipient autocomplete from known_recipients, drafts autosave, sanitized HTML + blocked remote images, operator search (full §8.2 grammar) on PG FTS, WS live updates. ✅ Accept: add gmail.com (via the app-password helper panel: deep-link → paste → auto-test → Connected ✓), outlook.com, and one autoconfig-only domain through the wizard; the Gmail panel's auth_failed branch offers the OAuth fallback when GOOGLE_OAUTH_ENABLED; reply from a thread lands correctly threaded in the recipient's client; all §8.2 operators pass integration tests; docker-compose up from clean checkout works.

M2 — Hardening / v1.0 OAuth2 (Google + Microsoft, XOAUTH2 for IMAP/SMTP, server-side refresh), QRESYNC fast resync, drafts mirrored to server, undo-send, attachment upload/preview, Meilisearch backend + instant search-as-you-type UI + saved searches as virtual mailboxes, session revocation, backfill progress UX, folder-name i18n heuristics. ✅ Accept: typo query (invocie) finds "invoice" mail < 200 ms on a 100k-message corpus; kill -9 the worker mid-backfill and restart resumes without duplicates (idempotency via (mailbox_id, uid) unique key); undo-send within window results in no SMTP traffic.

M3 — Plugin platform Loader, full SDK per §11 including plugin migrations (11.1), plugin jobs (11.2), and app.page (11.3); filters + labels shipped as plugins; plugin docs. ✅ Accept: both plugins install/uninstall cleanly with zero core diffs; a third trivial plugin ("Unsubscribe header button" on message.actions) can be written from docs/plugins.md alone; a skeleton page-plugin (route + nav entry + one plugin table + one cron job) round-trips install → run → uninstall --drop-data leaving no residue.

M4 — Calendar plugin (Appendix A) Phases A1–A2: local calendars + full page views + iMIP invite handling; then A3: CalDAV client sync + ICS subscriptions. ✅ Accept: criteria in Appendix A §A.7.

M5 — Contacts plugin (Appendix B) Phases B1–B2: contact books, full page, groups, add-from-mail, recipients.suggest capability, vCard import/export; then B3: CardDAV client sync. May be built in parallel with M4 — both depend only on the M3 SDK. ✅ Accept: criteria in Appendix B §B.7.

M6 (optional) — Hosted mailboxes (Appendix D) A Stalwart mail server beside the app: every user gets a mailbox at their handle (localpart@APP_DOMAIN) with the same password, added to their webmail as a permanent (non-removable) account; webmail stops keeping its own full copy of that mail. Phases D0 (hosting decision) → D1 (attachment de-duplication, useful on its own) → D2 (Stalwart spike) → D3 (native account) → D4 (on-demand bodies) → D5 (security) → D6 (production). ✅ Accept: criteria in Appendix D §D.9. Status: planned — awaiting the D0 decisions (§D.10).

M7 — Anti-spam plugin (Appendix C) May be built any time after M3 (depends only on the plugin SDK; parallelizable with M4/M5). Phases C1–C2 client-side; C3 activates at the M6 edge. ✅ Accept: criteria in Appendix C §C.8.

13. Testing & CI

• Unit: pytest + pytest-asyncio; threading vectors (§7.4); query parser (§8.2); autodiscovery resolver with respx-mocked HTTP + dnspython stub; MIME corpus in tests/fixtures/mime/ (agent: include ≥ 25 nasty real-world-style fixtures — encoded headers incl. RFC2047 Polish diacritics, nested multiparts, missing Message-ID, malformed dates, HTML-only, inline cid images, 8-bit subjects, winmail.dat placeholder).
• Integration: docker-compose.ci.yml with Greenmail (IMAP+SMTP) and Dovecot; full loop test: append message to Greenmail → appears in API < 10 s → reply via API → arrives in Greenmail SMTP sink with correct References.
• Frontend: vitest + React Testing Library; ThreadView ordering/anchor tests for both directions; Playwright smoke: login → add Greenmail account via wizard → read → inline reply → search.
• CI gates: ruff, mypy --strict, pytest, tsc, eslint, vitest, Playwright smoke; coverage floor 80% on threading_, search/queryparser, accounts/autodiscovery, indexer.

14. Configuration (env, complete list)

APP_DOMAIN, DATABASE_URL, REDIS_URL, MASTER_KEY, BLOB_ROOT or S3_*, SEARCH_BACKEND=pg|meili, MEILI_URL, MEILI_KEY, UNDO_SECONDS=10, SESSION_TTL_DAYS=30, OAUTH_GOOGLE_CLIENT_ID/SECRET, OAUTH_MS_CLIENT_ID/SECRET, GOOGLE_OAUTH_ENABLED=false (hides the Gmail OAuth option until a Cloud project is configured; app-password onboarding is the default Gmail path — §6.2.1), MAX_ATTACHMENT_MB=25, LOG_LEVEL; sync engine v2 (§6.3): WORKER_ROLE=jobs|sync, MAX_ACCOUNTS_PER_WORKER=400, LEASE_TTL=60, LEASE_RENEW=20, POLL_WARM=180, POLL_COLD=900, TIER_HOT_ACTIVITY_H=24, TIER_SESSION_GRACE_MIN=30, TIER_COLD_AFTER_D=3, HOT_BUDGET_PER_WORKER=250, PER_HOST_CONN_CAP=50, BULK_CONCURRENCY=2, PARSE_CONCURRENCY=4, STREAM_PARSE_MB=8. All read once in config.py (pydantic-settings); no os.environ elsewhere.

---

Appendix A — Calendar plugin (webmail_plugin_calendar)

The calendar is the reference "heavy plugin": it lives in the same app and repo, uses the §11 SDK only, and must build with zero core diffs. It ships as a page (/calendar) beside Mail — one product, one login.

A.1 Scope by phase

• A1 — Local calendar: create/edit/delete events, multiple calendars per user, month/week/day/agenda views, recurrence, reminders (WS notification), timezone-correct display.
• A2 — Invites over mail (iMIP): detect text/calendar parts on message.new; render an invite card in the thread (message.actions + inline summary); Accept/Tentative/Decline sends an iTIP REPLY through the outbox; organizer flow sends REQUEST/CANCEL to attendees; updates (SEQUENCE bumps) modify the stored event.
• A3 — External sources: CalendarProvider interface with CalDAV client (first implementation; covers iCloud, Fastmail, Nextcloud) and read-only ICS URL subscriptions (holidays, team feeds); sync loop as a plugin job.
• A4 (deferred, spec later): Google/Microsoft API providers reusing M2 OAuth machinery; secret ICS export feed GET /api/plugins/calendar/feed/{token}.ics; CalDAV server endpoint for native phone sync — architecturally anticipated, not built now.

A.2 Dependencies

icalendar ~5.0 (parse/serialize RFC 5545), python-dateutil (RRULE expansion), caldav ~1.3 (A3), tzdata. No other additions.

A.3 Data model (plugin migrations, prefix plg_calendar_)

CREATE TABLE plg_calendar_calendars (
  id uuid PRIMARY KEY DEFAULT gen_random_uuid(),
  user_id uuid NOT NULL,
  name text NOT NULL,
  color text NOT NULL DEFAULT '#3b82f6',
  kind text NOT NULL DEFAULT 'local',     -- local|caldav|ics_subscription
  source_config jsonb NOT NULL DEFAULT '{}',  -- caldav url/collection, ics url
  secret_ct bytea,                        -- caldav credentials, §10.2 vault scheme
  sync_token text,                        -- CalDAV sync-token / ICS etag
  read_only boolean NOT NULL DEFAULT false,
  UNIQUE (user_id, name)
);

CREATE TABLE plg_calendar_events (
  id uuid PRIMARY KEY DEFAULT gen_random_uuid(),
  user_id uuid NOT NULL,
  calendar_id uuid NOT NULL REFERENCES plg_calendar_calendars(id) ON DELETE CASCADE,
  ical_uid text NOT NULL,                 -- UID from iCalendar; join key for iMIP updates
  sequence int NOT NULL DEFAULT 0,
  summary text NOT NULL DEFAULT '',
  description text NOT NULL DEFAULT '',
  location text NOT NULL DEFAULT '',
  dtstart timestamptz NOT NULL,
  dtend timestamptz,
  all_day boolean NOT NULL DEFAULT false,
  tz text NOT NULL DEFAULT 'UTC',         -- original TZID; render in user tz, store UTC
  rrule text,                             -- raw RRULE string, NULL = single
  exdates timestamptz[] NOT NULL DEFAULT '{}',
  recurrence_id timestamptz,              -- non-NULL = detached exception instance
  organizer jsonb,                        -- {name,email}
  attendees jsonb NOT NULL DEFAULT '[]',  -- [{name,email,partstat,role}]
  status text NOT NULL DEFAULT 'confirmed',   -- confirmed|tentative|cancelled
  my_partstat text,                       -- needs-action|accepted|declined|tentative
  reminders int[] NOT NULL DEFAULT '{}',  -- minutes before start
  origin_message_id uuid,                 -- link back to the invite mail thread
  raw_ical text,                          -- round-trip fidelity for iMIP/CalDAV
  UNIQUE (calendar_id, ical_uid, recurrence_id)
);
CREATE INDEX ON plg_calendar_events (user_id, dtstart);

Recurrence rule: store the RRULE string; expand instances at query time with dateutil.rrule over the requested view range only (no materialization table). Exceptions: exdates remove instances; a row with recurrence_id overrides one instance. Range queries must account for events starting before the window but recurring into it (expand from dtstart, cap iterations at 1000 per event).

A.4 CalendarProvider interface (inside the plugin, mirrors §5.1)

# PLUGIN INTERFACE VERSION: 1
class CalendarProvider(Protocol):
    async def verify(self) -> None: ...
    async def list_collections(self) -> list[RemoteCalendar]: ...
    async def pull(self, collection: str, *, sync_token: str | None
                   ) -> tuple[list[RemoteEvent], list[str], str]: ...
        # returns (changed_events_as_ical, deleted_uids, new_sync_token)
    async def push(self, collection: str, ical: str, *, uid: str,
                   etag: str | None) -> str: ...        # returns new etag
    async def delete(self, collection: str, uid: str, etag: str | None) -> None: ...

Implementations in A3: CaldavProvider, IcsSubscriptionProvider (pull-only, push/delete raise ReadOnlyCalendar). Sync loop = plugin cron job (schedule="*/5 * * * *", singleton=True per user) doing pull → upsert by (calendar_id, ical_uid, recurrence_id) → local-dirty push with etag conflict handling (conflict ⇒ server wins, local copy saved as a duplicate marked status='tentative', user notified).

A.5 iMIP flow (A2)

• Inbound: hook on message.new; if a text/calendar part exists, parse METHOD: REQUEST → upsert event (match ical_uid; apply only if SEQUENCE ≥ stored) into the user's default calendar with my_partstat='needs-action', set origin_message_id; CANCEL → status='cancelled'; REPLY → update that attendee's partstat on the organizer's copy.
• RSVP: invite card buttons call POST /api/plugins/calendar/events/{id}/rsvp {partstat} → update row → build iTIP REPLY ics → send via SDK outbox helper to the organizer (subject Accepted: {summary} etc.) → publish plugin.calendar.event_updated.
• Organizer: creating an event with attendees sends METHOD:REQUEST to each; edits bump SEQUENCE and re-send; deletion sends CANCEL. All outgoing invite mail uses the account whose address matches the organizer identity (picker if ambiguous).

A.6 UI (page /calendar, nav icon between Mail and Settings)

Routes: /calendar (month default), /calendar/{month|week|day|agenda}/:date, /calendar/event/:id, /calendar/settings (calendars CRUD, CalDAV/ICS wizard reusing the account-wizard UX pattern: URL+credentials → test → collection picker). Interactions: click-drag create, drag-move/resize (writes optimistically, reverts on API error), event popover with edit/RSVP, mini-month navigator, calendar visibility toggles (client state), t = today, arrows = prev/next period. Invite linkage both ways: invite card in thread links to /calendar/event/:id; event view links back to the mail thread when origin_message_id is set. Live updates via WS plugin.calendar.* events. Timezone: render in user setting settings.tz (add to §9.1 settings; default from browser at registration).

A.7 Acceptance criteria

1. Plugin installs/uninstalls (--drop-data) with zero diffs in app/ and frontend/src/ outside plugins//slots/ loader code.
2. A weekly-recurring event with two EXDATEs and one moved instance renders correctly in month and week views across a DST boundary (Europe/Warsaw fixtures).
3. Round-trip with Google Calendar via iMIP mail only: receive a Google invite → Accept in our UI → organizer's Google Calendar shows attendee accepted; our organizer invite → Google attendee accepts → our event shows partstat=accepted.
4. CalDAV sync against a Nextcloud/Radicale test container: create/edit/delete propagate both directions in ≤ 5 min (one cron cycle); conflict case produces the A.4 tentative-duplicate, never silent loss.
5. ICS subscription to a public holiday feed renders read-only events; editing is blocked in UI and API (403 read_only_calendar).
6. Killing the worker mid-sync and restarting causes no duplicate events (upsert key discipline).

---

Appendix B — Contacts plugin (webmail_plugin_contacts)

The second heavy plugin: same rules as the calendar — §11 SDK only, zero core diffs, own page. Division of labor with core: core owns implicit recipient history (known_recipients, §4/§9.1) so autocomplete works even without this plugin; the Contacts plugin owns explicit, user-curated contact records and enriches suggestions via the recipients.suggest capability (§11.3).

B.1 Scope by phase

• B1 — Local contacts: address books, contact CRUD with structured fields, groups, photo, search, dedicated page /contacts; "Add sender to contacts" from the thread view; suggestion capability wired into the composer.
• B2 — Portability: vCard 3.0/4.0 import (file, multi-card) and export (single contact / book), duplicate detection + merge tool.
• B3 — External sources: ContactProvider interface with CardDAV client sync (iCloud, Fastmail, Nextcloud, Radicale); per-book sync job.
• B4 (deferred, spec later): Google People / Microsoft Graph providers (reusing M2 OAuth), CardDAV server endpoint for native phone sync, book sharing.

B.2 Dependencies

vobject ~0.9 (vCard parse/serialize). Photos stored via the SDK blob store (JPEG/PNG, re-encoded server-side, ≤ 512×512). Nothing else.

B.3 Data model (plugin migrations, prefix plg_contacts_)

CREATE TABLE plg_contacts_books (
  id uuid PRIMARY KEY DEFAULT gen_random_uuid(),
  user_id uuid NOT NULL,
  name text NOT NULL,
  kind text NOT NULL DEFAULT 'local',       -- local|carddav
  source_config jsonb NOT NULL DEFAULT '{}',-- carddav url/collection
  secret_ct bytea,                          -- carddav credentials, §10.2 vault scheme
  sync_token text,
  read_only boolean NOT NULL DEFAULT false,
  UNIQUE (user_id, name)
);

CREATE TABLE plg_contacts_contacts (
  id uuid PRIMARY KEY DEFAULT gen_random_uuid(),
  user_id uuid NOT NULL,
  book_id uuid NOT NULL REFERENCES plg_contacts_books(id) ON DELETE CASCADE,
  vcard_uid text NOT NULL,                  -- UID property; join key for CardDAV
  etag text,                                -- server etag (carddav books)
  name_display text NOT NULL DEFAULT '',    -- FN
  name_parts jsonb NOT NULL DEFAULT '{}',   -- {given,family,middle,prefix,suffix} (N)
  emails jsonb NOT NULL DEFAULT '[]',       -- [{value,type,pref}]
  phones jsonb NOT NULL DEFAULT '[]',
  addresses jsonb NOT NULL DEFAULT '[]',    -- structured ADR
  org text NOT NULL DEFAULT '',
  title text NOT NULL DEFAULT '',
  birthday date,
  urls jsonb NOT NULL DEFAULT '[]',
  notes text NOT NULL DEFAULT '',
  photo_sha256 text,
  raw_vcard text,                           -- round-trip fidelity: preserve unknown props
  updated_at timestamptz NOT NULL DEFAULT now(),
  UNIQUE (book_id, vcard_uid)
);
CREATE INDEX ON plg_contacts_contacts (user_id, name_display);
CREATE INDEX ON plg_contacts_contacts USING gin (emails jsonb_path_ops);

CREATE TABLE plg_contacts_groups (
  id uuid PRIMARY KEY DEFAULT gen_random_uuid(),
  user_id uuid NOT NULL,
  name text NOT NULL,
  UNIQUE (user_id, name)
);
CREATE TABLE plg_contacts_group_members (
  group_id uuid NOT NULL REFERENCES plg_contacts_groups(id) ON DELETE CASCADE,
  contact_id uuid NOT NULL REFERENCES plg_contacts_contacts(id) ON DELETE CASCADE,
  PRIMARY KEY (group_id, contact_id)
);

vCard mapping discipline: parse known properties into columns, keep the full source in raw_vcard, and on export/push regenerate from columns merged over raw_vcard so properties we don't model (e.g. X- extensions) survive round-trips.

B.4 ContactProvider interface (inside the plugin, mirrors A.4)

# PLUGIN INTERFACE VERSION: 1
class ContactProvider(Protocol):
    async def verify(self) -> None: ...
    async def list_books(self) -> list[RemoteBook]: ...
    async def pull(self, book: str, *, sync_token: str | None
                   ) -> tuple[list[RemoteVcard], list[str], str]: ...
        # (changed vcards, deleted uids, new sync_token)
    async def push(self, book: str, vcard: str, *, uid: str,
                   etag: str | None) -> str: ...
    async def delete(self, book: str, uid: str, etag: str | None) -> None: ...

Sync loop = plugin cron job (*/5 * * * *, singleton per user), upsert by (book_id, vcard_uid), etag conflict ⇒ server wins + local copy kept as a duplicate flagged for the merge tool (never silent loss) — same policy as A.4.

B.5 API & capability (/api/plugins/contacts/)

GET /contacts?book=&group=&q=&cursor= · POST /contacts · GET/PUT/DELETE /contacts/{id} · POST /contacts/from-message {message_id} (prefill from From header) · GET/POST/DELETE /books, /groups, group membership ops · POST /import (vCard file → report {created, updated, skipped}) · GET /export?book=|contact= (.vcf) · GET /duplicates (candidate pairs: same email, or same normalized name + fuzzy) · POST /merge {keep_id, remove_id} (field union, keep wins conflicts, group memberships merged) · GET /suggest?q= — the recipients.suggest capability endpoint: matches name/email prefix and substring, returns [{name, email, source:"contact", contact_id, photo_url?}]. Composer merge rule (client-side, defined here so behavior is deterministic): contact hits rank above known_recipients hits at equal prefix quality; dedupe by email, contact entry wins and carries the photo/display name.

B.6 UI

Page /contacts (nav entry after Calendar): three-pane layout — books/groups sidebar, alphabetically grouped virtualized list with search-as-you-type, detail pane with edit-in-place form; import/export and merge review under /contacts/tools; CardDAV book wizard reusing the account-wizard UX pattern (URL + credentials → test → book picker). Mail integration via message.actions slot: "Add to Contacts" (hidden when the sender already matches a contact; then it becomes "View contact" linking to /contacts/{id}). Group compose: selecting a group in the composer expands to its members' preferred emails.

B.7 Acceptance criteria

1. Zero core diffs; install/uninstall --drop-data round-trip clean.
2. Import a 500-card vCard file mixing v3.0/v4.0, quoted-printable encoding, and Polish diacritics in names (Grzegorz Brzęczyszczykiewicz) — all names, emails, and photos correct; re-export → re-import is lossless including X- properties (raw_vcard merge discipline).
3. Composer typing a 3-char prefix returns merged suggestions in < 100 ms p95 (contacts ranked above history, deduped by email, photos shown for contacts).
4. CardDAV sync against Radicale: create/edit/delete propagate both directions within one cron cycle; concurrent edit conflict produces a flagged duplicate, never silent loss; worker kill mid-sync ⇒ no duplicate contacts on restart.
5. "Add sender to contacts" from a thread creates a prefilled contact; the same sender's messages immediately render the contact's display name, and the action flips to "View contact".
6. Merge tool: merging two contacts unions emails/phones/groups, keeps the chosen primary's conflicting fields, and repoints nothing else in the system (calendar attendees and mail remain keyed by email, by design).

---

Appendix C — Anti-spam plugin (webmail_plugin_antispam)

Third heavy plugin; §11 SDK only, zero core diffs. Architectural context: for synced accounts the upstream provider has usually already filtered spam, so this plugin is a second-pass filter (weak-provider accounts, unified experience, per-user trainable Bayes). It becomes the first-pass filter only for M6 hosted mailboxes, where the same engine also runs at the SMTP edge (§C.7).

C.1 Scope by phase

• C1 — Classification & actions: score every newly synced inbox message; move high-scoring mail to Junk (through the existing move machinery, so the verdict propagates to the IMAP server); banner low-confidence suspects; per-user allow/deny lists; per-account enable toggle.
• C2 — Learning loop: explicit "Spam"/"Not spam" actions train per-user Bayes; passive learning from user moves in/out of Junk and (sampled, capped) from provider-classified Junk; stats page.
• C3 (M6 era): the same Rspamd configuration/expertise deployed at the mail edge (VPS), where full IP/SPF/DKIM/DMARC/greylisting checks are actually meaningful; hosted-mailbox delivery is filtered before it reaches the index.

C.2 Engine decision

Rspamd as a service container (mature, fast, HTTP API) with a small dedicated Redis container for its Bayes/fuzzy storage (isolated from the app Redis on purpose — different persistence and flush semantics). No Python spam engine is written from scratch; the plugin is glue + policy. Compose additions: rspamd (rspamd/rspamd:3.x) + redis-rspamd (redis:7, --save 300 1). Rspamd's network checks (RBL/SURBL/fuzzy lookups leak message-derived hashes/URLs to third parties) are off by default, one settings toggle to enable (privacy default matches the product's self-hosted ethos).

C.3 Data model (plugin migrations, prefix plg_antispam_)

CREATE TABLE plg_antispam_settings (
  user_id uuid PRIMARY KEY,
  enabled_accounts uuid[] NOT NULL DEFAULT '{}',   -- empty = all accounts
  move_threshold real NOT NULL DEFAULT 12.0,       -- score >= : move to Junk
  banner_threshold real NOT NULL DEFAULT 6.0,      -- score >= : warn banner
  network_checks boolean NOT NULL DEFAULT false,
  learn_from_provider_junk boolean NOT NULL DEFAULT true
);
CREATE TABLE plg_antispam_rules (
  id uuid PRIMARY KEY DEFAULT gen_random_uuid(),
  user_id uuid NOT NULL,
  kind text NOT NULL,          -- allow|deny
  pattern citext NOT NULL,     -- email or @domain
  UNIQUE (user_id, kind, pattern)
);
CREATE TABLE plg_antispam_verdicts (
  message_id uuid PRIMARY KEY,
  user_id uuid NOT NULL,
  score real NOT NULL,
  symbols jsonb NOT NULL DEFAULT '[]',   -- rspamd symbol list for explainability
  action text NOT NULL,        -- none|banner|moved|allow_listed|deny_listed
  learned text                 -- spam|ham|NULL
);

C.4 Classification flow

Hook on message.new; process only messages landing in special_use='inbox' mailboxes of enabled accounts (never Sent/Drafts/Junk/Trash; never re-classify on flag changes). Order: (1) deny list → move to Junk, action deny_listed; (2) allow list → action allow_listed, stop; (3) POST raw blob to Rspamd /checkv2 with Deliver-To-style per-user context header (per-user Bayes statistics via Rspamd user settings); (4) apply thresholds → move via the existing move job (verdict thus syncs to the server's Junk folder) or record banner; (5) write verdict row, publish plugin.antispam.classified. Rspamd unavailability = fail-open (deliver normally, log, retry queue for learning only — spam filtering must never delay or lose mail).

C.5 Learning

• Explicit: message.actions buttons "Spam" / "Not spam" → POST /api/plugins/antispam/learn {message_id, verdict} → Rspamd /learnspam|/learnham with the per-user context → move message accordingly → update verdicts.learned.
• Passive: hook on flags.changed/move events — user moves message out of Junk ⇒ learn ham; into Junk ⇒ learn spam (dedup via verdicts.learned).
• Provider bootstrap (per account, if learn_from_provider_junk): during/after backfill, sample up to 500 provider-Junk messages as spam corpus and 500 aged inbox messages as ham, rate-limited background job. Gives per-user Bayes a warm start.

C.6 API & UI

GET/PATCH /api/plugins/antispam/settings · GET/POST/DELETE /rules · POST /learn · GET /stats (verdict counts, top symbols, Bayes confidence). UI: settings.page (thresholds with plain-language labels, per-account toggles, lists editor, privacy toggle, stats); message.banner for the suspect zone ("Looks like spam — [Move to Junk] [Not spam] [Always trust sender]"); message.actions Spam/Not-spam. Junk folder view is core (SPECIAL-USE) and needs no changes.

C.7 M6 edge integration (assumption, specified fully with M6)

At the hosted-mail phase the VPS edge (Stalwart's built-in filter or Postfix+Rspamd) performs first-pass filtering with the checks that only make sense at SMTP time (connecting IP, SPF/DKIM/DMARC verification, greylisting, rate limits); accepted mail is tagged with its edge score in a header, and this plugin trusts that header for hosted accounts instead of re-scanning — one user-visible spam system, two enforcement points.

C.8 Acceptance criteria

1. Zero core diffs; install/uninstall --drop-data clean; with the plugin absent, mail flow is byte-identical to v1.4 core.
2. Rspamd container stopped ⇒ new mail still indexes and renders normally within normal latency (fail-open proven by test).
3. Verdict propagation: a moved-to-Junk verdict appears in the provider's Junk folder via IMAP within one sync cycle; "Not spam" moves it back and the same message is not re-classified.
4. Learning: after 20 spam + 20 ham teachings on a fixture corpus, a held-out obvious-spam fixture scores above move_threshold for that user and a ham fixture below banner_threshold (per-user isolation: a second user's scores are unaffected).
5. Deny-listed sender is moved without engine involvement; allow-listed sender bypasses even a high engine score.
6. Privacy: with network_checks=false (default), a network-capture test during classification shows zero outbound connections from Rspamd.

Appendix D — Hosted mailboxes (M6, Stalwart)

Every webmail user gets a real mailbox on a Stalwart server running beside the app. The address is the user's handle (localpart@APP_DOMAIN, already mail-grade per §4); mail apps sign in to it with app passwords only (§D.5) — the webmail password never opens it. That mailbox appears in webmail as an account the user cannot remove. Because Stalwart holds the mail, webmail stops keeping its own full copy of it (§D.6).

D.1 Hosting (decide first — this, not code, is the hard part)

Receiving needs inbound port 25, an MX record and a fixed address; many home ISPs block port 25. Sending from a home IP is commonly spam-foldered or refused, and needs reverse DNS (PTR), SPF, DKIM and DMARC.

Option	Inbound	Outbound	Notes
A. All on the NAS	MX → home IP, port 25 forwarded	direct	Only with a static IP, settable PTR, unblocked port 25.
B. NAS + relay (recommended)	Small VPS holds the MX, forwards to the NAS over WireGuard	Relay service (SMTP2GO / Postmark / SES) or the VPS	Mail stays on the NAS; reliable delivery; the VPS is the SMTP-time spam edge (§C.7).
C. Stalwart on a VPS	VPS	VPS	Mail stored off the NAS; webmail reaches it over the internet.

Chosen (2026-10-10): option A — everything on the QNAP; see docs/mail-server-network.md for the network, DNS and port plan and the connection checks.

D.2 Infrastructure

• stalwart service in both compose stacks (dev webmail, prod webmail-prod — never share the project name), image pinned to an exact version: stalwartlabs/stalwart:v0.16.19.
• Stores: data store = a separate stalwart database on the existing Postgres, owned by its own role stalwart (STALWART_DB_PASSWORD); blob store = filesystem (/var/lib/stalwart/blobs, bind mount under DATA_ROOT/stalwart in production). backup.sh and webmail.sh update (backup → migrate → roll → health-gate) then cover it; no RocksDB to snapshot separately.
• Configuration (built, D2): in 0.16 config.json holds only the data store (docker/stalwart/config.json, committed: its password is read from the environment, {"@type":"EnvironmentVariable","variableName":"STALWART_DB_PASSWORD"}); everything else lives in that database and is managed through the JMAP management API (x:Domain, x:Directory, x:Role, …). app/hosted/provision.py (python -m app.hosted.provision, idempotent) sets it all up: the role and database (the server restart-loops until they exist), the domain and host name, logging to the console, mail on disk (switching the blob store takes a restart — the script says so), submission on 587 beside smtp 25 / submissions 465 / imaps 993, plain folder names for new mailboxes (Trash, Junk, Sent, Drafts, Archive), the SQL directory, the worker's role and sign-in, and the native account of every existing user. The management API is signed in to as the recovery admin (STALWART_RECOVERY_ADMIN=admin:$STALWART_ADMIN_SECRET).
• Stalwart 0.16.19 quirks: three built-in SURBL spam-list rules fail its own validator (bit_and), which makes every ReloadSettings fail; they never load anyway, so provisioning removes them. A blocked account's IMAP sign-in is not answered — the connection is dropped after a minute.
• TLS: Stalwart's built-in ACME with DNS-01 (nginx owns 80/443), or the existing certificates mounted read-only. The webmail itself reaches IMAP/SMTP over the Docker network with TLS but no certificate check.
• Spam: SMTP-time filtering at the edge (Stalwart's filter or Rspamd) tags an edge-score header; the anti-spam plugin trusts it for hosted accounts (§C.7). The existing Rspamd keeps per-user learning.

D.3 Same username, no password copying

Stalwart uses a SQL directory reading the view mail_directory (migration 0025), which the stalwart role may SELECT and nothing else in the app's database:

CREATE VIEW mail_directory AS
  SELECT handle AS name,
         NULL::text AS secret,  -- users: app passwords only (§D.5)
         'individual' AS type, handle AS description,
         blocked_at IS NULL AS active
    FROM users
  UNION ALL
  SELECT name, secret, 'individual', description, true FROM mail_service_accounts;

Sign-in query: … WHERE name = $1 AND active; recipient query: … WHERE name = $1 (a blocked user still receives mail).

• Decided (2026-10-11): the webmail password never opens the mailbox — not even with 2FA off. A mail app's password is stored on a phone or laptop and sent on every connection; an app password limits a leak to the mailbox (not the webmail account) and can be revoked on its own. The view still lists every user (for delivery and for the worker's impersonation), with no secret.
• Built (D2): the SQL directory returns one secret per account (Stalwart reads the first row only), so app passwords cannot be extra rows of this view — see §D.5.
• Block: user_admin.set_blocked also takes away the mailbox account's authenticate permission on the mail server (app passwords and the worker included); unblocking restores it. The mail server goes first: if it cannot be told, nothing changes (502).
• User deletion first deletes the mailbox and its mail through the management API — if that fails nothing is deleted, since a mailbox left behind would be handed to whoever registers the address next.
• The worker's sign-in (built): Stalwart's master-user login user@domain%webmail-worker@domain. The worker account is a row of mail_service_accounts (argon2id of STALWART_MASTER_SECRET); on the mail server its only role is impersonate + authenticate, so it cannot open a session as itself. Webmail stores no user's password (secret_ct holds nothing for native accounts). Accounts never signed in to are brought into Stalwart's registry by that first impersonated sign-in.
• Reserved handles (postmaster, abuse, …) stay unregistrable; route them to the administrator. webmail-worker is taken by the service account.

D.4 The permanent account in webmail — built (D3)

• Migration 0025: accounts.kind text NOT NULL DEFAULT 'external' (external | native), at most one native per user. app/hosted/native.py: the native account is created at registration, by provisioning for users that already exist, and when the user removes an account for their own address (such an account — the same address at its old provider, e.g. home.pl during the move — is left alone: no native account is added while it exists).
• DELETE /api/accounts/{id} on a native account → 409 native_account; PATCH with server settings or a password → 400 native_account. Lists carry kind; the UI shows a "Your mailbox" badge and hides Remove and the server settings; signature, colour and display name stay editable.
• provider_factory: native accounts connect to STALWART_IMAP_HOST:993 / STALWART_SMTP_HOST:465 (from the settings, not the row) as the worker acting for the user, outside the per-host connection cap. The account syncs Junk (the mail server's own filter files there).
• Sync: the IMAP provider unchanged (IDLE, CONDSTORE, QRESYNC, BINARY, MOVE, UIDPLUS all advertised). Optional later: a JMAP provider with push. Found on the way: UIDVALIDITY values above 2^31 did not fit the integer columns — migration 0026 makes UIDVALIDITY, MODSEQ and UIDs bigint.
• Sending goes through Stalwart, which refuses a From address the user does not own; Sent copy, retries and the Scheduled folder behave as for any account.

D.5 External mail clients and 2FA

Mail apps (IMAP/SMTP) sign in with app passwords only, whether 2FA is on or not: the webmail password would bypass TOTP, and is better never typed into a mail app. The directory view returns no password hash for users (verified: the password fails, app passwords and the worker work). Stalwart's login rate limits / auto-ban on.

App passwords — built (v1.3.37, ahead of D5): table user_app_passwords (migration 0024: id, user_id, name, password_hash, created_at, last_used_at), app/identity/app_passwords.py, routes GET/POST /api/auth/app-passwords, DELETE /api/auth/app-passwords/{id}, Settings → Security → App passwords.

• Without hosted mailboxes: 16 random lowercase letters (~75 bits), for the DAV server only.
• With hosted mailboxes (built, D2): the mail server makes the password — Stalwart's own app-password credential, app_ + 36 base32 characters naming the credential and 144 random bits — and checks it for IMAP/SMTP itself; its argon2id hash is kept in user_app_passwords too (for DAV), with the server's id in server_id (migration 0025) to revoke it there. One password works for mail and DAV. Revoking goes to the mail server first (502 mail_server_unavailable if it does not answer).
• Shown once in groups of four, copied and entered without spaces. Creating one takes the account password; at most 25 per user; each is revoked on its own; a password reset revokes them all (as Google does). The webmail's own sign-in never accepts one. Settings → Security shows the mail-app settings (mail_host on /api/auth/me: IMAP 993 SSL, SMTP 465 SSL or 587 STARTTLS, user name = the address).
• Still for D5: last_used_at (Stalwart's app-password credential records no last use).

D.6 Storing mail once

Today the indexer keeps every message in full (raw .eml blob + sanitized body in body_parts + search index), so external accounts are already a second copy, and with Stalwart both copies would sit on the same disk. Ideas, best first:

1. Attachments are stored twice inside webmail today. indexer.py stores each attachment as its own blob and it is inside the raw blob (base64), ≈2.3× its size; attachments are most of mail's bytes. Fix: keep only the raw message, extract an attachment when downloaded, keep thumbnails cached. Cuts attachment storage by about 40% (≈1.37× instead of ≈2.37× the file size) for all accounts, independent of Stalwart. A background pass frees existing duplicates. Done (v1.3.34): app/indexer/attachments.py, migration 0023.
2. Native accounts: no raw copy in webmail. Stalwart is the master copy; webmail keeps metadata, sanitized body, snippet and search index (~5–10% of the size). Full message, attachments and "show original" are fetched on demand over the local network (IMAP BODY[part] or JMAP blob download), with a small cache for repeat opens and thumbnails. Built (D4): app/indexer/rawstore.py — a native message's raw source goes into {BLOB_ROOT}/{user}/rawcache/{sha256} instead of the blob store, bounded per user (RAW_CACHE_MB, default 256; least recently used out first, anything unused for 30 days; pruned every 50 writes and by the nightly clean-up). Readers find it with rawstore.locate (blob store, then cache); request handlers call rawstore.ensure(message) first, which fetches the whole message (UID FETCH BODY.PEEK[], ~0.3 s on the same host) into the cache when missing. Covered: attachments (download, zip, thumbnails, inline images, forward, reopening a draft), re-sanitizing the HTML, the plugins' header/raw access. The nightly clean-up no longer keeps a blob-store file only native messages refer to. Thumbnails and taken-out attachments keep their own caches, as for every account.
3. External accounts (optional setting): age out old raw copies. Keep raw blobs for N days or the newest X GB; refetch older ones over IMAP when needed. Readable bodies stay local; old attachments then need the external server reachable.
4. Filesystem: at least lz4 compression on the ZFS data share (QuTS hero). ZFS dedup would merge identical copies but costs a lot of RAM and only hits uncompressed byte-identical data — last resort.
5. Rejected — MTA-only Stalwart delivering straight into webmail with no IMAP mailbox: zero duplication, but phones/Thunderbird lose access unless webmail implements IMAP. Items 1 + 2 give nearly the same saving.

Recommendation: item 1 now, item 2 together with Stalwart, item 3 later as a setting.

D.7 Phases

• D0 — Decisions: hosting option (§D.1), domain, whether external IMAP clients are allowed (§D.5).
• D1 — Attachment de-duplication (§D.6 item 1): independent, can ship any time.
• D2 — Spike (dev): Stalwart container, Postgres stores, SQL-directory view, master-user login, IMAP sync of a test user. Done (2026-10-11) — dev compose service stalwart, app/hosted/.
• D3 — Native account: accounts.kind, creation at registration + for existing users, undeletable in API and UI, provider wiring, delete/block hooks; tests against a Stalwart container (tests/hosted/, skipped when it is not running). Done.
• D4 — On-demand bodies for native accounts (§D.6 item 2). Done.
• D5 — Security: app passwords only for mail apps (done), rate limits, last_used_at.
• D6 — Production: compose, DNS (MX/SPF/DKIM/DMARC/PTR), relay or VPS per §D.1, backup, docs/nas-install.md runbook, release.

D.8 Configuration additions (§14)

HOSTED_MAILBOXES (on/off; off = M6 inert, byte-identical to pre-M6 behaviour), STALWART_URL (management API, default http://stalwart:8080), STALWART_ADMIN_SECRET (its recovery admin), STALWART_MASTER_SECRET (the worker's sign-in), STALWART_DB_PASSWORD (its Postgres role), STALWART_DB_NAME (default stalwart), STALWART_PG_HOST (Postgres as the mail server sees it, default postgres), STALWART_IMAP_HOST / STALWART_IMAP_PORT (993), STALWART_SMTP_HOST / STALWART_SMTP_PORT (465), MAIL_HOSTNAME (public name, default mail.APP_DOMAIN), RAW_CACHE_MB (256). Dev values are in .env.dev; the dev Stalwart publishes 8180 (API), 1025 (SMTP in), 1465, 1587, 1993 on loopback.

D.9 Acceptance criteria

1. A newly registered user can receive mail from an outside address at their handle and sees it in webmail within one sync cycle; mail they send arrives at Gmail not in spam (SPF/DKIM/DMARC pass).
2. The handle + an app password signs in to IMAP/SMTP; the webmail password never does; revoking the app password (or a password reset) stops it at once; a blocked user cannot sign in to IMAP.
3. The native account cannot be removed via API (409) or UI; deleting the user removes the Stalwart mailbox.
4. Webmail stores no copy of the account's password (secret_ct unused for native accounts).
5. After D1, an attachment-heavy fixture mailbox uses ≤ 60% of the pre-D1 blob storage, and every attachment still downloads byte-identical.
6. After D4, a native message's raw blob is absent from BLOB_ROOT, yet opening, downloading attachments and "show original" all work; with Stalwart stopped, list and bodies still render from local data.
7. Backup → restore on a fresh host brings back both webmail and Stalwart mail intact.

D.10 Open questions (owner: product)

1. Hosting option A, B or C? Static IP and open port 25 available?
2. Which domain do the addresses use?
3. May external mail clients (IMAP/SMTP) use these mailboxes?
4. Ship D1 ahead of the rest?

Appendix E — CalDAV / CardDAV server (phones, Thunderbird, macOS)

Ultimo serves the calendars and address books it already holds to standard clients — iPhone/iPad and macOS Calendar & Contacts, Thunderbird, Android via DAVx5, GNOME/Evolution — so they sync two-way with the web app. No second store: clients read and write the calendar/contacts plugins' own tables. (Chosen over Stalwart's DAV, which would make Stalwart the store and the plugins its clients, §D.)

E.1 Shape

• Core app/dav/ — the protocol: routes, Basic sign-in, discovery, principal, XML (PROPFIND / PROPPATCH / REPORT multistatus), ETag preconditions, sync tokens (RFC 6578). No knowledge of events or contacts.
• Providers in the plugins — Plugin(dav=...): the calendar plugin serves calendar collections (iCalendar, RFC 4791), the contacts plugin address books (vCard, RFC 6352). Interface: list collections; list resources (name + ETag); get; put (If-Match / If-None-Match); delete; changes since a sync token; create / update / delete a collection; calendar time-range query. Each runs in the user's RLS session (sdk.session).
• URLs — /.well-known/caldav, /.well-known/carddav → 301 /dav/; principal /dav/principals/{user_id}/; homes /dav/calendars/{user_id}/, /dav/addressbooks/{user_id}/; a collection …/{collection_id}/; a resource …/{name}.ics|.vcf. nginx proxies /dav/ and the two well-known paths to the API.

E.2 Sign-in

HTTP Basic over HTTPS: the handle (full address or local part) and an app password (§D.5) — never the account password, so 2FA cannot be bypassed and each device can be revoked. A successful check is cached briefly (keyed by a hash of the password, invalidated when app passwords change) so a sync's many requests don't each cost an argon2id verify; failures are rate-limited per address and IP. last_used_at is set (at most every few minutes).

E.3 Calendars

• A resource is one iCalendar object per UID: the series plus its moved/cancelled occurrences (RECURRENCE-ID). Name: what the client PUT it as, else derived from the UID.
• Fidelity: what a client PUTs is kept verbatim and served back unchanged while the event's modelled fields are still what it said; once the web app (or iMIP, or an upstream sync) changes them, the object is regenerated from the fields — SUMMARY, DESCRIPTION, LOCATION, DTSTART/DTEND with TZID and VTIMEZONE, all-day dates, RRULE, EXDATE, overrides, STATUS, TRANSP, ORGANIZER, ATTENDEEs with PARTSTAT, VALARMs from reminders, COLOR, the video link.
• Writes go through the same columns the web app uses, so reminders, invitations and upstream CalDAV write-back (the dav_dirty trigger, tombstones on synced calendars) behave identically. Subscribed (ICS) and read-only calendars are served read-only.
• Change tracking by triggers on the plugin tables (any write path counts) into a per-calendar change log; a calendar's sync token / CTag is its latest change number; deletions are reported by name. Old log entries are pruned; a client holding a pruned token gets valid-sync-token and resyncs.
• REPORTs: calendar-query (time-range on VEVENT), calendar-multiget, sync-collection. MKCALENDAR and PROPPATCH (name, colour) create / restyle local calendars; DELETE of a collection deletes a local calendar.

E.4 Contacts

Address books = plg_contacts_books; resources = contacts as vCard (3.0 served; 3.0 and 4.0 accepted): FN, N, NICKNAME, EMAIL (type, pref), TEL, ADR, URL, ORG/department, TITLE, BDAY, NOTE, CATEGORIES ↔ groups, the VIP star as an X- property. Same fidelity rule (raw_vcard kept until the web app edits), same change tracking. REPORTs: addressbook-query (basic), addressbook-multiget, sync-collection.

E.5 Phases

• E1 core + discovery + sign-in + calendars read/write (tested with a CalDAV client library);
• E2 vCard read/write + address books;
• E3 deployment (nginx, docs: how to add the account on each client), then checks with real devices.

---

End of specification. Ambiguities: prefer the simplest behavior consistent with the invariants in §1, leave a TODO(spec) comment, and list open questions in docs/open-questions.md rather than guessing silently.