mail UX: restore-to-folder, draft/attachment fixes, printing, sidebar collapse
Backend: - Restore returns each message to its original folder (e.g. Sent/Archive) instead of always the Inbox: new messages.restore_mailbox_id (migration 0010) + restore_thread_job. - Drafts: expunge reconciliation for the Drafts poll (provider.list_uids / _reconcile_drafts) clears phantom "remote" drafts; send + discard drop the mirrored DB row; adopting a remote draft carries its real attachments into the compose flow. - Thread detail/list are folder-aware: individually trashed/spam messages are hidden outside their folder, Trash/Spam show only their own messages, and the mail-list message count matches the view. Frontend: - Thread reading panel: fix newest-first card overlap/spacing (expand-all and single expand), hide Reply/Forward on trashed messages, close the Actions menu on message-body (iframe) clicks, scroll to the inline composer, and prompt Save draft / Discard on the X close. - Composer: remount when switching drafts, dirty-tracking to stop duplicate IMAP mirrors, and close the From/More dropdowns on outside click. - Sidebar: collapse/expand toggle (icons-only ~60px rail) and a log-out icon on Sign out. - Mail-standard printing in a new window with reliable page margins (lib/print.ts). Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01U72FZiF1W1pb8XxyWBTREA
This commit is contained in:
Claude Opus 4.8
@@ -0,0 +1,811 @@
|
||||
# Webmail Client — Implementation Specification (v1.4, agent-ready)
|
||||
|
||||
This document is a complete, self-contained build specification. It is written to be handed to an AI coding agent. Read it fully before writing code. The companion document `webmail-client-concept.md` gives background; where they differ, **this spec wins**.
|
||||
|
||||
---
|
||||
|
||||
## 0. Instructions to the implementing agent
|
||||
|
||||
1. **Build milestone by milestone** (§12), in order. Do not start a milestone before the previous one's acceptance criteria pass.
|
||||
2. **Do not add features beyond this spec.** Anything not specified here is out of scope; leave `TODO(spec)` comments instead of inventing behavior.
|
||||
3. **Definition of done** for every task: code + type hints + unit tests + passing `ruff check`, `ruff format --check`, `mypy --strict` (backend) / `tsc --noEmit`, `eslint` (frontend) + updated docs in `docs/` if an interface changed.
|
||||
4. **Stability over novelty:** use exactly the pinned dependencies in §3. Do not swap libraries or add dependencies without a `# DEPENDENCY:` comment explaining why, and never for something the stdlib does adequately.
|
||||
5. **All interfaces in §5 are frozen contracts.** Implementations may evolve; signatures may not (without bumping the version comment at the top of the interface file).
|
||||
6. **Security rules in §10 are hard requirements**, not suggestions.
|
||||
7. Every module publishing or consuming events must use the event names in §8 verbatim.
|
||||
8. Commit in small, working increments; every commit must leave the test suite green.
|
||||
|
||||
## 1. Architecture invariants (never violate)
|
||||
|
||||
- **I-1 Sync-first:** the web UI and REST API never open IMAP connections. Only sync workers talk to mail servers. The API reads exclusively from Postgres/Meilisearch.
|
||||
- **I-2 Independent identity:** login uses app accounts (`localpart@our-web-client.example`) with Argon2id hashes. Mailbox credentials are payload data, encrypted at rest, decrypted only inside sync workers.
|
||||
- **I-3 One index, many views:** unified inbox, per-account views, virtual mailboxes, and search results are all parameterized queries over the same index. No duplicated storage per view.
|
||||
- **I-4 Threading is ours:** JWZ in our backend (§7). Never use the IMAP `THREAD` extension.
|
||||
- **I-5 Direction-agnostic thread UI:** thread ordering (oldest-first/newest-first) is a data-ordering concern. One layout, one composer-anchor slot (§9.4).
|
||||
- **I-6 Plugins extend, core stays closed:** new features beyond this spec must be expressible through the plugin surfaces (§11).
|
||||
- **I-7 Multi-user isolation:** every query (SQL and search) carries a mandatory `user_id` filter. No exceptions, including admin tooling.
|
||||
- **I-8 Mail HTML is hostile input:** sanitize with nh3 server-side at index time AND render in a sandboxed iframe. Both, always.
|
||||
|
||||
## 2. Repository layout (monorepo)
|
||||
|
||||
```
|
||||
webmail/
|
||||
├── backend/
|
||||
│ ├── pyproject.toml
|
||||
│ ├── app/
|
||||
│ │ ├── main.py # FastAPI app factory
|
||||
│ │ ├── config.py # pydantic-settings; all config via env
|
||||
│ │ ├── db.py # SQLAlchemy async engine/session
|
||||
│ │ ├── events.py # EventBus (Redis pub/sub + arq enqueue)
|
||||
│ │ ├── identity/ # §6.1 auth, sessions, handles, TOTP
|
||||
│ │ ├── accounts/ # §6.2 wizard, autodiscovery, credential vault
|
||||
│ │ ├── providers/
|
||||
│ │ │ ├── base.py # MailProvider interface (§5.1)
|
||||
│ │ │ └── imap_smtp/ # first implementation
|
||||
│ │ ├── sync/ # §6.3 backfill, IDLE loops, reconciliation
|
||||
│ │ ├── indexer/ # §6.4 MIME parsing → DB rows + blobs
|
||||
│ │ ├── threading_/ # §7 JWZ (trailing _ avoids stdlib clash)
|
||||
│ │ ├── search/
|
||||
│ │ │ ├── base.py # SearchBackend interface (§5.2)
|
||||
│ │ │ ├── queryparser.py # operator grammar (§8.2)
|
||||
│ │ │ ├── pg.py # Postgres FTS backend (M0–M1)
|
||||
│ │ │ └── meili.py # Meilisearch backend (M2)
|
||||
│ │ ├── outbox/ # §6.5 drafts, MIME assembly, send queue, undo
|
||||
│ │ ├── api/ # routers, one file per §9.1 group
|
||||
│ │ ├── ws.py # WebSocket hub (§9.2)
|
||||
│ │ └── plugins/ # §11 loader + plugin packages
|
||||
│ ├── workers/worker.py # arq worker entrypoint
|
||||
│ ├── migrations/ # alembic
|
||||
│ └── tests/
|
||||
├── frontend/
|
||||
│ ├── package.json
|
||||
│ └── src/
|
||||
│ ├── api/ # generated client from OpenAPI + WS layer
|
||||
│ ├── routes/ # §9.3
|
||||
│ ├── components/
|
||||
│ │ ├── thread/ThreadView.tsx # §9.4 direction-agnostic
|
||||
│ │ ├── thread/Composer.tsx
|
||||
│ │ ├── list/MessageList.tsx # virtualized
|
||||
│ │ └── slots/ # plugin UI slots (§11)
|
||||
│ └── lib/
|
||||
├── docker/
|
||||
│ ├── docker-compose.yml # app, worker, postgres:16, redis:7, meilisearch:v1.8, dovecot (test)
|
||||
│ └── docker-compose.ci.yml # + greenmail for integration tests
|
||||
└── docs/
|
||||
```
|
||||
|
||||
## 3. Pinned stack
|
||||
|
||||
**Backend (Python 3.11):** fastapi ~0.115, uvicorn ~0.30, sqlalchemy ~2.0 (async, asyncpg), alembic ~1.13, pydantic ~2.8, pydantic-settings, arq ~0.26, redis ~5.0, aioimaplib ~1.1, imapclient ~3.0 (utility/fallback), aiosmtplib ~3.0, argon2-cffi ~23.1, pyotp ~2.9, dnspython ~2.6, httpx ~0.27, nh3 ~0.2, cryptography ~43.0, mail-parser ~4.0 (helpers only; primary parsing via stdlib `email.message.EmailMessage` with `policy.default`).
|
||||
|
||||
**Frontend:** TypeScript ~5.5, React ~18.3, Vite ~5, TanStack Query ~5, TanStack Virtual ~3, Radix UI, Tailwind ~3.4, zod (WS payload validation). API client generated from OpenAPI (`openapi-typescript`).
|
||||
|
||||
**Infra:** PostgreSQL 16 (with `citext` extension), Redis 7, Meilisearch v1.8 (M2+), MinIO or plain filesystem for the blob store (config-switchable; filesystem default).
|
||||
|
||||
## 4. Database schema (Postgres 16, authoritative DDL)
|
||||
|
||||
All PKs are `uuid DEFAULT gen_random_uuid()`. All tables carrying user data include `user_id` with an index. Timestamps are `timestamptz`.
|
||||
|
||||
```sql
|
||||
CREATE EXTENSION IF NOT EXISTS citext;
|
||||
|
||||
CREATE TABLE users (
|
||||
id uuid PRIMARY KEY DEFAULT gen_random_uuid(),
|
||||
handle citext UNIQUE NOT NULL, -- full form: localpart@<APP_DOMAIN>
|
||||
password_hash text NOT NULL, -- argon2id
|
||||
totp_secret bytea, -- encrypted; NULL = 2FA off
|
||||
recovery_email citext,
|
||||
settings jsonb NOT NULL DEFAULT '{}', -- incl. {"thread_order":"asc"|"desc"}
|
||||
created_at timestamptz NOT NULL DEFAULT now()
|
||||
);
|
||||
-- Handle rules (enforce in code, mail-grade from day one):
|
||||
-- localpart: 3–30 chars, [a-z0-9._-], no leading/trailing/double dot,
|
||||
-- case-insensitive, reserved list (postmaster, abuse, admin, root, noreply, ...).
|
||||
|
||||
CREATE TABLE accounts (
|
||||
id uuid PRIMARY KEY DEFAULT gen_random_uuid(),
|
||||
user_id uuid NOT NULL REFERENCES users(id) ON DELETE CASCADE,
|
||||
email citext NOT NULL,
|
||||
display_name text NOT NULL DEFAULT '',
|
||||
color text NOT NULL DEFAULT '#888888',
|
||||
imap_config jsonb NOT NULL, -- {host,port,tls:"ssl"|"starttls",auth:"password"|"oauth2"}
|
||||
smtp_config jsonb NOT NULL, -- same shape
|
||||
secret_ct bytea NOT NULL, -- envelope-encrypted password or OAuth token bundle (§10.2)
|
||||
discovery_src text NOT NULL, -- preset|autoconfig|ispdb|srv|probe|manual
|
||||
state text NOT NULL DEFAULT 'backfilling', -- backfilling|live|error|disabled
|
||||
state_detail text NOT NULL DEFAULT '',
|
||||
last_sync_at timestamptz,
|
||||
created_at timestamptz NOT NULL DEFAULT now(),
|
||||
UNIQUE (user_id, email)
|
||||
);
|
||||
|
||||
CREATE TABLE mailboxes (
|
||||
id uuid PRIMARY KEY DEFAULT gen_random_uuid(),
|
||||
account_id uuid NOT NULL REFERENCES accounts(id) ON DELETE CASCADE,
|
||||
user_id uuid NOT NULL REFERENCES users(id) ON DELETE CASCADE,
|
||||
path text NOT NULL, -- IMAP mailbox name, UTF-7 decoded
|
||||
special_use text, -- inbox|sent|drafts|trash|junk|archive|NULL
|
||||
uidvalidity bigint NOT NULL DEFAULT 0,
|
||||
highestmodseq bigint, -- NULL if no CONDSTORE
|
||||
last_seen_uid bigint NOT NULL DEFAULT 0,
|
||||
UNIQUE (account_id, path)
|
||||
);
|
||||
|
||||
CREATE TABLE threads (
|
||||
id uuid PRIMARY KEY DEFAULT gen_random_uuid(),
|
||||
user_id uuid NOT NULL REFERENCES users(id) ON DELETE CASCADE,
|
||||
subject_norm text NOT NULL, -- normalized subject (§7.3)
|
||||
snippet text NOT NULL DEFAULT '',
|
||||
participants jsonb NOT NULL DEFAULT '[]', -- [{name,email}], deduped, ordered by recency
|
||||
account_ids uuid[] NOT NULL DEFAULT '{}',
|
||||
message_count int NOT NULL DEFAULT 0,
|
||||
unread_count int NOT NULL DEFAULT 0,
|
||||
has_attachments boolean NOT NULL DEFAULT false,
|
||||
last_message_at timestamptz NOT NULL
|
||||
);
|
||||
CREATE INDEX ON threads (user_id, last_message_at DESC);
|
||||
|
||||
CREATE TABLE messages (
|
||||
id uuid PRIMARY KEY DEFAULT gen_random_uuid(),
|
||||
user_id uuid NOT NULL REFERENCES users(id) ON DELETE CASCADE,
|
||||
account_id uuid NOT NULL REFERENCES accounts(id) ON DELETE CASCADE,
|
||||
mailbox_id uuid NOT NULL REFERENCES mailboxes(id) ON DELETE CASCADE,
|
||||
uid bigint NOT NULL, -- IMAP UID within mailbox
|
||||
thread_id uuid REFERENCES threads(id),
|
||||
message_id_hdr text, -- Message-ID, angle brackets stripped
|
||||
in_reply_to text,
|
||||
references_hdr text[] NOT NULL DEFAULT '{}',
|
||||
subject text NOT NULL DEFAULT '',
|
||||
from_addr jsonb NOT NULL DEFAULT '[]',
|
||||
to_addr jsonb NOT NULL DEFAULT '[]',
|
||||
cc_addr jsonb NOT NULL DEFAULT '[]',
|
||||
date_hdr timestamptz,
|
||||
internal_date timestamptz NOT NULL,
|
||||
flags text[] NOT NULL DEFAULT '{}', -- \Seen,\Flagged,\Answered,\Draft,\Deleted
|
||||
size_bytes int NOT NULL DEFAULT 0,
|
||||
has_attachments boolean NOT NULL DEFAULT false,
|
||||
snippet text NOT NULL DEFAULT '', -- first ~160 chars of text body
|
||||
blob_sha256 text NOT NULL, -- raw RFC822 in blob store (§6.4)
|
||||
body_indexed tsvector, -- Postgres FTS (M0–M1 SearchBackend)
|
||||
UNIQUE (mailbox_id, uid)
|
||||
);
|
||||
CREATE INDEX ON messages (user_id, thread_id);
|
||||
CREATE INDEX ON messages (account_id, message_id_hdr);
|
||||
CREATE INDEX ON messages USING gin (body_indexed);
|
||||
|
||||
CREATE TABLE body_parts (
|
||||
id uuid PRIMARY KEY DEFAULT gen_random_uuid(),
|
||||
message_id uuid NOT NULL REFERENCES messages(id) ON DELETE CASCADE,
|
||||
kind text NOT NULL, -- text|html_sanitized
|
||||
content text NOT NULL
|
||||
);
|
||||
|
||||
CREATE TABLE attachments (
|
||||
id uuid PRIMARY KEY DEFAULT gen_random_uuid(),
|
||||
message_id uuid NOT NULL REFERENCES messages(id) ON DELETE CASCADE,
|
||||
user_id uuid NOT NULL,
|
||||
filename text NOT NULL,
|
||||
mime_type text NOT NULL,
|
||||
size_bytes int NOT NULL,
|
||||
content_id text, -- for inline cid: images
|
||||
blob_sha256 text NOT NULL
|
||||
);
|
||||
|
||||
CREATE TABLE drafts (
|
||||
id uuid PRIMARY KEY DEFAULT gen_random_uuid(),
|
||||
user_id uuid NOT NULL REFERENCES users(id) ON DELETE CASCADE,
|
||||
account_id uuid NOT NULL REFERENCES accounts(id) ON DELETE CASCADE,
|
||||
thread_id uuid REFERENCES threads(id), -- NULL = new mail
|
||||
reply_to_message_id uuid REFERENCES messages(id),
|
||||
kind text NOT NULL DEFAULT 'reply', -- reply|reply_all|forward|new
|
||||
payload jsonb NOT NULL, -- {to,cc,bcc,subject,html,text,attachment_ids}
|
||||
updated_at timestamptz NOT NULL DEFAULT now()
|
||||
);
|
||||
|
||||
CREATE TABLE outbox (
|
||||
id uuid PRIMARY KEY DEFAULT gen_random_uuid(),
|
||||
user_id uuid NOT NULL,
|
||||
draft_id uuid NOT NULL,
|
||||
state text NOT NULL DEFAULT 'undo_window', -- undo_window|sending|sent|failed|cancelled
|
||||
send_after timestamptz NOT NULL, -- now() + undo window
|
||||
error text
|
||||
);
|
||||
|
||||
CREATE TABLE saved_searches (
|
||||
id uuid PRIMARY KEY DEFAULT gen_random_uuid(),
|
||||
user_id uuid NOT NULL REFERENCES users(id) ON DELETE CASCADE,
|
||||
name text NOT NULL,
|
||||
query text NOT NULL, -- raw operator-syntax string (§8.2)
|
||||
pinned boolean NOT NULL DEFAULT true,
|
||||
UNIQUE (user_id, name)
|
||||
);
|
||||
|
||||
-- Core recipient autocomplete (Gmail-style, derived from mail history).
|
||||
-- The Contacts plugin (Appendix B) ENRICHES suggestions; this table works without it.
|
||||
CREATE TABLE known_recipients (
|
||||
user_id uuid NOT NULL REFERENCES users(id) ON DELETE CASCADE,
|
||||
email citext NOT NULL,
|
||||
name text NOT NULL DEFAULT '', -- last seen display name
|
||||
times_used int NOT NULL DEFAULT 0, -- weighted: sent-to counts 3, received-from counts 1
|
||||
last_used_at timestamptz NOT NULL,
|
||||
PRIMARY KEY (user_id, email)
|
||||
);
|
||||
CREATE INDEX ON known_recipients (user_id, email text_pattern_ops);
|
||||
```
|
||||
|
||||
**Blob store:** raw RFC822 messages and attachment payloads are stored content-addressed at `${BLOB_ROOT}/{user_id}/{sha256[:2]}/{sha256}` (filesystem) or the equivalent object key (MinIO). Postgres holds metadata only. Deleting a user removes their blob subtree.
|
||||
|
||||
**Sessions** live in Redis (`sess:{token}` → JSON `{user_id, created, ua}`, TTL 30 days sliding), not Postgres.
|
||||
|
||||
## 5. Frozen core interfaces (Python)
|
||||
|
||||
### 5.1 `providers/base.py` — MailProvider
|
||||
|
||||
```python
|
||||
# INTERFACE VERSION: 1
|
||||
from dataclasses import dataclass
|
||||
from typing import AsyncIterator, Protocol
|
||||
from datetime import datetime
|
||||
|
||||
@dataclass(frozen=True)
|
||||
class RemoteMailbox:
|
||||
path: str
|
||||
special_use: str | None # "inbox"|"sent"|"drafts"|"trash"|"junk"|"archive"|None
|
||||
uidvalidity: int
|
||||
|
||||
@dataclass(frozen=True)
|
||||
class RemoteMessage:
|
||||
uid: int
|
||||
raw: bytes # full RFC822
|
||||
flags: frozenset[str]
|
||||
internal_date: datetime
|
||||
|
||||
@dataclass(frozen=True)
|
||||
class FlagChange:
|
||||
uid: int
|
||||
flags: frozenset[str]
|
||||
|
||||
@dataclass(frozen=True)
|
||||
class MailboxDelta:
|
||||
new: list[RemoteMessage]
|
||||
flag_changes: list[FlagChange]
|
||||
expunged_uids: list[int]
|
||||
new_uidvalidity: int | None # non-None => UIDVALIDITY changed, caller must full-resync
|
||||
new_highestmodseq: int | None
|
||||
|
||||
class MailProvider(Protocol):
|
||||
async def verify(self) -> None: ...
|
||||
# Raises ProviderAuthError / ProviderConnectError with a human-readable .detail
|
||||
|
||||
async def list_mailboxes(self) -> list[RemoteMailbox]: ...
|
||||
|
||||
def backfill(self, mailbox: str, *, newest_first: bool = True,
|
||||
batch_size: int = 200) -> AsyncIterator[list[RemoteMessage]]: ...
|
||||
# Yields batches until the mailbox is exhausted. Headers+body in one pass.
|
||||
|
||||
async def poll(self, mailbox: str, *, last_seen_uid: int,
|
||||
highestmodseq: int | None) -> MailboxDelta: ...
|
||||
# One reconciliation pass. Uses QRESYNC/CONDSTORE when highestmodseq is not None,
|
||||
# otherwise UID SEARCH + FETCH FLAGS diff.
|
||||
|
||||
async def idle(self, mailbox: str) -> None: ...
|
||||
# Blocks until the server signals change or 25 min elapse (re-issue IDLE). May raise
|
||||
# ProviderNotSupported => sync loop falls back to poll() every POLL_INTERVAL (default 120 s).
|
||||
|
||||
async def push_flags(self, mailbox: str, uid: int,
|
||||
add: frozenset[str], remove: frozenset[str]) -> None: ...
|
||||
async def move(self, uid: int, src: str, dst: str) -> int: ... # returns new UID
|
||||
async def append(self, mailbox: str, raw: bytes,
|
||||
flags: frozenset[str]) -> int: ... # Sent/Drafts upload
|
||||
async def send(self, raw: bytes, envelope_from: str,
|
||||
envelope_to: list[str]) -> None: ... # SMTP submission
|
||||
```
|
||||
|
||||
`ImapSmtpProvider` is the only M0–M2 implementation. A future `NativeProvider` (hosted mailboxes) or `JmapProvider` implements the same protocol — nothing above this interface may import IMAP libraries.
|
||||
|
||||
### 5.2 `search/base.py` — SearchBackend
|
||||
|
||||
```python
|
||||
# INTERFACE VERSION: 1
|
||||
@dataclass(frozen=True)
|
||||
class SearchQuery:
|
||||
text: str | None # free-text remainder after operator extraction
|
||||
filters: dict[str, object] # parsed operators, keys per §8.2
|
||||
user_id: UUID # ALWAYS set (invariant I-7)
|
||||
|
||||
@dataclass(frozen=True)
|
||||
class SearchHit:
|
||||
message_id: UUID
|
||||
thread_id: UUID
|
||||
score: float
|
||||
highlights: dict[str, str] # field -> HTML-escaped snippet with <mark>
|
||||
|
||||
class SearchBackend(Protocol):
|
||||
async def index(self, doc: SearchDoc) -> None: ...
|
||||
async def delete(self, message_id: UUID) -> None: ...
|
||||
async def delete_user(self, user_id: UUID) -> None: ...
|
||||
async def query(self, q: SearchQuery, *, limit: int, cursor: str | None
|
||||
) -> tuple[list[SearchHit], str | None]: ...
|
||||
```
|
||||
|
||||
`SearchDoc` fields: `message_id, thread_id, user_id, account_id, mailbox_special_use, from_ (name+email), to, cc, subject, body_text, attachment_names[], has_attachments, is_unread, is_starred, date (unix), size_bytes`. Both backends (pg.py, meili.py) index the same doc. Meilisearch: one index `messages`, filterable = every non-body field incl. `user_id`, searchable = `subject, body_text, from_, to, cc, attachment_names`, sortable = `date`; ranking rules default + `date:desc` tiebreak.
|
||||
|
||||
### 5.3 `events.py` — EventBus
|
||||
|
||||
```python
|
||||
# INTERFACE VERSION: 1
|
||||
class EventBus(Protocol):
|
||||
async def publish(self, event: str, payload: dict) -> None:
|
||||
# Fan-out to Redis pub/sub channel "evt:{event}" AND enqueue arq job
|
||||
# "on_{event}" for durable consumers (indexer, plugins).
|
||||
def subscribe(self, event: str, handler: Handler) -> None: ...
|
||||
```
|
||||
|
||||
Payloads always include `user_id`. Event names are listed in §8.3 and are the only ones allowed.
|
||||
|
||||
## 6. Backend module specifications
|
||||
|
||||
### 6.1 Identity (`app/identity/`)
|
||||
- Registration: validate localpart per §4 rules against reserved list in `identity/reserved.py`; store full handle `localpart@{APP_DOMAIN}` (env `APP_DOMAIN`).
|
||||
- Login: constant-time verify (argon2id, `time_cost=3, memory_cost=64MiB, parallelism=4`); if `totp_secret` set, require valid TOTP (±1 window). Rate limit: 10 attempts / 15 min per handle AND per IP (Redis counters); lockout returns 429 with `Retry-After`.
|
||||
- Session cookie: `__Host-session`, httpOnly, Secure, SameSite=Lax. CSRF: double-submit token header `X-CSRF` for all mutating requests.
|
||||
- TOTP setup: `POST /auth/totp/setup` returns provisioning URI + QR payload; enable requires one valid code; secret stored envelope-encrypted.
|
||||
- Password reset: only if `recovery_email` verified; single-use token, 30 min TTL, Redis-backed.
|
||||
|
||||
### 6.2 Accounts & autodiscovery (`app/accounts/`)
|
||||
`discover(email) -> list[CandidateConfig]` runs the chain, **returning on first success** but recording every step's outcome in the response (`steps: [{method, status, detail}]`) for wizard UX:
|
||||
|
||||
1. **Presets** — `accounts/presets.yaml` (config, not code): match by domain; ships with gmail.com, googlemail.com, outlook.com/hotmail/live, office365, yahoo, icloud/me.com, gmx, wp.pl, onet.pl, o2.pl, interia.pl. Each entry: imap/smtp host+port+tls, auth kind, oauth provider id if applicable.
|
||||
2. **Autoconfig** — GET `https://autoconfig.{domain}/mail/config-v1.1.xml?emailaddress={email}` then `https://{domain}/.well-known/autoconfig/mail/config-v1.1.xml`; parse Thunderbird XML (incomingServer type=imap, outgoingServer type=smtp; `%EMAILADDRESS%`/`%EMAILLOCALPART%` substitution).
|
||||
3. **ISPDB** — GET `https://autoconfig.thunderbird.net/v1.1/{domain}`; same XML format.
|
||||
4. **DNS SRV (RFC 6186)** — `_imaps._tcp.{domain}` (SSL 993), `_imap._tcp` (STARTTLS 143), `_submissions._tcp` (465), `_submission._tcp` (587); lowest priority wins.
|
||||
5. **Probe** — try `{imap,mail}.{domain}` × ports 993(ssl)/143(starttls) and `{smtp,mail}.{domain}` × 465(ssl)/587(starttls); TCP connect + TLS handshake + IMAP/SMTP greeting, 5 s timeout each, all concurrently.
|
||||
|
||||
All HTTP/DNS lookups: 5 s timeout, no redirects off-domain, HTTPS only. `test(config, credentials)`: full IMAP LOGIN + `LIST`, and SMTP EHLO+AUTH (no message sent); errors mapped to precise user-facing strings (`auth_failed`, `tls_failed`, `connect_timeout`, `unknown_host`). On save: encrypt credentials (§10.2), create account row (`state=backfilling`), enqueue `sync.bootstrap_account`.
|
||||
|
||||
### 6.3 Sync engine v2 (`app/sync/`) — sharded, lease-owned, tiered
|
||||
(Authoritative detail: `sync-engine-v2-refactor.md`; this section summarizes the binding rules.)
|
||||
|
||||
- **Two worker roles** (`WORKER_ROLE`): `jobs` = stateless arq consumer, N instances; `sync` = lease claimer + per-account runners, 1..N instances. Compose services `worker-jobs` / `worker-sync`, `nofile` ulimit 65536.
|
||||
- **Lease ownership:** exactly one sync worker owns each live account. Redis SET `sync:live` (API-maintained, DB-reconciled every 300 s); lease `sync:lease:{account_id}` claimed `SET NX EX LEASE_TTL`, renewed every `LEASE_RENEW` via owner-checked Lua; **fencing** — on renewal failure, cancel that account's runner before any further provider/DB op. Worker heartbeats in `sync:workers:{id}`; rebalance releases excess above `target*1.2+5`. Crash failover = lease expiry + reclaim (≤ ~90 s), no coordinator.
|
||||
- **Event-driven lifecycle:** `account.state_changed` adds/removes ids from `sync:live` at runtime — accounts go live ≤ 30 s after creation and stop ≤ 30 s after deletion with no restarts; runners re-verify `state='live'` each cycle (zombie-proof). New core event **`user.active`** (API publishes on login/WS connect) drives tier promotion.
|
||||
- **Bootstrap (unchanged from v1, now a `bulk` job):** `list_mailboxes()` → upsert `mailboxes` (SPECIAL-USE mapping with name heuristics — Sent/Wysłane, Trash/Kosz, etc. in `sync/folder_names.py`); backfill newest-first per mailbox (INBOX first, then sent, then rest); `sync.progress` published per batch; on completion set `state='live'` → the API adds the id to `sync:live` and a sync worker claims it.
|
||||
- **AccountRunner per tier:** *hot* = persistent INBOX IDLE + transient CONDSTORE poll of special folders every `POLL_WARM`; *warm* = transient poll of INBOX+specials every `POLL_WARM`; *cold* = every `POLL_COLD`. Promotion to hot on `user.active` (≤ 5 s) or new mail within `TIER_HOT_ACTIVITY_H`; demotion by inactivity; `HOT_BUDGET_PER_WORKER` degrades least-active non-session accounts. Delta application, UIDVALIDITY wipe-and-rebackfill, expunge soft-handling, and write-then-enqueue outbound flags are unchanged from v1.
|
||||
- **Connection budgets:** ≤ 2 sync-side connections per account (4 total incl. jobs, shared semaphore); process-local per-destination-host semaphore cap `PER_HOST_CONN_CAP`.
|
||||
- **Queues:** `interactive` (send/flags/move/draft jobs, timeout 120 s, concurrency 20) and `bulk` (backfills, timeout 3600 s, concurrency `BULK_CONCURRENCY`) — bulk work can never delay a send.
|
||||
- **Memory bounds:** parse semaphore `PARSE_CONCURRENCY`; raw messages > `STREAM_PARSE_MB` stream to blob and parse from file — peak RAM ≈ `PARSE_CONCURRENCY × STREAM_PARSE_MB` per process regardless of mailbox size.
|
||||
- **Shutdown:** SIGTERM → stop claimer → cancel runners → release leases → drain in < 10 s.
|
||||
|
||||
### 6.4 Indexer (`app/indexer/`)
|
||||
Input: `RemoteMessage`. Steps: store raw blob (sha256, dedupe) → parse with `email` stdlib (`policy.default`) → extract: headers (decode RFC2047), `references_hdr` = parsed `References` + `In-Reply-To` message-ids in order; best text body and best HTML body (multipart/alternative preference: html for display, text for indexing; if only HTML, derive text via nh3 strip); sanitize HTML with nh3 (allowlist: structural tags, inline styles filtered, `cid:` images rewritten to `/api/messages/{id}/cid/{content_id}`, remote `src` rewritten to blocked-placeholder attributes for FR-23) → write `messages`, `body_parts` (kinds `text` and `html_sanitized`), `attachments` → hand to Threader (§7) → `bus.publish("message.new", ...)` (search indexing + thread denormalization are consumers). For non-junk received messages, upsert the `From` address into `known_recipients` (weight 1).
|
||||
|
||||
### 6.5 Outbox (`app/outbox/`)
|
||||
- Drafts: autosave endpoint upserts `drafts` (client debounces 2 s); a background job mirrors drafts to the account's Drafts folder via `append()` at most every 60 s per draft (replace previous UID).
|
||||
- Send: `POST /send {draft_id}` → build MIME (html + auto-generated text alternative via html-to-text; `In-Reply-To`/`References` from `reply_to_message_id`; attachments from blob store) → create `outbox` row with `send_after = now()+UNDO_SECONDS` (env, default 10) → schedule arq job at `send_after`. Undo endpoint flips state to `cancelled` if still in window. The job: SMTP `send()`, then `append()` to Sent (skip for Gmail preset — Gmail self-appends), mark `sent`, publish `message.sent`, and upsert every To/Cc/Bcc address into `known_recipients` (weight 3). Failure → `failed` + WS notification, draft preserved.
|
||||
|
||||
## 7. Threading specification (JWZ, `app/threading_/`)
|
||||
|
||||
### 7.1 Normalization
|
||||
- `message_id_hdr`, `in_reply_to`, each `References` entry: strip angle brackets and whitespace, lowercase the domain part only. Discard obviously invalid ids (no `@`).
|
||||
- Missing/duplicate `Message-ID`: synthesize `sha256(raw)[:32]@synthetic.local` and mark it non-authoritative (never used to join other messages to it).
|
||||
|
||||
### 7.2 Incremental algorithm (per new message)
|
||||
1. Candidate set = all normalized ids in `references_hdr` + `in_reply_to` + own `message_id_hdr`.
|
||||
2. `SELECT DISTINCT thread_id FROM messages WHERE user_id=$1 AND message_id_hdr = ANY($2) AND thread_id IS NOT NULL`.
|
||||
3. 0 threads → subject fallback (§7.3); still none → create new thread.
|
||||
4. 1 thread → attach.
|
||||
5. ≥2 threads → **merge**: keep the thread with the oldest `last_message_at` as canonical, repoint messages of the others, delete emptied thread rows, publish `thread.updated` for all involved.
|
||||
6. Recompute denormalized thread fields (count, unread, participants, snippet = newest message's snippet, has_attachments, account_ids, last_message_at) in one UPDATE.
|
||||
|
||||
Threads are cross-account by design: the candidate lookup is scoped by `user_id`, not `account_id` (invariant I-3). Sent-folder messages join threads exactly like received ones (FR-15).
|
||||
|
||||
### 7.3 Subject fallback
|
||||
Only when reference joining fails: normalize subject (strip repeated `Re:/Fw:/Fwd:/Odp:/PD:` prefixes case-insensitively, collapse whitespace); join to a thread with the same `subject_norm` that has activity within 7 days AND shares ≥1 participant email; never join two messages that both lack references AND have empty subjects.
|
||||
|
||||
### 7.4 Test vectors (must pass in `tests/threading/`)
|
||||
Linear chain; chain arriving out of order; reply missing `In-Reply-To` but carrying `References`; thread merge on late-arriving parent; broken client (subject-only) joins; distinct-topic same-subject ("hi") does NOT join; cross-account merge (message in account A references message in account B); duplicate Message-ID; synthetic-ID isolation.
|
||||
|
||||
## 8. Search specification
|
||||
|
||||
### 8.1 Pipeline
|
||||
Consumer of `message.new` / `message.changed` / `message.deleted` → builds `SearchDoc` → active `SearchBackend` (env `SEARCH_BACKEND=pg|meili`). Backfill indexes inline with sync (per batch), never as a separate pass the user waits on (FR-30).
|
||||
|
||||
### 8.2 Query grammar (`search/queryparser.py`, engine-independent)
|
||||
Tokenization: whitespace-split, respecting `"quoted phrases"`. A token matching `key:value` (or `-key:value`) with a known key becomes a filter; everything else joins free text. `-` prefix negates a term or filter. `OR` (uppercase) groups adjacent terms/filters into a disjunction; default combinator is AND.
|
||||
|
||||
| Operator | Filter semantics |
|
||||
|---|---|
|
||||
| `from:` `to:` `cc:` | substring match on name or email of that field |
|
||||
| `subject:` | match constrained to subject |
|
||||
| `account:` | account by `display_name` or email substring |
|
||||
| `in:` | `inbox\|sent\|drafts\|trash\|junk\|archive` (special_use) or folder path |
|
||||
| `is:` | `unread\|read\|starred\|answered` |
|
||||
| `has:attachment` | boolean |
|
||||
| `filename:` | attachment name substring |
|
||||
| `before:` `after:` | `YYYY-MM-DD` |
|
||||
| `newer_than:` `older_than:` | `Nd\|Nw\|Nm\|Ny` |
|
||||
| `larger:` `smaller:` | `NK\|NM` bytes |
|
||||
|
||||
Parser output = `SearchQuery` (§5.2). Unknown `key:` tokens are treated as free text (forward compatibility). Unit tests must cover every operator, negation, quoting, `OR`, and garbage input (parser never raises).
|
||||
|
||||
### 8.3 Event catalog (complete)
|
||||
`message.new`, `message.changed`, `message.deleted`, `message.sent`, `flags.changed`, `thread.updated`, `sync.progress`, `account.state_changed`, `draft.saved`, `outbox.state_changed`, `user.active` (published by the API on login and WS connect; consumed by sync workers for tier promotion — not forwarded to browsers). Payloads: `{user_id, ...ids, minimal-diff fields}` — documented in `docs/events.md` (agent: generate it from code).
|
||||
|
||||
Plugins may additionally publish events namespaced `plugin.{name}.*` (e.g. `plugin.calendar.event_updated`); core code never subscribes to plugin events, but the WS hub forwards them to the owning user like any other event.
|
||||
|
||||
## 9. API, realtime, frontend
|
||||
|
||||
### 9.1 REST API (prefix `/api`, session-cookie auth, JSON; errors: `{error:{code,message}}`)
|
||||
|
||||
**Auth:** `POST /auth/register` · `POST /auth/login` · `POST /auth/logout` · `GET /auth/me` · `GET /auth/sessions` · `DELETE /auth/sessions/{id}` · `POST /auth/totp/setup` · `POST /auth/totp/enable` · `POST /auth/password/reset-request` · `POST /auth/password/reset`
|
||||
|
||||
**Accounts:** `POST /accounts/discover {email}` → `{candidates:[{imap,smtp,source}], steps:[...]}` · `POST /accounts/test {config, credentials}` → `{ok}|{error:{code,detail}}` · `POST /accounts` · `GET /accounts` (incl. `state`, sync progress) · `PATCH /accounts/{id}` · `DELETE /accounts/{id}` (cascades rows, blobs, search docs)
|
||||
|
||||
**Mail:** `GET /threads?view=unified|account:{id}|mailbox:{id}|search&q=&cursor=&limit=50` → thread summaries (denormalized fields only; cursor = `last_message_at,id` keyset) · `GET /threads/{id}?sort=asc|desc` → ordered full messages (bodies of the N newest expanded per client hint `expand=`; older = envelope only) · `POST /threads/{id}/flags {add,remove}` (whole thread) · `POST /messages/{id}/flags` · `POST /messages/{id}/move {mailbox_id}` · `GET /messages/{id}/html` (sanitized, for iframe `srcdoc`) · `GET /messages/{id}/cid/{content_id}` · `GET /attachments/{id}` (Content-Disposition attachment; images/PDF also inline for preview) · `POST /messages/{id}/allow-remote-images` (per-sender allowlist in user settings)
|
||||
|
||||
**Compose:** `POST /drafts` · `PUT /drafts/{id}` · `DELETE /drafts/{id}` · `POST /send {draft_id}` → `{outbox_id, undo_until}` · `POST /send/{outbox_id}/undo` · `GET /recipients/suggest?q=` → top 10 from `known_recipients` (prefix match on email or name, ranked by `times_used` decayed by recency); the composer merges these with any plugin capability providers (§11.3)
|
||||
|
||||
**Search:** `GET /search?q=&cursor=` → `{threads:[...], hits_by_thread, highlights}` (grouped by thread, FR-28; `&group=message` for flat) · `GET/POST/DELETE /saved-searches`
|
||||
|
||||
**Settings:** `GET/PATCH /settings` (incl. `thread_order: "asc"|"desc"`, `tz` IANA timezone — default from browser at registration, undo window, per-sender image allowlist)
|
||||
|
||||
### 9.2 WebSocket `/api/ws`
|
||||
Auth via session cookie. Server → client frames: `{event, payload}` for every §8.3 event belonging to the user (that's the whole contract — the UI refetches via TanStack Query invalidation keyed on ids in the payload). Client → server: `{type:"ping"}` only. Zod-validate all frames.
|
||||
|
||||
### 9.3 Frontend routes
|
||||
`/login`, `/register`, `/mail` (unified inbox), `/mail/a/:accountId`, `/mail/a/:accountId/f/:mailboxId`, `/mail/t/:threadId` (thread pane; on desktop rendered beside the list), `/mail/search?q=`, `/settings/{profile|security|accounts|appearance}`; `/settings/accounts/new` = wizard.
|
||||
|
||||
**Account wizard flow:** step 1 email (+ password field revealed unless preset says oauth) → calls `discover` while typing pauses; step 2 shows resolved config with source badge ("found via ISPDB") and a Test button (auto-run); failure expands the manual form pre-filled with the best guess and the precise error; step 3 = created, live backfill progress bar (WS `sync.progress`) with "start reading now" (list fills as backfill proceeds).
|
||||
|
||||
### 9.4 ThreadView contract (invariant I-5)
|
||||
```
|
||||
ThreadView(props: { messages: Message[] /* ASC always */, order: 'asc'|'desc',
|
||||
composerState, ... })
|
||||
```
|
||||
- Rendering order = `order==='asc' ? messages : [...messages].reverse()`. No other branching on `order` is allowed in layout code.
|
||||
- **Composer anchor** = a slot component rendered adjacent to the newest message: after it when `asc` (bottom of thread), directly after it when `desc` (top of thread, above older messages). Same component, one placement rule: "composer is always attached to the newest message."
|
||||
- Initial scroll: to the newest message (bottom for asc, top for desc). Collapse: all read messages except the newest are collapsed; "Expand all" toggles. Unread messages are never collapsed.
|
||||
- Keyboard: `j`/`k` move in *visual* order; `o`/`Enter` expand; `r` reply, `a` reply-all, `f` forward (open composer at anchor); `e` archive; `/` focus search; `gi` go to unified inbox; `esc` close composer (draft persists). Toolbar has the per-thread order toggle; it updates local state only, and long-presses/menu offers "make default" (PATCH /settings).
|
||||
- Message HTML renders in `<iframe sandbox="" srcdoc={...}>` sized via ResizeObserver postMessage shim; remote images show a "load images / always from this sender" bar when the sanitizer flagged any.
|
||||
|
||||
## 10. Security requirements (hard)
|
||||
|
||||
1. Argon2id params per §6.1; no other password hashing anywhere.
|
||||
2. **Credential vault:** mailbox secrets encrypted with AES-256-GCM under per-account data keys, wrapped by a master key from env `MASTER_KEY` (32 B, base64) — `cryptography` library, key id embedded for future rotation. Decryption code path exists only in `workers/` and `accounts/test`. Secrets never appear in logs, API responses, or exceptions (custom repr).
|
||||
3. CSRF double-submit on all mutating routes; `__Host-` cookie prefix; CSP: `default-src 'self'; frame-src 'self'; img-src 'self' data:` (remote mail images are proxied? **No** — v1 loads allowed remote images inside the sandboxed iframe only, which has its own relaxed CSP via srcdoc; document this in code).
|
||||
4. nh3 sanitization at index time; re-sanitize on render if sanitizer version changed (store `sanitizer_v` on body_parts).
|
||||
5. Rate limits: login (§6.1), `POST /accounts/discover` 10/min/user, `POST /send` 30/h/user (env-tunable).
|
||||
6. All outbound autodiscovery HTTP: HTTPS only, 5 s timeout, response size cap 256 KB, no off-domain redirects (SSRF guard: resolve + reject private IP ranges).
|
||||
7. Every SQL query and search query includes `user_id` (enforced by repository-layer helpers; raw session use in API code is a review error).
|
||||
8. Uploads (attachments to drafts): 25 MB/file, 40 MB/message total, content-type sniffed server-side, stored to blob store, never executed/served with original content type unless in safe list (images/pdf/text) — others force `application/octet-stream` + attachment disposition.
|
||||
|
||||
## 11. Plugin API (freeze in M3 — including the heavy-plugin surfaces below)
|
||||
|
||||
A plugin = Python package `webmail_plugin_<name>` exposing:
|
||||
|
||||
```python
|
||||
plugin = Plugin(
|
||||
name, version,
|
||||
hooks = {event: handler}, # subscribe to §8.3 events
|
||||
router = APIRouter | None, # mounted at /api/plugins/{name}
|
||||
ui_manifest = UiManifest | None, # slots + pages, see below
|
||||
migrations = "migrations/" | None, # 11.1
|
||||
jobs = [JobSpec, ...] | None, # 11.2
|
||||
)
|
||||
```
|
||||
|
||||
Loader discovers plugins via entry points group `webmail.plugins`, forwards declared events, and serves `GET /api/plugins/ui-manifest` for the frontend loader. Plugins may not import from `app.*` internals other than `app.plugins.sdk` (which exposes: DB session factory scoped to plugin tables, EventBus publish under `plugin.{name}.*`, blob store, outbox draft/send helpers, settings accessor, HTTP client with the §10.6 SSRF guard built in).
|
||||
|
||||
Every plugin gets the namespaced KV store (`plugin_kv(plugin, user_id, key)`) for light state. Heavy plugins additionally get:
|
||||
|
||||
### 11.1 Plugin-owned tables
|
||||
A plugin may ship Alembic migrations in its own version branch (`alembic -n plugin_{name}`). All tables MUST be prefixed `plg_{name}_` and every user-data table MUST carry `user_id` (invariant I-7 applies to plugins). Core migrations never reference plugin tables and vice versa. Uninstalling a plugin offers `--drop-data` to run its downgrade chain.
|
||||
|
||||
### 11.2 Plugin background jobs
|
||||
`JobSpec(func, schedule: cron_str | None, singleton: bool)` registers tasks on the shared arq worker: cron-scheduled (e.g. a sync loop trigger) or enqueued on demand from hooks/routes. `singleton=True` guarantees at most one running instance per (plugin, user) via a Redis lock. Plugin jobs run in the same worker process pool with the same retry semantics as core jobs and are visible in worker logs under `plg.{name}.*`.
|
||||
|
||||
### 11.3 Frontend surfaces
|
||||
- **Slots** (lazy-loaded ESM injected into mail UI): `thread.toolbar`, `message.actions`, `message.banner` (a strip rendered above a message body — used for spam warnings, calendar invite cards, and similar per-message notices), `sidebar.section`, `settings.page`.
|
||||
- **Pages** (`app.page`): a plugin may declare full views: `{route: "/calendar", nav: {icon, label, order}, bundle}`. The shell adds the nav entry, and the route (including sub-routes under it) lazy-loads the bundle into the **entire content area** — no mail list/thread components are rendered. Page bundles receive the SDK context (API client, WS event stream filtered to `plugin.{name}.*` + requested core events, design-system components) and are sandboxed to their route: importing mail-component internals is a review error. Removing the plugin removes the nav entry and route cleanly.
|
||||
- **Capabilities**: the manifest may declare named capabilities mapping to plugin endpoints, e.g. `capabilities: {"recipients.suggest": "/api/plugins/contacts/suggest"}`. Core UI features query all providers of a capability they support and merge results client-side (defined capabilities in v1: `recipients.suggest` — used by the composer To/Cc/Bcc fields and by any other plugin needing person-picking, such as calendar attendees). This keeps core backend code plugin-unaware.
|
||||
|
||||
Acceptance test for the API: implement **filters** (rules on `message.new`: match → move/flag) and **labels** entirely as plugins using only slots + KV; then implement the **calendar** (Appendix A) using tables + jobs + a page — all with zero core diffs.
|
||||
|
||||
## 12. Milestones with acceptance criteria
|
||||
|
||||
**M0 — Architecture spike**
|
||||
Login/register (no 2FA), one hardcoded-config IMAP account, backfill+IDLE for INBOX, JWZ, read-only unified list + thread view with working asc/desc toggle and correct composer-anchor placeholder, Postgres FTS free-text search.
|
||||
✅ Accept: sync a real 10k-message Gmail account via app password; threads match Gmail's grouping on a 50-thread sample ≥ 90%; toggle order and the newest message + anchor swap ends correctly; search returns in < 500 ms.
|
||||
|
||||
**M1 — MVP**
|
||||
Full identity (2FA, sessions UI, reset), account wizard with full discovery chain, multi-account + unified inbox with account chips, flags/move/archive both directions, inline reply + reply-all/forward + SMTP send + Sent append, recipient autocomplete from `known_recipients`, drafts autosave, sanitized HTML + blocked remote images, operator search (full §8.2 grammar) on PG FTS, WS live updates.
|
||||
✅ Accept: add gmail.com, outlook.com, and one autoconfig-only domain through the wizard typing only address+password; reply from a thread lands correctly threaded in the recipient's client; all §8.2 operators pass integration tests; `docker-compose up` from clean checkout works.
|
||||
|
||||
**M2 — Hardening / v1.0**
|
||||
OAuth2 (Google + Microsoft, XOAUTH2 for IMAP/SMTP, server-side refresh), QRESYNC fast resync, drafts mirrored to server, undo-send, attachment upload/preview, Meilisearch backend + instant search-as-you-type UI + saved searches as virtual mailboxes, session revocation, backfill progress UX, folder-name i18n heuristics.
|
||||
✅ Accept: typo query (`invocie`) finds "invoice" mail < 200 ms on a 100k-message corpus; kill -9 the worker mid-backfill and restart resumes without duplicates (idempotency via `(mailbox_id, uid)` unique key); undo-send within window results in no SMTP traffic.
|
||||
|
||||
**M3 — Plugin platform**
|
||||
Loader, full SDK per §11 **including** plugin migrations (11.1), plugin jobs (11.2), and `app.page` (11.3); filters + labels shipped as plugins; plugin docs.
|
||||
✅ Accept: both plugins install/uninstall cleanly with zero core diffs; a third trivial plugin ("Unsubscribe header button" on `message.actions`) can be written from `docs/plugins.md` alone; a skeleton page-plugin (route + nav entry + one plugin table + one cron job) round-trips install → run → uninstall `--drop-data` leaving no residue.
|
||||
|
||||
**M4 — Calendar plugin (Appendix A)**
|
||||
Phases A1–A2: local calendars + full page views + iMIP invite handling; then A3: CalDAV client sync + ICS subscriptions.
|
||||
✅ Accept: criteria in Appendix A §A.7.
|
||||
|
||||
**M5 — Contacts plugin (Appendix B)**
|
||||
Phases B1–B2: contact books, full page, groups, add-from-mail, `recipients.suggest` capability, vCard import/export; then B3: CardDAV client sync. May be built in parallel with M4 — both depend only on the M3 SDK.
|
||||
✅ Accept: criteria in Appendix B §B.7.
|
||||
|
||||
**M6 (optional) — Hosted mailboxes**
|
||||
Inbound MTA (Stalwart) delivering to a `NativeProvider`; handle becomes receiving address. Specified separately when reached.
|
||||
|
||||
**M7 — Anti-spam plugin (Appendix C)**
|
||||
May be built any time after M3 (depends only on the plugin SDK; parallelizable with M4/M5). Phases C1–C2 client-side; C3 activates at the M6 edge.
|
||||
✅ Accept: criteria in Appendix C §C.8.
|
||||
|
||||
## 13. Testing & CI
|
||||
|
||||
- **Unit:** pytest + pytest-asyncio; threading vectors (§7.4); query parser (§8.2); autodiscovery resolver with `respx`-mocked HTTP + `dnspython` stub; MIME corpus in `tests/fixtures/mime/` (agent: include ≥ 25 nasty real-world-style fixtures — encoded headers incl. RFC2047 Polish diacritics, nested multiparts, missing Message-ID, malformed dates, HTML-only, inline cid images, 8-bit subjects, winmail.dat placeholder).
|
||||
- **Integration:** docker-compose.ci.yml with Greenmail (IMAP+SMTP) and Dovecot; full loop test: append message to Greenmail → appears in API < 10 s → reply via API → arrives in Greenmail SMTP sink with correct References.
|
||||
- **Frontend:** vitest + React Testing Library; ThreadView ordering/anchor tests for both directions; Playwright smoke: login → add Greenmail account via wizard → read → inline reply → search.
|
||||
- **CI gates:** ruff, mypy --strict, pytest, tsc, eslint, vitest, Playwright smoke; coverage floor 80% on `threading_`, `search/queryparser`, `accounts/autodiscovery`, `indexer`.
|
||||
|
||||
## 14. Configuration (env, complete list)
|
||||
|
||||
`APP_DOMAIN`, `DATABASE_URL`, `REDIS_URL`, `MASTER_KEY`, `BLOB_ROOT` or `S3_*`, `SEARCH_BACKEND=pg|meili`, `MEILI_URL`, `MEILI_KEY`, `UNDO_SECONDS=10`, `SESSION_TTL_DAYS=30`, `OAUTH_GOOGLE_CLIENT_ID/SECRET`, `OAUTH_MS_CLIENT_ID/SECRET`, `MAX_ATTACHMENT_MB=25`, `LOG_LEVEL`; sync engine v2 (§6.3): `WORKER_ROLE=jobs|sync`, `MAX_ACCOUNTS_PER_WORKER=400`, `LEASE_TTL=60`, `LEASE_RENEW=20`, `POLL_WARM=180`, `POLL_COLD=900`, `TIER_HOT_ACTIVITY_H=24`, `TIER_SESSION_GRACE_MIN=30`, `TIER_COLD_AFTER_D=3`, `HOT_BUDGET_PER_WORKER=250`, `PER_HOST_CONN_CAP=50`, `BULK_CONCURRENCY=2`, `PARSE_CONCURRENCY=4`, `STREAM_PARSE_MB=8`. All read once in `config.py` (pydantic-settings); no `os.environ` elsewhere.
|
||||
|
||||
---
|
||||
|
||||
## Appendix A — Calendar plugin (`webmail_plugin_calendar`)
|
||||
|
||||
The calendar is the reference "heavy plugin": it lives in the same app and repo, uses the §11 SDK only, and must build with **zero core diffs**. It ships as a page (`/calendar`) beside Mail — one product, one login.
|
||||
|
||||
### A.1 Scope by phase
|
||||
- **A1 — Local calendar:** create/edit/delete events, multiple calendars per user, month/week/day/agenda views, recurrence, reminders (WS notification), timezone-correct display.
|
||||
- **A2 — Invites over mail (iMIP):** detect `text/calendar` parts on `message.new`; render an invite card in the thread (`message.actions` + inline summary); Accept/Tentative/Decline sends an iTIP REPLY through the outbox; organizer flow sends REQUEST/CANCEL to attendees; updates (SEQUENCE bumps) modify the stored event.
|
||||
- **A3 — External sources:** `CalendarProvider` interface with **CalDAV client** (first implementation; covers iCloud, Fastmail, Nextcloud) and read-only **ICS URL subscriptions** (holidays, team feeds); sync loop as a plugin job.
|
||||
- **A4 (deferred, spec later):** Google/Microsoft API providers reusing M2 OAuth machinery; secret **ICS export feed** `GET /api/plugins/calendar/feed/{token}.ics`; CalDAV **server** endpoint for native phone sync — architecturally anticipated, not built now.
|
||||
|
||||
### A.2 Dependencies
|
||||
`icalendar ~5.0` (parse/serialize RFC 5545), `python-dateutil` (RRULE expansion), `caldav ~1.3` (A3), `tzdata`. No other additions.
|
||||
|
||||
### A.3 Data model (plugin migrations, prefix `plg_calendar_`)
|
||||
```sql
|
||||
CREATE TABLE plg_calendar_calendars (
|
||||
id uuid PRIMARY KEY DEFAULT gen_random_uuid(),
|
||||
user_id uuid NOT NULL,
|
||||
name text NOT NULL,
|
||||
color text NOT NULL DEFAULT '#3b82f6',
|
||||
kind text NOT NULL DEFAULT 'local', -- local|caldav|ics_subscription
|
||||
source_config jsonb NOT NULL DEFAULT '{}', -- caldav url/collection, ics url
|
||||
secret_ct bytea, -- caldav credentials, §10.2 vault scheme
|
||||
sync_token text, -- CalDAV sync-token / ICS etag
|
||||
read_only boolean NOT NULL DEFAULT false,
|
||||
UNIQUE (user_id, name)
|
||||
);
|
||||
|
||||
CREATE TABLE plg_calendar_events (
|
||||
id uuid PRIMARY KEY DEFAULT gen_random_uuid(),
|
||||
user_id uuid NOT NULL,
|
||||
calendar_id uuid NOT NULL REFERENCES plg_calendar_calendars(id) ON DELETE CASCADE,
|
||||
ical_uid text NOT NULL, -- UID from iCalendar; join key for iMIP updates
|
||||
sequence int NOT NULL DEFAULT 0,
|
||||
summary text NOT NULL DEFAULT '',
|
||||
description text NOT NULL DEFAULT '',
|
||||
location text NOT NULL DEFAULT '',
|
||||
dtstart timestamptz NOT NULL,
|
||||
dtend timestamptz,
|
||||
all_day boolean NOT NULL DEFAULT false,
|
||||
tz text NOT NULL DEFAULT 'UTC', -- original TZID; render in user tz, store UTC
|
||||
rrule text, -- raw RRULE string, NULL = single
|
||||
exdates timestamptz[] NOT NULL DEFAULT '{}',
|
||||
recurrence_id timestamptz, -- non-NULL = detached exception instance
|
||||
organizer jsonb, -- {name,email}
|
||||
attendees jsonb NOT NULL DEFAULT '[]', -- [{name,email,partstat,role}]
|
||||
status text NOT NULL DEFAULT 'confirmed', -- confirmed|tentative|cancelled
|
||||
my_partstat text, -- needs-action|accepted|declined|tentative
|
||||
reminders int[] NOT NULL DEFAULT '{}', -- minutes before start
|
||||
origin_message_id uuid, -- link back to the invite mail thread
|
||||
raw_ical text, -- round-trip fidelity for iMIP/CalDAV
|
||||
UNIQUE (calendar_id, ical_uid, recurrence_id)
|
||||
);
|
||||
CREATE INDEX ON plg_calendar_events (user_id, dtstart);
|
||||
```
|
||||
|
||||
**Recurrence rule:** store the RRULE string; expand instances at query time with `dateutil.rrule` over the requested view range only (no materialization table). Exceptions: `exdates` remove instances; a row with `recurrence_id` overrides one instance. Range queries must account for events starting before the window but recurring into it (expand from `dtstart`, cap iterations at 1000 per event).
|
||||
|
||||
### A.4 CalendarProvider interface (inside the plugin, mirrors §5.1)
|
||||
```python
|
||||
# PLUGIN INTERFACE VERSION: 1
|
||||
class CalendarProvider(Protocol):
|
||||
async def verify(self) -> None: ...
|
||||
async def list_collections(self) -> list[RemoteCalendar]: ...
|
||||
async def pull(self, collection: str, *, sync_token: str | None
|
||||
) -> tuple[list[RemoteEvent], list[str], str]: ...
|
||||
# returns (changed_events_as_ical, deleted_uids, new_sync_token)
|
||||
async def push(self, collection: str, ical: str, *, uid: str,
|
||||
etag: str | None) -> str: ... # returns new etag
|
||||
async def delete(self, collection: str, uid: str, etag: str | None) -> None: ...
|
||||
```
|
||||
Implementations in A3: `CaldavProvider`, `IcsSubscriptionProvider` (pull-only, `push`/`delete` raise `ReadOnlyCalendar`). Sync loop = plugin cron job (`schedule="*/5 * * * *"`, `singleton=True` per user) doing pull → upsert by `(calendar_id, ical_uid, recurrence_id)` → local-dirty push with etag conflict handling (conflict ⇒ server wins, local copy saved as a duplicate marked `status='tentative'`, user notified).
|
||||
|
||||
### A.5 iMIP flow (A2)
|
||||
- **Inbound:** hook on `message.new`; if a `text/calendar` part exists, parse METHOD:
|
||||
`REQUEST` → upsert event (match `ical_uid`; apply only if `SEQUENCE` ≥ stored) into the user's default calendar with `my_partstat='needs-action'`, set `origin_message_id`; `CANCEL` → `status='cancelled'`; `REPLY` → update that attendee's `partstat` on the organizer's copy.
|
||||
- **RSVP:** invite card buttons call `POST /api/plugins/calendar/events/{id}/rsvp {partstat}` → update row → build iTIP REPLY ics → send via SDK outbox helper to the organizer (subject `Accepted: {summary}` etc.) → publish `plugin.calendar.event_updated`.
|
||||
- **Organizer:** creating an event with attendees sends METHOD:REQUEST to each; edits bump `SEQUENCE` and re-send; deletion sends CANCEL. All outgoing invite mail uses the account whose address matches the organizer identity (picker if ambiguous).
|
||||
|
||||
### A.6 UI (page `/calendar`, nav icon between Mail and Settings)
|
||||
Routes: `/calendar` (month default), `/calendar/{month|week|day|agenda}/:date`, `/calendar/event/:id`, `/calendar/settings` (calendars CRUD, CalDAV/ICS wizard reusing the account-wizard UX pattern: URL+credentials → test → collection picker). Interactions: click-drag create, drag-move/resize (writes optimistically, reverts on API error), event popover with edit/RSVP, mini-month navigator, calendar visibility toggles (client state), `t` = today, arrows = prev/next period. Invite linkage both ways: invite card in thread links to `/calendar/event/:id`; event view links back to the mail thread when `origin_message_id` is set. Live updates via WS `plugin.calendar.*` events. Timezone: render in user setting `settings.tz` (add to §9.1 settings; default from browser at registration).
|
||||
|
||||
### A.7 Acceptance criteria
|
||||
1. Plugin installs/uninstalls (`--drop-data`) with zero diffs in `app/` and `frontend/src/` outside `plugins/`/`slots/` loader code.
|
||||
2. A weekly-recurring event with two EXDATEs and one moved instance renders correctly in month and week views across a DST boundary (Europe/Warsaw fixtures).
|
||||
3. Round-trip with Google Calendar **via iMIP mail only**: receive a Google invite → Accept in our UI → organizer's Google Calendar shows attendee accepted; our organizer invite → Google attendee accepts → our event shows `partstat=accepted`.
|
||||
4. CalDAV sync against a Nextcloud/Radicale test container: create/edit/delete propagate both directions in ≤ 5 min (one cron cycle); conflict case produces the A.4 tentative-duplicate, never silent loss.
|
||||
5. ICS subscription to a public holiday feed renders read-only events; editing is blocked in UI and API (403 `read_only_calendar`).
|
||||
6. Killing the worker mid-sync and restarting causes no duplicate events (upsert key discipline).
|
||||
|
||||
---
|
||||
|
||||
## Appendix B — Contacts plugin (`webmail_plugin_contacts`)
|
||||
|
||||
The second heavy plugin: same rules as the calendar — §11 SDK only, zero core diffs, own page. Division of labor with core: core owns *implicit* recipient history (`known_recipients`, §4/§9.1) so autocomplete works even without this plugin; the Contacts plugin owns *explicit*, user-curated contact records and enriches suggestions via the `recipients.suggest` capability (§11.3).
|
||||
|
||||
### B.1 Scope by phase
|
||||
- **B1 — Local contacts:** address books, contact CRUD with structured fields, groups, photo, search, dedicated page `/contacts`; "Add sender to contacts" from the thread view; suggestion capability wired into the composer.
|
||||
- **B2 — Portability:** vCard 3.0/4.0 import (file, multi-card) and export (single contact / book), duplicate detection + merge tool.
|
||||
- **B3 — External sources:** `ContactProvider` interface with **CardDAV client** sync (iCloud, Fastmail, Nextcloud, Radicale); per-book sync job.
|
||||
- **B4 (deferred, spec later):** Google People / Microsoft Graph providers (reusing M2 OAuth), CardDAV **server** endpoint for native phone sync, book sharing.
|
||||
|
||||
### B.2 Dependencies
|
||||
`vobject ~0.9` (vCard parse/serialize). Photos stored via the SDK blob store (JPEG/PNG, re-encoded server-side, ≤ 512×512). Nothing else.
|
||||
|
||||
### B.3 Data model (plugin migrations, prefix `plg_contacts_`)
|
||||
```sql
|
||||
CREATE TABLE plg_contacts_books (
|
||||
id uuid PRIMARY KEY DEFAULT gen_random_uuid(),
|
||||
user_id uuid NOT NULL,
|
||||
name text NOT NULL,
|
||||
kind text NOT NULL DEFAULT 'local', -- local|carddav
|
||||
source_config jsonb NOT NULL DEFAULT '{}',-- carddav url/collection
|
||||
secret_ct bytea, -- carddav credentials, §10.2 vault scheme
|
||||
sync_token text,
|
||||
read_only boolean NOT NULL DEFAULT false,
|
||||
UNIQUE (user_id, name)
|
||||
);
|
||||
|
||||
CREATE TABLE plg_contacts_contacts (
|
||||
id uuid PRIMARY KEY DEFAULT gen_random_uuid(),
|
||||
user_id uuid NOT NULL,
|
||||
book_id uuid NOT NULL REFERENCES plg_contacts_books(id) ON DELETE CASCADE,
|
||||
vcard_uid text NOT NULL, -- UID property; join key for CardDAV
|
||||
etag text, -- server etag (carddav books)
|
||||
name_display text NOT NULL DEFAULT '', -- FN
|
||||
name_parts jsonb NOT NULL DEFAULT '{}', -- {given,family,middle,prefix,suffix} (N)
|
||||
emails jsonb NOT NULL DEFAULT '[]', -- [{value,type,pref}]
|
||||
phones jsonb NOT NULL DEFAULT '[]',
|
||||
addresses jsonb NOT NULL DEFAULT '[]', -- structured ADR
|
||||
org text NOT NULL DEFAULT '',
|
||||
title text NOT NULL DEFAULT '',
|
||||
birthday date,
|
||||
urls jsonb NOT NULL DEFAULT '[]',
|
||||
notes text NOT NULL DEFAULT '',
|
||||
photo_sha256 text,
|
||||
raw_vcard text, -- round-trip fidelity: preserve unknown props
|
||||
updated_at timestamptz NOT NULL DEFAULT now(),
|
||||
UNIQUE (book_id, vcard_uid)
|
||||
);
|
||||
CREATE INDEX ON plg_contacts_contacts (user_id, name_display);
|
||||
CREATE INDEX ON plg_contacts_contacts USING gin (emails jsonb_path_ops);
|
||||
|
||||
CREATE TABLE plg_contacts_groups (
|
||||
id uuid PRIMARY KEY DEFAULT gen_random_uuid(),
|
||||
user_id uuid NOT NULL,
|
||||
name text NOT NULL,
|
||||
UNIQUE (user_id, name)
|
||||
);
|
||||
CREATE TABLE plg_contacts_group_members (
|
||||
group_id uuid NOT NULL REFERENCES plg_contacts_groups(id) ON DELETE CASCADE,
|
||||
contact_id uuid NOT NULL REFERENCES plg_contacts_contacts(id) ON DELETE CASCADE,
|
||||
PRIMARY KEY (group_id, contact_id)
|
||||
);
|
||||
```
|
||||
|
||||
vCard mapping discipline: parse known properties into columns, keep the full source in `raw_vcard`, and on export/push regenerate from columns **merged over** `raw_vcard` so properties we don't model (e.g. `X-` extensions) survive round-trips.
|
||||
|
||||
### B.4 ContactProvider interface (inside the plugin, mirrors A.4)
|
||||
```python
|
||||
# PLUGIN INTERFACE VERSION: 1
|
||||
class ContactProvider(Protocol):
|
||||
async def verify(self) -> None: ...
|
||||
async def list_books(self) -> list[RemoteBook]: ...
|
||||
async def pull(self, book: str, *, sync_token: str | None
|
||||
) -> tuple[list[RemoteVcard], list[str], str]: ...
|
||||
# (changed vcards, deleted uids, new sync_token)
|
||||
async def push(self, book: str, vcard: str, *, uid: str,
|
||||
etag: str | None) -> str: ...
|
||||
async def delete(self, book: str, uid: str, etag: str | None) -> None: ...
|
||||
```
|
||||
Sync loop = plugin cron job (`*/5 * * * *`, singleton per user), upsert by `(book_id, vcard_uid)`, etag conflict ⇒ server wins + local copy kept as a duplicate flagged for the merge tool (never silent loss) — same policy as A.4.
|
||||
|
||||
### B.5 API & capability (`/api/plugins/contacts/`)
|
||||
`GET /contacts?book=&group=&q=&cursor=` · `POST /contacts` · `GET/PUT/DELETE /contacts/{id}` · `POST /contacts/from-message {message_id}` (prefill from From header) · `GET/POST/DELETE /books`, `/groups`, group membership ops · `POST /import` (vCard file → report `{created, updated, skipped}`) · `GET /export?book=|contact=` (`.vcf`) · `GET /duplicates` (candidate pairs: same email, or same normalized name + fuzzy) · `POST /merge {keep_id, remove_id}` (field union, keep wins conflicts, group memberships merged) · **`GET /suggest?q=`** — the `recipients.suggest` capability endpoint: matches name/email prefix and substring, returns `[{name, email, source:"contact", contact_id, photo_url?}]`. Composer merge rule (client-side, defined here so behavior is deterministic): contact hits rank above `known_recipients` hits at equal prefix quality; dedupe by email, contact entry wins and carries the photo/display name.
|
||||
|
||||
### B.6 UI
|
||||
Page `/contacts` (nav entry after Calendar): three-pane layout — books/groups sidebar, alphabetically grouped virtualized list with search-as-you-type, detail pane with edit-in-place form; import/export and merge review under `/contacts/tools`; CardDAV book wizard reusing the account-wizard UX pattern (URL + credentials → test → book picker). Mail integration via `message.actions` slot: "Add to Contacts" (hidden when the sender already matches a contact; then it becomes "View contact" linking to `/contacts/{id}`). Group compose: selecting a group in the composer expands to its members' preferred emails.
|
||||
|
||||
### B.7 Acceptance criteria
|
||||
1. Zero core diffs; install/uninstall `--drop-data` round-trip clean.
|
||||
2. Import a 500-card vCard file mixing v3.0/v4.0, quoted-printable encoding, and Polish diacritics in names (`Grzegorz Brzęczyszczykiewicz`) — all names, emails, and photos correct; re-export → re-import is lossless including `X-` properties (raw_vcard merge discipline).
|
||||
3. Composer typing a 3-char prefix returns merged suggestions in < 100 ms p95 (contacts ranked above history, deduped by email, photos shown for contacts).
|
||||
4. CardDAV sync against Radicale: create/edit/delete propagate both directions within one cron cycle; concurrent edit conflict produces a flagged duplicate, never silent loss; worker kill mid-sync ⇒ no duplicate contacts on restart.
|
||||
5. "Add sender to contacts" from a thread creates a prefilled contact; the same sender's messages immediately render the contact's display name, and the action flips to "View contact".
|
||||
6. Merge tool: merging two contacts unions emails/phones/groups, keeps the chosen primary's conflicting fields, and repoints nothing else in the system (calendar attendees and mail remain keyed by email, by design).
|
||||
|
||||
---
|
||||
|
||||
## Appendix C — Anti-spam plugin (`webmail_plugin_antispam`)
|
||||
|
||||
Third heavy plugin; §11 SDK only, zero core diffs. Architectural context: for **synced** accounts the upstream provider has usually already filtered spam, so this plugin is a *second-pass* filter (weak-provider accounts, unified experience, per-user trainable Bayes). It becomes the *first-pass* filter only for M6 hosted mailboxes, where the same engine also runs at the SMTP edge (§C.7).
|
||||
|
||||
### C.1 Scope by phase
|
||||
- **C1 — Classification & actions:** score every newly synced inbox message; move high-scoring mail to Junk (through the existing move machinery, so the verdict propagates to the IMAP server); banner low-confidence suspects; per-user allow/deny lists; per-account enable toggle.
|
||||
- **C2 — Learning loop:** explicit "Spam"/"Not spam" actions train per-user Bayes; passive learning from user moves in/out of Junk and (sampled, capped) from provider-classified Junk; stats page.
|
||||
- **C3 (M6 era):** the same Rspamd configuration/expertise deployed at the mail edge (VPS), where full IP/SPF/DKIM/DMARC/greylisting checks are actually meaningful; hosted-mailbox delivery is filtered *before* it reaches the index.
|
||||
|
||||
### C.2 Engine decision
|
||||
**Rspamd** as a service container (mature, fast, HTTP API) with a small **dedicated Redis** container for its Bayes/fuzzy storage (isolated from the app Redis on purpose — different persistence and flush semantics). No Python spam engine is written from scratch; the plugin is glue + policy. Compose additions: `rspamd` (`rspamd/rspamd:3.x`) + `redis-rspamd` (`redis:7`, `--save 300 1`). Rspamd's network checks (RBL/SURBL/fuzzy lookups leak message-derived hashes/URLs to third parties) are **off by default**, one settings toggle to enable (privacy default matches the product's self-hosted ethos).
|
||||
|
||||
### C.3 Data model (plugin migrations, prefix `plg_antispam_`)
|
||||
```sql
|
||||
CREATE TABLE plg_antispam_settings (
|
||||
user_id uuid PRIMARY KEY,
|
||||
enabled_accounts uuid[] NOT NULL DEFAULT '{}', -- empty = all accounts
|
||||
move_threshold real NOT NULL DEFAULT 12.0, -- score >= : move to Junk
|
||||
banner_threshold real NOT NULL DEFAULT 6.0, -- score >= : warn banner
|
||||
network_checks boolean NOT NULL DEFAULT false,
|
||||
learn_from_provider_junk boolean NOT NULL DEFAULT true
|
||||
);
|
||||
CREATE TABLE plg_antispam_rules (
|
||||
id uuid PRIMARY KEY DEFAULT gen_random_uuid(),
|
||||
user_id uuid NOT NULL,
|
||||
kind text NOT NULL, -- allow|deny
|
||||
pattern citext NOT NULL, -- email or @domain
|
||||
UNIQUE (user_id, kind, pattern)
|
||||
);
|
||||
CREATE TABLE plg_antispam_verdicts (
|
||||
message_id uuid PRIMARY KEY,
|
||||
user_id uuid NOT NULL,
|
||||
score real NOT NULL,
|
||||
symbols jsonb NOT NULL DEFAULT '[]', -- rspamd symbol list for explainability
|
||||
action text NOT NULL, -- none|banner|moved|allow_listed|deny_listed
|
||||
learned text -- spam|ham|NULL
|
||||
);
|
||||
```
|
||||
|
||||
### C.4 Classification flow
|
||||
Hook on `message.new`; process only messages landing in `special_use='inbox'` mailboxes of enabled accounts (never Sent/Drafts/Junk/Trash; never re-classify on flag changes). Order: (1) deny list → move to Junk, action `deny_listed`; (2) allow list → action `allow_listed`, stop; (3) POST raw blob to Rspamd `/checkv2` with `Deliver-To`-style per-user context header (per-user Bayes statistics via Rspamd user settings); (4) apply thresholds → move via the existing move job (verdict thus syncs to the server's Junk folder) or record `banner`; (5) write verdict row, publish `plugin.antispam.classified`. Rspamd unavailability = fail-open (deliver normally, log, retry queue for learning only — spam filtering must never delay or lose mail).
|
||||
|
||||
### C.5 Learning
|
||||
- Explicit: `message.actions` buttons "Spam" / "Not spam" → `POST /api/plugins/antispam/learn {message_id, verdict}` → Rspamd `/learnspam|/learnham` with the per-user context → move message accordingly → update `verdicts.learned`.
|
||||
- Passive: hook on `flags.changed`/move events — user moves message out of Junk ⇒ learn ham; into Junk ⇒ learn spam (dedup via `verdicts.learned`).
|
||||
- Provider bootstrap (per account, if `learn_from_provider_junk`): during/after backfill, sample up to 500 provider-Junk messages as spam corpus and 500 aged inbox messages as ham, rate-limited background job. Gives per-user Bayes a warm start.
|
||||
|
||||
### C.6 API & UI
|
||||
`GET/PATCH /api/plugins/antispam/settings` · `GET/POST/DELETE /rules` · `POST /learn` · `GET /stats` (verdict counts, top symbols, Bayes confidence). UI: `settings.page` (thresholds with plain-language labels, per-account toggles, lists editor, privacy toggle, stats); `message.banner` for the suspect zone ("Looks like spam — [Move to Junk] [Not spam] [Always trust sender]"); `message.actions` Spam/Not-spam. Junk folder view is core (SPECIAL-USE) and needs no changes.
|
||||
|
||||
### C.7 M6 edge integration (assumption, specified fully with M6)
|
||||
At the hosted-mail phase the VPS edge (Stalwart's built-in filter or Postfix+Rspamd) performs first-pass filtering with the checks that only make sense at SMTP time (connecting IP, SPF/DKIM/DMARC verification, greylisting, rate limits); accepted mail is tagged with its edge score in a header, and this plugin trusts that header for hosted accounts instead of re-scanning — one user-visible spam system, two enforcement points.
|
||||
|
||||
### C.8 Acceptance criteria
|
||||
1. Zero core diffs; install/uninstall `--drop-data` clean; with the plugin absent, mail flow is byte-identical to v1.4 core.
|
||||
2. Rspamd container stopped ⇒ new mail still indexes and renders normally within normal latency (fail-open proven by test).
|
||||
3. Verdict propagation: a moved-to-Junk verdict appears in the provider's Junk folder via IMAP within one sync cycle; "Not spam" moves it back and the same message is not re-classified.
|
||||
4. Learning: after 20 spam + 20 ham teachings on a fixture corpus, a held-out obvious-spam fixture scores above `move_threshold` for that user and a ham fixture below `banner_threshold` (per-user isolation: a second user's scores are unaffected).
|
||||
5. Deny-listed sender is moved without engine involvement; allow-listed sender bypasses even a high engine score.
|
||||
6. Privacy: with `network_checks=false` (default), a network-capture test during classification shows zero outbound connections from Rspamd.
|
||||
|
||||
---
|
||||
*End of specification. Ambiguities: prefer the simplest behavior consistent with the invariants in §1, leave a `TODO(spec)` comment, and list open questions in `docs/open-questions.md` rather than guessing silently.*
|
||||
Reference in New Issue
Block a user