Spec: M6 hosted mailboxes (Stalwart) plan — Appendix D
The M6 placeholder becomes a plan: a Stalwart mailbox for every user at their handle with the webmail password (SQL directory over users.password_hash, master-user login, no password copying), added as a permanent account; hosting options (NAS / NAS + relay / VPS); app passwords for 2FA users; ways to stop storing mail twice; phases, acceptance criteria and the open decisions. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
This commit is contained in:
Claude Opus 5.5
@@ -559,8 +559,9 @@ Phases A1–A2: local calendars + full page views + iMIP invite handling; then A
|
||||
Phases B1–B2: contact books, full page, groups, add-from-mail, `recipients.suggest` capability, vCard import/export; then B3: CardDAV client sync. May be built in parallel with M4 — both depend only on the M3 SDK.
|
||||
✅ Accept: criteria in Appendix B §B.7.
|
||||
|
||||
**M6 (optional) — Hosted mailboxes**
|
||||
Inbound MTA (Stalwart) delivering to a `NativeProvider`; handle becomes receiving address. Specified separately when reached.
|
||||
**M6 (optional) — Hosted mailboxes (Appendix D)**
|
||||
A Stalwart mail server beside the app: every user gets a mailbox at their handle (`localpart@APP_DOMAIN`) with the same password, added to their webmail as a permanent (non-removable) account; webmail stops keeping its own full copy of that mail. Phases D0 (hosting decision) → D1 (attachment de-duplication, useful on its own) → D2 (Stalwart spike) → D3 (native account) → D4 (on-demand bodies) → D5 (security) → D6 (production).
|
||||
✅ Accept: criteria in Appendix D §D.9. **Status: planned — awaiting the D0 decisions (§D.10).**
|
||||
|
||||
**M7 — Anti-spam plugin (Appendix C)**
|
||||
May be built any time after M3 (depends only on the plugin SDK; parallelizable with M4/M5). Phases C1–C2 client-side; C3 activates at the M6 edge.
|
||||
@@ -829,5 +830,87 @@ At the hosted-mail phase the VPS edge (Stalwart's built-in filter or Postfix+Rsp
|
||||
5. Deny-listed sender is moved without engine involvement; allow-listed sender bypasses even a high engine score.
|
||||
6. Privacy: with `network_checks=false` (default), a network-capture test during classification shows zero outbound connections from Rspamd.
|
||||
|
||||
## Appendix D — Hosted mailboxes (M6, Stalwart)
|
||||
|
||||
Every webmail user gets a real mailbox on a **Stalwart** server running beside the app. The address is the user's handle (`localpart@APP_DOMAIN`, already mail-grade per §4), and the password is the webmail password. That mailbox appears in webmail as an account the user cannot remove. Because Stalwart holds the mail, webmail stops keeping its own full copy of it (§D.6).
|
||||
|
||||
### D.1 Hosting (decide first — this, not code, is the hard part)
|
||||
Receiving needs inbound port 25, an MX record and a fixed address; many home ISPs block port 25. Sending from a home IP is commonly spam-foldered or refused, and needs reverse DNS (PTR), SPF, DKIM and DMARC.
|
||||
|
||||
| Option | Inbound | Outbound | Notes |
|
||||
|---|---|---|---|
|
||||
| **A. All on the NAS** | MX → home IP, port 25 forwarded | direct | Only with a static IP, settable PTR, unblocked port 25. |
|
||||
| **B. NAS + relay (recommended)** | Small VPS holds the MX, forwards to the NAS over WireGuard | Relay service (SMTP2GO / Postmark / SES) or the VPS | Mail stays on the NAS; reliable delivery; the VPS is the SMTP-time spam edge (§C.7). |
|
||||
| **C. Stalwart on a VPS** | VPS | VPS | Mail stored off the NAS; webmail reaches it over the internet. |
|
||||
|
||||
### D.2 Infrastructure
|
||||
- `stalwart` service in both compose stacks (dev `webmail`, prod `webmail-prod` — never share the project name), image pinned to an exact version.
|
||||
- **Stores:** data store = a separate `stalwart` database on the existing Postgres; blob store = filesystem under `DATA_ROOT/stalwart` (bind mount, not a named volume). `backup.sh` and `webmail.sh update` (backup → migrate → roll → health-gate) then cover it; no RocksDB to snapshot separately.
|
||||
- **TLS:** Stalwart's built-in ACME with DNS-01 (nginx owns 80/443), or the existing certificates mounted read-only.
|
||||
- **Spam:** SMTP-time filtering at the edge (Stalwart's filter or Rspamd) tags an edge-score header; the anti-spam plugin trusts it for hosted accounts (§C.7). The existing Rspamd keeps per-user learning.
|
||||
|
||||
### D.3 Same username and password — no password copying
|
||||
Stalwart uses a **SQL directory** reading a read-only view over `users` (dedicated read-only role):
|
||||
|
||||
```sql
|
||||
CREATE VIEW mail_principals AS
|
||||
SELECT handle AS name, handle AS email, password_hash AS secret,
|
||||
quota_bytes AS quota, (blocked_at IS NOT NULL) AS disabled
|
||||
FROM users;
|
||||
```
|
||||
|
||||
- Stalwart verifies argon2id PHC hashes, which is what `identity/passwords.py` produces, so registration, password change, reset, block and quota apply **instantly** with no provisioning or sync step.
|
||||
- User deletion additionally purges the mailbox through Stalwart's management API.
|
||||
- Webmail never stores the user's password: the worker signs in as a **master user** (`handle%master`, secret in `.env` as `STALWART_MASTER_SECRET`), not via `accounts.secret_ct`. *Verify the master-user feature and syntax in the pinned version during D2.*
|
||||
- Reserved handles (`postmaster`, `abuse`, …) stay unregistrable; route them to the administrator.
|
||||
|
||||
### D.4 The permanent account in webmail
|
||||
- Migration: `accounts.kind text NOT NULL DEFAULT 'external'` (`external` | `native`). The native account is created in `register_user`; a one-off command creates it for users that already exist.
|
||||
- `DELETE /api/accounts/{id}` on a native account → `409 {error:{code:"native_account"}}`. UI hides Remove and the server-settings fields; signature, colour and display name stay editable.
|
||||
- `provider_factory`: native accounts connect to `stalwart:993` / `stalwart:465` on the Docker network with the master credentials.
|
||||
- **Sync:** first reuse the IMAP provider unchanged (IDLE + QRESYNC already work). Optional later: a JMAP provider with push (Stalwart's native protocol).
|
||||
- Sending goes through Stalwart; Sent copy, retries and the Scheduled folder behave as for any account.
|
||||
|
||||
### D.5 External mail clients and 2FA
|
||||
If phones/desktop clients may use the mailbox over IMAP/SMTP, a password alone would bypass TOTP. Rule: users with 2FA enabled must use **app passwords** for IMAP/SMTP — a `user_app_passwords` table (argon2id hashes) added to the directory view, plus a Settings screen to create/revoke them. Stalwart's login rate limits / auto-ban on. If external clients are not wanted, IMAP/SMTP submission is exposed only on the Docker network and this section shrinks to nothing.
|
||||
|
||||
### D.6 Storing mail once
|
||||
Today the indexer keeps every message in full (raw `.eml` blob + sanitized body in `body_parts` + search index), so external accounts are already a second copy, and with Stalwart both copies would sit on the same disk. Ideas, best first:
|
||||
|
||||
1. **Attachments are stored twice inside webmail today.** `indexer.py` stores each attachment as its own blob *and* it is inside the raw blob (base64), ≈2.3× its size; attachments are most of mail's bytes. Fix: keep only the raw message, extract an attachment when downloaded, keep thumbnails cached. Roughly halves storage for **all** accounts, independent of Stalwart. A cleanup script frees existing duplicates.
|
||||
2. **Native accounts: no raw copy in webmail.** Stalwart is the master copy; webmail keeps metadata, sanitized body, snippet and search index (~5–10% of the size). Full message, attachments and "show original" are fetched on demand over the local network (IMAP `BODY[part]` or JMAP blob download), with a small cache for repeat opens and thumbnails.
|
||||
3. **External accounts (optional setting): age out old raw copies.** Keep raw blobs for N days or the newest X GB; refetch older ones over IMAP when needed. Readable bodies stay local; old attachments then need the external server reachable.
|
||||
4. **Filesystem:** at least lz4 compression on the ZFS data share (QuTS hero). ZFS dedup would merge identical copies but costs a lot of RAM and only hits uncompressed byte-identical data — last resort.
|
||||
5. **Rejected — MTA-only Stalwart** delivering straight into webmail with no IMAP mailbox: zero duplication, but phones/Thunderbird lose access unless webmail implements IMAP. Items 1 + 2 give nearly the same saving.
|
||||
|
||||
**Recommendation:** item 1 now, item 2 together with Stalwart, item 3 later as a setting.
|
||||
|
||||
### D.7 Phases
|
||||
- **D0 — Decisions:** hosting option (§D.1), domain, whether external IMAP clients are allowed (§D.5).
|
||||
- **D1 — Attachment de-duplication** (§D.6 item 1): independent, can ship any time.
|
||||
- **D2 — Spike (dev):** Stalwart container, Postgres stores, SQL-directory view, master-user login, IMAP sync of a test user.
|
||||
- **D3 — Native account:** `accounts.kind`, creation at registration + for existing users, undeletable in API and UI, provider wiring, delete/block hooks; tests against a Stalwart container (may replace greenmail).
|
||||
- **D4 — On-demand bodies for native accounts** (§D.6 item 2).
|
||||
- **D5 — Security:** app passwords + 2FA rule, rate limits.
|
||||
- **D6 — Production:** compose, DNS (MX/SPF/DKIM/DMARC/PTR), relay or VPS per §D.1, backup, `docs/nas-install.md` runbook, release.
|
||||
|
||||
### D.8 Configuration additions (§14)
|
||||
`STALWART_URL` (management API), `STALWART_IMAP_HOST` / `STALWART_SMTP_HOST`, `STALWART_MASTER_SECRET`, `STALWART_ADMIN_SECRET`, `HOSTED_MAILBOXES` (on/off; off = M6 inert, byte-identical to pre-M6 behaviour).
|
||||
|
||||
### D.9 Acceptance criteria
|
||||
1. A newly registered user can receive mail from an outside address at their handle and sees it in webmail within one sync cycle; mail they send arrives at Gmail not in spam (SPF/DKIM/DMARC pass).
|
||||
2. The same handle + password signs in to IMAP (when §D.5 allows it); after a webmail password change or reset, the old password fails and the new one works with no other step; a blocked user cannot sign in to IMAP.
|
||||
3. The native account cannot be removed via API (409) or UI; deleting the user removes the Stalwart mailbox.
|
||||
4. Webmail stores no copy of the account's password (`secret_ct` unused for native accounts).
|
||||
5. After D1, an attachment-heavy fixture mailbox uses ≤ 55% of the pre-D1 blob storage, and every attachment still downloads byte-identical.
|
||||
6. After D4, a native message's raw blob is absent from `BLOB_ROOT`, yet opening, downloading attachments and "show original" all work; with Stalwart stopped, list and bodies still render from local data.
|
||||
7. Backup → restore on a fresh host brings back both webmail and Stalwart mail intact.
|
||||
|
||||
### D.10 Open questions (owner: product)
|
||||
1. Hosting option A, B or C? Static IP and open port 25 available?
|
||||
2. Which domain do the addresses use?
|
||||
3. May external mail clients (IMAP/SMTP) use these mailboxes?
|
||||
4. Ship D1 ahead of the rest?
|
||||
|
||||
---
|
||||
*End of specification. Ambiguities: prefer the simplest behavior consistent with the invariants in §1, leave a `TODO(spec)` comment, and list open questions in `docs/open-questions.md` rather than guessing silently.*
|
||||
|
||||
Reference in New Issue
Block a user