lukas/webmail-client
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

M6 hosted mailboxes (off by default); dark message mode; fixes

Browse Source 
Hosted mailboxes (SPEC App. D), behind HOSTED_MAILBOXES (off: unchanged):
- Stalwart mail server, provisioned by `python -m app.hosted.provision`;
  directory read from a Postgres view, app passwords only for mail apps
- native account per user, the worker signs in as master for the user
- raw messages of hosted mailboxes only cached, fetched back on demand
- migrations 0025 (accounts.kind, directory view), 0026 (IMAP ids bigint)

Mail:
- dark message mode switch in the reading pane (Thunderbird's algorithm)
- moves and folder creation quote the folder name ("Deleted Items"):
  swipe-to-delete / Trash no longer bounce back
- thread unread counters: lock before recount, sorted lock order,
  UIDVALIDITY re-import recounts, nightly repair_threads
- sidebar accounts sorted alphabetically, more space above headings

Settings: Security headings renamed; antispam accounts are explicit
(none ticked = none filtered), input widths, Activity hidden.
Plugin bundles served with Cache-Control: no-cache.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
This commit is contained in:
lukas
2026-10-12 19:25:10 +02:00
co-authored by Claude Opus 5.5
parent d45fbf2c44
commit b88855931e
62 changed files with 2427 additions and 188 deletions
Download Patch File Download Diff File Expand all files Collapse all files
webmail-client-spec_v1.5.md
+42 -31
View File
@@ -832,7 +832,7 @@ At the hosted-mail phase the VPS edge (Stalwart's built-in filter or Postfix+Rsp
## Appendix D — Hosted mailboxes (M6, Stalwart)
Every webmail user gets a real mailbox on a **Stalwart** server running beside the app. The address is the user's handle (`localpart@APP_DOMAIN`, already mail-grade per §4), and the password is the webmail password. That mailbox appears in webmail as an account the user cannot remove. Because Stalwart holds the mail, webmail stops keeping its own full copy of it (§D.6).
Every webmail user gets a real mailbox on a **Stalwart** server running beside the app. The address is the user's handle (`localpart@APP_DOMAIN`, already mail-grade per §4); mail apps sign in to it with **app passwords only** (§D.5) — the webmail password never opens it. That mailbox appears in webmail as an account the user cannot remove. Because Stalwart holds the mail, webmail stops keeping its own full copy of it (§D.6).
### D.1 Hosting (decide first — this, not code, is the hard part)
Receiving needs inbound port 25, an MX record and a fixed address; many home ISPs block port 25. Sending from a home IP is commonly spam-foldered or refused, and needs reverse DNS (PTR), SPF, DKIM and DMARC.
@@ -846,46 +846,57 @@ Receiving needs inbound port 25, an MX record and a fixed address; many home ISP
**Chosen (2026-10-10): option A** — everything on the QNAP; see `docs/mail-server-network.md` for the network, DNS and port plan and the connection checks.
### D.2 Infrastructure
- `stalwart` service in both compose stacks (dev `webmail`, prod `webmail-prod` — never share the project name), image pinned to an exact version.
- **Stores:** data store = a separate `stalwart` database on the existing Postgres; blob store = filesystem under `DATA_ROOT/stalwart` (bind mount, not a named volume). `backup.sh` and `webmail.sh update` (backup → migrate → roll → health-gate) then cover it; no RocksDB to snapshot separately.
- **TLS:** Stalwart's built-in ACME with DNS-01 (nginx owns 80/443), or the existing certificates mounted read-only.
- `stalwart` service in both compose stacks (dev `webmail`, prod `webmail-prod` — never share the project name), image pinned to an exact version: **`stalwartlabs/stalwart:v0.16.19`**.
- **Stores:** data store = a separate `stalwart` database on the existing Postgres, owned by its own role `stalwart` (`STALWART_DB_PASSWORD`); blob store = filesystem (`/var/lib/stalwart/blobs`, bind mount under `DATA_ROOT/stalwart` in production). `backup.sh` and `webmail.sh update` (backup → migrate → roll → health-gate) then cover it; no RocksDB to snapshot separately.
- **Configuration (built, D2):** in 0.16 `config.json` holds only the data store (`docker/stalwart/config.json`, committed: its password is read from the environment, `{"@type":"EnvironmentVariable","variableName":"STALWART_DB_PASSWORD"}`); everything else lives in that database and is managed through the JMAP management API (`x:Domain`, `x:Directory`, `x:Role`, …). `app/hosted/provision.py` (`python -m app.hosted.provision`, idempotent) sets it all up: the role and database (the server restart-loops until they exist), the domain and host name, logging to the console, mail on disk (switching the blob store takes a **restart** — the script says so), submission on 587 beside smtp 25 / submissions 465 / imaps 993, plain folder names for new mailboxes (Trash, Junk, Sent, Drafts, Archive), the SQL directory, the worker's role and sign-in, and the native account of every existing user. The management API is signed in to as the recovery admin (`STALWART_RECOVERY_ADMIN=admin:$STALWART_ADMIN_SECRET`).
- **Stalwart 0.16.19 quirks:** three built-in SURBL spam-list rules fail its own validator (`bit_and`), which makes every *ReloadSettings* fail; they never load anyway, so provisioning removes them. A blocked account's IMAP sign-in is not answered — the connection is dropped after a minute.
- **TLS:** Stalwart's built-in ACME with DNS-01 (nginx owns 80/443), or the existing certificates mounted read-only. The webmail itself reaches IMAP/SMTP over the Docker network with TLS but no certificate check.
- **Spam:** SMTP-time filtering at the edge (Stalwart's filter or Rspamd) tags an edge-score header; the anti-spam plugin trusts it for hosted accounts (§C.7). The existing Rspamd keeps per-user learning.
### D.3 Same username and password — no password copying
Stalwart uses a **SQL directory** reading a read-only view over `users` (dedicated read-only role):
### D.3 Same username, no password copying
Stalwart uses a **SQL directory** reading the view `mail_directory` (migration 0025), which the `stalwart` role may SELECT and nothing else in the app's database:
```sql
CREATE VIEW mail_principals AS
SELECT handle AS name, handle AS email, password_hash AS secret,
quota_bytes AS quota, (blocked_at IS NOT NULL) AS disabled
FROM users;
CREATE VIEW mail_directory AS
SELECT handle AS name,
NULL::text AS secret, -- users: app passwords only (§D.5)
'individual' AS type, handle AS description,
blocked_at IS NULL AS active
FROM users
UNION ALL
SELECT name, secret, 'individual', description, true FROM mail_service_accounts;
```
- Stalwart verifies argon2id PHC hashes, which is what `identity/passwords.py` produces, so registration, password change, reset, block and quota apply **instantly** with no provisioning or sync step.
- User deletion additionally purges the mailbox through Stalwart's management API.
- Webmail never stores the user's password: the worker signs in as a **master user** (`handle%master`, secret in `.env` as `STALWART_MASTER_SECRET`), not via `accounts.secret_ct`. *Verify the master-user feature and syntax in the pinned version during D2.*
- Reserved handles (`postmaster`, `abuse`, …) stay unregistrable; route them to the administrator.
Sign-in query: `… WHERE name = $1 AND active`; recipient query: `… WHERE name = $1` (a blocked user still receives mail).
### D.4 The permanent account in webmail
- Migration: `accounts.kind text NOT NULL DEFAULT 'external'` (`external` | `native`). The native account is created in `register_user`; a one-off command creates it for users that already exist.
- `DELETE /api/accounts/{id}` on a native account → `409 {error:{code:"native_account"}}`. UI hides Remove and the server-settings fields; signature, colour and display name stay editable.
- `provider_factory`: native accounts connect to `stalwart:993` / `stalwart:465` on the Docker network with the master credentials.
- **Sync:** first reuse the IMAP provider unchanged (IDLE + QRESYNC already work). Optional later: a JMAP provider with push (Stalwart's native protocol).
- Sending goes through Stalwart; Sent copy, retries and the Scheduled folder behave as for any account.
- **Decided (2026-10-11): the webmail password never opens the mailbox** — not even with 2FA off. A mail app's password is stored on a phone or laptop and sent on every connection; an app password limits a leak to the mailbox (not the webmail account) and can be revoked on its own. The view still lists every user (for delivery and for the worker's impersonation), with no secret.
- **Built (D2):** the SQL directory returns **one secret per account** (Stalwart reads the first row only), so app passwords cannot be extra rows of this view — see §D.5.
- **Block:** `user_admin.set_blocked` also takes away the mailbox account's *authenticate* permission on the mail server (app passwords and the worker included); unblocking restores it. The mail server goes first: if it cannot be told, nothing changes (502).
- **User deletion** first deletes the mailbox and its mail through the management API — if that fails nothing is deleted, since a mailbox left behind would be handed to whoever registers the address next.
- **The worker's sign-in (built):** Stalwart's master-user login `user@domain%webmail-worker@domain`. The worker account is a row of `mail_service_accounts` (argon2id of `STALWART_MASTER_SECRET`); on the mail server its only role is *impersonate* + *authenticate*, so it cannot open a session as itself. Webmail stores no user's password (`secret_ct` holds nothing for native accounts). Accounts never signed in to are brought into Stalwart's registry by that first impersonated sign-in.
- Reserved handles (`postmaster`, `abuse`, …) stay unregistrable; route them to the administrator. `webmail-worker` is taken by the service account.
### D.4 The permanent account in webmail — built (D3)
- Migration 0025: `accounts.kind text NOT NULL DEFAULT 'external'` (`external` | `native`), at most one native per user. `app/hosted/native.py`: the native account is created at registration, by provisioning for users that already exist, and when the user removes an account for their own address (such an account — the same address at its old provider, e.g. home.pl during the move — is left alone: no native account is added while it exists).
- `DELETE /api/accounts/{id}` on a native account → `409 native_account`; `PATCH` with server settings or a password → `400 native_account`. Lists carry `kind`; the UI shows a "Your mailbox" badge and hides Remove and the server settings; signature, colour and display name stay editable.
- `provider_factory`: native accounts connect to `STALWART_IMAP_HOST:993` / `STALWART_SMTP_HOST:465` (from the settings, not the row) as the worker acting for the user, outside the per-host connection cap. The account syncs Junk (the mail server's own filter files there).
- **Sync:** the IMAP provider unchanged (IDLE, CONDSTORE, QRESYNC, BINARY, MOVE, UIDPLUS all advertised). Optional later: a JMAP provider with push. Found on the way: UIDVALIDITY values above 2^31 did not fit the `integer` columns — migration 0026 makes UIDVALIDITY, MODSEQ and UIDs `bigint`.
- Sending goes through Stalwart, which refuses a From address the user does not own; Sent copy, retries and the Scheduled folder behave as for any account.
### D.5 External mail clients and 2FA
If phones/desktop clients may use the mailbox over IMAP/SMTP, a password alone would bypass TOTP. Rule: users with 2FA enabled must use **app passwords** for IMAP/SMTP — a `user_app_passwords` table (argon2id hashes) added to the directory view, plus a Settings screen to create/revoke them. Stalwart's login rate limits / auto-ban on. If external clients are not wanted, IMAP/SMTP submission is exposed only on the Docker network and this section shrinks to nothing.
Mail apps (IMAP/SMTP) sign in with **app passwords only**, whether 2FA is on or not: the webmail password would bypass TOTP, and is better never typed into a mail app. The directory view returns no password hash for users (verified: the password fails, app passwords and the worker work). Stalwart's login rate limits / auto-ban on.
**App passwords — built (v1.3.37, ahead of D5):** table `user_app_passwords` (migration 0024: `id, user_id, name, password_hash, created_at, last_used_at`), `app/identity/app_passwords.py`, routes `GET/POST /api/auth/app-passwords`, `DELETE /api/auth/app-passwords/{id}`, Settings → Security → App passwords.
- 16 random lowercase letters (~75 bits), shown once in groups of four, copied and entered **without spaces**: the hash is of the bare letters, because the mail server compares exactly what an app sends.
- Creating one takes the account password; at most 25 per user; each is revoked on its own; a password reset revokes them all (as Google does). The webmail's own sign-in never accepts one.
- Still for D2/D5: add the hashes to the directory's secrets query (a user's secrets = `users.password_hash` unless 2FA is on, plus every `user_app_passwords.password_hash`), and set `last_used_at` from the mail server's successful sign-ins.
- Without hosted mailboxes: 16 random lowercase letters (~75 bits), for the DAV server only.
- **With hosted mailboxes (built, D2):** the mail server makes the password — Stalwart's own app-password credential, `app_` + 36 base32 characters naming the credential and 144 random bits — and checks it for IMAP/SMTP itself; its argon2id hash is kept in `user_app_passwords` too (for DAV), with the server's id in `server_id` (migration 0025) to revoke it there. One password works for mail and DAV. Revoking goes to the mail server first (502 `mail_server_unavailable` if it does not answer).
- Shown once in groups of four, copied and entered **without spaces**. Creating one takes the account password; at most 25 per user; each is revoked on its own; a password reset revokes them all (as Google does). The webmail's own sign-in never accepts one. Settings → Security shows the mail-app settings (`mail_host` on `/api/auth/me`: IMAP 993 SSL, SMTP 465 SSL or 587 STARTTLS, user name = the address).
- Still for D5: `last_used_at` (Stalwart's app-password credential records no last use).
### D.6 Storing mail once
Today the indexer keeps every message in full (raw `.eml` blob + sanitized body in `body_parts` + search index), so external accounts are already a second copy, and with Stalwart both copies would sit on the same disk. Ideas, best first:
1. **Attachments are stored twice inside webmail today.** `indexer.py` stores each attachment as its own blob *and* it is inside the raw blob (base64), ≈2.3× its size; attachments are most of mail's bytes. Fix: keep only the raw message, extract an attachment when downloaded, keep thumbnails cached. Cuts attachment storage by about 40% (≈1.37× instead of ≈2.37× the file size) for **all** accounts, independent of Stalwart. A background pass frees existing duplicates. **Done (v1.3.34):** `app/indexer/attachments.py`, migration 0023.
2. **Native accounts: no raw copy in webmail.** Stalwart is the master copy; webmail keeps metadata, sanitized body, snippet and search index (~5–10% of the size). Full message, attachments and "show original" are fetched on demand over the local network (IMAP `BODY[part]` or JMAP blob download), with a small cache for repeat opens and thumbnails.
2. **Native accounts: no raw copy in webmail.** Stalwart is the master copy; webmail keeps metadata, sanitized body, snippet and search index (~5–10% of the size). Full message, attachments and "show original" are fetched on demand over the local network (IMAP `BODY[part]` or JMAP blob download), with a small cache for repeat opens and thumbnails. **Built (D4):** `app/indexer/rawstore.py` — a native message's raw source goes into `{BLOB_ROOT}/{user}/rawcache/{sha256}` instead of the blob store, bounded per user (`RAW_CACHE_MB`, default 256; least recently used out first, anything unused for 30 days; pruned every 50 writes and by the nightly clean-up). Readers find it with `rawstore.locate` (blob store, then cache); request handlers call `rawstore.ensure(message)` first, which fetches the whole message (`UID FETCH BODY.PEEK[]`, ~0.3 s on the same host) into the cache when missing. Covered: attachments (download, zip, thumbnails, inline images, forward, reopening a draft), re-sanitizing the HTML, the plugins' header/raw access. The nightly clean-up no longer keeps a blob-store file only native messages refer to. Thumbnails and taken-out attachments keep their own caches, as for every account.
3. **External accounts (optional setting): age out old raw copies.** Keep raw blobs for N days or the newest X GB; refetch older ones over IMAP when needed. Readable bodies stay local; old attachments then need the external server reachable.
4. **Filesystem:** at least lz4 compression on the ZFS data share (QuTS hero). ZFS dedup would merge identical copies but costs a lot of RAM and only hits uncompressed byte-identical data — last resort.
5. **Rejected — MTA-only Stalwart** delivering straight into webmail with no IMAP mailbox: zero duplication, but phones/Thunderbird lose access unless webmail implements IMAP. Items 1 + 2 give nearly the same saving.
@@ -895,18 +906,18 @@ Today the indexer keeps every message in full (raw `.eml` blob + sanitized body
### D.7 Phases
- **D0 — Decisions:** hosting option (§D.1), domain, whether external IMAP clients are allowed (§D.5).
- **D1 — Attachment de-duplication** (§D.6 item 1): independent, can ship any time.
- **D2 — Spike (dev):** Stalwart container, Postgres stores, SQL-directory view, master-user login, IMAP sync of a test user.
- **D3 — Native account:** `accounts.kind`, creation at registration + for existing users, undeletable in API and UI, provider wiring, delete/block hooks; tests against a Stalwart container (may replace greenmail).
- **D4 — On-demand bodies for native accounts** (§D.6 item 2).
- **D5 — Security:** app passwords + 2FA rule, rate limits.
- **D2 — Spike (dev):** Stalwart container, Postgres stores, SQL-directory view, master-user login, IMAP sync of a test user. **Done (2026-10-11)** — dev compose service `stalwart`, `app/hosted/`.
- **D3 — Native account:** `accounts.kind`, creation at registration + for existing users, undeletable in API and UI, provider wiring, delete/block hooks; tests against a Stalwart container (`tests/hosted/`, skipped when it is not running). **Done.**
- **D4 — On-demand bodies for native accounts** (§D.6 item 2). **Done.**
- **D5 — Security:** app passwords only for mail apps (**done**), rate limits, `last_used_at`.
- **D6 — Production:** compose, DNS (MX/SPF/DKIM/DMARC/PTR), relay or VPS per §D.1, backup, `docs/nas-install.md` runbook, release.
### D.8 Configuration additions (§14)
`STALWART_URL` (management API), `STALWART_IMAP_HOST` / `STALWART_SMTP_HOST`, `STALWART_MASTER_SECRET`, `STALWART_ADMIN_SECRET`, `HOSTED_MAILBOXES` (on/off; off = M6 inert, byte-identical to pre-M6 behaviour).
`HOSTED_MAILBOXES` (on/off; off = M6 inert, byte-identical to pre-M6 behaviour), `STALWART_URL` (management API, default `http://stalwart:8080`), `STALWART_ADMIN_SECRET` (its recovery admin), `STALWART_MASTER_SECRET` (the worker's sign-in), `STALWART_DB_PASSWORD` (its Postgres role), `STALWART_DB_NAME` (default `stalwart`), `STALWART_PG_HOST` (Postgres as the mail server sees it, default `postgres`), `STALWART_IMAP_HOST` / `STALWART_IMAP_PORT` (993), `STALWART_SMTP_HOST` / `STALWART_SMTP_PORT` (465), `MAIL_HOSTNAME` (public name, default `mail.APP_DOMAIN`), `RAW_CACHE_MB` (256). Dev values are in `.env.dev`; the dev Stalwart publishes 8180 (API), 1025 (SMTP in), 1465, 1587, 1993 on loopback.
### D.9 Acceptance criteria
1. A newly registered user can receive mail from an outside address at their handle and sees it in webmail within one sync cycle; mail they send arrives at Gmail not in spam (SPF/DKIM/DMARC pass).
2. The same handle + password signs in to IMAP (when §D.5 allows it); after a webmail password change or reset, the old password fails and the new one works with no other step; a blocked user cannot sign in to IMAP.
2. The handle + an app password signs in to IMAP/SMTP; the webmail password never does; revoking the app password (or a password reset) stops it at once; a blocked user cannot sign in to IMAP.
3. The native account cannot be removed via API (409) or UI; deleting the user removes the Stalwart mailbox.
4. Webmail stores no copy of the account's password (`secret_ct` unused for native accounts).
5. After D1, an attachment-heavy fixture mailbox uses ≤ 60% of the pre-D1 blob storage, and every attachment still downloads byte-identical.