Hosted mailboxes (SPEC App. D), behind HOSTED_MAILBOXES (off: unchanged):
- Stalwart mail server, provisioned by `python -m app.hosted.provision`;
directory read from a Postgres view, app passwords only for mail apps
- native account per user, the worker signs in as master for the user
- raw messages of hosted mailboxes only cached, fetched back on demand
- migrations 0025 (accounts.kind, directory view), 0026 (IMAP ids bigint)
Mail:
- dark message mode switch in the reading pane (Thunderbird's algorithm)
- moves and folder creation quote the folder name ("Deleted Items"):
swipe-to-delete / Trash no longer bounce back
- thread unread counters: lock before recount, sorted lock order,
UIDVALIDITY re-import recounts, nightly repair_threads
- sidebar accounts sorted alphabetically, more space above headings
Settings: Security headings renamed; antispam accounts are explicit
(none ticked = none filtered), input widths, Activity hidden.
Plugin bundles served with Cache-Control: no-cache.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
A reminder notification links to its day, event and occurrence
(/p/calendar?date=…&event=…&start=…). The calendar opens on that day with the
event's details showing — the full calendar on a computer; on a phone any
calendar link leads to Mail with the Calendar panel slid in (the big-screen
page never opens there), scrolled to the event, details open.
The panel's agenda kept no scroll position across re-renders: opening an
event's details lower down jumped the list back to the top. It stays put now.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
App passwords (SPEC App. D §D.5): one generated password per app, shown once,
revocable on its own, revoked on a password reset; never accepted by the web
sign-in. Settings → Security → App passwords (migration 0024).
CalDAV / CardDAV server (SPEC Appendix E), built in: /dav/ with
/.well-known/caldav|carddav, HTTP Basic with app passwords only (cached checks,
throttled failures). The calendar and contacts plugins serve their own tables as
collections (Plugin(dav=...)):
- calendars as iCalendar: series with overrides, time zones, all-day, attendees
and RSVP, alarms <-> reminders; MKCALENDAR / PROPPATCH / DELETE of calendars;
subscriptions read-only; writes to upstream-synced calendars flow on upstream;
- address books as vCard (3.0 served, 3.0/4.0 read), groups as CATEGORIES;
- what a client stores is served back verbatim until the web app edits it;
- sync tokens / CTags from change logs filled by triggers (calendar0008,
contacts0002), so every write path counts; trimmed nightly after 90 days.
REPORTs: calendar-query (time range), calendar-/addressbook-multiget,
addressbook-query, sync-collection. nginx and the Vite proxy route /dav/.
Settings shows the server and user name to enter; install guide §5.4.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
A job every minute finds the event reminders that fell due — repeating events
and moved occurrences expanded as the calendar view does — and pushes each
once to every device with notifications on (sdk.push, new: plugins can notify
through the new-mail push delivery). A reminder missed while the worker was down
is still sent within 10 minutes; cancelled events and declined invitations stay
quiet. Title, then "In 15 minutes, 14:00 · Room 2" in the user's zone and clock
format; it stays until dismissed and shows even with the app open; clicking it
opens that day (/p/calendar?date=…, new deep link).
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Deleting an event the user organizes, with other guests, asks "Would you like
to inform the participants that this event is canceled?" (Yes, inform them and
delete / No, just delete / Cancel; for a recurring one after which occurrences).
Yes sends each guest an iTIP CANCEL from the organizer's mailbox — of one
occurrence via RECURRENCE-ID — or, for "this and following", the shortened
series. Invitations now carry their RRULE, so recurring events repeat for
guests too.
An organizer's cancellation is applied as the email arrives (message.new hook)
or, for older mail, when the card is opened — always, whatever the settings:
only from that event's organizer and not older than the calendar's copy; one
occurrence becomes an EXDATE. The card says "Event was removed from calendar".
Settings → Calendars → Invitations now work, defaulting to "Do not
automatically update my calendar": "Only update events that I have already
added" applies the organizer's changes (keeping the user's answer), "Add new
events…" also adds new invitations, unanswered — from contacts or anyone — to
the chosen calendar, and "mark as read and file" archives such messages. With
no automatic updates, a changed invitation says so on its card.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Invitations received by email (iMIP): above the message body, a card with
when, what, where (or "Join video call"), the organizer, what else is in the
calendar then (a conflict, else the previous event that day), and Yes / No /
Maybe. Yes and Maybe put the event in the calendar (or update it), No takes
it out — and is still shown as the answer given; each sends the organizer
the iTIP REPLY, then a "Reply sent" toast, and an open calendar refreshes.
A cancellation offers "Remove from calendar". It is the calendar plugin's
message.banner slot (ui/invite.js, imip.py, /invitations endpoints); the SDK
gains the user's own accounts and settings. An Outlook-style invitation sent
only as text/calendar is now kept as invite.ics too.
Calendar: clicking an empty day in the side panel's agenda opens a new event
on that date; a video-call link can be removed (✕ beside Copy); the page
guests see after answering, and our invitations' footer, say Ultimo Calendar
(with the logo).
Attachments: videos (mp4, mov, webm, …) play in the full-screen preview,
paused until played; attachments are now served from disk with HTTP Range,
so a video starts at once and can be seeked.
People tab: everyone in the thread, the user included, A–Z, under the name
any message gives them. Initials circles share one colour rule in the list,
the reading pane and the People tab. The contact view drops its duplicate
"Message" button. A general corner toast (webmail:toast) joins Undo's.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Found on the NAS, where nothing had been learned in weeks while the UI said
"Marked as spam": the source tree predated v1.3.2, so api/worker-* had no
controller password and every lesson got 401 — and the trail of state that
left behind kept it broken after the password was fixed.
- /learn records a lesson only when rspamd accepted it. Recording refused ones
made the menu claim "Marked as spam" and its dedup blocked every retry.
- Bootstrap: done-markers move to a new key. The old one may have been written
by the "-0" bug (a total outage read as a clean run of zero), which left an
account marked done with nothing taught and never retried; every older
install re-runs the bootstrap once, harmlessly (rspamd skips what it knows).
- "Learn from my Spam folder now" in Settings > Antispam filtering queues the
bootstrap for the user's accounts at once, instead of the hourly sweep.
- webmail.sh update now recreates rspamd: it reads the controller password only
when created, so a changed .env reached the app but never the engine.
- doctor's advice for the 401 names both real causes (a pre-v1.3.2 compose
file, or an rspamd started with an old password) and the fix for each; the
old "update v1.3.2" advice could not help.
- docs/nas-install.md: how to refresh the source tree on a tarball install.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
`_teach` signalled failure by negating the count taught so far. That
works for every case but the one that matters: when the FIRST message is
refused nothing has been taught yet, and `-0` is `0` — not negative. The
caller's `< 0` test then read a total engine outage as a clean run of
zero, recorded `_bootstrapped` for the account, and the hourly sweep
skipped it from then on. The account most in need of a warm start was
the one guaranteed never to get one.
Which is precisely what happened alongside the missing controller
password: every learn was refused 401, and each account was quietly
retired on its first sweep.
Return None for "the engine stopped accepting these" so failure cannot
be confused with a count, and decline to mark an account whose samples
were all unreadable.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_018SqJYt7jD7oUEkoY68sBZo
Rspamd answers on two ports and only one of them asks for a password:
the scan worker takes /checkv2 anonymously, the controller demands
`Password:` for /stat and /learnspam. `webmail.sh init` generates
RSPAMD_CONTROLLER_PASSWORD and docker-compose hands it to the rspamd
container — but the shared x-backend-env block that feeds api and
worker-* never carried it. The plugin reads it from the environment,
found "", and sent no header.
So classification worked and learning never did. Every learn returned
401 and the plugin swallowed it, as it is built to: an engine problem
must not lose mail. Bootstrap logged "engine unavailable", passive
learning from Inbox->Spam moves taught nothing, and the single visible
trace was `"bayes": null` on GET /api/plugins/antispam/stats, on an
install whose verdict counts were ticking up normally.
Verified against a live engine: with no password /stat and /learnspam
both 401 and rspamd.learn() returns False; with it, stats returns and
learn returns True.
- pass RSPAMD_CONTROLLER_PASSWORD to the app containers, in both the
production and dev compose files
- doctor now makes the real call from inside the api container, with
the password that container actually holds, and says plainly that
nothing will be learned if it fails
- a test reads both compose files, because this is a missing line that
review already let through once
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_018SqJYt7jD7oUEkoY68sBZo
Warming Bayes hung on a single event — `account.state_changed` reaching "live".
That covers an account added from now on and nothing else. An account that went
live before this plugin existed never gets one; neither does one whose Junk
folder was empty at the time and filled up later, or one whose owner enabled
spam-folder sync afterwards. A worker restarting at the wrong moment loses it
too. The result was a feature that quietly required someone to run it by hand
per account, which is not a feature.
An hourly sweep now walks every live account and offers each a bootstrap. That
works because the per-account job was already cheap to repeat: it returns at once
for an account already done, and declines — without marking it done — when the
corpus is still too small to learn from. So the sweep simply keeps asking until
there is something worth learning, then does it once.
`s.accounts.live()` is the new SDK surface it needs: cross-user by design,
because a sweep runs for the instance rather than a request. Ids and state only,
so reading anything still goes through the user-scoped accessors.
One failing account cannot stop the sweep — the mailbox that raises is logged and
the rest still get their turn.
docs/plugins.md now says this in general terms next to `JobSpec`: a state-change
hook is one-shot, so pair it with a sweep when the work must eventually happen
for everything.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_018SqJYt7jD7oUEkoY68sBZo
docs/plugins.md described an SDK that no longer exists. It now covers
messages.raw/sample, actions.move_to_special, the envelope's account_id /
mailbox_special_use / account_state, the message.changed transition payload, the
message.actions.menu slot, Slot.title and .wm-menu-item — with the reasons, not
just the signatures. M3's acceptance is that a plugin can be written from this
file alone, and it had drifted.
Acceptance 4 (§C.8): a test that teaches 20 spam + 20 ham to a real engine and
checks the classifier generalises to held-out mail, keeps ham under the banner
threshold, and does not leak between users. It skips when no engine is reachable.
It does NOT assert the literal move_threshold — a synthetic fixture cannot reach
12.0 once the SMTP-time symbols are zeroed, which is the whole point of the
tuning. Recorded as OQ-8 rather than quietly asserting something weaker.
Bayes ignores training until min_learns, stock 200. For a personal mailbox that
means every correction appears to be ignored for months, so it is now 20.
The bootstrap would have made this filter worse. Accounts do not sync their
provider's spam folder by default, so the real corpus here is 3 junk beside 1,294
aged inbox — teaching both as found trains "everything is ham". It now refuses
below 10 spam samples (leaving the account unmarked so a later run can do it
properly) and otherwise caps ham at 3x the spam count.
Run against the live account, it correctly declined:
"only 3 spam samples (need 10); enable spam-folder sync to build a corpus"
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_018SqJYt7jD7oUEkoY68sBZo
"Mark as Spam" moved the message but left it on screen. Two causes, both fixed
generally rather than for this one action.
`move_job` published message.changed without a thread_id, so the WS handler
invalidated the message list but never the thread the user was reading — the
panel had no reason to refetch. It now names the thread.
And a thread whose last visible message has gone should not linger. MailLayout
closes the panel when the open thread comes back empty, whoever emptied it —
core action or plugin.
`move_to_special` now also publishes the move immediately rather than waiting
for the IMAP job, since a plugin acting on the user's behalf has no client-side
handle to update optimistically the way the core endpoint's caller does.
That exposed an ordering bug: the optimistic event reaches the passive-learning
hook, which would re-teach the lesson /learn had just taught. /learn now records
`learned` before moving, so the hook's dedup stands down — and records it with
an upsert, because provider-filed spam has no verdict row of ours to update.
One real mistake on the way: the event first reported the destination folder as
the source, which would have made every move look like a no-op to anything
reading the pair.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_018SqJYt7jD7oUEkoY68sBZo
A button pinned to every message header is a persistent control; marking spam is
a verb you reach for occasionally, like Reply or Delete. It now lives in the
Actions menu, between Labels and Delete.
That needed a slot inside the dropdown, which the catalog had no name for:
`message.actions.menu`. The host wraps it so any click closes the menu, leaving
a bundle to do nothing but act.
The UI kit gains `.wm-menu-item`, @applying the exact utilities ThreadView's own
menu items use — the same trick the sidebar shell already relies on, so a plugin
verb is indistinguishable from a core one rather than approximately similar.
Labels follow the folder as before: "Mark as Spam" in the inbox or archive,
"Mark as not spam" in the spam folder, nothing in Sent/Drafts/Trash. A message
already taught shows that, disabled, instead of offering the lesson again.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_018SqJYt7jD7oUEkoY68sBZo
Spam / Not spam appeared on every message in every thread. "Not spam" on an
inbox message is nonsense — it was never filed as spam — and a pair of buttons
everywhere is noise that trains nobody.
The slot now shows a single action chosen by where the message actually is:
"Spam" in the inbox or archive, "Not spam" in the spam folder, nothing at all in
Sent, Drafts or Trash, where no move is a verdict about spam. A message already
taught says so instead of offering the same lesson again.
That needs the message's folder, which the slot was never told. Rather than a
core change to the slot props, /verdict/{id} becomes /context/{id} and returns
the folder alongside the verdict — one round-trip, answered for any message
including ones never classified.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_018SqJYt7jD7oUEkoY68sBZo
Explicit teaching shipped with C1 (the Spam / Not spam buttons). This adds the
two halves that need no buttons at all.
Passive learning (§C.5). Moving mail into Spam says "this is spam" more honestly
than any button, because the user is doing it for their own reasons rather than
to train a filter; moving it back out says the opposite. Only those two
transitions teach — Spam to Trash is someone tidying up, Inbox to Archive says
nothing about spam. Teaching the same message the same thing twice is skipped,
since users move things back and forth, but reversing a lesson is allowed. If
the engine is down nothing is recorded, so a later move can teach it again.
This needed the transition, not just the destination: `message.changed` carried
only where a message landed, so `move_job` now also publishes
previous_mailbox_id and both special_use values.
Provider bootstrap (§C.5). An account going live enqueues a job that teaches
Bayes from mail the user already owns — up to 500 messages from the provider's
Junk folder as spam, and up to 500 Inbox messages older than 30 days as ham.
This is the constructive use of the history that classification deliberately
refuses to act on: the same backfill, read instead of judged. It sleeps between
messages so live classification never queues behind training, stops rather than
grinding if the engine disappears, and marks the account done in KV so it runs
once.
That needed `PluginMessages.sample`: the SDK could fetch a message by id but not
enumerate a folder. Ids only — bodies are pulled one at a time, so a 500-message
sample never materialises 500 bodies.
Verified against the running engine: teaching moved its `learned` counter 0 -> 2.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_018SqJYt7jD7oUEkoY68sBZo
Dev compose gains rspamd + redis-rspamd, matching production. The dev API and
worker run on the host rather than in the compose network, so the ports are
published on loopback and RSPAMD_URL points at 127.0.0.1 — documented in
.env.example.
Contacts now cap a verdict at "banner": mail from someone in the user's address
book is never moved to Junk. Capped rather than exempted, because the way this
fails in practice is a contact's account being compromised and spamming its own
address book — the warning still has value even when the move would be wrong.
An explicit deny rule still wins; that is the user overruling us deliberately.
The lookup reads the contacts plugin's table directly and forgives its absence:
no contacts plugin means no contacts to check, not an error. Addresses are
stored as the user typed them, so both sides are lower-cased rather than relying
on the jsonb containment index, and secondary addresses in the emails array
count as well as the primary. It runs only when the verdict would otherwise be a
move, so the common path pays nothing.
Also fixes a test that asserted stats degrade "without an engine" by relying on
no engine being present — true until this commit. It now simulates the absence.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_018SqJYt7jD7oUEkoY68sBZo
message.new fires for imported history as well as new arrivals, and EventBus
awaits hooks inline. As written, adding an account would have scored every
message in its backfill — filing years-old inbox mail into Junk on the server,
and adding ~70 ms of serialized HTTP per message to the backfill loop (twelve
minutes on a 10k-message account). It is also when the filter is least able to
judge, because Bayes has learned nothing yet.
The envelope now carries account_state, and the hook returns unless it is
"live". The backfill window belongs to the C2 learning bootstrap (§C.5), which
reads that history as training data instead of acting on it.
Tests pin the gating: backfilling accounts are not even read from the blob
store, non-inbox folders are ignored, a live inbox arrival still classifies, and
an unreachable engine still delivers.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_018SqJYt7jD7oUEkoY68sBZo
Completes phase C1 — until now the plugin classified mail that nobody could see
or steer.
Router (§C.6): settings with merge semantics, allow/deny rules, a per-message
verdict lookup, learn, and stats. Two behaviours worth naming. A banner
threshold above the move threshold is silently capped rather than refused,
because a banner past the move score can never be seen — everything that far up
is already in Junk. And /learn moves the message even when the engine is
unreachable: teaching is best-effort, but filing is what the user actually
asked for.
UI: settings.page (thresholds in plain language, per-account toggles, the two
lists, the privacy switch, activity); message.banner for the uncertain middle,
offering Move to Junk / Not spam / Always trust sender; message.actions with
Spam / Not spam. The banner mounts on every message but renders only for a
`banner` verdict — a spam warning on ordinary mail is worse than no warning.
Bundle serving is allow-listed by filename, so /ui/../hook.py is a 404 rather
than a file read.
Tests cover the defaults, merge semantics, the threshold cap, rule
normalisation and idempotency, stats degrading to empty when Rspamd is absent,
and RLS isolation: one user cannot see another's rules or settings, nor delete a
rule whose id they know.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_018SqJYt7jD7oUEkoY68sBZo
Second-pass spam filtering per SPEC Appendix C. The engine is Rspamd in its own
container; this plugin is glue and policy, never a spam engine of its own.
Classification (§C.4) hooks message.new, acts only on inbox arrivals of enabled
accounts, and runs deny list -> allow list -> engine -> thresholds. Every
failure path ends in "deliver normally, unclassified": an unreachable engine, a
missing blob, an account with no Junk folder, a failed verdict write. Nothing
here may delay or lose mail, so no service declares depends_on: rspamd.
Two containers, started with everything else rather than behind a profile:
rspamd, and a redis of its own. The separate redis is not incidental — the app's
redis holds sync leases and arq queues under noeviction, and an ever-growing
Bayes corpus must never be able to fail a job enqueue. This one takes RDB
snapshots and evicts, because its data is derived and retrainable.
Network checks (RBL/SURBL/fuzzy) are off: they leak message-derived hashes and
URLs to third parties, which a self-hosted server's users did not ask for. The
engine makes no outbound connections in the default configuration.
The controller password is generated into .env like MASTER_KEY. It is internal;
humans will reach Rspamd through the admin panel, authenticated as the ordinary
administrator, rather than typing it.
local.d/groups.conf zeroes six SMTP-time symbols — DATE_IN_PAST,
PREVIOUSLY_DELIVERED, HFILTER_HOSTNAME_UNKNOWN, RCVD_NO_TLS_LAST, ARC_NA,
DMARC_NA — and greylisting is disabled. Measured against 398 real delivered
messages, those checks were firing on 44-71% of ordinary mail and pushing 11.3%
of it over the banner threshold; scoring mail at rest for facts true of every
delivered message is simply the wrong question. After: median 3.09 -> 0.70,
p95 7.00 -> 3.40, banner rate 11.3% -> 1.0%. The spec's 12.0/6.0 thresholds were
never the problem and are unchanged. Symbols are zeroed rather than removed so
verdicts stay explainable, and they matter again at the M6 edge (§C.7).
Still to come in C1: the §C.6 router and UI slots. C2 adds the learning loop —
until Bayes is trained, genuine scams and legitimate bulk mail overlap in the
5-8 band and no threshold separates them.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_018SqJYt7jD7oUEkoY68sBZo
CalDAV account connect (Settings → Calendars) with RFC 6764 discovery,
per-calendar vaulted credentials, and full two-way sync verified live
against NextCloud:
- caldav.py: discovery, calendar-query REPORT (pull), PUT/DELETE with
quoted ETags (If-Match/If-None-Match), iCloud partition-redirect
handling, ICS build/parse; clear error codes (auth/private-host/
unreachable) surfaced to the connect UI.
- sync.py: ctag-gated pull + push of local changes; dirty rows protected
from being clobbered; 15-min cron + immediate enqueue on every
create/edit/delete so changes reach the server in ~1s.
- Dirty tracking via a BEFORE INSERT/UPDATE trigger (migration 0007),
suppressed during sync with a txn-local GUC; deletes tombstone then
DELETE server-side. Writability now driven by server privileges.
- migration 0006: event uniqueness NULLS NOT DISTINCT — fixes duplicate
events on every full re-pull (NULL recurrence_id never matched ON CONFLICT).
- 5 CalDAV parsing unit tests.
Calendar UI (page.js): all-day events now render in Day/3-days/Week views
(were month-only). They share the timed-event column layout so overlapping
events sit side-by-side, fill the day column vertically with a sticky label,
and are matched by start day (fixes UTC-midnight events spilling into the
next local day).
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Calendar plugin (webmail_plugin_calendar, non-removable core plugin):
- Local calendars/events CRUD with RLS isolation; recurrence expanded at
query time (DST-correct), EXDATE + detached-override merge.
- Calendar app bundle: month/week/day/3-day views, mini-month, event form
(color, show-as, timezone combobox, repeat, reminders, guests w/ recipient
autocomplete, Jitsi video call), event popover with RSVP badges, resizable/
collapsible left panel + theme toggle, overlap layout.
- iMIP invites: sdk.mail sends REQUEST from the organizer's mailbox (HTML +
text/calendar + invite.ics); public /rsvp endpoint (vault-sealed token)
records accepted/declined/tentative per attendee.
- "This and following" edit + scoped delete (this / this-and-following), with
a delete confirmation dialog for recurring events.
Core:
- Settings: time_format (12/24h, default 24h) + jitsi_server; new Integrations
section; date order follows time format (en-GB vs en-US) across Mail.
- Plugin SDK: mail (send from a user's account), vault (envelope encrypt),
multi-method SSRF-guarded http; AppSwitcher product switcher; full-bleed
PluginPage.
Mail HTML rendering:
- Sanitizer allows <style> blocks + class (CSS sanitized: drop script-like,
strip @import/url() when remote images blocked); SANITIZER_VERSION 3 so
already-indexed mail re-sanitizes on view.
- Attachments: a Content-Disposition: attachment part is a real attachment
even when it also carries a Content-ID (fixes hidden invite.ics).
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
lukasandClaude Opus 5.5