System mail: the password-reset email (and the admin panel's test email) now
opens with the Ultimo logo, centered. It is a PNG rendered from
icons/ultimo-logo.svg and embedded in the message (multipart/related, cid:),
so it shows where remote images are blocked and in clients that do not
render SVG. The PNG is listed in package-data so the Docker image carries it.
Trusted devices: a 2FA sign-in offers "Don't ask for the code on this device
in future". Ticked with a correct code, the browser gets its own HttpOnly
trusted-device cookie (token in Redis, 90 days, renewed on use); later
sign-ins to the same account from that browser need only the password. Other
accounts and browsers are unaffected. The trust is revoked on a password
reset, when 2FA is turned off, and when an admin blocks or deletes the user;
signing out keeps it.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Branding: the page titles and app name are now "Ultimo Email".
Admin panel (now two columns with a divider):
- Users: search by login or by a mail account's address or name (results
only, nothing listed until something is typed). A user opens a window to
change their quota, block/unblock them, or delete them; block and delete ask
for confirmation, and an admin cannot do either to their own account.
- Blocking (users.blocked_at, migration 0019) refuses sign-in with a clear
message and revokes every session; a new user.signed_out event sends the
user's open tabs to the sign-in screen at once. Any "not signed in" answer
now does the same, not only /auth/me. Deleting removes mail accounts (sync
stops), plugin rows (cleared inside the user's RLS context, since plugin
tables have no FK to users), blobs, the search index, sessions and the user.
- Outgoing system mail (migration 0020): SMTP server, port, security, login,
password (vault-encrypted, never sent back), From address and the public
address reset links point to, plus a "Send test" that reports the server's
error.
Password reset:
- "Forgot password?" on the sign-in page asks for the user name and always
answers "If this account exists you will receive an email to your recovery
email address"; the mail goes out after the response, so neither the reply
nor its timing reveals an account, and it is capped at 3 per hour.
- The emailed link (token in the URL fragment, single use, 30 minutes) opens a
page to set a new password. With 2-step verification on, the code is checked
before anything changes; five wrong codes burn the link.
- Settings > Profile: the recovery email can be added, changed or removed,
confirmed with the current password, which must be typed — autofill and
paste are refused (TypedPasswordInput).
Also adds frontend/icons/ultimo-logo-bimi.svg, an SVG Tiny PS version of the
logo for a BIMI record.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
A missing avatar was a 404, which the browser reports as an error for every
such sender on every page. It is now an empty 204: the <img> still fails to
decode it and the initials show through, but nothing is logged, and the miss
carries the same one-day browser cache as a hit so it stops being re-fetched.
On a phone the mail list's rows sit 20px apart instead of 7px.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The shared Redis client decodes replies as UTF-8 text, but the avatar cache
holds raw image bytes and a b"none" miss marker: an image could not be read
back at all, and a miss came back as str and crashed on .partition(b"\n").
Only a sender's first request worked; after that, a 500. The cache now goes
through a binary-safe client, get_redis_bytes().
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
<title> is not an allowed tag, so nh3 unwrapped it and kept its text —
"mailer", "Email template" or the subject rendered before the content.
It is now dropped with its content, both in the sanitized HTML and in the
text derived from HTML-only mail (snippet and search). SANITIZER_VERSION
goes to 4 so stored bodies re-sanitize from the raw blob on next render.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The sender column showed participants[0], and participants are rebuilt
newest message first — so replying to a thread put the user in that slot
and the list named them instead of the person who wrote in.
Thread summaries (list and search) now carry `senders`: the distinct From
addresses of the thread's visible messages, oldest first, with the same
Trash/Spam visibility as the message count. The list joins them and shows
the user's own account addresses as "me" ("me, Alice" for a thread the
user started), and the avatar prefers the other side of the conversation.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
`_teach` signalled failure by negating the count taught so far. That
works for every case but the one that matters: when the FIRST message is
refused nothing has been taught yet, and `-0` is `0` — not negative. The
caller's `< 0` test then read a total engine outage as a clean run of
zero, recorded `_bootstrapped` for the account, and the hourly sweep
skipped it from then on. The account most in need of a warm start was
the one guaranteed never to get one.
Which is precisely what happened alongside the missing controller
password: every learn was refused 401, and each account was quietly
retired on its first sweep.
Return None for "the engine stopped accepting these" so failure cannot
be confused with a count, and decline to mark an account whose samples
were all unreadable.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_018SqJYt7jD7oUEkoY68sBZo
Rspamd answers on two ports and only one of them asks for a password:
the scan worker takes /checkv2 anonymously, the controller demands
`Password:` for /stat and /learnspam. `webmail.sh init` generates
RSPAMD_CONTROLLER_PASSWORD and docker-compose hands it to the rspamd
container — but the shared x-backend-env block that feeds api and
worker-* never carried it. The plugin reads it from the environment,
found "", and sent no header.
So classification worked and learning never did. Every learn returned
401 and the plugin swallowed it, as it is built to: an engine problem
must not lose mail. Bootstrap logged "engine unavailable", passive
learning from Inbox->Spam moves taught nothing, and the single visible
trace was `"bayes": null` on GET /api/plugins/antispam/stats, on an
install whose verdict counts were ticking up normally.
Verified against a live engine: with no password /stat and /learnspam
both 401 and rspamd.learn() returns False; with it, stats returns and
learn returns True.
- pass RSPAMD_CONTROLLER_PASSWORD to the app containers, in both the
production and dev compose files
- doctor now makes the real call from inside the api container, with
the password that container actually holds, and says plainly that
nothing will be learned if it fails
- a test reads both compose files, because this is a missing line that
review already let through once
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_018SqJYt7jD7oUEkoY68sBZo
Reported from the running app: a message showed as unread in the list — bold,
blue dot — while the reading pane showed it as read, and only a manual "mark as
read" reconciled them.
The two views read different things. The pane reads the message's own flags; the
list reads `threads.unread_count`, denormalized from them. `apply_delta` updated
message flags with a bare UPDATE and never recomputed the thread, so a flag set
anywhere but in this app — read on a phone, filed by a server-side rule — moved
one and not the other. Auto-mark-on-read could not fix it either: it sees a
message that is already \Seen and correctly does nothing, leaving the counter
wrong forever.
Expunges had the same hole, which also left `message_count` overstated.
`apply_delta` now collects the affected threads and re-denormalizes them after
the deletes, so a thread emptied by an expunge is removed rather than recounted.
Migration 0018 repairs rows that already drifted — recomputed from the messages,
so it is a no-op on a healthy database. One thread was stale here; installs that
have been syncing longer will have more.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_018SqJYt7jD7oUEkoY68sBZo
Warming Bayes hung on a single event — `account.state_changed` reaching "live".
That covers an account added from now on and nothing else. An account that went
live before this plugin existed never gets one; neither does one whose Junk
folder was empty at the time and filled up later, or one whose owner enabled
spam-folder sync afterwards. A worker restarting at the wrong moment loses it
too. The result was a feature that quietly required someone to run it by hand
per account, which is not a feature.
An hourly sweep now walks every live account and offers each a bootstrap. That
works because the per-account job was already cheap to repeat: it returns at once
for an account already done, and declines — without marking it done — when the
corpus is still too small to learn from. So the sweep simply keeps asking until
there is something worth learning, then does it once.
`s.accounts.live()` is the new SDK surface it needs: cross-user by design,
because a sweep runs for the instance rather than a request. Ids and state only,
so reading anything still goes through the user-scoped accessors.
One failing account cannot stop the sweep — the mailbox that raises is logged and
the rest still get their turn.
docs/plugins.md now says this in general terms next to `JobSpec`: a state-change
hook is one-shot, so pair it with a sweep when the work must eventually happen
for everything.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_018SqJYt7jD7oUEkoY68sBZo
docs/plugins.md described an SDK that no longer exists. It now covers
messages.raw/sample, actions.move_to_special, the envelope's account_id /
mailbox_special_use / account_state, the message.changed transition payload, the
message.actions.menu slot, Slot.title and .wm-menu-item — with the reasons, not
just the signatures. M3's acceptance is that a plugin can be written from this
file alone, and it had drifted.
Acceptance 4 (§C.8): a test that teaches 20 spam + 20 ham to a real engine and
checks the classifier generalises to held-out mail, keeps ham under the banner
threshold, and does not leak between users. It skips when no engine is reachable.
It does NOT assert the literal move_threshold — a synthetic fixture cannot reach
12.0 once the SMTP-time symbols are zeroed, which is the whole point of the
tuning. Recorded as OQ-8 rather than quietly asserting something weaker.
Bayes ignores training until min_learns, stock 200. For a personal mailbox that
means every correction appears to be ignored for months, so it is now 20.
The bootstrap would have made this filter worse. Accounts do not sync their
provider's spam folder by default, so the real corpus here is 3 junk beside 1,294
aged inbox — teaching both as found trains "everything is ham". It now refuses
below 10 spam samples (leaving the account unmarked so a later run can do it
properly) and otherwise caps ham at 3x the spam count.
Run against the live account, it correctly declined:
"only 3 spam samples (need 10); enable spam-folder sync to build a corpus"
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_018SqJYt7jD7oUEkoY68sBZo
"Mark as Spam" moved the message but left it on screen. Two causes, both fixed
generally rather than for this one action.
`move_job` published message.changed without a thread_id, so the WS handler
invalidated the message list but never the thread the user was reading — the
panel had no reason to refetch. It now names the thread.
And a thread whose last visible message has gone should not linger. MailLayout
closes the panel when the open thread comes back empty, whoever emptied it —
core action or plugin.
`move_to_special` now also publishes the move immediately rather than waiting
for the IMAP job, since a plugin acting on the user's behalf has no client-side
handle to update optimistically the way the core endpoint's caller does.
That exposed an ordering bug: the optimistic event reaches the passive-learning
hook, which would re-teach the lesson /learn had just taught. /learn now records
`learned` before moving, so the hook's dedup stands down — and records it with
an upsert, because provider-filed spam has no verdict row of ours to update.
One real mistake on the way: the event first reported the destination folder as
the source, which would have made every move look like a no-op to anything
reading the pair.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_018SqJYt7jD7oUEkoY68sBZo
A button pinned to every message header is a persistent control; marking spam is
a verb you reach for occasionally, like Reply or Delete. It now lives in the
Actions menu, between Labels and Delete.
That needed a slot inside the dropdown, which the catalog had no name for:
`message.actions.menu`. The host wraps it so any click closes the menu, leaving
a bundle to do nothing but act.
The UI kit gains `.wm-menu-item`, @applying the exact utilities ThreadView's own
menu items use — the same trick the sidebar shell already relies on, so a plugin
verb is indistinguishable from a core one rather than approximately similar.
Labels follow the folder as before: "Mark as Spam" in the inbox or archive,
"Mark as not spam" in the spam folder, nothing in Sent/Drafts/Trash. A message
already taught shows that, disabled, instead of offering the lesson again.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_018SqJYt7jD7oUEkoY68sBZo
Spam / Not spam appeared on every message in every thread. "Not spam" on an
inbox message is nonsense — it was never filed as spam — and a pair of buttons
everywhere is noise that trains nobody.
The slot now shows a single action chosen by where the message actually is:
"Spam" in the inbox or archive, "Not spam" in the spam folder, nothing at all in
Sent, Drafts or Trash, where no move is a verdict about spam. A message already
taught says so instead of offering the same lesson again.
That needs the message's folder, which the slot was never told. Rather than a
core change to the slot props, /verdict/{id} becomes /context/{id} and returns
the folder alongside the verdict — one round-trip, answered for any message
including ones never classified.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_018SqJYt7jD7oUEkoY68sBZo
Explicit teaching shipped with C1 (the Spam / Not spam buttons). This adds the
two halves that need no buttons at all.
Passive learning (§C.5). Moving mail into Spam says "this is spam" more honestly
than any button, because the user is doing it for their own reasons rather than
to train a filter; moving it back out says the opposite. Only those two
transitions teach — Spam to Trash is someone tidying up, Inbox to Archive says
nothing about spam. Teaching the same message the same thing twice is skipped,
since users move things back and forth, but reversing a lesson is allowed. If
the engine is down nothing is recorded, so a later move can teach it again.
This needed the transition, not just the destination: `message.changed` carried
only where a message landed, so `move_job` now also publishes
previous_mailbox_id and both special_use values.
Provider bootstrap (§C.5). An account going live enqueues a job that teaches
Bayes from mail the user already owns — up to 500 messages from the provider's
Junk folder as spam, and up to 500 Inbox messages older than 30 days as ham.
This is the constructive use of the history that classification deliberately
refuses to act on: the same backfill, read instead of judged. It sleeps between
messages so live classification never queues behind training, stops rather than
grinding if the engine disappears, and marks the account done in KV so it runs
once.
That needed `PluginMessages.sample`: the SDK could fetch a message by id but not
enumerate a folder. Ids only — bodies are pulled one at a time, so a 500-message
sample never materialises 500 bodies.
Verified against the running engine: teaching moved its `learned` counter 0 -> 2.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_018SqJYt7jD7oUEkoY68sBZo
Dev compose gains rspamd + redis-rspamd, matching production. The dev API and
worker run on the host rather than in the compose network, so the ports are
published on loopback and RSPAMD_URL points at 127.0.0.1 — documented in
.env.example.
Contacts now cap a verdict at "banner": mail from someone in the user's address
book is never moved to Junk. Capped rather than exempted, because the way this
fails in practice is a contact's account being compromised and spamming its own
address book — the warning still has value even when the move would be wrong.
An explicit deny rule still wins; that is the user overruling us deliberately.
The lookup reads the contacts plugin's table directly and forgives its absence:
no contacts plugin means no contacts to check, not an error. Addresses are
stored as the user typed them, so both sides are lower-cased rather than relying
on the jsonb containment index, and secondary addresses in the emails array
count as well as the primary. It runs only when the verdict would otherwise be a
move, so the common path pays nothing.
Also fixes a test that asserted stats degrade "without an engine" by relying on
no engine being present — true until this commit. It now simulates the absence.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_018SqJYt7jD7oUEkoY68sBZo
message.new fires for imported history as well as new arrivals, and EventBus
awaits hooks inline. As written, adding an account would have scored every
message in its backfill — filing years-old inbox mail into Junk on the server,
and adding ~70 ms of serialized HTTP per message to the backfill loop (twelve
minutes on a 10k-message account). It is also when the filter is least able to
judge, because Bayes has learned nothing yet.
The envelope now carries account_state, and the hook returns unless it is
"live". The backfill window belongs to the C2 learning bootstrap (§C.5), which
reads that history as training data instead of acting on it.
Tests pin the gating: backfilling accounts are not even read from the blob
store, non-inbox folders are ignored, a live inbox arrival still classifies, and
an unreachable engine still delivers.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_018SqJYt7jD7oUEkoY68sBZo
Completes phase C1 — until now the plugin classified mail that nobody could see
or steer.
Router (§C.6): settings with merge semantics, allow/deny rules, a per-message
verdict lookup, learn, and stats. Two behaviours worth naming. A banner
threshold above the move threshold is silently capped rather than refused,
because a banner past the move score can never be seen — everything that far up
is already in Junk. And /learn moves the message even when the engine is
unreachable: teaching is best-effort, but filing is what the user actually
asked for.
UI: settings.page (thresholds in plain language, per-account toggles, the two
lists, the privacy switch, activity); message.banner for the uncertain middle,
offering Move to Junk / Not spam / Always trust sender; message.actions with
Spam / Not spam. The banner mounts on every message but renders only for a
`banner` verdict — a spam warning on ordinary mail is worse than no warning.
Bundle serving is allow-listed by filename, so /ui/../hook.py is a 404 rather
than a file read.
Tests cover the defaults, merge semantics, the threshold cap, rule
normalisation and idempotency, stats degrading to empty when Rspamd is absent,
and RLS isolation: one user cannot see another's rules or settings, nor delete a
rule whose id they know.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_018SqJYt7jD7oUEkoY68sBZo
Second-pass spam filtering per SPEC Appendix C. The engine is Rspamd in its own
container; this plugin is glue and policy, never a spam engine of its own.
Classification (§C.4) hooks message.new, acts only on inbox arrivals of enabled
accounts, and runs deny list -> allow list -> engine -> thresholds. Every
failure path ends in "deliver normally, unclassified": an unreachable engine, a
missing blob, an account with no Junk folder, a failed verdict write. Nothing
here may delay or lose mail, so no service declares depends_on: rspamd.
Two containers, started with everything else rather than behind a profile:
rspamd, and a redis of its own. The separate redis is not incidental — the app's
redis holds sync leases and arq queues under noeviction, and an ever-growing
Bayes corpus must never be able to fail a job enqueue. This one takes RDB
snapshots and evicts, because its data is derived and retrainable.
Network checks (RBL/SURBL/fuzzy) are off: they leak message-derived hashes and
URLs to third parties, which a self-hosted server's users did not ask for. The
engine makes no outbound connections in the default configuration.
The controller password is generated into .env like MASTER_KEY. It is internal;
humans will reach Rspamd through the admin panel, authenticated as the ordinary
administrator, rather than typing it.
local.d/groups.conf zeroes six SMTP-time symbols — DATE_IN_PAST,
PREVIOUSLY_DELIVERED, HFILTER_HOSTNAME_UNKNOWN, RCVD_NO_TLS_LAST, ARC_NA,
DMARC_NA — and greylisting is disabled. Measured against 398 real delivered
messages, those checks were firing on 44-71% of ordinary mail and pushing 11.3%
of it over the banner threshold; scoring mail at rest for facts true of every
delivered message is simply the wrong question. After: median 3.09 -> 0.70,
p95 7.00 -> 3.40, banner rate 11.3% -> 1.0%. The spec's 12.0/6.0 thresholds were
never the problem and are unchanged. Symbols are zeroed rather than removed so
verdicts stay explainable, and they matter again at the M6 edge (§C.7).
Still to come in C1: the §C.6 router and UI slots. C2 adds the learning loop —
until Bayes is trained, genuine scams and legitimate bulk mail overlap in the
5-8 band and no threshold separates them.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_018SqJYt7jD7oUEkoY68sBZo
Three strands that only make sense together, so they land together.
Registration gate (migration 0016). POST /auth/register was open to anyone
who could reach the port. Now the first account on an empty instance is
always allowed, becomes the administrator, and closes registration behind
itself; after that it takes an admin reopening it. The decision is made
under a row lock on app_settings so two simultaneous first signups cannot
both see an empty table and both claim administrator. Existing installs
keep their oldest account as admin and start closed.
Instance default quota (migration 0017). users.quota_bytes is snapshotted
at creation from app_settings.default_quota_bytes, so changing the default
affects later accounts only and never resizes a mailbox that exists.
The panel is a separate app, not a page in the mail client. admin.html is
its own Vite entry (5.8 KB next to the client's 1.7 MB; they share only the
vendor chunk), with its own administrator sign-in. One nginx now serves two
listeners: :80 is the mail client and 404s both /admin.html and
/api/admin/*, :81 is the panel. docker-compose publishes them as WEB_PORT
and ADMIN_PORT. The boundary is nginx refusing paths on one listener, not
process isolation — CurrentAdmin still guards every admin route — so
ADMIN_PORT must be restricted to the LAN, which the runbook now covers for
both DSM and QNAP.
Also QNAP: deploy/webmail.sh finds Container Station's docker under
/share/*/.qpkg/container-station/bin, which is off an SSH session's PATH,
before anything shells out to it. docs/nas-install.md gains a QuTS hero
runbook (§2.6), splits the DSM firewall section into container-to-container
(§8.1) and the egress case that makes account-add time out while
autodiscovery appears to work (§8.2), and scopes both to Synology.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_018SqJYt7jD7oUEkoY68sBZo
CalDAV account connect (Settings → Calendars) with RFC 6764 discovery,
per-calendar vaulted credentials, and full two-way sync verified live
against NextCloud:
- caldav.py: discovery, calendar-query REPORT (pull), PUT/DELETE with
quoted ETags (If-Match/If-None-Match), iCloud partition-redirect
handling, ICS build/parse; clear error codes (auth/private-host/
unreachable) surfaced to the connect UI.
- sync.py: ctag-gated pull + push of local changes; dirty rows protected
from being clobbered; 15-min cron + immediate enqueue on every
create/edit/delete so changes reach the server in ~1s.
- Dirty tracking via a BEFORE INSERT/UPDATE trigger (migration 0007),
suppressed during sync with a txn-local GUC; deletes tombstone then
DELETE server-side. Writability now driven by server privileges.
- migration 0006: event uniqueness NULLS NOT DISTINCT — fixes duplicate
events on every full re-pull (NULL recurrence_id never matched ON CONFLICT).
- 5 CalDAV parsing unit tests.
Calendar UI (page.js): all-day events now render in Day/3-days/Week views
(were month-only). They share the timed-event column layout so overlapping
events sit side-by-side, fill the day column vertically with a sticky label,
and are matched by start day (fixes UTC-midnight events spilling into the
next local day).
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Sender avatars (privacy-safe, server-fetched, cached):
- New GET /api/avatar resolves Gravatar (d=404) then the sender domain's favicon
under the SSRF guard, Redis-cached (hit+miss); gated by SENDER_AVATARS_ENABLED.
GuardedResponse gains raw bytes. Shown in the mail list, reading pane, People
panel, and Contacts (overlay that falls back to initials on a miss).
PDF preview:
- Password-protected PDFs now prompt in-place and retry (pdf.js PasswordException),
with a clear error for other load failures instead of a blank overlay. The
password is remembered per attachment until the thread is left; clicking the
field no longer dismisses the overlay.
Contacts:
- Autosave reply recipients: message.sent now carries to/cc (name+email); the
contacts plugin subscribes and adds any non-contact recipients to the Autosaved
group. Contact avatars show the gravatar/favicon via /api/avatar.
Mail views / retention:
- Clicking an account shows all its mail except Trash/Spam (was inbox-only).
- Daily cron autopurge_trash_spam permanently deletes Trash/Spam older than
TRASH_RETENTION_DAYS (default 30; 0 disables) — server EXPUNGE + local rows.
UI polish:
- Ultimo signet logo in the Mail/Calendar/Contacts sidebar header and collapsed
rail; consistent 11px top margin and vertically-aligned header row across states
(fixed-size flex icon buttons); SVG favicon on the browser tab.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01U72FZiF1W1pb8XxyWBTREA
Gmail sync (SPEC: Gmail sync):
- Provider fetches X-GM-LABELS when X-GM-EXT-1 is advertised; parses custom labels
(system labels dropped, mUTF-7 decoded) onto RemoteMessage.
- Engine syncs only system folders + All Mail for Gmail, skips every label folder,
and dedupes the All Mail pass by Message-ID — kills the INBOX/All-Mail duplication.
- Custom labels ride the message.new event; the labels plugin subscribes and maps
them to chips as "<name> [Gmail]" (per-message + thread aggregate + known set),
so labels fill in gradually with the sync window. Non-ASCII labels decode right.
Labels self-healing (message-driven reconcile):
- Per-message keys are the source of truth; t: aggregates and _labels are derived.
Reconcile drops labels of deleted messages, regroups survivors by their current
thread (fixing JWZ thread merges), rewrites changed t: keys, and rebuilds _labels
from live messages — keeping user labels, dropping orphaned Gmail ones.
- New SDK reads: sdk.threads.existing / sdk.messages.threads_of. Sidebar refreshes
live via a plugin.labels.changed WS event on new labels.
Account deletion: prune threads left empty after the cascade (they were user-scoped,
so only messages cascaded) + de-list the account from surviving cross-account threads.
Sync SPAM: opt-in per-account sync_spam (migration 0015, default off) — the engine
skips the Junk folder in backfill and poll unless enabled; wizard checkbox added.
Robustness / UX:
- WebSocket auto-reconnects with backoff and resyncs on (re)connect, and account
add/delete invalidates threads/labels — no more hard refresh for live updates.
- Backfill banner + sidebar ring also require done < total, so a completed account
can't leave a stuck "X of X".
- Scrub lone UTF-16 surrogates from parsed headers/bodies so bad messages no longer
crash the backfill on Postgres text/JSONB.
- Ultimo signet logo in the Mail/Calendar/Contacts sidebar header (and at the top of
the collapsed rail), via the shared plugin-ui.css + MailLayout.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01U72FZiF1W1pb8XxyWBTREA
Calendar plugin (webmail_plugin_calendar, non-removable core plugin):
- Local calendars/events CRUD with RLS isolation; recurrence expanded at
query time (DST-correct), EXDATE + detached-override merge.
- Calendar app bundle: month/week/day/3-day views, mini-month, event form
(color, show-as, timezone combobox, repeat, reminders, guests w/ recipient
autocomplete, Jitsi video call), event popover with RSVP badges, resizable/
collapsible left panel + theme toggle, overlap layout.
- iMIP invites: sdk.mail sends REQUEST from the organizer's mailbox (HTML +
text/calendar + invite.ics); public /rsvp endpoint (vault-sealed token)
records accepted/declined/tentative per attendee.
- "This and following" edit + scoped delete (this / this-and-following), with
a delete confirmation dialog for recurring events.
Core:
- Settings: time_format (12/24h, default 24h) + jitsi_server; new Integrations
section; date order follows time format (en-GB vs en-US) across Mail.
- Plugin SDK: mail (send from a user's account), vault (envelope encrypt),
multi-method SSRF-guarded http; AppSwitcher product switcher; full-bleed
PluginPage.
Mail HTML rendering:
- Sanitizer allows <style> blocks + class (CSS sanitized: drop script-like,
strip @import/url() when remote images blocked); SANITIZER_VERSION 3 so
already-indexed mail re-sanitizes on view.
- Attachments: a Content-Disposition: attachment part is a real attachment
even when it also carries a Content-ID (fixes hidden invite.ics).
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Server-side deletions now propagate to the DB. Previously poll() detected new
messages and CONDSTORE flag deltas but never expunges (the old TODO), so mail
deleted in another client lingered as phantoms outside the small Drafts reconcile.
- provider: enable QRESYNC when advertised (_maybe_enable_qresync + qresync_enabled
property). poll() harvests VANISHED (expunged UIDs) and changed FLAGS in one pass
via a raw `SELECT <mbox> (QRESYNC (uidvalidity modseq))` — issued as a low-level
SELECT because aioimaplib only surfaces untagged VANISHED lines on a sync command,
not on an async UID FETCH. CONDSTORE-only servers (e.g. Gmail) keep the CHANGEDSINCE
flag delta and fall back to the engine reconcile.
- parsing: parse_vanished() for `VANISHED [(EARLIER)] <uid-set>` with range expansion.
- engine: apply_delta now drops expunged rows from the search index, un-links any
local draft mirroring a vanished UID, and publishes message.deleted for the UI.
- tests: parse_vanished unit cases + apply_delta expunge (removal, draft un-link,
modseq advance).
Verified live: the worker issues `SELECT "INBOX" (QRESYNC (uidvalidity modseq))`
against QRESYNC-capable accounts. ruff + mypy --strict clean; 296 tests pass.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Rcj3TpydMTzTSmU7kunzf8
Frontend:
- Full-screen attachment preview overlay (AttachmentPreview) for image/pdf/txt
plus office docs: .docx via docx-preview (rendered live into the sandboxed
iframe so header/footer VML images survive), .xlsx via xlsx-preview, and
.odt/.ods/.odp/.odg via a JSZip+DOMParser ODF walker (officeConvert).
- Type-aware attachment icons (green spreadsheet, blue document, red PDF,
image/archive); new document/image/spreadsheet/print SVGs.
- Composer: drag-drop attachments (body drop inserts images inline, else
attaches); inserted images are selectable/resizable/deletable (RichTextEditor).
- Undo-send: "Sending…" banner with live "N seconds to Undo" countdown.
- Search result list: pin/archive/delete update correctly; grey-out only for
search hits; scoped "Search in Trash/Spam".
Backend:
- Search: prefix (search-as-you-type) matching, email-address search, and
address-first result ordering; Trash/Junk excluded with scoped folder search.
- Attachments: RFC 6266 Content-Disposition for non-ASCII filenames; one-time
inline image loading.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Rcj3TpydMTzTSmU7kunzf8
Search (PG backend):
- '@' terms match address substrings (from/to/cc), so a partial email like
"dyrektor@wfdif" resolves and narrows predictably as you type
- index each address split into parts, so a bare domain/local word ("wfdif")
finds the address (indexed as one lexeme otherwise); dotted terms fall back to
an address substring
- rank sender/recipient name+address matches above body/subject-only ones
(addr_hit tier, then recency); cursor extended to keep keyset pagination correct
- scripts/reindex_address_parts.py backfills address tokens into existing rows
Inline images:
- extract attachments via a full MIME walk instead of shallow iter_attachments(),
which missed inline images nested in mixed>related>alternative (cid: rendered
broken); has_attachments counts only real (non-cid) attachments
- scripts/backfill_inline_attachments.py stores previously-missed inline images
Folder-name i18n (SPEC §12 M2):
- accent/punctuation-insensitive, multi-delimiter (/ \ .) special-use inference
across ~13 languages; curated names (no substring false positives)
Link security (reading panel):
- links never navigate the message iframe: a parent-side click interceptor opens
http/https in a new tab (noopener) and mailto in the composer (recipient
pre-filled); all other schemes blocked
Backend 292 tests green, ruff + mypy --strict clean; frontend tsc/eslint/vitest clean.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01U72FZiF1W1pb8XxyWBTREA
Search:
- Meilisearch backend (app/search/meili.py) + factory wiring; pinned SDK 2.7
- Search-as-you-type: debounced query + Postgres prefix matching, built from
the DB tokenizer so full email addresses match (email stays one lexeme)
- Saved searches API + sidebar virtual mailboxes; save affordance in the box
Account sync windows (SPEC §12 M2):
- Choose at setup: All (recent 30d first, then remainder), Last 3 months, New only
- Per-account sync_window + "Sync remaining mail" from settings (idempotent re-run)
- Provider windowed backfill (SINCE/BEFORE), message_count, newest_uid
- migration 0012: accounts.sync_window + backfill_total/backfill_done
Backfill progress UX:
- Floating progress card (auto-dissolves after 10s) + spinning sidebar ring that
fills with the account colour; driven by sync.progress over WS
Thread list pagination:
- Server-side offset + accurate grand total, with filter/order in SQL; working
prev/next across all synced mail
Reading panel:
- "Load images" one-off now actually renders images (?images=1) without trusting
the sender; envelope carries images_allowed so a trusted sender shows no prompt
Threading:
- Subject fallback no longer merges messages that declare an explicit (different)
parent; true siblings still group via a shared-parent reference match. Fixes
unrelated auto-replies with the same subject landing in one thread
Backend 257 tests green, ruff + mypy --strict clean; frontend tsc/eslint/vitest clean.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01U72FZiF1W1pb8XxyWBTREA
Backend:
- Collapse identical cross-account deliveries in a thread: new messages.delivery_id
(migration 0011) preserves the real Message-ID even when message_id_hdr is
synthesized (§7.1); get_thread groups by it and reports also_in_account_ids; the
list count counts distinct deliveries. Backfill script for existing rows.
Frontend:
- Bundle Lato (SIL OFL) as the UI font; unread mail-list rows use Black weight.
- Mail-list selection: hover-avatar checkbox, tri-state "select all", bulk
Archive/Pin/Delete (Restore/Delete in Archive/Trash/Spam) with a count in the
reading pane; unread blue dot; "also delivered to" note in the thread.
- Spacing/size tuning: sender-column width, icon gaps, avatar size, bar heights,
and a round of -1px font sizes across the sidebar, mail list, and reading pane.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01U72FZiF1W1pb8XxyWBTREA
Backend:
- Restore returns each message to its original folder (e.g. Sent/Archive) instead
of always the Inbox: new messages.restore_mailbox_id (migration 0010) +
restore_thread_job.
- Drafts: expunge reconciliation for the Drafts poll (provider.list_uids /
_reconcile_drafts) clears phantom "remote" drafts; send + discard drop the
mirrored DB row; adopting a remote draft carries its real attachments into the
compose flow.
- Thread detail/list are folder-aware: individually trashed/spam messages are
hidden outside their folder, Trash/Spam show only their own messages, and the
mail-list message count matches the view.
Frontend:
- Thread reading panel: fix newest-first card overlap/spacing (expand-all and
single expand), hide Reply/Forward on trashed messages, close the Actions menu
on message-body (iframe) clicks, scroll to the inline composer, and prompt
Save draft / Discard on the X close.
- Composer: remount when switching drafts, dirty-tracking to stop duplicate IMAP
mirrors, and close the From/More dropdowns on outside click.
- Sidebar: collapse/expand toggle (icons-only ~60px rail) and a log-out icon on
Sign out.
- Mail-standard printing in a new window with reliable page margins (lib/print.ts).
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01U72FZiF1W1pb8XxyWBTREA
Assert bulk backfills route to the bulk queue and interactive user actions to
the interactive queue, and that enqueue_job stamps _queue_name accordingly — the
mechanism that keeps a backfill burst from starving sends/flag changes.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
- Heartbeat JSON now carries {owned, sockets_open, tiers:{hot,warm,cold}, ts} per
§2 so peers and operators can see each worker's load and socket footprint.
- run.py runs the sync coordinator directly (not via a placeholder arq worker) so
SIGINT/SIGTERM can stop the claimer, cancel runners, and release every lease
explicitly within a 10s drain budget — peers adopt the accounts immediately
instead of waiting out the lease TTL. Removed the sync:control-queue hack and
the supervisor indirection (start/stop_supervisor, sync_on_startup/shutdown).
- Acceptance: graceful-shutdown test proves a peer adopts released accounts within
seconds under a 30s TTL (i.e. via release, not expiry).
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
- concurrency.loop_semaphore: per-event-loop semaphores (safe across pytest loops
and process restarts via a WeakKeyDictionary keyed by the running loop).
- provider: holds a per-account (≤4) and per-host (PER_HOST_CONN_CAP) slot for the
whole lifetime of an IMAP socket, released on close; SMTP send takes a per-host
slot for its duration. A worker can no longer open hundreds of sockets to one
host/account.
- indexer: parsing goes through a PARSE_CONCURRENCY semaphore, and a message over
STREAM_PARSE_MB is parsed from its blob file (parse_message_file) instead of an
in-memory tree, so peak RAM ~= PARSE_CONCURRENCY x STREAM_PARSE_MB regardless of
mailbox sizes. Backfill parallelism stays bounded by the bulk queue only.
- config: PER_HOST_CONN_CAP, PARSE_CONCURRENCY, STREAM_PARSE_MB.
- tests: per-account cap blocks the 5th connection / distinct accounts independent;
file-vs-bytes parse equivalence + large-message streaming routing.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Each owned account is tiered by the coordinator; the runner shapes its work to
match — correctness identical, only latency/socket cost differ.
- hot: persistent INBOX IDLE + secondary poll every POLL_WARM.
- warm: no persistent socket; transient INBOX+special poll every POLL_WARM.
- cold: same transient poll every POLL_COLD.
- engine: poll_all_loop (transient full poll); secondary_poll_loop interval is
now a parameter; POLL_INTERVAL removed in favor of POLL_WARM/POLL_COLD.
- coordinator: tier eval every 120s from session heat (sess:active:{user}) +
recent mail (max internal_date within TIER_HOT_ACTIVITY_H), with the
HOT_BUDGET_PER_WORKER guard shedding least-recently-active session-less hot
accounts; user.active event promotes a user's accounts to hot at once (≤5s).
- API: login + WS connect/ping publish user.active and refresh sess:active
(TTL TIER_SESSION_GRACE_MIN). New sync:tier:{id} key, default warm on claim.
- config: POLL_WARM/POLL_COLD/TIER_*/HOT_BUDGET_PER_WORKER.
- tests: 5 tier tests (session/recent-mail promotion, cold, user.active,
budget demotion) alongside the lease/lifecycle acceptance tests.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
- Coordinator subscribes to account.state_changed over Redis pub/sub: going
live marks the account claimable at once; disabled/deleted/error tears the
owning runner down immediately (≤30s) instead of waiting for the lease TTL.
- delete_account now publishes account.state_changed:deleted after dropping the
account from sync:live, so teardown is event-driven, not just fencing.
- AccountRunner gains a verify loop (defense in depth): it periodically confirms
the account is still live and flags itself dead if not; the coordinator reaps
dead runners and releases their lease. Zombie loops are structurally impossible.
- New core event user.active (SPEC §8.3), published on login and on every WS
connect; consumed for tier promotion in step 5.
- Acceptance: added event-driven teardown test (isolated from fencing/verify).
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Ownership of an account now moves through a Redis lease so any number of sync
workers can run against the same account set with no double-IDLE.
- leases.py: sync:live registry + sync:lease:{id} (SET NX EX) with owner-checked
Lua renew/release; mark_live / mark_not_live helpers.
- coordinator.py: SyncCoordinator claims from sync:live up to MAX_ACCOUNTS_PER_
WORKER, renews every LEASE_RENEW (fences the runner on lease loss), reconciles
sync:live against Postgres, heartbeats a worker key, and rebalances when it
holds well above a fair share.
- supervisor start/stop now drive a coordinator; engine._set_state and
delete_account maintain sync:live membership (+ lease drop = fencing teardown).
- config: MAX_ACCOUNTS_PER_WORKER, LEASE_TTL, LEASE_RENEW.
- tests: acceptance 1-3 (single ownership, crash failover, create/delete
lifecycle) driving the real coordinator + Redis with faked IMAP loops.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Backend
- Sent copies: persist locally on send + periodic multi-folder CONDSTORE poll
(Sent/Archive/Trash/Junk) so they thread and show without an IDLE-only INBOX.
- Bcc: transmitted copy omits Bcc, Sent copy keeps it; store/display bcc_addr.
- Drafts: mirror local drafts to the IMAP Drafts folder (append/replace/expunge);
inbound sync of foreign drafts (un-threaded) with adopt-to-edit + dedup by UID.
- Per-user storage quota (5 GiB default) and manual "sync all folders" action.
- Account editing (display/color/server/password) and per-account HTML signature.
- Account view scoped to that account's inbox; new migrations 0006–0009.
Frontend
- Mail-list pagination, folder filters (All/Unread/Pinned), sort toggle, refresh.
- Aligned + widened mail-list margins; reading-pane bars fixed height.
- Rich-text composer: color palette, highlight, image; attachments; expand/pop-out.
- Signatures: auto-insert for new mail, replace on From change, reply insert.
- Restore/Delete actions per folder; drafts view merges local + remote, date-sorted.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
lukasandClaude Opus 5.5