Commit Graph
52 Commits
Author SHA1 Message Date
lukasandClaude Opus 5 ff64a5dfe7 Deploy: create the data directories instead of assuming Docker will
"Bind mount failed: '/volume1/docker/webmail/data/postgres' doesn't exists" on
Synology after the data directory was removed. Docker on ordinary Linux creates
a missing bind-mount source silently; DSM's daemon refuses and fails the
container. Every service mounted its own subdirectory, so every one of them
needed to already exist — including the `init` service whose whole job was to
create them, which could therefore never run.

- init now mounts DATA_ROOT itself, once, and creates postgres/blobs/backups/
  redis/meili/state inside it. Only DATA_ROOT has to pre-exist.
- postgres gained the init dependency it was missing, so its data directory is
  never demanded before init has made it.
- init only runs `chown -R` when ownership is actually wrong, rather than
  walking the whole blob store on every start.
- webmail.sh up creates DATA_ROOT and, if it cannot, says so with the mkdir to
  run rather than leaving the daemon to produce the error above.

Verified from an empty DATA_ROOT: init creates all six directories, postgres
starts after it, migrations apply, stack serves.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Nux3LCu4zP9nih4BRhPyHw
2026-09-14 20:41:09 +02:00
lukasandClaude Opus 5 8f96a1ecae Deploy: fix the initdb race that made migrate time out
On a real Synology first run, `migrate` exited 1 with a bare asyncpg
TimeoutError while the postgres log showed a perfectly healthy server.

The healthcheck was `pg_isready` with no -h, so it checked the UNIX SOCKET.
During first-time initialisation the postgres image runs a temporary server
that listens on the socket only (listen_addresses=''), so pg_isready answered
and the container was marked healthy while nothing was listening on TCP yet.
Dependents then connected to postgres:5432, hung, and timed out. It only bites
on the very first start of a fresh data directory, which is exactly the run an
operator is doing when they have least idea what is wrong.

- healthcheck uses `pg_isready -h 127.0.0.1 -p 5432`, so health means TCP is
  actually accepting.
- `migrate` additionally waits for the port itself (DB_WAIT_SECONDS, default
  180) before running alembic. It is a one-shot with no restart policy, so a
  few seconds of slow NAS disk should not fail the whole bring-up; the API and
  workers already self-heal through their restart policy.

Verified with a first-time bring-up against a fresh data directory: migrate
logs "database accepting connections", applies core + all four plugin
branches, and the stack comes up serving.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Nux3LCu4zP9nih4BRhPyHw
2026-09-14 20:31:03 +02:00
lukasandClaude Opus 5 d74e1f97b2 Deploy: make a missing .env name its own fix
With no .env found, compose fell back to a sentinel path and the daemon
reported: Bind mount failed: 'DATA_ROOT-is-unset/postgres' does not exist —
accurate, but it tells the operator nothing about what to do, and this is the
most common way the install goes wrong (the file is generated by `init`, not
shipped in the repo, so any re-copy of the folder loses it).

The sentinel is now /NO-.env-FOUND-RUN-deploy_webmail.sh-init-FIRST, so the
daemon's own error states the fix. doctor also checks for a missing or empty
.env before anything else and says the same thing.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Nux3LCu4zP9nih4BRhPyHw
2026-09-14 20:15:19 +02:00
lukasandClaude Opus 5 2f03bee45b Deploy: drop cpus: limits and stop assuming docker compose exists
Two NAS realities, both hit on a real Synology box.

- "NanoCPUs" error part-way through the project build: `cpus:` maps to the
  NanoCPUs field, which DSM's Docker daemon rejects. Removed from all three
  services (and from .env.prod.example). mem_limit uses the much older Memory
  field and works, so memory caps remain the throttle. Noted in the header
  next to the other DSM constraints.
- `docker compose build` on DSM: "'compose' is not a docker command". DSM and
  QTS ship the standalone `docker-compose` binary with no plugin subcommand,
  so every one of the script's 21 compose invocations was wrong for the target
  platform. The script now detects which is present, and only passes `--wait`
  when it finds v2 — on v1 it falls back to the health_gate it already had,
  which is also now run after `up` on both paths.

That the build reached container creation before failing says their compose
parses the rest of the file (anchors, profiles, service_completed_successfully,
top-level name), so this was a daemon-level rejection, not a schema one.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Nux3LCu4zP9nih4BRhPyHw
2026-09-14 20:09:32 +02:00
lukasandClaude Opus 5 f3dcd938b1 Deploy: fix the two things that broke the Container Manager build
DSM reported "invalid interpolation format for x-backend-env.MASTER_KEY" and
"required variable MASTER_KEY is missing a value". Both were mine.

1. DSM's bundled Compose cannot parse `${VAR:?message}`. Those guards are gone;
   interpolation now sticks to `${VAR}` and `${VAR:-default}`, the forms every
   Compose understands. Bind mounts keep a `${DATA_ROOT:-/DATA_ROOT-is-unset}`
   sentinel so a missing config still cannot scatter directories across the
   root of the NAS filesystem, and the header says to keep it this way.
2. Config was at docker/.env.prod with a `.env` symlink beside it. Container
   Manager did not pick that up. The real file is now `.env` in the project
   folder — what every compose front end reads without being told — and
   docker/.env.prod becomes a symlink to it. `init` and `require_env` migrate
   an existing install in place, keeping its MASTER_KEY.

Also, while testing the migration: `init` only generated secrets when creating
a file from scratch, so a migrated or hand-written .env with an empty
POSTGRES_PASSWORD started Postgres in trust-auth mode. It now tops up empty
secrets — and never touches a MASTER_KEY that already has a value, since
replacing it would make every stored credential undecryptable. doctor flags an
empty password too.

Verified: parses cleanly with no .env at all (the failure DSM hit), migration
preserves an existing key, and `docker compose up -d --wait` from the project
folder brings the stack up healthy and serving.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Nux3LCu4zP9nih4BRhPyHw
2026-09-14 19:49:03 +02:00
lukasandClaude Opus 5 5c444a6fd3 Deploy: complete the compose flattening (paths + script pointer)
Fixes the previous commit, which went out incomplete: a stale pathspec made
its `git add` abort, so the file move landed without the edits that make the
moved file work.

- docker-compose.yml: build contexts `..` -> `.` and the backup sidecar mounts
  ./docker/backup.sh, both now correct for a file at the repo root. Header
  explains why it lives here and to keep newer compose features out of it.
- deploy/webmail.sh: COMPOSE_FILE points at the root file, not the deleted
  docker/docker-compose.prod.yml.

8ddac22 on its own could not start: every build context and the backup mount
pointed one directory too high, and the control script referenced a file that
no longer exists.

Re-verified after this commit with `docker compose up -d --wait` from the repo
root: all services healthy, app serving, backup sidecar resolving its script.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Nux3LCu4zP9nih4BRhPyHw
2026-09-14 19:37:58 +02:00
lukasandClaude Opus 5 8ddac220b8 Deploy: flatten the prod stack into one root compose file
Synology Container Manager rejects the compose file with "Property include is
not allowed": DSM bundles an older Compose than the one that introduced
`include:` (v2.20). The root docker-compose.yml existed purely so the NAS GUIs
would find production rather than the dev service stack, so using a feature
the GUI cannot parse defeated its only purpose.

docker/docker-compose.prod.yml is now the root docker-compose.yml itself — one
plain, flat file, no indirection. Build contexts go from `..` to `.`, the
backup sidecar mounts ./docker/backup.sh, and webmail.sh points at the new
path. Nothing else changed: same services, same project name webmail-prod,
same guards.

The header now says why the file lives at the root and to keep it free of
newer compose features, so this does not get "tidied up" back into something
DSM refuses.

Verified with `docker compose up -d --wait` from the repo root as the GUI
invokes it: all services healthy, account created, backup sidecar resolving
its script at the new path, and webmail.sh status/doctor/down driving the same
stack.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Nux3LCu4zP9nih4BRhPyHw
2026-09-14 19:37:23 +02:00
lukasandClaude Opus 5 7f6b76ec12 Deploy: explain the root-owned env file instead of leaking "Permission denied"
init runs as root on a NAS, so docker/.env.prod is mode 600 owned by root and
the data directories are root-owned. Running any subcommand as a normal user
produced a bare redirect error naming an internal line number, with no hint
about the cause.

require_env now checks readability first and says who you are running as, the
exact sudo command to repeat, and that `sudo -i` avoids it for the session.
Docs note it beside the show-key step and above the command reference.

Reproduced with a root-owned 600 env file read as a normal user; the message
is what that case now prints.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Nux3LCu4zP9nih4BRhPyHw
2026-09-14 19:29:22 +02:00
lukasandClaude Opus 5 24031f1daa Deploy: make a no-SSH install possible from the DSM GUI
The install guide was SSH-first and left the GUI as a footnote, but the two
things it could not do were both fixable rather than inherent.

- `init` takes --data-root / --domain / --tz / --port / --bind, so configuring
  is one command instead of hand-editing a dotfile that File Station hides by
  default. Re-running with flags updates an existing .env.prod.
- `show-key` prints MASTER_KEY, so it can be backed up without opening that
  hidden file — intended to be redirected to a file, copied into a password
  manager, and deleted.
- usage() printed a hardcoded line range that spilled into the code once the
  header grew; it now prints the leading comment block however long it gets.
- docs §2.5 is a complete GUI walkthrough: Gitea "Download ZIP" + File Station
  (with the chmod a zip download needs), two one-off Task Scheduler jobs for
  config and key backup, then the Container Manager project on the repo root
  named webmail-prod. Plus a table for running update/rollback/backup/restore
  /doctor/reindex as Task Scheduler jobs, and why the UI's own Build action is
  not a substitute for `update` (no pre-backup, no health gate, no rollback).

Verified by running the documented sequence verbatim: flag-driven init,
show-key, then `docker compose up -d --wait` from the project folder as the
GUI does — all services healthy, account created, backup and doctor tasks
working against the result.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Nux3LCu4zP9nih4BRhPyHw
2026-09-14 19:18:41 +02:00
lukasandClaude Opus 5 c472404486 Add stray editor backup of the document icon
Committed on request as part of "commit everything". This looks like an
editor save-a-copy artefact from 2026-07-26 rather than an intended asset:
nothing references the filename, and it differs from the document.svg it sits
beside (1426 vs 420 bytes). Safe to revert this commit alone if so.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Nux3LCu4zP9nih4BRhPyHw
2026-09-14 19:08:35 +02:00
lukasandClaude Opus 5 07fd0a7adf Docs: add the deployment/release system build brief
The Gitea-hosted CI/CD brief (SemVer tags, Actions pipelines, registry,
deploy/rollback scripts, operator runbooks). Written before the NAS work and
kept as-is; note that §5.2's pgBackRest/ZFS snapshot step does not apply to a
Synology or QNAP target, where docs/nas-install.md substitutes verified
pg_dump plus Btrfs snapshots and Hyper Backup/HBS3.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Nux3LCu4zP9nih4BRhPyHw
2026-09-14 19:08:35 +02:00
lukasandClaude Opus 5 cc26ae7eaf Deploy: make the NAS GUI path work, and stop sourcing the env file
Synology Container Manager and QNAP Container Station build a project from a
folder and look for `docker-compose.yml` in it. Pointing either at `docker/`
finds the DEV service stack and starts the wrong thing, so production now has
a root-level entry point.

- docker-compose.yml at the repo root: `include`s the prod stack (relative
  paths resolve against the included file's directory, so the build context
  and backup.sh still resolve) and pins `name: webmail-prod`. A project name
  typed into the GUI overrides the file's, so the docs say to type the same
  one.
- webmail.sh init now drops a `.env` symlink to docker/.env.prod in the repo
  root: the GUI reads `.env` from the project folder, and a symlink keeps one
  file to edit instead of two copies that drift.
- Required-variable guards on DATA_ROOT / MASTER_KEY / POSTGRES_PASSWORD /
  APP_DOMAIN. A bare `docker compose up` with no env file used to interpolate
  DATA_ROOT to "" and bind-mount /postgres at the root of the filesystem; it
  now aborts with a message naming the variable.
- require_env no longer sources the env file. Sourcing executed it as shell:
  ENABLED_PLUGINS=calendar contacts labels filters ran `contacts` as a command
  and broke every subcommand, and anything else in the file that holds the
  master key would have been executed too. Replaced with a compose-compatible
  KEY=VALUE parser; the template quotes the value as well.
- doctor: no longer aborts on a not-yet-created DATA_ROOT (set -e + pipefail
  killed it inside a `df` assignment — on exactly the pre-install run where
  its NFS/SMB check is the one that matters), and probes the nearest existing
  ancestor so that check works before the directory exists.
- docs §2.5 rewritten with what the GUI can and cannot do: no compose
  profiles (so no Meilisearch), no health-gated update, no restore; plus the
  Task Scheduler route for generating MASTER_KEY without enabling SSH, and
  why init/migrate showing "Exited (0)" is correct.

Verified by running `docker compose up -d --wait` from the repo root with no
-f and no --env-file (what the GUI does): all services healthy, migrations
applied, app served, and deploy/webmail.sh manages the same project.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Nux3LCu4zP9nih4BRhPyHw
2026-09-14 19:03:42 +02:00
lukasandClaude Opus 5 5553b5febe Deploy: production Docker stack for a single host (Synology/QNAP NAS)
Everything needed to run the app on an Intel NAS that is also doing other
jobs: one published port, no host networking, memory/CPU caps and log
rotation on every container, and all state in bind mounts under DATA_ROOT
so mail survives image updates, project deletion in the NAS UI, and full
reinstalls — and so Hyper Backup/HBS3 can actually see it.

- docker/Dockerfile.backend: multi-stage, non-root (uid 10001), tini for
  the <10s graceful lease release. Installs the four plugins as entry-point
  distributions. One image, several roles via docker/entrypoint.sh
  (api | worker-sync | worker-jobs | migrate | reindex | plugins).
- docker/Dockerfile.frontend + nginx.conf: SPA + /api reverse proxy with
  the WebSocket upgrade for /api/ws, and X-Forwarded-Proto preserved so
  the API still emits __Host-/Secure cookies when TLS is terminated by
  the NAS reverse proxy (identity/cookies.py keys off request scheme).
- docker/docker-compose.prod.yml: project name webmail-prod, NOT webmail —
  docker-compose.yml (dev services) owns that name, and two compose files
  sharing one project name fight over each other's containers.
  Meilisearch is an opt-in profile; SEARCH_BACKEND=pg is the default so a
  small NAS pays nothing for search.
- migrate runs as a one-shot (core Alembic head + every plugin branch)
  that everything else waits on, so a bad migration can never leave new
  code running against an old schema.
- docker/backup.sh: nightly pg_dump verified with `pg_restore --list`
  before it counts as a backup, with retention and a baseline on first
  start.
- deploy/webmail.sh: init/doctor/up/down/status/logs/update/rollback/
  backup/restore/reindex/psql/shell. `update` is backup -> images ->
  migrate -> roll workers, api, web -> health-gate, with automatic
  rollback to the previous version if the gate fails. `doctor` refuses to
  be quiet when DATA_ROOT is on NFS/SMB (silent Postgres corruption).
- deploy/build-push.sh: build images off-NAS (the frontend build wants
  ~2GB RAM) and push to Gitea, or --save a tar to carry over.
- scripts/reindex_search.py: rebuild the search index from Postgres alone.
  Every SearchDoc field is already stored, so the index is derived data
  that never needs backing up — this closes the only real objection to
  running Meilisearch on a box whose volumes may be wiped.
- pyproject: package-data for accounts/presets.yaml and the shared plugin
  Alembic env. Without it a non-editable install (i.e. any image) silently
  ships a package that fails at runtime.
- GET /api/version, used by the deploy health gate.

Verified end to end on a real stack: fresh install, all services healthy,
core + 4 plugin migration branches applied, login/SPA/plugin bundles
through nginx, WS upgrade returning 101, __Host- cookies under a
simulated TLS proxy, backup -> restore round-trip preserving user and
plugin schema, failed-update auto-rollback (both a bad migration and a
boot-crashing API), and Meilisearch profile serving a live query.
ruff + mypy --strict clean, 363/363 backend tests pass.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Nux3LCu4zP9nih4BRhPyHw
2026-09-14 18:53:50 +02:00
lukasandClaude Opus 4.8 a27755a4d3 Calendar: CalDAV two-way sync (iCloud/generic) + all-day timegrid fix
CalDAV account connect (Settings → Calendars) with RFC 6764 discovery,
per-calendar vaulted credentials, and full two-way sync verified live
against NextCloud:

- caldav.py: discovery, calendar-query REPORT (pull), PUT/DELETE with
  quoted ETags (If-Match/If-None-Match), iCloud partition-redirect
  handling, ICS build/parse; clear error codes (auth/private-host/
  unreachable) surfaced to the connect UI.
- sync.py: ctag-gated pull + push of local changes; dirty rows protected
  from being clobbered; 15-min cron + immediate enqueue on every
  create/edit/delete so changes reach the server in ~1s.
- Dirty tracking via a BEFORE INSERT/UPDATE trigger (migration 0007),
  suppressed during sync with a txn-local GUC; deletes tombstone then
  DELETE server-side. Writability now driven by server privileges.
- migration 0006: event uniqueness NULLS NOT DISTINCT — fixes duplicate
  events on every full re-pull (NULL recurrence_id never matched ON CONFLICT).
- 5 CalDAV parsing unit tests.

Calendar UI (page.js): all-day events now render in Day/3-days/Week views
(were month-only). They share the timed-event column layout so overlapping
events sit side-by-side, fill the day column vertically with a sticky label,
and are matched by start day (fixes UTC-midnight events spilling into the
next local day).

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
2026-08-08 10:27:21 +02:00
lukasandClaude Opus 4.8 719eeef378 Backfill start-window indicator, account-delete spinner, app-password Help popup
- Backfill: show the sidebar spinner + "Preparing…" banner during the brief window
  after account creation when state=backfilling but total is not yet counted
  (total===0), so a freshly-added account shows progress immediately instead of
  looking done until a refresh. The stuck-at-100% guard is preserved.
- Settings: a centered spinner overlay ("Removing account…") covers the screen from
  confirm until the (slow) server cascade + accounts refetch complete, so the row
  no longer appears frozen before it disappears.
- Add-account wizard: the manual password hint now includes Outlook and gains a
  "Help" link opening a popup with per-provider direct links to enable 2-step
  verification and generate the app password (Gmail, Outlook, Yahoo, iCloud).

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01U72FZiF1W1pb8XxyWBTREA
2026-08-07 07:59:26 +02:00
lukasandClaude Opus 4.8 12a67825cc Sender avatars, PDF password preview, contacts autosave, account-view + Trash/Spam auto-purge, UI polish
Sender avatars (privacy-safe, server-fetched, cached):
- New GET /api/avatar resolves Gravatar (d=404) then the sender domain's favicon
  under the SSRF guard, Redis-cached (hit+miss); gated by SENDER_AVATARS_ENABLED.
  GuardedResponse gains raw bytes. Shown in the mail list, reading pane, People
  panel, and Contacts (overlay that falls back to initials on a miss).

PDF preview:
- Password-protected PDFs now prompt in-place and retry (pdf.js PasswordException),
  with a clear error for other load failures instead of a blank overlay. The
  password is remembered per attachment until the thread is left; clicking the
  field no longer dismisses the overlay.

Contacts:
- Autosave reply recipients: message.sent now carries to/cc (name+email); the
  contacts plugin subscribes and adds any non-contact recipients to the Autosaved
  group. Contact avatars show the gravatar/favicon via /api/avatar.

Mail views / retention:
- Clicking an account shows all its mail except Trash/Spam (was inbox-only).
- Daily cron autopurge_trash_spam permanently deletes Trash/Spam older than
  TRASH_RETENTION_DAYS (default 30; 0 disables) — server EXPUNGE + local rows.

UI polish:
- Ultimo signet logo in the Mail/Calendar/Contacts sidebar header and collapsed
  rail; consistent 11px top margin and vertically-aligned header row across states
  (fixed-size flex icon buttons); SVG favicon on the browser tab.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01U72FZiF1W1pb8XxyWBTREA
2026-08-06 19:23:27 +02:00
lukasandClaude Opus 4.8 4c200ab94a Gmail label sync + no-dup backfill, live label/UI updates, account-delete cleanup, Sync-SPAM toggle, sidebar logo
Gmail sync (SPEC: Gmail sync):
- Provider fetches X-GM-LABELS when X-GM-EXT-1 is advertised; parses custom labels
  (system labels dropped, mUTF-7 decoded) onto RemoteMessage.
- Engine syncs only system folders + All Mail for Gmail, skips every label folder,
  and dedupes the All Mail pass by Message-ID — kills the INBOX/All-Mail duplication.
- Custom labels ride the message.new event; the labels plugin subscribes and maps
  them to chips as "<name> [Gmail]" (per-message + thread aggregate + known set),
  so labels fill in gradually with the sync window. Non-ASCII labels decode right.

Labels self-healing (message-driven reconcile):
- Per-message keys are the source of truth; t: aggregates and _labels are derived.
  Reconcile drops labels of deleted messages, regroups survivors by their current
  thread (fixing JWZ thread merges), rewrites changed t: keys, and rebuilds _labels
  from live messages — keeping user labels, dropping orphaned Gmail ones.
- New SDK reads: sdk.threads.existing / sdk.messages.threads_of. Sidebar refreshes
  live via a plugin.labels.changed WS event on new labels.

Account deletion: prune threads left empty after the cascade (they were user-scoped,
so only messages cascaded) + de-list the account from surviving cross-account threads.

Sync SPAM: opt-in per-account sync_spam (migration 0015, default off) — the engine
skips the Junk folder in backfill and poll unless enabled; wizard checkbox added.

Robustness / UX:
- WebSocket auto-reconnects with backoff and resyncs on (re)connect, and account
  add/delete invalidates threads/labels — no more hard refresh for live updates.
- Backfill banner + sidebar ring also require done < total, so a completed account
  can't leave a stuck "X of X".
- Scrub lone UTF-16 surrogates from parsed headers/bodies so bad messages no longer
  crash the backfill on Postgres text/JSONB.
- Ultimo signet logo in the Mail/Calendar/Contacts sidebar header (and at the top of
  the collapsed rail), via the shared plugin-ui.css + MailLayout.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01U72FZiF1W1pb8XxyWBTREA
2026-08-05 18:04:49 +02:00
lukasandClaude Opus 4.8 e5a87b37b1 Gmail (app-password) onboarding: auth_strategy routing + credential-helper panel
Implements SPEC v1.5 §6.2.1–6.2.3 so Gmail (and Yahoo/iCloud/Outlook-personal)
onboard via a provider app password on the existing plain-password sync path —
no OAuth, no new sync code. Gmail was oauth2-only with OAuth unimplemented, so it
was previously un-addable.

Backend:
- presets.yaml: each preset declares an ordered `auth_strategy`; Gmail becomes
  `auth: password` + `[app_password, oauth2]`. App-password providers ship a
  data-driven `help:` block (deep_link, app_name_hint, i18n step keys, docs_url).
- discovery/types: CandidateConfig carries `auth_strategy` + `HelpBlock`; presets
  load them and autoconfig candidates derive a default from the detected auth.
- schemas/api: CandidateOut exposes auth_strategy + help; DiscoverResponse adds
  `google_oauth_enabled`.
- config: GOOGLE_OAUTH_ENABLED flag (default false) gates the OAuth fallback.

Frontend:
- ProviderCredentialPanel (wizard step 1.5): deep-link popup (user gesture),
  rendered steps, single paste field that strips whitespace, instant verify
  (auto POST /accounts/test at 16 chars → Connected ✓ / mapped error), and the
  auth_failed recovery branch with a gated "Sign in with Google instead" button.
- AccountWizard routes app-password providers to the panel, then a details step.
- i18n keys (en) for the panel + provider steps.

Honors the §6.2.2 prohibitions: no page automation, no real-password capture,
no scraping — deep-link + guide + paste + test only. Adds the v1.5 spec.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01U72FZiF1W1pb8XxyWBTREA
2026-08-04 19:59:52 +02:00
lukasandClaude Opus 4.8 0f686bbd5c Calendar/Contacts/Mail: pill resize handles, day hover, locale-safe date picker
- Panel resize handles: replace the tall bar with a small centered pill/capsule
  (elevated fill + border, accent on hover) across Mail, Calendar and Contacts.
- Calendar month/week/3-day views: brighten a day block on hover.
- Calendar month grid: shade the current month's days theme-aware — a subtle dark
  overlay in light mode (they were only distinguished in dark mode before), white
  overlay in dark mode.
- Calendar event form: replace the native <input type=date> (whose order follows the
  browser locale) with a custom field driven by the 24h/12h preference — an editable
  text box (typed dd/mm/yyyy for 24h) backed by a hidden ISO input, plus a popup
  month picker. Applies to the start date and the recurrence "Ends on date".

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01U72FZiF1W1pb8XxyWBTREA
2026-08-04 19:23:44 +02:00
lukasandClaude Opus 4.8 0a478a6b26 Labels in Mail UI (chips + Actions + sidebar), plain-text Lato, calendar-panel polish
Labels:
- Moved labelling out of the "+label" message slot into Mail: label chips on the
  thread subject bar (removable ✕) and before list subjects, and a Labels submenu
  in the per-message Actions menu (multi-select checkboxes + "Create new…").
  Per-message storage kept; the reading pane writes a thread-union aggregate so the
  list (no message ids) can show chips.
- Sidebar LABELS section rewritten: aligned monocolor tag rows, collapsible
  (collapsed by default), inline "+" create popover, hover "−" delete with a confirm
  popover, and click-to-filter the mail list by label.
- Backend: batch message/thread label reads, thread aggregate, create + delete
  (delete unhooks from all mail), and GET /labels/{name}/threads. Core GET /threads
  gained an `ids` param to power the label view. Removed the plugin-pages list from
  the Mail sidebar.

Plain-text mail now renders in Lato (self-hosted at /fonts, @font-face'd into the
sandboxed message iframe) instead of the monospace <pre> default.

Calendar right-panel (embed mode): restored the "…" view-options menu, pinned the
toolbar + mini-calendar (only the agenda scrolls), plain "All calendars" button
(inline resets), and the event popover now closes on any outside click.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01U72FZiF1W1pb8XxyWBTREA
2026-08-03 20:53:35 +02:00
lukasandClaude Opus 4.8 1d1a621ce2 Mail right panel: Calendar / People / Attachments (shared calendar forms)
New slide-out right panel in Mail (hidden by default, toggle in the reading pane,
resizable, persisted). Three tabs:
- Calendar: mounts the calendar plugin bundle in a new embed:"panel" mode, so the
  agenda, mini-month, event popover, event form and search results are the calendar
  app's OWN code — future changes propagate to both. Events click → real popover;
  New event / Edit / "Invite all" open the shared event form in the panel; search
  shows the calendar's results panel; All-calendars filter + open-in-new-tab.
- People: conversation senders / contact books (All/Recent/VIPs/Autosaved) drilling
  into a contact detail (Call/Message/Compose/Invite, related mail, find-all);
  "Compose to all" / "Invite all to new event".
- Attachments: every attachment in the thread, grouped by sender, with download and
  clickable previews (reuses the mail AttachmentPreview overlay for supported types).

Calendar bundle: mount(el, ctx, props) reads embed/newEventGuests; renderEmbed() +
panelBar/panelFoot; embed-aware agenda (TODAY); form/popover repositioned for the
narrow panel; guest deep-link accepts comma-separated emails.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01U72FZiF1W1pb8XxyWBTREA
2026-08-03 08:38:44 +02:00
lukasandClaude Opus 4.8 421b44a8ab Mail mark-all-read + iframe fixes; Contacts detail/selection/groups + resizable panels
Mail:
- New "More" (⋯) menu in the list sorting bar with "Mark all as read": POST
  /threads/mark-read marks every unread thread in the current view (unified /
  folder:<special> / account:<uuid>) as \Seen, pushes + publishes updates.
- Message iframe: no longer shows a spurious vertical scrollbar (display:block,
  getBoundingClientRect+ceil sizing, overflow-y:hidden on the doc) and has
  rounded corners.
- Deep links from Contacts: /mail?compose=1&to= opens a maximized prefilled
  composer; /mail?q= prefills the search box.

Contacts:
- Detail pane shows all populated fields (work subtitle, phones, address, urls,
  nickname, birthday). Company name is used as the display name when there's no
  personal name (existing rows too). Editable "Add notes".
- Selection: checkbox reveals on avatar hover only; Groups/More bulk menus in
  select mode; "N contacts selected" panel; filled yellow VIP star in list +
  detail; pagination bottom bar; Lato pinned on form controls.
- Groups: "+" on the GROUPS header and "Create new" in the group menus (bulk
  add/remove via POST /groups/{id}/add|remove, ownership-guarded).
- Rounded Mail-style list rows; removed the ⋯ button from the search bar;
  Compose→Mail, Find-all-conversations→Mail search; Message button removed;
  delete confirmation; resizable sidebar + list/detail split.

Calendar: the right panel (agenda / search) is now resizable.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01U72FZiF1W1pb8XxyWBTREA
2026-08-02 21:34:22 +02:00
lukasandClaude Opus 4.8 a393939628 Contacts: full detail pane, search, selection UX, empty illustration; Calendar invite deep-link
Contacts plugin:
- Detail pane now renders ALL populated fields (job/company subtitle, every email,
  phones, addresses, websites, nickname, birthday, notes) — not just the email.
- Company fallback for the display name: personal name → company → email, applied at
  read time so existing contacts whose name_display holds only the email now show the
  company without a re-save; name_display no longer stores emails going forward.
- Search box in the list top bar (like Calendar): debounced, matches name/email/company,
  keeps focus + caret across re-renders.
- Mail-style selection: hovering a row's avatar reveals a square checkbox; once anything
  is checked every row shows one (select mode), the bar "select all" matches and aligns
  with the row checkboxes, and the detail pane shows "N contact(s) selected".
- Delete contact now confirms via a modal.
- Empty reading pane shows contacts-illustration.svg (served from the plugin's ui/ dir),
  mirroring Mail's email-illustration empty state.

Calendar:
- Invite from a contact opens /p/calendar?guest=<email>, which pops a new-event form with
  that guest already added, then strips the param.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01U72FZiF1W1pb8XxyWBTREA
2026-08-02 11:57:39 +02:00
lukasandClaude Opus 4.8 a985b112b1 M5 Contacts plugin (builtin) + collapsed icon-rail for Calendar
Contacts (SPEC Appendix B, phase B1) — new builtin plugin `webmail_plugin_contacts`:
- Tables plg_contacts_{books,contacts,groups,group_members} (RLS-isolated,
  migration contacts0001): vCard-style structured contact fields, address books,
  groups with membership.
- API /api/plugins/contacts/: list (filter all|vip|recent|nogroup, group, q),
  get/create/update/delete, star toggle, groups CRUD, and the recipients.suggest
  capability endpoint (enriches the composer). Display name derived server-side
  from name parts so search + suggestions work regardless of client.
- Frontend ui/page.js: three-pane app — shared .wm-side-* sidebar template with a
  "Contacts" header, alphabetically grouped list, detail pane (avatar, VIP star,
  Call/Message/Compose/Invite, Contact details, live Mail search, Events stub) and
  a full edit form (name, org, birthday, repeatable emails/phones, address, notes).

Collapsed sidebar: both Contacts and Calendar now show a 60px icon-only rail (like
Mail) with expand toggle, primary button, quick nav and Settings/Sign out — instead
of hiding the panel (Contacts) / a top-bar button (Calendar).

Deferred: Gravatar photos, vCard import/export + merge (B2), CardDAV sync (B3),
"Add sender to contacts".

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01U72FZiF1W1pb8XxyWBTREA
2026-08-02 10:12:45 +02:00
lukasandClaude Opus 4.8 57a1f1562f M4 Calendar: reminders form, invitations, short publish links, settings UX
Calendar settings (Settings > Calendars):
- Reminders are now {method, minutes}: Add-reminder opens a boxed form with a
  method dropdown (Show an alert / Email me) + time dropdown, bell icons.
  Backend normalizes the legacy list[int] transparently.
- New Invitations section (auto-add emailed events): policy radios, "Add new
  events from" (My contacts / Anyone), "Add to calendar", mark-read+file toggle.
  Persisted to user settings (inbound iMIP processing still deferred).
- Time zone options show GMT codes, e.g. "Central European Summer Time (GMT+2)".
- Default calendar now defaults to "Personal" instead of a generic placeholder.
- Colour picker is a circular-swatch-only dropdown (sized so swatches don't overlap).
- Delete calendar / remove subscription now confirm via a modal.
- Create form: dropped the breadcrumb; heading stays "Create calendar" even after
  a Publish toggle auto-creates the calendar.

Publish links: replaced the 244-char vault-sealed token with a 16-char random
slug (secrets.token_urlsafe). New sdk.public_session() (privileged, non-RLS) lets
the public /public/{token} feed resolve the slug directly with explicit filters.

Settings shell: "<- Mail" is now a context-aware "<- Back" via ?from=; Calendar
opens Settings with from=/p/calendar so Back returns to the calendar.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01U72FZiF1W1pb8XxyWBTREA
2026-08-02 08:48:33 +02:00
lukasandClaude Opus 4.8 e2bef3c41d M4 Calendar: publish public .ics links, subscriptions, multi-calendar settings
Calendar (builtin plugin):
- Publish read-only public .ics feed per calendar via vault-sealed token
  (full-detail or free/busy-only); unauthenticated GET /public/{token},
  404 on bad/absent token. Migration calendar0004 adds publish_token,
  publish_busy, subscription_url + partial unique index.
- Subscribe to external iCal (.ics) URLs (SSRF-guarded) as read-only
  calendars; CSRF-guarded POST /subscriptions.
- Settings > Calendars: create/edit/delete, publish toggles with live
  shareable link + copy, Add-account stubs (Google/iCloud/CalDAV, deferred),
  Subscriptions section; default calendar + default reminders; auto-detected
  time zone; color-dropdown contrast fix; 3x-larger edit arrow.
- Left panel: shared Mail-parity sidebar template (.wm-side-*), calendar
  search panel, font/spacing tuning, per-section "+" create shortcut.

Frontend: CalendarOut publish fields, api publish/subscribe, settings keys.
Backend: settings default_calendar + reminder lists.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01U72FZiF1W1pb8XxyWBTREA
2026-08-01 20:22:53 +02:00
lukasandClaude Opus 4.8 8c634f7967 M4: Calendar plugin (builtin) + mail HTML/attachment fixes
Calendar plugin (webmail_plugin_calendar, non-removable core plugin):
- Local calendars/events CRUD with RLS isolation; recurrence expanded at
  query time (DST-correct), EXDATE + detached-override merge.
- Calendar app bundle: month/week/day/3-day views, mini-month, event form
  (color, show-as, timezone combobox, repeat, reminders, guests w/ recipient
  autocomplete, Jitsi video call), event popover with RSVP badges, resizable/
  collapsible left panel + theme toggle, overlap layout.
- iMIP invites: sdk.mail sends REQUEST from the organizer's mailbox (HTML +
  text/calendar + invite.ics); public /rsvp endpoint (vault-sealed token)
  records accepted/declined/tentative per attendee.
- "This and following" edit + scoped delete (this / this-and-following), with
  a delete confirmation dialog for recurring events.

Core:
- Settings: time_format (12/24h, default 24h) + jitsi_server; new Integrations
  section; date order follows time format (en-GB vs en-US) across Mail.
- Plugin SDK: mail (send from a user's account), vault (envelope encrypt),
  multi-method SSRF-guarded http; AppSwitcher product switcher; full-bleed
  PluginPage.

Mail HTML rendering:
- Sanitizer allows <style> blocks + class (CSS sanitized: drop script-like,
  strip @import/url() when remote images blocked); SANITIZER_VERSION 3 so
  already-indexed mail re-sanitizes on view.
- Attachments: a Content-Disposition: attachment part is a real attachment
  even when it also carries a Content-ID (fixes hidden invite.ics).

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
2026-07-28 16:33:34 +02:00
lukasandClaude Opus 4.8 d46ad13b7b M3 plugin platform: frontend host, reference plugins, UI kit + docs
Completes M3 (SPEC §11 + §12 acceptance) on top of the backend platform (4e077f1).

Frontend host (§11.3):
- src/plugins/: manifest loader, runtime ESM bundle loader (framework-agnostic
  mount contract), PluginSlot, PluginPage (app.page at /p/*), plugin context
  (scoped api/events/i18n), capabilities hook. Wired slots into ThreadView
  (message.actions/message.banner), MailLayout sidebar (sidebar.section + page nav)
  and Settings→Accounts (settings.page); ws.ts fans out core + plugin.* events.
- plugin-ui.css: theme-aware .wm-* classes so bundles don't re-derive inline styles.

SDK additions:
- CurrentUserId + require_csrf (plugin-router auth), messages.envelope/header,
  actions.set_flags/trash, sdk.enqueue.

Reference plugins (plugins/, installed via webmail.plugins entry points):
- hello: skeleton page-plugin (app.page + plg_hello_note w/ RLS + cron job) —
  install→run→uninstall --drop-data leaves zero residue.
- labels: message.actions + sidebar.section slots, KV only, zero core diffs.
- filters: message.new hook -> match (subject/from/to) -> Mark read / Pin / Delete
  (delete = move thread to Trash via the core path).

docs/plugins.md: full authoring guide + a complete "Unsubscribe button" worked
example (the §12 "writable from docs alone" bar).

Also: conftest resets the arq pool between tests (loop-bound global, like engine/redis).

Gates: backend ruff + mypy --strict (107) + full suite 337; frontend tsc + eslint
+ vitest (11) + vite build.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Rcj3TpydMTzTSmU7kunzf8
2026-07-27 08:30:24 +02:00
lukasandClaude Opus 4.8 4e077f1a37 M3 plugin platform: SDK + loader, RLS-isolated tables, migrations, jobs
Backend foundation for the plugin platform (SPEC §11), Phases 1-3.

Phase 1 — SDK + loader:
- app/plugins/types.py: frozen Plugin/UiManifest/Slot/Page/JobSpec contract.
- app/plugins/sdk.py: sdk(name) facade — the only app.* import allowed for plugins
  (scoped session, plugin.{name}.* publish, blob, outbox, settings, SSRF http_get, KV, enqueue).
- app/plugins/loader.py: entry-point discovery (webmail.plugins), hook forwarding
  (API + worker), router mounting at /api/plugins/{name}, ui-manifest aggregation.
- plugin_kv core table (migration 0013) + GET /api/plugins/ui-manifest.

Phase 2 — non-removable core plugins + DB-enforced privacy:
- builtin flag (calendar/contacts will be builtin=True; uninstall refuses them).
- RLS per-user isolation on plg_* tables: least-privilege webmail_plugin role
  (migration 0014), plugin_session(user_id) sets LOCAL ROLE + app.current_user_id,
  rls.enable_rls() (ENABLE+FORCE RLS + USING/WITH CHECK policy). Mail untouched.
- Per-plugin Alembic runner (shared env, dedicated version table) + install/uninstall
  --drop-data CLI; skeleton fixture proves install -> run -> uninstall --drop-data
  leaves zero residue (§12 M3 acceptance).

Phase 3 — plugin jobs (§11.2):
- JobSpec registered as arq function plg.{plugin}.{func} + cron (5-field parser);
  singleton=True guarded by a per-(plugin,user) Redis lock; wired onto the shared
  interactive worker.

Gates: ruff, mypy --strict (94 files + tests), full suite 326 passed. RLS isolation,
migration round-trip, and singleton lock covered by tests.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Rcj3TpydMTzTSmU7kunzf8
2026-07-26 21:42:35 +02:00
lukasandClaude Opus 4.8 4d5a30cffb sync: QRESYNC VANISHED expunge detection (SPEC §6.3 M2 fast resync)
Server-side deletions now propagate to the DB. Previously poll() detected new
messages and CONDSTORE flag deltas but never expunges (the old TODO), so mail
deleted in another client lingered as phantoms outside the small Drafts reconcile.

- provider: enable QRESYNC when advertised (_maybe_enable_qresync + qresync_enabled
  property). poll() harvests VANISHED (expunged UIDs) and changed FLAGS in one pass
  via a raw `SELECT <mbox> (QRESYNC (uidvalidity modseq))` — issued as a low-level
  SELECT because aioimaplib only surfaces untagged VANISHED lines on a sync command,
  not on an async UID FETCH. CONDSTORE-only servers (e.g. Gmail) keep the CHANGEDSINCE
  flag delta and fall back to the engine reconcile.
- parsing: parse_vanished() for `VANISHED [(EARLIER)] <uid-set>` with range expansion.
- engine: apply_delta now drops expunged rows from the search index, un-links any
  local draft mirroring a vanished UID, and publishes message.deleted for the UI.
- tests: parse_vanished unit cases + apply_delta expunge (removal, draft un-link,
  modseq advance).

Verified live: the worker issues `SELECT "INBOX" (QRESYNC (uidvalidity modseq))`
against QRESYNC-capable accounts. ruff + mypy --strict clean; 296 tests pass.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Rcj3TpydMTzTSmU7kunzf8
2026-07-26 18:28:09 +02:00
lukasandClaude Opus 4.8 dd0b089049 attachments preview overlay, docx/xlsx/odf rendering, composer drag-drop & undo-send UX
Frontend:
- Full-screen attachment preview overlay (AttachmentPreview) for image/pdf/txt
  plus office docs: .docx via docx-preview (rendered live into the sandboxed
  iframe so header/footer VML images survive), .xlsx via xlsx-preview, and
  .odt/.ods/.odp/.odg via a JSZip+DOMParser ODF walker (officeConvert).
- Type-aware attachment icons (green spreadsheet, blue document, red PDF,
  image/archive); new document/image/spreadsheet/print SVGs.
- Composer: drag-drop attachments (body drop inserts images inline, else
  attaches); inserted images are selectable/resizable/deletable (RichTextEditor).
- Undo-send: "Sending…" banner with live "N seconds to Undo" countdown.
- Search result list: pin/archive/delete update correctly; grey-out only for
  search hits; scoped "Search in Trash/Spam".

Backend:
- Search: prefix (search-as-you-type) matching, email-address search, and
  address-first result ordering; Trash/Junk excluded with scoped folder search.
- Attachments: RFC 6266 Content-Disposition for non-ASCII filenames; one-time
  inline image loading.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Rcj3TpydMTzTSmU7kunzf8
2026-07-26 16:15:49 +02:00
lukasandClaude Opus 4.8 c023cfa15e search: email address search + address-first ranking; inline images; folder i18n; link security
Search (PG backend):
- '@' terms match address substrings (from/to/cc), so a partial email like
  "dyrektor@wfdif" resolves and narrows predictably as you type
- index each address split into parts, so a bare domain/local word ("wfdif")
  finds the address (indexed as one lexeme otherwise); dotted terms fall back to
  an address substring
- rank sender/recipient name+address matches above body/subject-only ones
  (addr_hit tier, then recency); cursor extended to keep keyset pagination correct
- scripts/reindex_address_parts.py backfills address tokens into existing rows

Inline images:
- extract attachments via a full MIME walk instead of shallow iter_attachments(),
  which missed inline images nested in mixed>related>alternative (cid: rendered
  broken); has_attachments counts only real (non-cid) attachments
- scripts/backfill_inline_attachments.py stores previously-missed inline images

Folder-name i18n (SPEC §12 M2):
- accent/punctuation-insensitive, multi-delimiter (/ \ .) special-use inference
  across ~13 languages; curated names (no substring false positives)

Link security (reading panel):
- links never navigate the message iframe: a parent-side click interceptor opens
  http/https in a new tab (noopener) and mailto in the composer (recipient
  pre-filled); all other schemes blocked

Backend 292 tests green, ruff + mypy --strict clean; frontend tsc/eslint/vitest clean.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01U72FZiF1W1pb8XxyWBTREA
2026-07-26 09:28:48 +02:00
lukasandClaude Opus 4.8 49738740c0 M2: Meili search + saved searches, sync windows + backfill progress, threading & image fixes
Search:
- Meilisearch backend (app/search/meili.py) + factory wiring; pinned SDK 2.7
- Search-as-you-type: debounced query + Postgres prefix matching, built from
  the DB tokenizer so full email addresses match (email stays one lexeme)
- Saved searches API + sidebar virtual mailboxes; save affordance in the box

Account sync windows (SPEC §12 M2):
- Choose at setup: All (recent 30d first, then remainder), Last 3 months, New only
- Per-account sync_window + "Sync remaining mail" from settings (idempotent re-run)
- Provider windowed backfill (SINCE/BEFORE), message_count, newest_uid
- migration 0012: accounts.sync_window + backfill_total/backfill_done

Backfill progress UX:
- Floating progress card (auto-dissolves after 10s) + spinning sidebar ring that
  fills with the account colour; driven by sync.progress over WS

Thread list pagination:
- Server-side offset + accurate grand total, with filter/order in SQL; working
  prev/next across all synced mail

Reading panel:
- "Load images" one-off now actually renders images (?images=1) without trusting
  the sender; envelope carries images_allowed so a trusted sender shows no prompt

Threading:
- Subject fallback no longer merges messages that declare an explicit (different)
  parent; true siblings still group via a shared-parent reference match. Fixes
  unrelated auto-replies with the same subject landing in one thread

Backend 257 tests green, ruff + mypy --strict clean; frontend tsc/eslint/vitest clean.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01U72FZiF1W1pb8XxyWBTREA
2026-07-23 23:00:08 +02:00
lukasandClaude Opus 4.8 b4e07f11aa collapse cross-account dupes, Lato UI font, mail-list selection & density tuning
Backend:
- Collapse identical cross-account deliveries in a thread: new messages.delivery_id
  (migration 0011) preserves the real Message-ID even when message_id_hdr is
  synthesized (§7.1); get_thread groups by it and reports also_in_account_ids; the
  list count counts distinct deliveries. Backfill script for existing rows.

Frontend:
- Bundle Lato (SIL OFL) as the UI font; unread mail-list rows use Black weight.
- Mail-list selection: hover-avatar checkbox, tri-state "select all", bulk
  Archive/Pin/Delete (Restore/Delete in Archive/Trash/Spam) with a count in the
  reading pane; unread blue dot; "also delivered to" note in the thread.
- Spacing/size tuning: sender-column width, icon gaps, avatar size, bar heights,
  and a round of -1px font sizes across the sidebar, mail list, and reading pane.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01U72FZiF1W1pb8XxyWBTREA
2026-07-22 21:49:48 +02:00
lukasandClaude Opus 4.8 fdb6ee002e mail list redesign, folder actions, empty-folder, unread badges & empty-state art
Frontend:
- Redesign the mail list rows (avatar + sender + subject/snippet + right rail),
  even-spaced attachment/pin/archive/trash icons, hover-reveal actions, pink
  always-on pin, monocolor paperclip, row gaps, wider sender-subject gap.
- Sidebar folder badges: Inbox unread + Drafts counts (live via WS + mutations).
- Trash/Spam/Archive: hover Restore icon (replaces Archive), permanent delete in
  Trash; "Empty trash" / "Empty spam" buttons on the sort bar.
- Reading pane empty state: email illustration above an enlarged
  "Select a conversation".
- Search field clear (X) button; mark-read after a 4s dwell instead of instantly.

Backend:
- GET /api/mailboxes/counts (inbox unread + drafts) for the sidebar badges.
- POST /api/folders/{special}/empty + empty_folder_job to permanently clear a
  Trash/Spam/Archive folder (server EXPUNGE + DB rows + thread cleanup).

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01U72FZiF1W1pb8XxyWBTREA
2026-07-22 07:40:29 +02:00
lukasandClaude Opus 4.8 d650f16feb mail UX: restore-to-folder, draft/attachment fixes, printing, sidebar collapse
Backend:
- Restore returns each message to its original folder (e.g. Sent/Archive) instead
  of always the Inbox: new messages.restore_mailbox_id (migration 0010) +
  restore_thread_job.
- Drafts: expunge reconciliation for the Drafts poll (provider.list_uids /
  _reconcile_drafts) clears phantom "remote" drafts; send + discard drop the
  mirrored DB row; adopting a remote draft carries its real attachments into the
  compose flow.
- Thread detail/list are folder-aware: individually trashed/spam messages are
  hidden outside their folder, Trash/Spam show only their own messages, and the
  mail-list message count matches the view.

Frontend:
- Thread reading panel: fix newest-first card overlap/spacing (expand-all and
  single expand), hide Reply/Forward on trashed messages, close the Actions menu
  on message-body (iframe) clicks, scroll to the inline composer, and prompt
  Save draft / Discard on the X close.
- Composer: remount when switching drafts, dirty-tracking to stop duplicate IMAP
  mirrors, and close the From/More dropdowns on outside click.
- Sidebar: collapse/expand toggle (icons-only ~60px rail) and a log-out icon on
  Sign out.
- Mail-standard printing in a new window with reliable page margins (lib/print.ts).

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01U72FZiF1W1pb8XxyWBTREA
2026-07-20 23:12:25 +02:00
lukasandClaude Opus 4.8 1a83e0fb9b frontend: redesign the thread reading panel
Rework the conversation view to match the new dark-theme mockups:
- Subject bar gains a print button and an expand/collapse-all toggle (default
  action expands all; collapsing leaves only the newest message open).
- Collapsed messages are compact single-line cards (40px avatar, colored sender
  name, snippet, relative date) that overlap into a stack; alignment follows the
  sort order (top edge for oldest-first, bottom edge for newest-first) and the
  always-open newest card overlaps its neighbour in both orders.
- Expanded messages show sender+address (whole header row toggles collapse), a
  detail disclosure (From/To/Cc/Subject/Date/Size), the message location
  (Inbox/Sent), a read/unread circle (per-message flag toggle), a quick
  Reply/Reply-all, and an Actions menu (reply/forward/print/delete). The
  Reply/Reply all/Forward bar now sits inside the newest card above a divider.
- Message body: the quoted original is hidden by default with a "•••" toggle
  injected after the "On … wrote:" attribution; the iframe now collapses-then-
  measures so it shrinks back when the quote is hidden.
- Avatars use the sender's IMAP-account colour when the address matches one of
  our accounts, else a stable per-sender hue; two initials for multi-word names.
- Brighter dark-theme border token; new print/expand-all/collapse-all icons.

Backend: add size_bytes to the message envelope (shown in message details).

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
2026-07-19 18:58:01 +02:00
lukasandClaude Opus 4.8 293945480b sync-v2 §10.7: fix +- sign in RSS-growth output (use :+.1f)
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
2026-07-19 11:59:35 +02:00
lukasandClaude Opus 4.8 3d962a183c sync-v2 §10.7: mail-injecting mode for the soak harness
Optional SOAK_MAIL=1 makes the mock deliver synthetic messages (unique UIDs,
well-formed RFC822) on INBOX polls, so the full parse -> blob -> DB -> index path
actually runs and CPU/RSS move under real throughput. Tunables: SOAK_MAIL_PER_POLL,
SOAK_MSG_KB, SOAK_BIG_MSG_PCT (percent above STREAM_PARSE_MB -> streaming/§8 path).
Blob store + search backend are isolated (temp BLOB_ROOT, SEARCH_BACKEND=pg) and
torn down. Output gains msgs + msg/s columns and a delivered/peak-RSS summary.

RSS check is now a trend (early-third vs late-third mean, not min-to-max) so it
detects a real leak without flaking on transient large-message parse spikes.

Validated: mail run delivers ~4.2 msg/s (incl. 5% 10MB messages); RSS oscillates
123-267MB but its mean is flat (+3.7% over the run) -> no leak, parse memory
bound holds (§8); tier re-promotion on new mail exercised; all §10.7 checks pass.
Full gate green (238 tests).

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
2026-07-19 09:28:48 +02:00
lukasandClaude Opus 4.8 ee4dd99ad1 sync-v2 §10.7: soak stability — even per-worker cap + zombie diagnostic
- Size MAX_ACCOUNTS_PER_WORKER to ~1.2x the fair share so the fleet fills each
  worker evenly at startup and the rebalancer barely sheds; without this, three
  coordinators racing for the whole fleet shed/reclaim in a thrash that reset
  reclaimed accounts' first-poll clocks and dragged the sync-lag ramp.
- Sample a "dead" count (owned runners whose work loops have stopped) to
  distinguish a real zombie-runner bug from a slow ramp. Validated: dead=0
  throughout; 400-account run holds lag ~48s under a 52s budget across 13 steady
  samples, persistent sockets == hot, RSS flat.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
2026-07-19 08:25:03 +02:00
lukasandClaude Opus 4.8 887ba7ade6 sync-v2 §10.7: mock-provider soak/load harness + poll jitter
- loadtest/soak_sync.py + mock_provider.py: fully isolated soak for acceptance
  criterion 7. Creates a throwaway webmail_soak DB + uses Redis DB 14 (both torn
  down), simulates N accounts (default 1000, 20% active sessions) across 3
  in-process coordinators with a mock MailProvider (no real IMAP). Asserts:
  all accounts owned, persistent sockets <= hot*1.05, oldest lag < POLL_COLD*1.15,
  RSS stable. Steady state = after every account has completed its first sync.
  Configurable via SOAK_* / POLL_* env for fast smoke runs.
- engine: poll_all_loop and secondary_poll_loop now jitter their waits (first at a
  random point in the interval, later +/-10%) so a fleet claimed together doesn't
  poll in lockstep — anti-thundering-herd, a real improvement surfaced by the soak.

Validated: 60-account smoke passes all four criteria; full gate green (238 tests).

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
2026-07-19 08:09:44 +02:00
lukasandClaude Opus 4.8 8405e2f9b7 sync-v2: queue-isolation routing test (acceptance §10.5)
Assert bulk backfills route to the bulk queue and interactive user actions to
the interactive queue, and that enqueue_job stamps _queue_name accordingly — the
mechanism that keeps a backfill burst from starving sends/flag changes.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
2026-07-19 07:20:59 +02:00
lukasandClaude Opus 4.8 6d83f7cde8 sync-v2 step 7: observability + graceful shutdown (§7)
- Heartbeat JSON now carries {owned, sockets_open, tiers:{hot,warm,cold}, ts} per
  §2 so peers and operators can see each worker's load and socket footprint.
- run.py runs the sync coordinator directly (not via a placeholder arq worker) so
  SIGINT/SIGTERM can stop the claimer, cancel runners, and release every lease
  explicitly within a 10s drain budget — peers adopt the accounts immediately
  instead of waiting out the lease TTL. Removed the sync:control-queue hack and
  the supervisor indirection (start/stop_supervisor, sync_on_startup/shutdown).
- Acceptance: graceful-shutdown test proves a peer adopts released accounts within
  seconds under a 30s TTL (i.e. via release, not expiry).

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
2026-07-19 07:18:19 +02:00
lukasandClaude Opus 4.8 9dbc5ed998 sync-v2 step 6: memory + connection bounds (§7, §4)
- concurrency.loop_semaphore: per-event-loop semaphores (safe across pytest loops
  and process restarts via a WeakKeyDictionary keyed by the running loop).
- provider: holds a per-account (≤4) and per-host (PER_HOST_CONN_CAP) slot for the
  whole lifetime of an IMAP socket, released on close; SMTP send takes a per-host
  slot for its duration. A worker can no longer open hundreds of sockets to one
  host/account.
- indexer: parsing goes through a PARSE_CONCURRENCY semaphore, and a message over
  STREAM_PARSE_MB is parsed from its blob file (parse_message_file) instead of an
  in-memory tree, so peak RAM ~= PARSE_CONCURRENCY x STREAM_PARSE_MB regardless of
  mailbox sizes. Backfill parallelism stays bounded by the bulk queue only.
- config: PER_HOST_CONN_CAP, PARSE_CONCURRENCY, STREAM_PARSE_MB.
- tests: per-account cap blocks the 5th connection / distinct accounts independent;
  file-vs-bytes parse equivalence + large-message streaming routing.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
2026-07-19 07:13:03 +02:00
lukasandClaude Opus 4.8 19335116da sync-v2 step 5: activity tiers (hot/warm/cold)
Each owned account is tiered by the coordinator; the runner shapes its work to
match — correctness identical, only latency/socket cost differ.

- hot: persistent INBOX IDLE + secondary poll every POLL_WARM.
- warm: no persistent socket; transient INBOX+special poll every POLL_WARM.
- cold: same transient poll every POLL_COLD.
- engine: poll_all_loop (transient full poll); secondary_poll_loop interval is
  now a parameter; POLL_INTERVAL removed in favor of POLL_WARM/POLL_COLD.
- coordinator: tier eval every 120s from session heat (sess:active:{user}) +
  recent mail (max internal_date within TIER_HOT_ACTIVITY_H), with the
  HOT_BUDGET_PER_WORKER guard shedding least-recently-active session-less hot
  accounts; user.active event promotes a user's accounts to hot at once (≤5s).
- API: login + WS connect/ping publish user.active and refresh sess:active
  (TTL TIER_SESSION_GRACE_MIN). New sync:tier:{id} key, default warm on claim.
- config: POLL_WARM/POLL_COLD/TIER_*/HOT_BUDGET_PER_WORKER.
- tests: 5 tier tests (session/recent-mail promotion, cold, user.active,
  budget demotion) alongside the lease/lifecycle acceptance tests.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
2026-07-19 07:03:54 +02:00
lukasandClaude Opus 4.8 245672d3f2 sync-v2 step 4: event-driven account lifecycle (§3)
- Coordinator subscribes to account.state_changed over Redis pub/sub: going
  live marks the account claimable at once; disabled/deleted/error tears the
  owning runner down immediately (≤30s) instead of waiting for the lease TTL.
- delete_account now publishes account.state_changed:deleted after dropping the
  account from sync:live, so teardown is event-driven, not just fencing.
- AccountRunner gains a verify loop (defense in depth): it periodically confirms
  the account is still live and flags itself dead if not; the coordinator reaps
  dead runners and releases their lease. Zombie loops are structurally impossible.
- New core event user.active (SPEC §8.3), published on login and on every WS
  connect; consumed for tier promotion in step 5.
- Acceptance: added event-driven teardown test (isolated from fencing/verify).

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
2026-07-19 06:49:48 +02:00
lukasandClaude Opus 4.8 bcf11fcecb sync-v2 step 3: lease-owned account claimer (replaces startup supervisor)
Ownership of an account now moves through a Redis lease so any number of sync
workers can run against the same account set with no double-IDLE.

- leases.py: sync:live registry + sync:lease:{id} (SET NX EX) with owner-checked
  Lua renew/release; mark_live / mark_not_live helpers.
- coordinator.py: SyncCoordinator claims from sync:live up to MAX_ACCOUNTS_PER_
  WORKER, renews every LEASE_RENEW (fences the runner on lease loss), reconciles
  sync:live against Postgres, heartbeats a worker key, and rebalances when it
  holds well above a fair share.
- supervisor start/stop now drive a coordinator; engine._set_state and
  delete_account maintain sync:live membership (+ lease drop = fencing teardown).
- config: MAX_ACCOUNTS_PER_WORKER, LEASE_TTL, LEASE_RENEW.
- tests: acceptance 1-3 (single ownership, crash failover, create/delete
  lifecycle) driving the real coordinator + Redis with faked IMAP loops.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
2026-07-18 20:34:22 +02:00
lukasandClaude Opus 4.8 23704f323c sync-v2 step 2: extract AccountRunner
Move the per-account INBOX-IDLE + secondary-poll loops out of the supervisor
into a self-contained AccountRunner (start/stop/teardown). The startup supervisor
now launches one runner per live account. Pure refactor, no behavior change.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
2026-07-18 19:56:06 +02:00
lukasandClaude Opus 4.8 e0a45c5a46 sync-v2 step 1: worker roles + interactive/bulk queue split
- config: WORKER_ROLE (sync|jobs|all), INTERACTIVE_CONCURRENCY, BULK_CONCURRENCY.
- queue: named queues (interactive/bulk) + enqueue_job() router; sync_bootstrap
  goes to bulk, all other jobs to interactive.
- workers.run: single entrypoint; role=jobs runs both arq consumers (per-queue
  timeout/concurrency), role=sync runs the supervisor, all=both (dev default).
- docker-compose: worker-sync + worker-jobs services (shared image, nofile
  ulimit); backend/Dockerfile; Makefile worker/worker-sync/worker-jobs targets.
- tests repointed to app.queue.get_arq_pool; supervisor behavior unchanged.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
2026-07-18 19:53:31 +02:00
lukasandClaude Opus 4.8 9cb946f75d docs: add sync-engine v2 refactor spec and spec_v2
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
2026-07-18 19:42:05 +02:00