webmail-client/worker-description.txt

How the worker functions (multi-user / multi-account)
=====================================================

The worker wears two hats (in one process, one event loop)
----------------------------------------------------------
It is a single worker process (`python -m workers.run`, WORKER_ROLE=all) running
one asyncio event loop. Inside it, two very different things run:

A) A job runner (the Redis queue side). The API enqueues discrete jobs; the
   worker pulls and runs them, up to `max_jobs = 10` concurrently,
   `job_timeout = 3600s`. The registered jobs:
   sync_bootstrap_account (backfill a new account), push_flags_job, move_job,
   move_thread_to_folder_job, delete_thread_job, send_job,
   push_draft_to_imap_job, remove_imap_draft_job, refresh_account_job.
   Each job builds a provider (which is where the mailbox credential is
   decrypted with the MASTER_KEY - worker-only), does the IMAP/SMTP op, and
   closes the connection. On failure they retry with backoff (~x5) and then
   revert or mark failed.

B) A live-sync supervisor (long-running loops). Started once in on_startup and
   is NOT part of the job queue.


The per-account model (the key point)
-------------------------------------
On startup, start_supervisor reads every account with state = "live" and, for
each account, spawns two persistent asyncio tasks:

1. INBOX IDLE loop (live_loop) - opens one IMAP connection and keeps it open,
   blocking on IMAP IDLE (server push). It reacts within seconds when mail
   arrives, and re-issues IDLE at least every 25 min. This is a *persistent*
   connection held 24/7.

2. Secondary poll loop (secondary_poll_loop) - every POLL_INTERVAL (120 s) it
   opens a fresh, transient connection, CONDSTORE-polls
   Sent/Drafts/Archive/Trash/Junk (cheap MODSEQ deltas), and closes it.

So the steady-state cost scales linearly with the total number of email
accounts across all users:

  - Persistent IMAP connections : ~1 per account (the INBOX IDLE) held
                                   continuously
  - asyncio tasks               : 2 per account
  - Transient connections       : one burst per account every 120 s (secondary
                                   poll) + per job

For 3 users x 3 accounts that is ~9 always-open IDLE connections and 18 tasks -
trivial. For 500 accounts it is ~500 sockets open 24/7 plus 1000 tasks, and you
will hit IMAP server per-user/global connection limits, file-descriptor limits,
and memory pressure long before anything else.


How updates reach the browsers
------------------------------
The worker never talks to browsers. When it persists a change it publishes to
Redis pub/sub (evt:{event}); the API's WebSocket hub relays each event to the
owning user's connected clients, which invalidate the relevant queries. So Redis
is both the job queue and the fan-out bus.


Honest scaling limitations
--------------------------
- Single process / single core. All those loops and jobs share one event loop.
  It is fine because IMAP work is I/O-bound (mostly waiting), but a heavy
  backfill or MIME parse briefly blocks everything else, and each raw message is
  loaded fully into memory to parse + blob-store - so messages x concurrency is
  your memory driver.

- The live-sync side does not horizontally scale. If you ran two workers for
  throughput, both would start IDLE loops for all live accounts -> duplicate
  connections and double-processing. The job side would shard fine across
  workers (that is what arq does), but the supervisor assumes exactly one
  instance.

- Account discovery is startup-only. New or re-configured accounts are not
  picked up as live until the worker restarts (the code says so explicitly). A
  new account does get its one-time backfill immediately (via the enqueued
  sync_bootstrap_account job), but its ongoing IDLE loop will not start until a
  restart. The same gap causes the "zombie loop" after account deletion.

- max_jobs = 10 is global, shared across all users - a burst of sends/moves
  queues behind that cap.


What a real multi-tenant deployment would add
---------------------------------------------
- A pool of sync workers with account sharding: each account owned by exactly
  one worker (e.g., a Redis lease/consistent hashing), so IDLE loops do not
  duplicate and you can scale out.

- React to account.state_changed to start/stop loops at runtime (no restart on
  add/edit/delete).

- Connection budgeting - e.g., keep IDLE only for recently-active mailboxes and
  poll the long tail - plus per-server connection caps and backpressure.

- Move sync_bootstrap_account (a large backfill) onto a separate, higher-timeout
  queue from the fast interactive jobs (send/flags/move).

Net: the current design is a clean, correct single-node model - great for a
handful of users with a few accounts each - but the per-account persistent IDLE
loop and the startup-only, single-supervisor assumptions are the things you
would re-architect before going multi-tenant at scale.