How the worker functions (multi-user / multi-account) ===================================================== The worker wears two hats (in one process, one event loop) ---------------------------------------------------------- It is a single arq process (`arq workers.worker.WorkerSettings`) running one asyncio event loop. Inside it, two very different things run: A) A job runner (the Redis queue side). The API enqueues discrete jobs; the worker pulls and runs them, up to `max_jobs = 10` concurrently, `job_timeout = 3600s`. The registered jobs: sync_bootstrap_account (backfill a new account), push_flags_job, move_job, move_thread_to_folder_job, delete_thread_job, send_job, push_draft_to_imap_job, remove_imap_draft_job, refresh_account_job. Each job builds a provider (which is where the mailbox credential is decrypted with the MASTER_KEY - worker-only), does the IMAP/SMTP op, and closes the connection. On failure they retry with backoff (~x5) and then revert or mark failed. B) A live-sync supervisor (long-running loops). Started once in on_startup and is NOT part of the job queue. The per-account model (the key point) ------------------------------------- On startup, start_supervisor reads every account with state = "live" and, for each account, spawns two persistent asyncio tasks: 1. INBOX IDLE loop (live_loop) - opens one IMAP connection and keeps it open, blocking on IMAP IDLE (server push). It reacts within seconds when mail arrives, and re-issues IDLE at least every 25 min. This is a *persistent* connection held 24/7. 2. Secondary poll loop (secondary_poll_loop) - every POLL_INTERVAL (120 s) it opens a fresh, transient connection, CONDSTORE-polls Sent/Drafts/Archive/Trash/Junk (cheap MODSEQ deltas), and closes it. So the steady-state cost scales linearly with the total number of email accounts across all users: - Persistent IMAP connections : ~1 per account (the INBOX IDLE) held continuously - asyncio tasks : 2 per account - Transient connections : one burst per account every 120 s (secondary poll) + per job For 3 users x 3 accounts that is ~9 always-open IDLE connections and 18 tasks - trivial. For 500 accounts it is ~500 sockets open 24/7 plus 1000 tasks, and you will hit IMAP server per-user/global connection limits, file-descriptor limits, and memory pressure long before anything else. How updates reach the browsers ------------------------------ The worker never talks to browsers. When it persists a change it publishes to Redis pub/sub (evt:{event}); the API's WebSocket hub relays each event to the owning user's connected clients, which invalidate the relevant queries. So Redis is both the job queue and the fan-out bus. Honest scaling limitations -------------------------- - Single process / single core. All those loops and jobs share one event loop. It is fine because IMAP work is I/O-bound (mostly waiting), but a heavy backfill or MIME parse briefly blocks everything else, and each raw message is loaded fully into memory to parse + blob-store - so messages x concurrency is your memory driver. - The live-sync side does not horizontally scale. If you ran two workers for throughput, both would start IDLE loops for all live accounts -> duplicate connections and double-processing. The job side would shard fine across workers (that is what arq does), but the supervisor assumes exactly one instance. - Account discovery is startup-only. New or re-configured accounts are not picked up as live until the worker restarts (the code says so explicitly). A new account does get its one-time backfill immediately (via the enqueued sync_bootstrap_account job), but its ongoing IDLE loop will not start until a restart. The same gap causes the "zombie loop" after account deletion. - max_jobs = 10 is global, shared across all users - a burst of sends/moves queues behind that cap. What a real multi-tenant deployment would add --------------------------------------------- - A pool of sync workers with account sharding: each account owned by exactly one worker (e.g., a Redis lease/consistent hashing), so IDLE loops do not duplicate and you can scale out. - React to account.state_changed to start/stop loops at runtime (no restart on add/edit/delete). - Connection budgeting - e.g., keep IDLE only for recently-active mailboxes and poll the long tail - plus per-server connection caps and backpressure. - Move sync_bootstrap_account (a large backfill) onto a separate, higher-timeout queue from the fast interactive jobs (send/flags/move). Net: the current design is a clean, correct single-node model - great for a handful of users with a few accounts each - but the per-account persistent IDLE loop and the startup-only, single-supervisor assumptions are the things you would re-architect before going multi-tenant at scale.