Unreachable mail servers: send retries in Scheduled, real action retries, account health

Sending: a send the server could not take (unreachable, timeout, 4xx) is no
longer failed at once. It waits in Scheduled ("retrying") and is tried again
after 1, 2, 5, 10, 30 min, then hourly, for 24 hours — on the server, so it
works with the browser closed. A permanent refusal (5xx, wrong password) is
"failed" at once, with the server's reason. GET /outbox lists both;
POST /outbox/{id}/retry (Retry now) and /cancel (Edit, Delete) act on them,
and they are left out of Drafts. Mail delivered but not filed in Sent now
counts as sent, and a conditional claim keeps two jobs from sending one
message twice. The app says when a send failed and when it finally went.

Actions: arq retries a job only on Retry, never on a plain exception, so the
"retry x5, then revert" for flags and moves never happened. Flag, move,
archive, restore, delete and empty-folder jobs now raise Retry while the
server is unreachable (~24 hours of growing gaps), skip a change a later one
replaced, stop at the first connection failure instead of 20 s per message,
and delete/empty keep the rows until the server has the messages gone.

Account health: 3 failed connections in a row mark an account unreachable
(amber in the sidebar and Settings, cleared on the next success); 3 refused
sign-ins stop its syncing as auth_failed until the password is entered
again, so the provider does not lock the account. Sync loops back off 30 s
doubling to 10 min and log one line per expected failure; a refused
connection fails at once with its reason (the connect task is kept), and a
silently dead IDLE connection is noticed within ~11 min instead of ~29.

Also: the empty reading pane is clipped to its width when dragged very
narrow; tests no longer leave live accounts and outbox rows in the database.

Migration 0021: outbox attempts / next_attempt_at / retry_until, accounts
sync_error / sync_error_since.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

backend/app/sync/runner.py

@@ -18,6 +18,7 @@ from __future__ import annotations
 import asyncio
 import contextlib
 import logging
+import random
 import uuid
 from collections.abc import Awaitable, Callable
 
@@ -26,11 +27,18 @@ from sqlalchemy import select
 from app.config import get_settings
 from app.db import session_scope
 from app.models import Account
+from app.providers.base import ProviderAuthError
 from app.sync.engine import live_loop, poll_all_loop, secondary_poll_loop
+from app.sync.health import AccountHealth, is_temporary
 
 logger = logging.getLogger(__name__)
 
+# After a loop fails: wait 30 s, doubling with each failure in a row up to 10 min
+# (±20%, so accounts on one dead server don't retry in step). A loop that ran a
+# good while before failing starts again from 30 s.
 _RECONNECT_BACKOFF = 30.0
+_MAX_BACKOFF = 600.0
+_HEALTHY_RUN_S = 300.0
 _VERIFY_INTERVAL = 60.0
 
 Tier = str  # "hot" | "warm" | "cold"
@@ -49,6 +57,8 @@ class AccountRunner:
         # coordinator reaps such runners and releases their lease.
         self.dead = asyncio.Event()
         self.verify_interval = _VERIFY_INTERVAL
+        # Failures in a row across this account's loops: unreachable / sign-in failed.
+        self.health = AccountHealth(account_id)
 
     def start(self) -> None:
         """Launch the verify task and the current tier's work loops."""
@@ -98,13 +108,15 @@ class AccountRunner:
             await asyncio.gather(*tasks, return_exceptions=True)
 
     async def _run_live(self) -> None:
-        await live_loop(self.account_id, "INBOX", self._stop)
+        await live_loop(self.account_id, "INBOX", self._stop, self.health)
 
     async def _run_secondary(self) -> None:
-        await secondary_poll_loop(self.account_id, self._stop, get_settings().poll_warm)
+        await secondary_poll_loop(
+            self.account_id, self._stop, get_settings().poll_warm, self.health
+        )
 
     async def _run_poll(self, interval: float) -> None:
-        await poll_all_loop(self.account_id, interval, self._stop)
+        await poll_all_loop(self.account_id, interval, self._stop, self.health)
 
     async def _verify_loop(self) -> None:
         """Periodically confirm the account is still live; flag ``dead`` if not.
@@ -129,13 +141,47 @@ class AccountRunner:
         return state == "live"
 
     async def _supervise(self, run: Callable[[], Awaitable[None]], label: str) -> None:
-        """Run a loop, restarting it (with backoff) until stopped or cancelled."""
+        """Run a loop, restarting it (with growing backoff) until stopped or cancelled."""
+        loop = asyncio.get_running_loop()
+        fails = 0
         while not self._stop.is_set():
+            started = loop.time()
             try:
                 await run()
             except asyncio.CancelledError:
                 raise
-            except Exception:
-                logger.exception("%s loop for account %s crashed; retrying", label, self.account_id)
+            except Exception as exc:
+                if loop.time() - started > _HEALTHY_RUN_S:
+                    fails = 0  # it had been working: a fresh failure, not the same outage
+                fails += 1
+                delay = min(_MAX_BACKOFF, _RECONNECT_BACKOFF * 2 ** (fails - 1))
+                delay *= random.uniform(0.8, 1.2)
+                try:
+                    await self.health.failed(exc)
+                except Exception:  # noqa: BLE001 - never let bookkeeping kill the loop
+                    logger.exception("account %s: recording the failure failed", self.account_id)
+                if self.health.gave_up:
+                    # Sign-in refused too often: not synced any more (the health
+                    # tracker logged why). Flag the runner so the coordinator
+                    # reaps it and releases the lease.
+                    self.dead.set()
+                    return
+                if is_temporary(exc) or isinstance(exc, ProviderAuthError):
+                    # Expected while a server is down or a password is wrong: one
+                    # line, not a traceback every half minute.
+                    logger.warning(
+                        "%s loop for account %s: %s; retrying in %.0fs",
+                        label,
+                        self.account_id,
+                        getattr(exc, "detail", None) or exc or type(exc).__name__,
+                        delay,
+                    )
+                else:
+                    logger.exception(
+                        "%s loop for account %s crashed; retrying in %.0fs",
+                        label,
+                        self.account_id,
+                        delay,
+                    )
                 with contextlib.suppress(TimeoutError):
-                    await asyncio.wait_for(self._stop.wait(), timeout=_RECONNECT_BACKOFF)
+                    await asyncio.wait_for(self._stop.wait(), timeout=delay)