Optional SOAK_MAIL=1 makes the mock deliver synthetic messages (unique UIDs,
well-formed RFC822) on INBOX polls, so the full parse -> blob -> DB -> index path
actually runs and CPU/RSS move under real throughput. Tunables: SOAK_MAIL_PER_POLL,
SOAK_MSG_KB, SOAK_BIG_MSG_PCT (percent above STREAM_PARSE_MB -> streaming/§8 path).
Blob store + search backend are isolated (temp BLOB_ROOT, SEARCH_BACKEND=pg) and
torn down. Output gains msgs + msg/s columns and a delivered/peak-RSS summary.
RSS check is now a trend (early-third vs late-third mean, not min-to-max) so it
detects a real leak without flaking on transient large-message parse spikes.
Validated: mail run delivers ~4.2 msg/s (incl. 5% 10MB messages); RSS oscillates
123-267MB but its mean is flat (+3.7% over the run) -> no leak, parse memory
bound holds (§8); tier re-promotion on new mail exercised; all §10.7 checks pass.
Full gate green (238 tests).
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
- Size MAX_ACCOUNTS_PER_WORKER to ~1.2x the fair share so the fleet fills each
worker evenly at startup and the rebalancer barely sheds; without this, three
coordinators racing for the whole fleet shed/reclaim in a thrash that reset
reclaimed accounts' first-poll clocks and dragged the sync-lag ramp.
- Sample a "dead" count (owned runners whose work loops have stopped) to
distinguish a real zombie-runner bug from a slow ramp. Validated: dead=0
throughout; 400-account run holds lag ~48s under a 52s budget across 13 steady
samples, persistent sockets == hot, RSS flat.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
- loadtest/soak_sync.py + mock_provider.py: fully isolated soak for acceptance
criterion 7. Creates a throwaway webmail_soak DB + uses Redis DB 14 (both torn
down), simulates N accounts (default 1000, 20% active sessions) across 3
in-process coordinators with a mock MailProvider (no real IMAP). Asserts:
all accounts owned, persistent sockets <= hot*1.05, oldest lag < POLL_COLD*1.15,
RSS stable. Steady state = after every account has completed its first sync.
Configurable via SOAK_* / POLL_* env for fast smoke runs.
- engine: poll_all_loop and secondary_poll_loop now jitter their waits (first at a
random point in the interval, later +/-10%) so a fleet claimed together doesn't
poll in lockstep — anti-thundering-herd, a real improvement surfaced by the soak.
Validated: 60-account smoke passes all four criteria; full gate green (238 tests).
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
lukasandClaude Opus 4.8