sync-v2 step 6: memory + connection bounds (§7, §4)

- concurrency.loop_semaphore: per-event-loop semaphores (safe across pytest loops
  and process restarts via a WeakKeyDictionary keyed by the running loop).
- provider: holds a per-account (≤4) and per-host (PER_HOST_CONN_CAP) slot for the
  whole lifetime of an IMAP socket, released on close; SMTP send takes a per-host
  slot for its duration. A worker can no longer open hundreds of sockets to one
  host/account.
- indexer: parsing goes through a PARSE_CONCURRENCY semaphore, and a message over
  STREAM_PARSE_MB is parsed from its blob file (parse_message_file) instead of an
  in-memory tree, so peak RAM ~= PARSE_CONCURRENCY x STREAM_PARSE_MB regardless of
  mailbox sizes. Backfill parallelism stays bounded by the bulk queue only.
- config: PER_HOST_CONN_CAP, PARSE_CONCURRENCY, STREAM_PARSE_MB.
- tests: per-account cap blocks the 5th connection / distinct accounts independent;
  file-vs-bytes parse equivalence + large-message streaming routing.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
This commit is contained in:
2026-07-19 07:13:03 +02:00
co-authored by Claude Opus 4.8
parent 19335116da
commit 9dbc5ed998
7 changed files with 254 additions and 34 deletions
+19 -2
View File
@@ -17,8 +17,10 @@ from sqlalchemy import select
from sqlalchemy.dialects.postgresql import insert as pg_insert
from sqlalchemy.ext.asyncio import AsyncSession
from app.concurrency import loop_semaphore
from app.config import get_settings
from app.indexer import blobstore
from app.indexer.mime import ParsedEmail, parse_message
from app.indexer.mime import ParsedEmail, parse_message, parse_message_file
from app.indexer.sanitize import sanitize_html
from app.models import Attachment, BodyPart, KnownRecipient, Message
from app.providers.base import RemoteMessage
@@ -115,6 +117,21 @@ async def _resolve_message_id(
return parsed_id
async def _parse_bounded(user_id: uuid.UUID, sha: str, raw: bytes) -> ParsedEmail:
"""Parse under the PARSE_CONCURRENCY cap; stream large messages from disk (§7).
Peak RAM stays ~PARSE_CONCURRENCY x STREAM_PARSE_MB: only a few parses run at
once, and a message above the threshold is parsed from its (already written)
blob file instead of being held as an in-memory tree beside its raw bytes.
"""
settings = get_settings()
threshold = settings.stream_parse_mb * 1024 * 1024
async with loop_semaphore("mime_parse", "parse", settings.parse_concurrency):
if len(raw) > threshold:
return parse_message_file(blobstore.blob_path(user_id, sha))
return parse_message(raw)
async def persist_message(
session: AsyncSession,
*,
@@ -132,7 +149,7 @@ async def persist_message(
return None
sha = blobstore.store(user_id, remote.raw)
parsed = parse_message(remote.raw)
parsed = await _parse_bounded(user_id, sha, remote.raw)
message_id_hdr = await _resolve_message_id(
session, user_id=user_id, account_id=account_id, parsed_id=parsed.message_id, raw=remote.raw