sync-v2 step 6: memory + connection bounds (§7, §4)
- concurrency.loop_semaphore: per-event-loop semaphores (safe across pytest loops and process restarts via a WeakKeyDictionary keyed by the running loop). - provider: holds a per-account (≤4) and per-host (PER_HOST_CONN_CAP) slot for the whole lifetime of an IMAP socket, released on close; SMTP send takes a per-host slot for its duration. A worker can no longer open hundreds of sockets to one host/account. - indexer: parsing goes through a PARSE_CONCURRENCY semaphore, and a message over STREAM_PARSE_MB is parsed from its blob file (parse_message_file) instead of an in-memory tree, so peak RAM ~= PARSE_CONCURRENCY x STREAM_PARSE_MB regardless of mailbox sizes. Backfill parallelism stays bounded by the bulk queue only. - config: PER_HOST_CONN_CAP, PARSE_CONCURRENCY, STREAM_PARSE_MB. - tests: per-account cap blocks the 5th connection / distinct accounts independent; file-vs-bytes parse equivalence + large-message streaming routing. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
This commit is contained in:
@@ -17,8 +17,10 @@ from sqlalchemy import select
|
||||
from sqlalchemy.dialects.postgresql import insert as pg_insert
|
||||
from sqlalchemy.ext.asyncio import AsyncSession
|
||||
|
||||
from app.concurrency import loop_semaphore
|
||||
from app.config import get_settings
|
||||
from app.indexer import blobstore
|
||||
from app.indexer.mime import ParsedEmail, parse_message
|
||||
from app.indexer.mime import ParsedEmail, parse_message, parse_message_file
|
||||
from app.indexer.sanitize import sanitize_html
|
||||
from app.models import Attachment, BodyPart, KnownRecipient, Message
|
||||
from app.providers.base import RemoteMessage
|
||||
@@ -115,6 +117,21 @@ async def _resolve_message_id(
|
||||
return parsed_id
|
||||
|
||||
|
||||
async def _parse_bounded(user_id: uuid.UUID, sha: str, raw: bytes) -> ParsedEmail:
|
||||
"""Parse under the PARSE_CONCURRENCY cap; stream large messages from disk (§7).
|
||||
|
||||
Peak RAM stays ~PARSE_CONCURRENCY x STREAM_PARSE_MB: only a few parses run at
|
||||
once, and a message above the threshold is parsed from its (already written)
|
||||
blob file instead of being held as an in-memory tree beside its raw bytes.
|
||||
"""
|
||||
settings = get_settings()
|
||||
threshold = settings.stream_parse_mb * 1024 * 1024
|
||||
async with loop_semaphore("mime_parse", "parse", settings.parse_concurrency):
|
||||
if len(raw) > threshold:
|
||||
return parse_message_file(blobstore.blob_path(user_id, sha))
|
||||
return parse_message(raw)
|
||||
|
||||
|
||||
async def persist_message(
|
||||
session: AsyncSession,
|
||||
*,
|
||||
@@ -132,7 +149,7 @@ async def persist_message(
|
||||
return None
|
||||
|
||||
sha = blobstore.store(user_id, remote.raw)
|
||||
parsed = parse_message(remote.raw)
|
||||
parsed = await _parse_bounded(user_id, sha, remote.raw)
|
||||
|
||||
message_id_hdr = await _resolve_message_id(
|
||||
session, user_id=user_id, account_id=account_id, parsed_id=parsed.message_id, raw=remote.raw
|
||||
|
||||
Reference in New Issue
Block a user