sync-v2 step 6: memory + connection bounds (§7, §4)
- concurrency.loop_semaphore: per-event-loop semaphores (safe across pytest loops and process restarts via a WeakKeyDictionary keyed by the running loop). - provider: holds a per-account (≤4) and per-host (PER_HOST_CONN_CAP) slot for the whole lifetime of an IMAP socket, released on close; SMTP send takes a per-host slot for its duration. A worker can no longer open hundreds of sockets to one host/account. - indexer: parsing goes through a PARSE_CONCURRENCY semaphore, and a message over STREAM_PARSE_MB is parsed from its blob file (parse_message_file) instead of an in-memory tree, so peak RAM ~= PARSE_CONCURRENCY x STREAM_PARSE_MB regardless of mailbox sizes. Backfill parallelism stays bounded by the bulk queue only. - config: PER_HOST_CONN_CAP, PARSE_CONCURRENCY, STREAM_PARSE_MB. - tests: per-account cap blocks the 5th connection / distinct accounts independent; file-vs-bytes parse equivalence + large-message streaming routing. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
This commit is contained in:
@@ -7,6 +7,7 @@ Header RFC2047 decoding is automatic under ``policy.default``.
|
||||
from __future__ import annotations
|
||||
|
||||
import email
|
||||
import os
|
||||
from dataclasses import dataclass, field
|
||||
from datetime import datetime
|
||||
from email.headerregistry import AddressHeader
|
||||
@@ -104,10 +105,21 @@ def _extract_attachments(msg: EmailMessage) -> list[ParsedAttachment]:
|
||||
|
||||
|
||||
def parse_message(raw: bytes) -> ParsedEmail:
|
||||
"""Parse a raw RFC822 message into a :class:`ParsedEmail`."""
|
||||
"""Parse a raw RFC822 message (in memory) into a :class:`ParsedEmail`."""
|
||||
msg = email.message_from_bytes(raw, policy=default_policy)
|
||||
assert isinstance(msg, EmailMessage)
|
||||
return _parse_email(msg)
|
||||
|
||||
|
||||
def parse_message_file(path: str | os.PathLike[str]) -> ParsedEmail:
|
||||
"""Parse a message straight from a blob file handle (large-message path, §7)."""
|
||||
with open(path, "rb") as handle:
|
||||
msg = email.message_from_binary_file(handle, policy=default_policy)
|
||||
assert isinstance(msg, EmailMessage)
|
||||
return _parse_email(msg)
|
||||
|
||||
|
||||
def _parse_email(msg: EmailMessage) -> ParsedEmail:
|
||||
message_id = normalize_message_id(msg.get("Message-ID"))
|
||||
in_reply_to_ids = normalize_reference_list(msg.get("In-Reply-To"))
|
||||
in_reply_to = in_reply_to_ids[0] if in_reply_to_ids else None
|
||||
|
||||
Reference in New Issue
Block a user