sync-v2 step 6: memory + connection bounds (§7, §4)

- concurrency.loop_semaphore: per-event-loop semaphores (safe across pytest loops
  and process restarts via a WeakKeyDictionary keyed by the running loop).
- provider: holds a per-account (≤4) and per-host (PER_HOST_CONN_CAP) slot for the
  whole lifetime of an IMAP socket, released on close; SMTP send takes a per-host
  slot for its duration. A worker can no longer open hundreds of sockets to one
  host/account.
- indexer: parsing goes through a PARSE_CONCURRENCY semaphore, and a message over
  STREAM_PARSE_MB is parsed from its blob file (parse_message_file) instead of an
  in-memory tree, so peak RAM ~= PARSE_CONCURRENCY x STREAM_PARSE_MB regardless of
  mailbox sizes. Backfill parallelism stays bounded by the bulk queue only.
- config: PER_HOST_CONN_CAP, PARSE_CONCURRENCY, STREAM_PARSE_MB.
- tests: per-account cap blocks the 5th connection / distinct accounts independent;
  file-vs-bytes parse equivalence + large-message streaming routing.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
This commit is contained in:
2026-07-19 07:13:03 +02:00
co-authored by Claude Opus 4.8
parent 19335116da
commit 9dbc5ed998
7 changed files with 254 additions and 34 deletions
+13 -1
View File
@@ -7,6 +7,7 @@ Header RFC2047 decoding is automatic under ``policy.default``.
from __future__ import annotations
import email
import os
from dataclasses import dataclass, field
from datetime import datetime
from email.headerregistry import AddressHeader
@@ -104,10 +105,21 @@ def _extract_attachments(msg: EmailMessage) -> list[ParsedAttachment]:
def parse_message(raw: bytes) -> ParsedEmail:
"""Parse a raw RFC822 message into a :class:`ParsedEmail`."""
"""Parse a raw RFC822 message (in memory) into a :class:`ParsedEmail`."""
msg = email.message_from_bytes(raw, policy=default_policy)
assert isinstance(msg, EmailMessage)
return _parse_email(msg)
def parse_message_file(path: str | os.PathLike[str]) -> ParsedEmail:
"""Parse a message straight from a blob file handle (large-message path, §7)."""
with open(path, "rb") as handle:
msg = email.message_from_binary_file(handle, policy=default_policy)
assert isinstance(msg, EmailMessage)
return _parse_email(msg)
def _parse_email(msg: EmailMessage) -> ParsedEmail:
message_id = normalize_message_id(msg.get("Message-ID"))
in_reply_to_ids = normalize_reference_list(msg.get("In-Reply-To"))
in_reply_to = in_reply_to_ids[0] if in_reply_to_ids else None