sync-v2 step 6: memory + connection bounds (§7, §4)

- concurrency.loop_semaphore: per-event-loop semaphores (safe across pytest loops
  and process restarts via a WeakKeyDictionary keyed by the running loop).
- provider: holds a per-account (≤4) and per-host (PER_HOST_CONN_CAP) slot for the
  whole lifetime of an IMAP socket, released on close; SMTP send takes a per-host
  slot for its duration. A worker can no longer open hundreds of sockets to one
  host/account.
- indexer: parsing goes through a PARSE_CONCURRENCY semaphore, and a message over
  STREAM_PARSE_MB is parsed from its blob file (parse_message_file) instead of an
  in-memory tree, so peak RAM ~= PARSE_CONCURRENCY x STREAM_PARSE_MB regardless of
  mailbox sizes. Backfill parallelism stays bounded by the bulk queue only.
- config: PER_HOST_CONN_CAP, PARSE_CONCURRENCY, STREAM_PARSE_MB.
- tests: per-account cap blocks the 5th connection / distinct accounts independent;
  file-vs-bytes parse equivalence + large-message streaming routing.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
This commit is contained in:
2026-07-19 07:13:03 +02:00
co-authored by Claude Opus 4.8
parent 19335116da
commit 9dbc5ed998
7 changed files with 254 additions and 34 deletions
+32
View File
@@ -0,0 +1,32 @@
"""Per-event-loop semaphores for bounded concurrency (refactor §4, §7).
Connection and parse caps are process-local but must be bound to the running
event loop: an :class:`asyncio.Semaphore` created on one loop cannot be awaited
on another (a real hazard under pytest, which uses a fresh loop per test). These
helpers hand back a semaphore keyed by ``(group, key)`` for the current loop,
creating it on first use and dropping the whole set when the loop is collected.
"""
from __future__ import annotations
import asyncio
import weakref
# group -> {loop: {key: Semaphore}}. WeakKeyDictionary drops a loop's semaphores
# once the loop is gone, so no state leaks across tests or restarts.
_registry: dict[
str, weakref.WeakKeyDictionary[asyncio.AbstractEventLoop, dict[str, asyncio.Semaphore]]
] = {}
def loop_semaphore(group: str, key: str, cap: int) -> asyncio.Semaphore:
"""Return the current loop's semaphore for ``(group, key)`` (cap on first use)."""
loop = asyncio.get_running_loop()
per_group = _registry.setdefault(group, weakref.WeakKeyDictionary())
per_loop = per_group.get(loop)
if per_loop is None:
per_loop = per_group[loop] = {}
sem = per_loop.get(key)
if sem is None:
sem = per_loop[key] = asyncio.Semaphore(cap)
return sem