sync-v2 step 6: memory + connection bounds (§7, §4)
- concurrency.loop_semaphore: per-event-loop semaphores (safe across pytest loops and process restarts via a WeakKeyDictionary keyed by the running loop). - provider: holds a per-account (≤4) and per-host (PER_HOST_CONN_CAP) slot for the whole lifetime of an IMAP socket, released on close; SMTP send takes a per-host slot for its duration. A worker can no longer open hundreds of sockets to one host/account. - indexer: parsing goes through a PARSE_CONCURRENCY semaphore, and a message over STREAM_PARSE_MB is parsed from its blob file (parse_message_file) instead of an in-memory tree, so peak RAM ~= PARSE_CONCURRENCY x STREAM_PARSE_MB regardless of mailbox sizes. Backfill parallelism stays bounded by the bulk queue only. - config: PER_HOST_CONN_CAP, PARSE_CONCURRENCY, STREAM_PARSE_MB. - tests: per-account cap blocks the 5th connection / distinct accounts independent; file-vs-bytes parse equivalence + large-message streaming routing. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
This commit is contained in:
@@ -50,6 +50,10 @@ class Settings(BaseSettings):
|
||||
tier_session_grace_min: int = Field(default=30, alias="TIER_SESSION_GRACE_MIN")
|
||||
tier_cold_after_d: int = Field(default=3, alias="TIER_COLD_AFTER_D")
|
||||
hot_budget_per_worker: int = Field(default=250, alias="HOT_BUDGET_PER_WORKER")
|
||||
# Memory/connection bounds (§7, §4): peak RAM ~= PARSE_CONCURRENCY x STREAM_PARSE_MB.
|
||||
per_host_conn_cap: int = Field(default=50, alias="PER_HOST_CONN_CAP")
|
||||
parse_concurrency: int = Field(default=4, alias="PARSE_CONCURRENCY")
|
||||
stream_parse_mb: int = Field(default=8, alias="STREAM_PARSE_MB")
|
||||
|
||||
oauth_google_client_id: str = Field(default="", alias="OAUTH_GOOGLE_CLIENT_ID")
|
||||
oauth_google_client_secret: str = Field(default="", alias="OAUTH_GOOGLE_CLIENT_SECRET")
|
||||
|
||||
Reference in New Issue
Block a user