lukasandClaude Opus 5.5
3957dd540a
Attachments are no longer stored apart from their message (they were kept twice: base64 inside the raw message and decoded on their own). The indexer records the MIME part (attachments.part_index, migration 0023); downloads, zip, inline images, thumbnails and forwarding take it out of the message; videos stream from a short-lived copy (kept a day, 1 GiB per user). A background pass on worker start removes the copies stored before, each only after checking it can be taken out again (spec App. D §D.6 item 1, criterion corrected to <= 60%). A nightly job (03:47, after the Trash/Spam purge) removes files nothing refers to any more — deleted or purged messages, previews of attachments gone, uploads no draft has used for a week, expired streaming copies. Files younger than an hour are left alone; a sent message waiting for its Sent copy is held until its job is done. Account removal keeps files a draft still uses. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
58 lines
1.7 KiB
Python
58 lines
1.7 KiB
Python
"""arq queue helpers (enqueue side, used by the API).
|
|
|
|
Two named queues (refactor §6): ``interactive`` (fast user actions) and ``bulk``
|
|
(backfills). Enqueue via :func:`enqueue_job`, which routes each job to its queue —
|
|
the API never needs to know queue names.
|
|
"""
|
|
|
|
from __future__ import annotations
|
|
|
|
from typing import Any
|
|
|
|
from arq import create_pool
|
|
from arq.connections import ArqRedis, RedisSettings
|
|
from arq.jobs import Job
|
|
|
|
from app.config import get_settings
|
|
|
|
_pool: ArqRedis | None = None
|
|
|
|
QUEUE_INTERACTIVE = "interactive"
|
|
QUEUE_BULK = "bulk"
|
|
|
|
# Jobs default to the interactive queue; only bulk backfills go to the bulk queue.
|
|
_BULK_JOBS = frozenset({"sync_bootstrap_account", "dedupe_existing_job"})
|
|
|
|
|
|
def queue_for(job_name: str) -> str:
|
|
"""Return the queue a job should run on."""
|
|
return QUEUE_BULK if job_name in _BULK_JOBS else QUEUE_INTERACTIVE
|
|
|
|
|
|
def redis_settings() -> RedisSettings:
|
|
"""Return arq RedisSettings derived from ``REDIS_URL``."""
|
|
return RedisSettings.from_dsn(get_settings().redis_url)
|
|
|
|
|
|
async def get_arq_pool() -> ArqRedis:
|
|
"""Return (creating on first use) the shared arq enqueue pool."""
|
|
global _pool
|
|
if _pool is None:
|
|
_pool = await create_pool(redis_settings())
|
|
return _pool
|
|
|
|
|
|
async def enqueue_job(name: str, *args: Any, **kwargs: Any) -> Job | None:
|
|
"""Enqueue ``name`` onto its routed queue (interactive/bulk)."""
|
|
pool = await get_arq_pool()
|
|
kwargs.setdefault("_queue_name", queue_for(name))
|
|
return await pool.enqueue_job(name, *args, **kwargs)
|
|
|
|
|
|
async def close_arq_pool() -> None:
|
|
"""Close the arq pool."""
|
|
global _pool
|
|
if _pool is not None:
|
|
await _pool.aclose()
|
|
_pool = None
|