Attachments stored once; nightly clean-up of unused files

Attachments are no longer stored apart from their message (they were kept
twice: base64 inside the raw message and decoded on their own). The indexer
records the MIME part (attachments.part_index, migration 0023); downloads,
zip, inline images, thumbnails and forwarding take it out of the message;
videos stream from a short-lived copy (kept a day, 1 GiB per user). A
background pass on worker start removes the copies stored before, each only
after checking it can be taken out again (spec App. D §D.6 item 1, criterion
corrected to <= 60%).

A nightly job (03:47, after the Trash/Spam purge) removes files nothing
refers to any more — deleted or purged messages, previews of attachments
gone, uploads no draft has used for a week, expired streaming copies. Files
younger than an hour are left alone; a sent message waiting for its Sent copy
is held until its job is done. Account removal keeps files a draft still uses.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
This commit is contained in:
2026-10-08 20:20:21 +02:00
co-authored by Claude Opus 5.5
parent 0514053252
commit 3957dd540a
17 changed files with 1063 additions and 27 deletions
+17 -5
View File
@@ -15,6 +15,7 @@ from __future__ import annotations
import io
import logging
import uuid
from collections.abc import Callable
from pathlib import Path
from app.indexer import blobstore
@@ -86,8 +87,16 @@ def _render(data: bytes, mime_type: str) -> bytes | None:
return out.getvalue()
def thumbnail(user_id: uuid.UUID, sha: str, mime_type: str) -> bytes | None:
"""The thumbnail of a stored attachment, or None if it cannot have one.
def thumbnail(
user_id: uuid.UUID,
sha: str,
mime_type: str,
size: int,
load: Callable[[], bytes],
) -> bytes | None:
"""The thumbnail of an attachment (content hash ``sha``, ``size`` bytes,
read with ``load`` only when it is not made yet), or None if it cannot
have one.
Blocking (decoding is CPU work): call it in a thread.
"""
@@ -97,11 +106,14 @@ def thumbnail(user_id: uuid.UUID, sha: str, mime_type: str) -> bytes | None:
none = _path(user_id, sha, ".none")
if none.exists() or not can_thumbnail(mime_type):
return None
path = blobstore.blob_path(user_id, sha)
if not path.exists() or path.stat().st_size > _MAX_INPUT_BYTES:
if size > _MAX_INPUT_BYTES:
return None
try:
thumb = _render(path.read_bytes(), mime_type)
data = load()
except FileNotFoundError:
return None
try:
thumb = _render(data, mime_type)
except Exception as exc: # noqa: BLE001 - any undecodable file: no thumbnail
logger.info("no thumbnail for blob %s (%s): %s", sha, mime_type, exc)
thumb = None