Attachments stored once; nightly clean-up of unused files
Attachments are no longer stored apart from their message (they were kept twice: base64 inside the raw message and decoded on their own). The indexer records the MIME part (attachments.part_index, migration 0023); downloads, zip, inline images, thumbnails and forwarding take it out of the message; videos stream from a short-lived copy (kept a day, 1 GiB per user). A background pass on worker start removes the copies stored before, each only after checking it can be taken out again (spec App. D §D.6 item 1, criterion corrected to <= 60%). A nightly job (03:47, after the Trash/Spam purge) removes files nothing refers to any more — deleted or purged messages, previews of attachments gone, uploads no draft has used for a week, expired streaming copies. Files younger than an hour are left alone; a sent message waiting for its Sent copy is held until its job is done. Account removal keeps files a draft still uses. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
This commit is contained in:
@@ -15,6 +15,7 @@ from __future__ import annotations
|
||||
import io
|
||||
import logging
|
||||
import uuid
|
||||
from collections.abc import Callable
|
||||
from pathlib import Path
|
||||
|
||||
from app.indexer import blobstore
|
||||
@@ -86,8 +87,16 @@ def _render(data: bytes, mime_type: str) -> bytes | None:
|
||||
return out.getvalue()
|
||||
|
||||
|
||||
def thumbnail(user_id: uuid.UUID, sha: str, mime_type: str) -> bytes | None:
|
||||
"""The thumbnail of a stored attachment, or None if it cannot have one.
|
||||
def thumbnail(
|
||||
user_id: uuid.UUID,
|
||||
sha: str,
|
||||
mime_type: str,
|
||||
size: int,
|
||||
load: Callable[[], bytes],
|
||||
) -> bytes | None:
|
||||
"""The thumbnail of an attachment (content hash ``sha``, ``size`` bytes,
|
||||
read with ``load`` only when it is not made yet), or None if it cannot
|
||||
have one.
|
||||
|
||||
Blocking (decoding is CPU work): call it in a thread.
|
||||
"""
|
||||
@@ -97,11 +106,14 @@ def thumbnail(user_id: uuid.UUID, sha: str, mime_type: str) -> bytes | None:
|
||||
none = _path(user_id, sha, ".none")
|
||||
if none.exists() or not can_thumbnail(mime_type):
|
||||
return None
|
||||
path = blobstore.blob_path(user_id, sha)
|
||||
if not path.exists() or path.stat().st_size > _MAX_INPUT_BYTES:
|
||||
if size > _MAX_INPUT_BYTES:
|
||||
return None
|
||||
try:
|
||||
thumb = _render(path.read_bytes(), mime_type)
|
||||
data = load()
|
||||
except FileNotFoundError:
|
||||
return None
|
||||
try:
|
||||
thumb = _render(data, mime_type)
|
||||
except Exception as exc: # noqa: BLE001 - any undecodable file: no thumbnail
|
||||
logger.info("no thumbnail for blob %s (%s): %s", sha, mime_type, exc)
|
||||
thumb = None
|
||||
|
||||
Reference in New Issue
Block a user