Attachments stored once; nightly clean-up of unused files
Attachments are no longer stored apart from their message (they were kept twice: base64 inside the raw message and decoded on their own). The indexer records the MIME part (attachments.part_index, migration 0023); downloads, zip, inline images, thumbnails and forwarding take it out of the message; videos stream from a short-lived copy (kept a day, 1 GiB per user). A background pass on worker start removes the copies stored before, each only after checking it can be taken out again (spec App. D §D.6 item 1, criterion corrected to <= 60%). A nightly job (03:47, after the Trash/Spam purge) removes files nothing refers to any more — deleted or purged messages, previews of attachments gone, uploads no draft has used for a week, expired streaming copies. Files younger than an hour are left alone; a sent message waiting for its Sent copy is held until its job is done. Account removal keeps files a draft still uses. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
This commit is contained in:
@@ -877,7 +877,7 @@ If phones/desktop clients may use the mailbox over IMAP/SMTP, a password alone w
|
||||
### D.6 Storing mail once
|
||||
Today the indexer keeps every message in full (raw `.eml` blob + sanitized body in `body_parts` + search index), so external accounts are already a second copy, and with Stalwart both copies would sit on the same disk. Ideas, best first:
|
||||
|
||||
1. **Attachments are stored twice inside webmail today.** `indexer.py` stores each attachment as its own blob *and* it is inside the raw blob (base64), ≈2.3× its size; attachments are most of mail's bytes. Fix: keep only the raw message, extract an attachment when downloaded, keep thumbnails cached. Roughly halves storage for **all** accounts, independent of Stalwart. A cleanup script frees existing duplicates.
|
||||
1. **Attachments are stored twice inside webmail today.** `indexer.py` stores each attachment as its own blob *and* it is inside the raw blob (base64), ≈2.3× its size; attachments are most of mail's bytes. Fix: keep only the raw message, extract an attachment when downloaded, keep thumbnails cached. Cuts attachment storage by about 40% (≈1.37× instead of ≈2.37× the file size) for **all** accounts, independent of Stalwart. A background pass frees existing duplicates. **Done (v1.3.34):** `app/indexer/attachments.py`, migration 0023.
|
||||
2. **Native accounts: no raw copy in webmail.** Stalwart is the master copy; webmail keeps metadata, sanitized body, snippet and search index (~5–10% of the size). Full message, attachments and "show original" are fetched on demand over the local network (IMAP `BODY[part]` or JMAP blob download), with a small cache for repeat opens and thumbnails.
|
||||
3. **External accounts (optional setting): age out old raw copies.** Keep raw blobs for N days or the newest X GB; refetch older ones over IMAP when needed. Readable bodies stay local; old attachments then need the external server reachable.
|
||||
4. **Filesystem:** at least lz4 compression on the ZFS data share (QuTS hero). ZFS dedup would merge identical copies but costs a lot of RAM and only hits uncompressed byte-identical data — last resort.
|
||||
@@ -902,7 +902,7 @@ Today the indexer keeps every message in full (raw `.eml` blob + sanitized body
|
||||
2. The same handle + password signs in to IMAP (when §D.5 allows it); after a webmail password change or reset, the old password fails and the new one works with no other step; a blocked user cannot sign in to IMAP.
|
||||
3. The native account cannot be removed via API (409) or UI; deleting the user removes the Stalwart mailbox.
|
||||
4. Webmail stores no copy of the account's password (`secret_ct` unused for native accounts).
|
||||
5. After D1, an attachment-heavy fixture mailbox uses ≤ 55% of the pre-D1 blob storage, and every attachment still downloads byte-identical.
|
||||
5. After D1, an attachment-heavy fixture mailbox uses ≤ 60% of the pre-D1 blob storage, and every attachment still downloads byte-identical.
|
||||
6. After D4, a native message's raw blob is absent from `BLOB_ROOT`, yet opening, downloading attachments and "show original" all work; with Stalwart stopped, list and bodies still render from local data.
|
||||
7. Backup → restore on a fresh host brings back both webmail and Stalwart mail intact.
|
||||
|
||||
|
||||
Reference in New Issue
Block a user