Deploy: production Docker stack for a single host (Synology/QNAP NAS)

Everything needed to run the app on an Intel NAS that is also doing other jobs: one published port, no host networking, memory/CPU caps and log rotation on every container, and all state in bind mounts under DATA_ROOT so mail survives image updates, project deletion in the NAS UI, and full reinstalls — and so Hyper Backup/HBS3 can actually see it. - docker/Dockerfile.backend: multi-stage, non-root (uid 10001), tini for the <10s graceful lease release. Installs the four plugins as entry-point distributions. One image, several roles via docker/entrypoint.sh (api | worker-sync | worker-jobs | migrate | reindex | plugins). - docker/Dockerfile.frontend + nginx.conf: SPA + /api reverse proxy with the WebSocket upgrade for /api/ws, and X-Forwarded-Proto preserved so the API still emits __Host-/Secure cookies when TLS is terminated by the NAS reverse proxy (identity/cookies.py keys off request scheme). - docker/docker-compose.prod.yml: project name webmail-prod, NOT webmail — docker-compose.yml (dev services) owns that name, and two compose files sharing one project name fight over each other's containers. Meilisearch is an opt-in profile; SEARCH_BACKEND=pg is the default so a small NAS pays nothing for search. - migrate runs as a one-shot (core Alembic head + every plugin branch) that everything else waits on, so a bad migration can never leave new code running against an old schema. - docker/backup.sh: nightly pg_dump verified with `pg_restore --list` before it counts as a backup, with retention and a baseline on first start. - deploy/webmail.sh: init/doctor/up/down/status/logs/update/rollback/ backup/restore/reindex/psql/shell. `update` is backup -> images -> migrate -> roll workers, api, web -> health-gate, with automatic rollback to the previous version if the gate fails. `doctor` refuses to be quiet when DATA_ROOT is on NFS/SMB (silent Postgres corruption). - deploy/build-push.sh: build images off-NAS (the frontend build wants ~2GB RAM) and push to Gitea, or --save a tar to carry over. - scripts/reindex_search.py: rebuild the search index from Postgres alone. Every SearchDoc field is already stored, so the index is derived data that never needs backing up — this closes the only real objection to running Meilisearch on a box whose volumes may be wiped. - pyproject: package-data for accounts/presets.yaml and the shared plugin Alembic env. Without it a non-editable install (i.e. any image) silently ships a package that fails at runtime. - GET /api/version, used by the deploy health gate. Verified end to end on a real stack: fresh install, all services healthy, core + 4 plugin migration branches applied, login/SPA/plugin bundles through nginx, WS upgrade returning 101, __Host- cookies under a simulated TLS proxy, backup -> restore round-trip preserving user and plugin schema, failed-update auto-rollback (both a bad migration and a boot-crashing API), and Meilisearch profile serving a live query. ruff + mypy --strict clean, 363/363 backend tests pass. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01Nux3LCu4zP9nih4BRhPyHw
2026-09-14 18:53:50 +02:00
parent a27755a4d3
commit 5553b5febe
17 changed files with 1810 additions and 0 deletions
@@ -0,0 +1,149 @@
+"""Rebuild the search index for every message, from Postgres alone.
+
+Both backends index the same :class:`SearchDoc` (SPEC §5.2), and every field of
+it is already stored in the database — subject/addresses/flags on ``messages``,
+the plain-text body on ``body_parts``, filenames on ``attachments``. So the
+index is fully derivable and never needs its own backup: if the Meilisearch
+volume is lost, or you switch SEARCH_BACKEND, run this.
+
+    python scripts/reindex_search.py                 # everything
+    python scripts/reindex_search.py --user <uuid>   # one user
+    python scripts/reindex_search.py --purge         # drop stale docs first
+
+In Docker:  ./deploy/webmail.sh reindex [--purge]
+
+``--purge`` clears each user's existing documents before rebuilding, which is
+what you want after deleting mail outside the app or when the index has drifted;
+without it the rebuild is a pure upsert (safe to re-run, safe to interrupt).
+"""
+
+from __future__ import annotations
+
+import argparse
+import asyncio
+import uuid
+from collections import defaultdict
+from datetime import datetime
+
+from sqlalchemy import select
+
+from app.db import get_sessionmaker
+from app.models import Attachment, BodyPart, Mailbox, Message
+from app.search.base import SearchBackend, SearchDoc
+from app.search.factory import get_search_backend
+
+# Rows pulled per round-trip, and how many documents are pushed to the backend
+# concurrently. Deliberately modest: this runs on a NAS alongside live sync.
+BATCH = 500
+CONCURRENCY = 8
+
+
+def _epoch(value: datetime) -> int:
+    return int(value.timestamp())
+
+
+async def _index_batch(backend: SearchBackend, docs: list[SearchDoc]) -> None:
+    sem = asyncio.Semaphore(CONCURRENCY)
+
+    async def one(doc: SearchDoc) -> None:
+        async with sem:
+            await backend.index(doc)
+
+    await asyncio.gather(*(one(d) for d in docs))
+
+
+async def main() -> None:
+    parser = argparse.ArgumentParser(description="Rebuild the search index from Postgres")
+    parser.add_argument("--user", help="only this user_id (UUID)")
+    parser.add_argument(
+        "--purge",
+        action="store_true",
+        help="delete each user's existing documents before rebuilding",
+    )
+    args = parser.parse_args()
+
+    only_user = uuid.UUID(args.user) if args.user else None
+    backend = get_search_backend()
+    sm = get_sessionmaker()
+
+    if args.purge:
+        async with sm() as session:
+            users = (
+                [only_user]
+                if only_user
+                else list(await session.scalars(select(Message.user_id).distinct()))
+            )
+        for user_id in users:
+            await backend.delete_user(user_id)
+        print(f"purged existing documents for {len(users)} user(s)")
+
+    total = 0
+    after: uuid.UUID | None = None
+    while True:
+        async with sm() as session:
+            stmt = (
+                select(Message, Mailbox.special_use)
+                .join(Mailbox, Mailbox.id == Message.mailbox_id)
+                .order_by(Message.id)
+                .limit(BATCH)
+            )
+            if after is not None:
+                stmt = stmt.where(Message.id > after)
+            if only_user is not None:
+                stmt = stmt.where(Message.user_id == only_user)
+            rows = list(await session.execute(stmt))
+            if not rows:
+                break
+
+            ids = [row[0].id for row in rows]
+            after = ids[-1]
+
+            # Two batched lookups instead of 2N per-message queries.
+            bodies: dict[uuid.UUID, str] = {}
+            for message_id, content in await session.execute(
+                select(BodyPart.message_id, BodyPart.content).where(
+                    BodyPart.message_id.in_(ids), BodyPart.kind == "text"
+                )
+            ):
+                bodies.setdefault(message_id, content)
+
+            names: dict[uuid.UUID, list[str]] = defaultdict(list)
+            for message_id, filename in await session.execute(
+                select(Attachment.message_id, Attachment.filename).where(
+                    Attachment.message_id.in_(ids),
+                    Attachment.content_id.is_(None),  # real attachments, not inline images
+                )
+            ):
+                names[message_id].append(filename)
+
+        docs = [
+            SearchDoc(
+                message_id=msg.id,
+                thread_id=msg.thread_id or msg.id,
+                user_id=msg.user_id,
+                account_id=msg.account_id,
+                mailbox_special_use=special_use,
+                from_=msg.from_addr,
+                to=msg.to_addr,
+                cc=msg.cc_addr,
+                subject=msg.subject,
+                body_text=bodies.get(msg.id, ""),
+                attachment_names=names.get(msg.id, []),
+                has_attachments=msg.has_attachments,
+                is_unread="\\Seen" not in msg.flags,
+                is_starred="\\Flagged" in msg.flags,
+                date=_epoch(msg.internal_date),
+                size_bytes=msg.size_bytes,
+            )
+            for msg, special_use in rows
+        ]
+
+        await _index_batch(backend, docs)
+        total += len(docs)
+        print(f"indexed {total} messages", flush=True)
+
+    print(f"done: {total} messages reindexed")
+
+
+if __name__ == "__main__":
+    asyncio.run(main())