From 5553b5febe5d99667260eb42ddc2d27852e30558 Mon Sep 17 00:00:00 2001 From: Lukas Remis Date: Mon, 14 Sep 2026 18:53:50 +0200 Subject: [PATCH] Deploy: production Docker stack for a single host (Synology/QNAP NAS) MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Everything needed to run the app on an Intel NAS that is also doing other jobs: one published port, no host networking, memory/CPU caps and log rotation on every container, and all state in bind mounts under DATA_ROOT so mail survives image updates, project deletion in the NAS UI, and full reinstalls — and so Hyper Backup/HBS3 can actually see it. - docker/Dockerfile.backend: multi-stage, non-root (uid 10001), tini for the <10s graceful lease release. Installs the four plugins as entry-point distributions. One image, several roles via docker/entrypoint.sh (api | worker-sync | worker-jobs | migrate | reindex | plugins). - docker/Dockerfile.frontend + nginx.conf: SPA + /api reverse proxy with the WebSocket upgrade for /api/ws, and X-Forwarded-Proto preserved so the API still emits __Host-/Secure cookies when TLS is terminated by the NAS reverse proxy (identity/cookies.py keys off request scheme). - docker/docker-compose.prod.yml: project name webmail-prod, NOT webmail — docker-compose.yml (dev services) owns that name, and two compose files sharing one project name fight over each other's containers. Meilisearch is an opt-in profile; SEARCH_BACKEND=pg is the default so a small NAS pays nothing for search. - migrate runs as a one-shot (core Alembic head + every plugin branch) that everything else waits on, so a bad migration can never leave new code running against an old schema. - docker/backup.sh: nightly pg_dump verified with `pg_restore --list` before it counts as a backup, with retention and a baseline on first start. - deploy/webmail.sh: init/doctor/up/down/status/logs/update/rollback/ backup/restore/reindex/psql/shell. `update` is backup -> images -> migrate -> roll workers, api, web -> health-gate, with automatic rollback to the previous version if the gate fails. `doctor` refuses to be quiet when DATA_ROOT is on NFS/SMB (silent Postgres corruption). - deploy/build-push.sh: build images off-NAS (the frontend build wants ~2GB RAM) and push to Gitea, or --save a tar to carry over. - scripts/reindex_search.py: rebuild the search index from Postgres alone. Every SearchDoc field is already stored, so the index is derived data that never needs backing up — this closes the only real objection to running Meilisearch on a box whose volumes may be wiped. - pyproject: package-data for accounts/presets.yaml and the shared plugin Alembic env. Without it a non-editable install (i.e. any image) silently ships a package that fails at runtime. - GET /api/version, used by the deploy health gate. Verified end to end on a real stack: fresh install, all services healthy, core + 4 plugin migration branches applied, login/SPA/plugin bundles through nginx, WS upgrade returning 101, __Host- cookies under a simulated TLS proxy, backup -> restore round-trip preserving user and plugin schema, failed-update auto-rollback (both a bad migration and a boot-crashing API), and Meilisearch profile serving a live query. ruff + mypy --strict clean, 363/363 backend tests pass. Co-Authored-By: Claude Opus 5 Claude-Session: https://claude.ai/code/session_01Nux3LCu4zP9nih4BRhPyHw --- .dockerignore | 30 +++ .gitignore | 2 + Makefile | 19 ++ backend/app/config.py | 3 + backend/app/main.py | 5 + backend/pyproject.toml | 7 + backend/scripts/reindex_search.py | 149 ++++++++++++ deploy/build-push.sh | 77 ++++++ deploy/webmail.sh | 378 ++++++++++++++++++++++++++++++ docker/.env.prod.example | 111 +++++++++ docker/Dockerfile.backend | 79 +++++++ docker/Dockerfile.frontend | 35 +++ docker/backup.sh | 88 +++++++ docker/docker-compose.prod.yml | 307 ++++++++++++++++++++++++ docker/entrypoint.sh | 62 +++++ docker/nginx.conf | 104 ++++++++ docs/nas-install.md | 354 ++++++++++++++++++++++++++++ 17 files changed, 1810 insertions(+) create mode 100644 .dockerignore create mode 100644 backend/scripts/reindex_search.py create mode 100755 deploy/build-push.sh create mode 100755 deploy/webmail.sh create mode 100644 docker/.env.prod.example create mode 100644 docker/Dockerfile.backend create mode 100644 docker/Dockerfile.frontend create mode 100755 docker/backup.sh create mode 100644 docker/docker-compose.prod.yml create mode 100755 docker/entrypoint.sh create mode 100644 docker/nginx.conf create mode 100644 docs/nas-install.md diff --git a/.dockerignore b/.dockerignore new file mode 100644 index 0000000..a561148 --- /dev/null +++ b/.dockerignore @@ -0,0 +1,30 @@ +# Build context hygiene: the backend image builds from the repo root (it needs +# backend/ *and* plugins/), so everything heavy or secret is excluded here. +.git +.github +.gitea +.claude +**/node_modules +**/dist +**/__pycache__ +**/*.py[cod] +**/.mypy_cache +**/.ruff_cache +**/.pytest_cache +**/*.egg-info +# Runtime data: mail content and local blobs must never enter an image layer. +var/ +backend/var/ +**/var/blobs +# Secrets +.env.dev +.env.*.local +gitea-server-credentials.md +docker/.env.prod +deploy/*.env +# Docs / screenshots / scratch +screenshots/ +docs/ +*.md +!README.md +claude-session.txt diff --git a/.gitignore b/.gitignore index 91caee1..b131317 100644 --- a/.gitignore +++ b/.gitignore @@ -1,5 +1,7 @@ # Secrets / local env .env.dev +docker/.env.prod +docker/.env .env.*.local gitea-server-credentials.md diff --git a/Makefile b/Makefile index 3815c37..495e093 100644 --- a/Makefile +++ b/Makefile @@ -21,3 +21,22 @@ test: cd backend && pytest -q && cd ../frontend && npx vitest run mail-test: ## start throwaway IMAP/SMTP server docker compose -f docker/docker-compose.yml up -d greenmail + +# --- production (NAS / single-host Docker deploy; see docs/nas-install.md) --- +prod-init: ## generate docker/.env.prod + data dirs + ./deploy/webmail.sh init +prod-up: ## start the production stack + ./deploy/webmail.sh up +prod-down: ## stop the production stack (data kept) + ./deploy/webmail.sh down +prod-status: ## what is running, versions, disk, backups + ./deploy/webmail.sh status +prod-update: ## backup -> migrate -> roll -> health-gate (auto-rollback on failure) + ./deploy/webmail.sh update +prod-backup: ## take a verified database dump now + ./deploy/webmail.sh backup +prod-doctor: ## pre-flight checks on a deployment + ./deploy/webmail.sh doctor +prod-images: ## build production images locally (no push) + docker build -f docker/Dockerfile.backend -t webmail/backend:local . + docker build -f docker/Dockerfile.frontend -t webmail/web:local . diff --git a/backend/app/config.py b/backend/app/config.py index a52b5e0..86b5622 100644 --- a/backend/app/config.py +++ b/backend/app/config.py @@ -32,6 +32,9 @@ class Settings(BaseSettings): session_ttl_days: int = Field(default=30, alias="SESSION_TTL_DAYS") max_attachment_mb: int = Field(default=25, alias="MAX_ATTACHMENT_MB") log_level: str = Field(default="INFO", alias="LOG_LEVEL") + # Release identifier baked into the container image at build time; surfaced + # by GET /api/version so a deploy can verify what is actually running. + app_version: str = Field(default="dev", alias="APP_VERSION") # --- Sync engine v2 (workers layer) -------------------------------------- # Worker role: "sync" (lease-owned account runners) | "jobs" (arq consumer) diff --git a/backend/app/main.py b/backend/app/main.py index 6ff12b9..1af8c5a 100644 --- a/backend/app/main.py +++ b/backend/app/main.py @@ -45,6 +45,11 @@ def create_app() -> FastAPI: await get_redis().ping() return {"status": "ok"} + @app.get("/api/version") + async def version() -> dict[str, str]: + """Report the running release, for deploy/rollback verification.""" + return {"version": get_settings().app_version} + from app.api import register_routers register_routers(app) diff --git a/backend/pyproject.toml b/backend/pyproject.toml index 33f0f36..7d2e1a9 100644 --- a/backend/pyproject.toml +++ b/backend/pyproject.toml @@ -52,6 +52,13 @@ dev = [ where = ["."] include = ["app*", "workers*"] +# Non-Python files that a *non-editable* install (the Docker image) must carry: +# the account presets config (SPEC §6.2) and the shared Alembic env used by the +# per-plugin migration runner (app/plugins/migrate.py). Without these, a +# `pip install ./backend` produces a package that fails at runtime. +[tool.setuptools.package-data] +app = ["accounts/presets.yaml", "plugins/alembic/*"] + [tool.ruff] line-length = 100 target-version = "py311" diff --git a/backend/scripts/reindex_search.py b/backend/scripts/reindex_search.py new file mode 100644 index 0000000..3e0c2f0 --- /dev/null +++ b/backend/scripts/reindex_search.py @@ -0,0 +1,149 @@ +"""Rebuild the search index for every message, from Postgres alone. + +Both backends index the same :class:`SearchDoc` (SPEC §5.2), and every field of +it is already stored in the database — subject/addresses/flags on ``messages``, +the plain-text body on ``body_parts``, filenames on ``attachments``. So the +index is fully derivable and never needs its own backup: if the Meilisearch +volume is lost, or you switch SEARCH_BACKEND, run this. + + python scripts/reindex_search.py # everything + python scripts/reindex_search.py --user # one user + python scripts/reindex_search.py --purge # drop stale docs first + +In Docker: ./deploy/webmail.sh reindex [--purge] + +``--purge`` clears each user's existing documents before rebuilding, which is +what you want after deleting mail outside the app or when the index has drifted; +without it the rebuild is a pure upsert (safe to re-run, safe to interrupt). +""" + +from __future__ import annotations + +import argparse +import asyncio +import uuid +from collections import defaultdict +from datetime import datetime + +from sqlalchemy import select + +from app.db import get_sessionmaker +from app.models import Attachment, BodyPart, Mailbox, Message +from app.search.base import SearchBackend, SearchDoc +from app.search.factory import get_search_backend + +# Rows pulled per round-trip, and how many documents are pushed to the backend +# concurrently. Deliberately modest: this runs on a NAS alongside live sync. +BATCH = 500 +CONCURRENCY = 8 + + +def _epoch(value: datetime) -> int: + return int(value.timestamp()) + + +async def _index_batch(backend: SearchBackend, docs: list[SearchDoc]) -> None: + sem = asyncio.Semaphore(CONCURRENCY) + + async def one(doc: SearchDoc) -> None: + async with sem: + await backend.index(doc) + + await asyncio.gather(*(one(d) for d in docs)) + + +async def main() -> None: + parser = argparse.ArgumentParser(description="Rebuild the search index from Postgres") + parser.add_argument("--user", help="only this user_id (UUID)") + parser.add_argument( + "--purge", + action="store_true", + help="delete each user's existing documents before rebuilding", + ) + args = parser.parse_args() + + only_user = uuid.UUID(args.user) if args.user else None + backend = get_search_backend() + sm = get_sessionmaker() + + if args.purge: + async with sm() as session: + users = ( + [only_user] + if only_user + else list(await session.scalars(select(Message.user_id).distinct())) + ) + for user_id in users: + await backend.delete_user(user_id) + print(f"purged existing documents for {len(users)} user(s)") + + total = 0 + after: uuid.UUID | None = None + while True: + async with sm() as session: + stmt = ( + select(Message, Mailbox.special_use) + .join(Mailbox, Mailbox.id == Message.mailbox_id) + .order_by(Message.id) + .limit(BATCH) + ) + if after is not None: + stmt = stmt.where(Message.id > after) + if only_user is not None: + stmt = stmt.where(Message.user_id == only_user) + rows = list(await session.execute(stmt)) + if not rows: + break + + ids = [row[0].id for row in rows] + after = ids[-1] + + # Two batched lookups instead of 2N per-message queries. + bodies: dict[uuid.UUID, str] = {} + for message_id, content in await session.execute( + select(BodyPart.message_id, BodyPart.content).where( + BodyPart.message_id.in_(ids), BodyPart.kind == "text" + ) + ): + bodies.setdefault(message_id, content) + + names: dict[uuid.UUID, list[str]] = defaultdict(list) + for message_id, filename in await session.execute( + select(Attachment.message_id, Attachment.filename).where( + Attachment.message_id.in_(ids), + Attachment.content_id.is_(None), # real attachments, not inline images + ) + ): + names[message_id].append(filename) + + docs = [ + SearchDoc( + message_id=msg.id, + thread_id=msg.thread_id or msg.id, + user_id=msg.user_id, + account_id=msg.account_id, + mailbox_special_use=special_use, + from_=msg.from_addr, + to=msg.to_addr, + cc=msg.cc_addr, + subject=msg.subject, + body_text=bodies.get(msg.id, ""), + attachment_names=names.get(msg.id, []), + has_attachments=msg.has_attachments, + is_unread="\\Seen" not in msg.flags, + is_starred="\\Flagged" in msg.flags, + date=_epoch(msg.internal_date), + size_bytes=msg.size_bytes, + ) + for msg, special_use in rows + ] + + await _index_batch(backend, docs) + total += len(docs) + print(f"indexed {total} messages", flush=True) + + print(f"done: {total} messages reindexed") + + +if __name__ == "__main__": + asyncio.run(main()) diff --git a/deploy/build-push.sh b/deploy/build-push.sh new file mode 100755 index 0000000..6f64278 --- /dev/null +++ b/deploy/build-push.sh @@ -0,0 +1,77 @@ +#!/usr/bin/env bash +# Build the two production images on THIS machine and either push them to a +# registry or export them as a file you can copy to the NAS. +# +# Why you'd use this instead of building on the NAS: the frontend build (Vite + +# tsc) wants ~2 GB of RAM and a lot of CPU. On a 2-4 GB NAS that is slow at best +# and an OOM at worst, and it competes with everything else the box is doing. +# +# ./deploy/build-push.sh v1.0.0 # build + push to $REGISTRY_PREFIX +# REGISTRY_PREFIX=192.168.1.185:8418/claude/webmail \ +# ./deploy/build-push.sh v1.0.0 +# ./deploy/build-push.sh v1.0.0 --save # build + write a .tar to load on the NAS +# +# On the NAS afterwards: +# push mode: set REGISTRY_PREFIX and IMAGE_TAG in docker/.env.prod, then +# ./deploy/webmail.sh update v1.0.0 +# save mode: docker load -i webmail-v1.0.0.tar && ./deploy/webmail.sh update v1.0.0 +set -euo pipefail + +ROOT_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")/.." && pwd)" +TAG="${1:-}" +MODE="${2:-push}" +REGISTRY_PREFIX="${REGISTRY_PREFIX:-webmail}" +PLATFORM="${PLATFORM:-linux/amd64}" # Synology and QNAP Intel boxes + +[ -n "$TAG" ] || { echo "usage: $0 [--save]" >&2; exit 1; } + +GIT_SHA="$(git -C "$ROOT_DIR" rev-parse --short HEAD 2>/dev/null || echo unknown)" +if ! git -C "$ROOT_DIR" diff --quiet 2>/dev/null; then + echo "warning: working tree has uncommitted changes; tagging ${TAG} anyway" >&2 +fi + +build() { + local name="$1" dockerfile="$2" + echo "== building ${REGISTRY_PREFIX}/${name}:${TAG}" + docker build \ + --platform "$PLATFORM" \ + -f "${ROOT_DIR}/docker/${dockerfile}" \ + --build-arg "APP_VERSION=${TAG}" \ + --build-arg "GIT_SHA=${GIT_SHA}" \ + -t "${REGISTRY_PREFIX}/${name}:${TAG}" \ + -t "${REGISTRY_PREFIX}/${name}:latest" \ + "$ROOT_DIR" +} + +build backend Dockerfile.backend +build web Dockerfile.frontend + +if [ "$MODE" = "--save" ]; then + out="${ROOT_DIR}/webmail-${TAG}.tar" + echo "== saving both images to ${out}" + docker save -o "$out" \ + "${REGISTRY_PREFIX}/backend:${TAG}" \ + "${REGISTRY_PREFIX}/web:${TAG}" + echo "done. Copy it to the NAS and run: docker load -i $(basename "$out")" + exit 0 +fi + +if [ "$REGISTRY_PREFIX" = "webmail" ]; then + cat >&2 <<'MSG' +REGISTRY_PREFIX is still the local default ("webmail"), so there is nothing to +push to. Either set it to your registry path, e.g. + + REGISTRY_PREFIX=192.168.1.185:8418/claude/webmail ./deploy/build-push.sh v1.0.0 + +(run `docker login 192.168.1.185:8418` once first), or re-run with --save to +produce a tar file instead. +MSG + exit 1 +fi + +echo "== pushing" +docker push "${REGISTRY_PREFIX}/backend:${TAG}" +docker push "${REGISTRY_PREFIX}/web:${TAG}" +docker push "${REGISTRY_PREFIX}/backend:latest" +docker push "${REGISTRY_PREFIX}/web:latest" +echo "pushed ${TAG}. On the NAS: ./deploy/webmail.sh update ${TAG}" diff --git a/deploy/webmail.sh b/deploy/webmail.sh new file mode 100755 index 0000000..2b227e2 --- /dev/null +++ b/deploy/webmail.sh @@ -0,0 +1,378 @@ +#!/usr/bin/env bash +# Webmail operator control script. One command per thing you might need to do. +# +# ./deploy/webmail.sh init create data dirs + a filled-in .env.prod +# ./deploy/webmail.sh up [--meili] start (builds images if missing) +# ./deploy/webmail.sh down stop (data is untouched) +# ./deploy/webmail.sh status what is running, and which version +# ./deploy/webmail.sh logs [svc] [-f] tail logs +# ./deploy/webmail.sh update [tag] backup -> new images -> migrate -> roll -> health-gate +# ./deploy/webmail.sh rollback [tag] go back to the previous version +# ./deploy/webmail.sh backup take a verified database dump now +# ./deploy/webmail.sh restore DESTRUCTIVE: restore a dump +# ./deploy/webmail.sh reindex [args] rebuild the search index from Postgres +# ./deploy/webmail.sh psql open a database shell +# ./deploy/webmail.sh shell [svc] shell inside a container (default: api) +# ./deploy/webmail.sh doctor check the install before it bites you +set -euo pipefail + +SCRIPT_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)" +ROOT_DIR="$(dirname "$SCRIPT_DIR")" +COMPOSE_FILE="${ROOT_DIR}/docker/docker-compose.prod.yml" +ENV_FILE="${ROOT_DIR}/docker/.env.prod" +ENV_EXAMPLE="${ROOT_DIR}/docker/.env.prod.example" + +# Services that must come back healthy after a roll. +HEALTH_SERVICES=(postgres redis api web) +HEALTH_TIMEOUT=120 + +red() { printf '\033[31m%s\033[0m\n' "$*"; } +grn() { printf '\033[32m%s\033[0m\n' "$*"; } +ylw() { printf '\033[33m%s\033[0m\n' "$*"; } +die() { red "error: $*" >&2; exit 1; } + +compose() { + docker compose --env-file "$ENV_FILE" -f "$COMPOSE_FILE" "$@" +} + +require_env() { + [ -f "$ENV_FILE" ] || die "$ENV_FILE not found. Run: ./deploy/webmail.sh init" + # shellcheck disable=SC1090 + set -a; . "$ENV_FILE"; set +a + [ -n "${DATA_ROOT:-}" ] || die "DATA_ROOT is not set in $ENV_FILE" + [ -n "${MASTER_KEY:-}" ] || die "MASTER_KEY is not set in $ENV_FILE" + [ -n "${POSTGRES_PASSWORD:-}" ] || die "POSTGRES_PASSWORD is not set in $ENV_FILE" +} + +state_dir() { echo "${DATA_ROOT}/state"; } + +# busybox sed (what you may get on DSM/QTS) has no -i suffix argument, so every +# in-place edit goes through a temp file. +set_env_value() { + local key="$1" value="$2" + sed "s|^${key}=.*|${key}=${value}|" "$ENV_FILE" > "${ENV_FILE}.tmp" \ + && mv "${ENV_FILE}.tmp" "$ENV_FILE" && chmod 600 "$ENV_FILE" +} + +record_version() { + local new="$1" cur + mkdir -p "$(state_dir)" 2>/dev/null || true + if ! { : > "$(state_dir)/.probe"; } 2>/dev/null; then + ylw "cannot write ${DATA_ROOT}/state — version history not recorded (run as the owner of DATA_ROOT)" + return 0 + fi + rm -f "$(state_dir)/.probe" + cur="$(cat "$(state_dir)/version" 2>/dev/null || echo "")" + [ -n "$cur" ] && [ "$cur" != "$new" ] && echo "$cur" > "$(state_dir)/previous" + echo "$new" > "$(state_dir)/version" +} + +# --- commands ---------------------------------------------------------------- + +cmd_init() { + if [ ! -f "$ENV_FILE" ]; then + cp "$ENV_EXAMPLE" "$ENV_FILE" + chmod 600 "$ENV_FILE" + local master pgpass + master="$(openssl rand -base64 32)" + pgpass="$(openssl rand -base64 24 | tr -d '/+=' | cut -c1-24)" + # Portable in-place edit (BSD and GNU sed disagree about -i). + sed "s|^MASTER_KEY=.*|MASTER_KEY=${master}|; s|^POSTGRES_PASSWORD=.*|POSTGRES_PASSWORD=${pgpass}|" \ + "$ENV_FILE" > "${ENV_FILE}.tmp" && mv "${ENV_FILE}.tmp" "$ENV_FILE" + chmod 600 "$ENV_FILE" + grn "created $ENV_FILE with freshly generated secrets" + ylw "" + ylw " >> BACK UP MASTER_KEY NOW. Copy it into your password manager. <<" + ylw " Without it, saved mail-account passwords cannot be decrypted even" + ylw " from a perfect database backup." + ylw "" + echo "Then edit $ENV_FILE and set at least: DATA_ROOT, APP_DOMAIN, TZ, WEB_PORT." + return 0 + fi + + ylw "$ENV_FILE already exists; leaving it alone." + require_env + mkdir -p "${DATA_ROOT}"/{postgres,blobs,redis,backups,meili,state} 2>/dev/null \ + || ylw "could not create every directory under ${DATA_ROOT} (the init container will)" + grn "data directories ready under ${DATA_ROOT}" +} + +cmd_doctor() { + require_env + local ok=0 + + echo "config:" + echo " DATA_ROOT ${DATA_ROOT}" + echo " APP_DOMAIN ${APP_DOMAIN:-}" + echo " WEB_PORT ${WEB_PORT:-8088}" + echo " SEARCH_BACKEND ${SEARCH_BACKEND:-pg}" + echo " IMAGE_TAG ${IMAGE_TAG:-local}" + echo + + if [ "$(printf '%s' "${MASTER_KEY}" | base64 -d 2>/dev/null | wc -c)" -ne 32 ]; then + red " MASTER_KEY is not 32 bytes of base64 — the app will refuse to start"; ok=1 + else + grn " MASTER_KEY looks well-formed" + fi + + if [ "$(stat -c %a "$ENV_FILE" 2>/dev/null || echo 600)" != "600" ]; then + ylw " $ENV_FILE is not chmod 600 (it holds your master key)" + fi + + # Missing directories are not a failure before the first start: `up` and the + # `init` container create them. Only say so, so a pre-install doctor is green. + local missing=0 + for d in postgres blobs redis backups state; do + [ -d "${DATA_ROOT}/${d}" ] || missing=1 + done + if [ "$missing" -eq 1 ]; then + ylw " data directories not created yet — 'up' will create them" + else + grn " data directories present" + fi + + # Probe the nearest existing ancestor: the NFS/SMB check has to work BEFORE + # the first install, which is the only time it can still save you. + local fstype probe="${DATA_ROOT}" + while [ ! -d "$probe" ] && [ "$probe" != "/" ] && [ -n "$probe" ]; do + probe="$(dirname "$probe")" + done + # `|| true`: with `set -e -o pipefail`, df failing on a not-yet-created + # DATA_ROOT would abort doctor on exactly the run where you need it most. + fstype="$(df -PT "$probe" 2>/dev/null | awk 'NR==2{print $2}' || true)" + [ -n "$fstype" ] || fstype="$(stat -f -c %T "$probe" 2>/dev/null || echo "")" + case "$fstype" in + nfs*|cifs|smbfs|fuse*) + red " ${DATA_ROOT} is on a network filesystem — Postgres WILL corrupt there." + red " Move DATA_ROOT to a local volume (btrfs/ext4)."; ok=1 ;; + "") ylw " could not determine the filesystem of ${DATA_ROOT}" ;; + *) grn " ${DATA_ROOT} is on a local filesystem" ;; + esac + + local avail + avail="$(df -Pm "$probe" 2>/dev/null | awk 'NR==2{print $4}' || true)" + [ -n "$avail" ] && { + if [ "$avail" -lt 5120 ]; then ylw " only ${avail} MB free on ${DATA_ROOT}" + else grn " ${avail} MB free on ${DATA_ROOT}"; fi + } + + if command -v ss >/dev/null 2>&1 && ss -ltn 2>/dev/null | grep -q ":${WEB_PORT:-8088} "; then + ylw " port ${WEB_PORT:-8088} already has a listener — change WEB_PORT if that is not us" + fi + + # chmod 700 and owned by uid 999, so count them as root in a container. + local recent + recent="$(docker run --rm -v "${DATA_ROOT}/backups:/b:ro" alpine:3.20 \ + sh -c 'find /b -maxdepth 1 -name "*.dump" -mtime -2 2>/dev/null | wc -l' 2>/dev/null || echo 0)" + if [ "${recent:-0}" -gt 0 ]; then grn " ${recent} database backup(s) from the last 48 h" + else ylw " no database backup in the last 48 h"; fi + + [ "$ok" -eq 0 ] && grn "doctor: no blocking problems" || red "doctor: problems found (above)" + return "$ok" +} + +cmd_up() { + require_env + local profile=() build=() + for arg in "$@"; do + case "$arg" in + --meili) profile=(--profile meili) ;; + --build) build=(--build) ;; + *) die "unknown option: $arg" ;; + esac + done + [ "${SEARCH_BACKEND:-pg}" = "meili" ] && [ ${#profile[@]} -eq 0 ] && { + ylw "SEARCH_BACKEND=meili — starting the Meilisearch container too" + profile=(--profile meili) + } + # Best-effort only: Docker creates any missing bind-mount path itself, and + # the `init` service fixes ownership. This line must not be fatal, because + # DATA_ROOT is typically root-owned while you may be running as yourself. + mkdir -p "${DATA_ROOT}"/{postgres,blobs,redis,backups,meili,state} 2>/dev/null || true + compose "${profile[@]}" up -d "${build[@]}" --wait + record_version "${IMAGE_TAG:-local}" + grn "up. open http://:${WEB_PORT:-8088}/" +} + +cmd_down() { require_env; compose --profile meili down; grn "stopped (data kept in ${DATA_ROOT})"; } +cmd_restart() { require_env; compose restart "${@:-}"; } + +cmd_status() { + require_env + compose --profile meili ps + echo + echo "recorded version : $(cat "$(state_dir)/version" 2>/dev/null || echo unknown)" + echo "previous version : $(cat "$(state_dir)/previous" 2>/dev/null || echo none)" + local running + running="$(compose exec -T api python -c \ + "import urllib.request,json;print(json.load(urllib.request.urlopen('http://127.0.0.1:8000/api/version'))['version'])" \ + 2>/dev/null || echo unreachable)" + echo "running version : ${running}" + echo + # The data directories are owned by the container users (postgres 999, + # backend 10001) and backups are chmod 700, so read them as root inside a + # throwaway container rather than reporting misleading zeroes. + docker run --rm -v "${DATA_ROOT}:/d:ro" alpine:3.20 sh -c ' + echo "data usage:" + du -sh /d/postgres /d/blobs /d/backups 2>/dev/null | sed "s|/d/| |" + echo + echo "latest backups:" + ls -lh /d/backups/*.dump 2>/dev/null | tail -3 | sed "s|/d/backups/| |" || echo " none yet" + ' +} + +cmd_logs() { + require_env + compose --profile meili logs --tail 200 "$@" +} + +health_gate() { + local deadline=$(( $(date +%s) + HEALTH_TIMEOUT )) svc cid state + while [ "$(date +%s)" -lt "$deadline" ]; do + local all_ok=1 + for svc in "${HEALTH_SERVICES[@]}"; do + cid="$(compose ps -q "$svc" 2>/dev/null || true)" + [ -n "$cid" ] || { all_ok=0; break; } + state="$(docker inspect -f '{{if .State.Health}}{{.State.Health.Status}}{{else}}{{.State.Status}}{{end}}' "$cid" 2>/dev/null || echo missing)" + case "$state" in + healthy|running) ;; + *) all_ok=0; break ;; + esac + done + [ "$all_ok" -eq 1 ] && return 0 + sleep 5 + done + return 1 +} + +cmd_backup() { + require_env + compose run --rm backup once +} + +cmd_update() { + require_env + local target="${1:-${IMAGE_TAG:-local}}" + local previous="${IMAGE_TAG:-local}" + + echo "== 1/5 backing up the database before anything changes" + cmd_backup || die "backup failed — refusing to update. Fix the backup first." + + echo "== 2/5 obtaining images for tag '${target}'" + if [ "${REGISTRY_PREFIX:-webmail}" = "webmail" ] || [ "$target" = "local" ]; then + IMAGE_TAG="$target" compose build + else + IMAGE_TAG="$target" compose pull || die "pull failed — nothing has changed, you are still on ${previous}" + fi + + echo "== 3/5 applying migrations" + IMAGE_TAG="$target" compose run --rm migrate \ + || die "migration failed — services untouched, still on ${previous}. Restore from ${DATA_ROOT}/backups if the schema is damaged." + + echo "== 4/5 rolling services" + # Workers first (leases transfer to the replacement within ~90 s; the lease + # protocol and idempotent (mailbox_id, uid) upserts make this lossless), + # then the API, then the web front end. + IMAGE_TAG="$target" compose up -d worker-sync worker-jobs + IMAGE_TAG="$target" compose up -d api web + + echo "== 5/5 health gate (${HEALTH_TIMEOUT}s)" + if health_gate; then + record_version "$target" + set_env_value IMAGE_TAG "$target" + grn "updated to ${target}" + else + red "health gate failed — rolling back to ${previous}" + IMAGE_TAG="$previous" compose up -d --wait || true + die "update to ${target} failed; production is back on ${previous}" + fi +} + +cmd_rollback() { + require_env + local target="${1:-$(cat "$(state_dir)/previous" 2>/dev/null || echo "")}" + [ -n "$target" ] || die "no previous version recorded; pass a tag: ./deploy/webmail.sh rollback v1.0.0" + ylw "rolling back to ${target}" + IMAGE_TAG="$target" compose up -d --wait + if health_gate; then + record_version "$target" + set_env_value IMAGE_TAG "$target" + grn "rolled back to ${target}" + else + die "rollback to ${target} did not come up healthy — check: ./deploy/webmail.sh logs" + fi +} + +cmd_restore() { + require_env + local arg="${1:-}" dir base + [ -n "$arg" ] || die "usage: ./deploy/webmail.sh restore (run \`status\` to list them)" + + # Backups are chmod 700 owned by uid 999, so a plain `[ -f ]` from an + # unprivileged shell says "missing" for a file that is right there. Resolve + # and verify inside a container instead. + if [ -f "$arg" ]; then + dir="$(cd "$(dirname "$arg")" && pwd)"; base="$(basename "$arg")" + else + dir="${DATA_ROOT}/backups"; base="$(basename "$arg")" + fi + if ! docker run --rm -v "${dir}:/b:ro" alpine:3.20 test -f "/b/${base}" 2>/dev/null; then + red "no such dump: ${base}" + echo "available in ${DATA_ROOT}/backups:" + docker run --rm -v "${DATA_ROOT}/backups:/b:ro" alpine:3.20 \ + sh -c 'ls -1 /b/*.dump 2>/dev/null | sed "s|/b/| |"' || true + exit 1 + fi + local dump="${dir}/${base}" + + red "THIS REPLACES THE CURRENT DATABASE with ${base}." + red "Mail synced since that dump will be gone (it re-syncs from the servers)." + printf 'Type RESTORE to continue: ' + read -r confirm + [ "$confirm" = "RESTORE" ] || die "aborted" + + echo "stopping application containers (postgres stays up)" + compose stop api worker-sync worker-jobs web backup + + echo "restoring $(basename "$dump")" + # --clean --if-exists drops and recreates objects; a single transaction so a + # failure leaves the old database intact. + docker run --rm -i --network webmail-prod \ + -e PGPASSWORD="${POSTGRES_PASSWORD}" \ + -v "$(dirname "$dump")":/restore:ro \ + postgres:16 \ + pg_restore --clean --if-exists --no-owner --single-transaction \ + -h postgres -U "${POSTGRES_USER:-webmail}" -d "${POSTGRES_DB:-webmail}" \ + "/restore/$(basename "$dump")" \ + || die "restore failed; the database was left as it was" + + echo "starting application containers" + compose up -d --wait + grn "restored. If the blob store was lost too, mail bodies will refill on next sync." +} + +cmd_reindex() { require_env; compose run --rm api reindex "$@"; } +cmd_psql() { require_env; compose exec postgres psql -U "${POSTGRES_USER:-webmail}" -d "${POSTGRES_DB:-webmail}"; } +cmd_shell() { require_env; compose exec "${1:-api}" /bin/sh; } + +usage() { sed -n '2,20p' "${BASH_SOURCE[0]}" | sed 's|^# \?||'; } + +cmd="${1:-}"; shift || true +case "$cmd" in + init) cmd_init "$@" ;; + doctor) cmd_doctor "$@" ;; + up) cmd_up "$@" ;; + down) cmd_down "$@" ;; + restart) cmd_restart "$@" ;; + status) cmd_status "$@" ;; + logs) cmd_logs "$@" ;; + update) cmd_update "$@" ;; + rollback) cmd_rollback "$@" ;; + backup) cmd_backup "$@" ;; + restore) cmd_restore "$@" ;; + reindex) cmd_reindex "$@" ;; + psql) cmd_psql "$@" ;; + shell) cmd_shell "$@" ;; + ""|-h|--help|help) usage ;; + *) die "unknown command: $cmd (try --help)" ;; +esac diff --git a/docker/.env.prod.example b/docker/.env.prod.example new file mode 100644 index 0000000..954cb0f --- /dev/null +++ b/docker/.env.prod.example @@ -0,0 +1,111 @@ +# ============================================================================ +# Webmail production configuration. +# +# Copy to docker/.env.prod, fill in the three REQUIRED values, keep it safe: +# cp docker/.env.prod.example docker/.env.prod && chmod 600 docker/.env.prod +# +# THIS FILE IS A BACKUP ARTEFACT. MASTER_KEY is not stored anywhere else: lose +# it and every saved mail-account password becomes permanently undecryptable, +# even from a perfect database backup. Keep a copy in a password manager. +# ============================================================================ + +# --- REQUIRED --------------------------------------------------------------- + +# Where all persistent data lives on the host. Use a real shared folder your NAS +# backup tool can see — not a Docker named volume. +# Synology: /volume1/docker/webmail/data +# QNAP: /share/Container/webmail/data +DATA_ROOT=/volume1/docker/webmail/data + +# 32 random bytes, base64. Generate ONCE, then never change it: +# openssl rand -base64 32 +MASTER_KEY= + +# Database password. Generate once: +# openssl rand -base64 24 | tr -d '/+=' | cut -c1-24 +POSTGRES_PASSWORD= + +# --- Identity --------------------------------------------------------------- +# The domain your user handles are minted under (user@APP_DOMAIN) and the issuer +# shown in 2FA authenticator apps. It does NOT have to be publicly resolvable. +APP_DOMAIN=mail.home.lan + +# --- Network ---------------------------------------------------------------- +# The single published port. Avoid 5000/5001 (Synology DSM) and 8080 (QNAP QTS). +WEB_PORT=8088 +# Bind to one NAS interface instead of every network, e.g. WEB_BIND=192.168.1.185 +WEB_BIND=0.0.0.0 + +TZ=Europe/Warsaw + +# --- Search ----------------------------------------------------------------- +# pg = Postgres full-text. No extra container, no extra RAM. Default. +# meili = Meilisearch: typo tolerance, better ranking, faster on large mailboxes. +# Costs a container and ~0.7-1 GB RAM. Start it with: +# ./deploy/webmail.sh up --meili +# and after switching an existing install, rebuild the index: +# ./deploy/webmail.sh reindex +SEARCH_BACKEND=pg +# Required if SEARCH_BACKEND=meili — generate with: openssl rand -base64 32 +MEILI_MASTER_KEY= + +# --- Images ----------------------------------------------------------------- +# `local` = built on this host from the repo. To deploy prebuilt images from +# your Gitea registry instead, set both of these (see deploy/build-push.sh): +# REGISTRY_PREFIX=192.168.1.185:8418/claude/webmail +# IMAGE_TAG=v1.0.0 +REGISTRY_PREFIX=webmail +IMAGE_TAG=local + +# --- Application ------------------------------------------------------------ +LOG_LEVEL=INFO +UNDO_SECONDS=10 +SESSION_TTL_DAYS=30 +MAX_ATTACHMENT_MB=25 +# Auto-purge Trash/Spam older than N days (0 disables). +TRASH_RETENTION_DAYS=30 +# Server-side sender avatars (Gravatar + favicon), fetched by the server so +# senders never see your IP. Set false to make zero outbound web requests. +SENDER_AVATARS_ENABLED=true + +GOOGLE_OAUTH_ENABLED=false +OAUTH_GOOGLE_CLIENT_ID= +OAUTH_GOOGLE_CLIENT_SECRET= +OAUTH_MS_CLIENT_ID= +OAUTH_MS_CLIENT_SECRET= + +# Plugins whose schema is migrated on every start. Remove one to stop shipping +# its tables; `hello` is the SDK demo and is intentionally not listed. +ENABLED_PLUGINS=calendar contacts labels filters + +# --- Backups ---------------------------------------------------------------- +BACKUP_AT=03:30 +BACKUP_KEEP_DAYS=14 + +# --- Sizing (defaults suit a personal install on a shared NAS) -------------- +# Peak worker RAM is roughly PARSE_CONCURRENCY x STREAM_PARSE_MB plus overhead. +INTERACTIVE_CONCURRENCY=8 +BULK_CONCURRENCY=1 +PARSE_CONCURRENCY=2 +STREAM_PARSE_MB=8 +PER_HOST_CONN_CAP=10 +HOT_BUDGET_PER_WORKER=50 + +# Container caps. Total without Meilisearch: ~2.2 GB of *limits*; real steady +# usage for one or two mail accounts is around 700 MB-1 GB. +PG_MEM_LIMIT=768m +PG_SHARED_BUFFERS=256MB +PG_MAX_CONNECTIONS=100 +PG_WORK_MEM=8MB +PG_MAINTENANCE_WORK_MEM=128MB +PG_EFFECTIVE_CACHE_SIZE=768MB +REDIS_MEM_LIMIT=160m +REDIS_MAXMEMORY=96mb +API_MEM_LIMIT=512m +API_CPUS=1.0 +WORKER_SYNC_MEM_LIMIT=512m +WORKER_JOBS_MEM_LIMIT=640m +WORKER_CPUS=1.5 +WEB_MEM_LIMIT=96m +BACKUP_MEM_LIMIT=256m +MEILI_MEM_LIMIT=1g diff --git a/docker/Dockerfile.backend b/docker/Dockerfile.backend new file mode 100644 index 0000000..8f4210c --- /dev/null +++ b/docker/Dockerfile.backend @@ -0,0 +1,79 @@ +# Production backend image: API, both worker roles, and the migration/admin CLI. +# +# One image, several commands (see docker/entrypoint.sh) — matches SPEC §6.3's +# WORKER_ROLE split. Build context is the REPO ROOT, because the image needs +# backend/ *and* plugins/ (the five plugins are separate distributions that the +# loader discovers through the `webmail.plugins` entry-point group, SPEC §11). +# +# docker build -f docker/Dockerfile.backend -t webmail/backend . + +# ---------- builder: resolve wheels into a self-contained venv ---------------- +FROM python:3.11-slim AS builder + +ENV PIP_DISABLE_PIP_VERSION_CHECK=1 \ + PIP_NO_CACHE_DIR=1 \ + PYTHONDONTWRITEBYTECODE=1 + +# Only needed if a pinned dep has no manylinux wheel for this platform; the +# whole stage is discarded, so it costs nothing in the runtime image. +RUN apt-get update \ + && apt-get install -y --no-install-recommends build-essential \ + && rm -rf /var/lib/apt/lists/* + +RUN python -m venv /opt/venv +ENV PATH="/opt/venv/bin:$PATH" + +# Backend first: its dependency set is the slowest layer and changes least often. +COPY backend/pyproject.toml /src/backend/pyproject.toml +COPY backend/app /src/backend/app +COPY backend/workers /src/backend/workers +RUN pip install /src/backend + +# Plugins are ordinary distributions installed on top (each owns its own deps). +COPY plugins /src/plugins +RUN pip install \ + /src/plugins/webmail_plugin_calendar \ + /src/plugins/webmail_plugin_contacts \ + /src/plugins/webmail_plugin_labels \ + /src/plugins/webmail_plugin_filters + +# ---------- runtime ----------------------------------------------------------- +FROM python:3.11-slim + +ARG APP_VERSION=dev +ARG GIT_SHA=unknown +LABEL org.opencontainers.image.title="webmail-backend" \ + org.opencontainers.image.version="${APP_VERSION}" \ + org.opencontainers.image.revision="${GIT_SHA}" + +ENV PYTHONUNBUFFERED=1 \ + PYTHONDONTWRITEBYTECODE=1 \ + PATH="/opt/venv/bin:$PATH" \ + APP_VERSION="${APP_VERSION}" \ + GIT_SHA="${GIT_SHA}" + +# tini reaps zombies and forwards SIGTERM, which the workers need for the <10 s +# graceful lease release in workers/run.py. +RUN apt-get update \ + && apt-get install -y --no-install-recommends tini \ + && rm -rf /var/lib/apt/lists/* \ + && useradd --system --uid 10001 --create-home --home-dir /home/webmail webmail + +COPY --from=builder /opt/venv /opt/venv + +WORKDIR /app +# Alembic's core tree lives outside the installed package, so it is copied in. +COPY backend/alembic.ini ./alembic.ini +COPY backend/migrations ./migrations +COPY backend/scripts ./scripts +COPY docker/entrypoint.sh /usr/local/bin/entrypoint +RUN chmod +x /usr/local/bin/entrypoint + +# Blob store mount point (BLOB_ROOT). Ownership is set by the compose `init` +# service, which runs before anything else touches the bind mount. +RUN mkdir -p /data/blobs && chown webmail:webmail /data/blobs +VOLUME ["/data/blobs"] + +USER webmail +ENTRYPOINT ["/usr/bin/tini", "--", "/usr/local/bin/entrypoint"] +CMD ["api"] diff --git a/docker/Dockerfile.frontend b/docker/Dockerfile.frontend new file mode 100644 index 0000000..040846f --- /dev/null +++ b/docker/Dockerfile.frontend @@ -0,0 +1,35 @@ +# Production web image: the built SPA served by nginx, which also reverse-proxies +# /api (including the /api/ws WebSocket) to the API container. Build context is +# the REPO ROOT so the build stage can see frontend/. +# +# docker build -f docker/Dockerfile.frontend -t webmail/web . + +# ---------- builder ----------------------------------------------------------- +FROM node:20-slim AS builder + +WORKDIR /src +COPY frontend/package.json frontend/package-lock.json* ./ +# `npm ci` when a lockfile is present (reproducible), `npm install` otherwise. +RUN if [ -f package-lock.json ]; then npm ci; else npm install; fi + +COPY frontend/ ./ +# Type-check + bundle. `npm run build` runs `tsc --noEmit && vite build`, so a +# type error fails the image build rather than shipping a broken bundle. +RUN npm run build + +# ---------- runtime ----------------------------------------------------------- +FROM nginx:1.27-alpine + +ARG APP_VERSION=dev +ARG GIT_SHA=unknown +LABEL org.opencontainers.image.title="webmail-web" \ + org.opencontainers.image.version="${APP_VERSION}" \ + org.opencontainers.image.revision="${GIT_SHA}" + +COPY docker/nginx.conf /etc/nginx/conf.d/default.conf +COPY --from=builder /src/dist /usr/share/nginx/html + +# Surface the running version to `deploy/webmail.sh status` without needing the API. +RUN printf '%s\n' "${APP_VERSION}" > /usr/share/nginx/html/version.txt + +EXPOSE 80 diff --git a/docker/backup.sh b/docker/backup.sh new file mode 100755 index 0000000..bbe9ef5 --- /dev/null +++ b/docker/backup.sh @@ -0,0 +1,88 @@ +#!/bin/bash +# Nightly logical backup of the webmail database. +# +# Runs as a long-lived sidecar: sleeps until BACKUP_AT, dumps, verifies, prunes, +# repeats. `docker compose run --rm backup once` takes a single backup now (this +# is what `deploy/webmail.sh backup` does, e.g. immediately before an update). +# +# Why logical dumps and not just a NAS snapshot: a btrfs/Hyper Backup snapshot of +# a *running* Postgres data directory is crash-consistent, which usually replays +# fine but is not guaranteed and cannot be restored into a different Postgres +# version. A pg_dump is version-portable and verifiable. Keep both. +set -euo pipefail + +BACKUP_DIR="${BACKUP_DIR:-/backups}" +BACKUP_AT="${BACKUP_AT:-03:30}" +BACKUP_KEEP_DAYS="${BACKUP_KEEP_DAYS:-14}" +DB="${PGDATABASE:-webmail}" + +log() { echo "[backup $(date -Is)] $*"; } + +take_backup() { + local stamp out globals + stamp="$(date +%Y%m%d-%H%M%S)" + out="${BACKUP_DIR}/${DB}-${stamp}.dump" + globals="${BACKUP_DIR}/globals-${stamp}.sql" + + log "dumping ${DB} -> $(basename "$out")" + # -Fc: compressed custom format, restorable selectively with pg_restore. + if ! pg_dump --format=custom --compress=6 --file="${out}.partial" "$DB"; then + log "ERROR: pg_dump failed; leaving no partial file" + rm -f "${out}.partial" + return 1 + fi + + # Verify the archive is readable before it counts as a backup. + if ! pg_restore --list "${out}.partial" >/dev/null 2>&1; then + log "ERROR: dump failed verification (pg_restore --list); discarding" + rm -f "${out}.partial" + return 1 + fi + + mv "${out}.partial" "$out" + chmod 600 "$out" + + # Roles/passwords live outside the database; tiny, and makes a restore onto a + # bare Postgres one step shorter. + pg_dumpall --globals-only --file="$globals" 2>/dev/null || true + [ -f "$globals" ] && chmod 600 "$globals" + + ln -sfn "$(basename "$out")" "${BACKUP_DIR}/latest.dump" + log "ok: $(du -h "$out" | cut -f1) $(basename "$out")" + + log "pruning dumps older than ${BACKUP_KEEP_DAYS} days" + find "$BACKUP_DIR" -maxdepth 1 -name "${DB}-*.dump" -mtime "+${BACKUP_KEEP_DAYS}" -print -delete || true + find "$BACKUP_DIR" -maxdepth 1 -name "globals-*.sql" -mtime "+${BACKUP_KEEP_DAYS}" -print -delete || true +} + +seconds_until() { + local target now next + target="$1" + now=$(date +%s) + next=$(date -d "today ${target}" +%s 2>/dev/null || echo 0) + if [ "$next" -le "$now" ]; then + next=$(date -d "tomorrow ${target}" +%s) + fi + echo $(( next - now )) +} + +if [ "${1:-}" = "once" ]; then + take_backup + exit $? +fi + +log "backup sidecar started; schedule ${BACKUP_AT} daily, retention ${BACKUP_KEEP_DAYS} days" + +# Baseline on first start: if there is no dump from the last 24 h, take one now +# rather than leaving a fresh install unprotected until the first scheduled run. +if ! find "$BACKUP_DIR" -maxdepth 1 -name "${DB}-*.dump" -mtime -1 | grep -q .; then + log "no recent dump found; taking a baseline backup" + take_backup || log "baseline backup failed; will retry on schedule" +fi + +while true; do + wait_s="$(seconds_until "$BACKUP_AT")" + log "next backup in ${wait_s}s (at ${BACKUP_AT})" + sleep "$wait_s" + take_backup || log "scheduled backup failed; continuing" +done diff --git a/docker/docker-compose.prod.yml b/docker/docker-compose.prod.yml new file mode 100644 index 0000000..dd56908 --- /dev/null +++ b/docker/docker-compose.prod.yml @@ -0,0 +1,307 @@ +# Production stack for a single box (Synology / QNAP / any Docker host). +# +# Data lives in bind mounts under ${DATA_ROOT} — NOT in named volumes — so it +# survives `docker compose down`, image updates, project deletion in the NAS +# UI, and a full reinstall, and so the NAS backup tools (Hyper Backup, +# Snapshot Replication, HBS3) can actually see it. +# +# Nothing is published except ${WEB_PORT}. Postgres, Redis, Meilisearch and the +# API are reachable only on the internal `webmail` network. +# +# Start: ./deploy/webmail.sh up +# Manual: docker compose --env-file docker/.env.prod -f docker/docker-compose.prod.yml up -d + +# Deliberately NOT "webmail": docker/docker-compose.yml (the dev service stack) +# uses that project name, and two compose files sharing one project name fight +# over each other's containers — running `make up` would stop production. +name: webmail-prod + +x-logging: &logging + # A shared NAS must never have its volume filled by container logs. + driver: json-file + options: { max-size: "10m", max-file: "3" } + +x-backend-image: &backend-image + image: ${REGISTRY_PREFIX:-webmail}/backend:${IMAGE_TAG:-local} + build: + context: .. + dockerfile: docker/Dockerfile.backend + args: + APP_VERSION: ${IMAGE_TAG:-local} + GIT_SHA: ${GIT_SHA:-unknown} + +x-backend-env: &backend-env + APP_DOMAIN: ${APP_DOMAIN} + DATABASE_URL: postgresql+asyncpg://${POSTGRES_USER:-webmail}:${POSTGRES_PASSWORD}@postgres:5432/${POSTGRES_DB:-webmail} + REDIS_URL: redis://redis:6379/0 + MASTER_KEY: ${MASTER_KEY} + BLOB_ROOT: /data/blobs + SEARCH_BACKEND: ${SEARCH_BACKEND:-pg} + MEILI_URL: ${MEILI_URL:-http://meilisearch:7700} + MEILI_KEY: ${MEILI_MASTER_KEY:-} + LOG_LEVEL: ${LOG_LEVEL:-INFO} + UNDO_SECONDS: ${UNDO_SECONDS:-10} + SESSION_TTL_DAYS: ${SESSION_TTL_DAYS:-30} + MAX_ATTACHMENT_MB: ${MAX_ATTACHMENT_MB:-25} + TRASH_RETENTION_DAYS: ${TRASH_RETENTION_DAYS:-30} + SENDER_AVATARS_ENABLED: ${SENDER_AVATARS_ENABLED:-true} + GOOGLE_OAUTH_ENABLED: ${GOOGLE_OAUTH_ENABLED:-false} + OAUTH_GOOGLE_CLIENT_ID: ${OAUTH_GOOGLE_CLIENT_ID:-} + OAUTH_GOOGLE_CLIENT_SECRET: ${OAUTH_GOOGLE_CLIENT_SECRET:-} + OAUTH_MS_CLIENT_ID: ${OAUTH_MS_CLIENT_ID:-} + OAUTH_MS_CLIENT_SECRET: ${OAUTH_MS_CLIENT_SECRET:-} + # Worker sizing — see docker/.env.prod.example for what these cost in RAM. + INTERACTIVE_CONCURRENCY: ${INTERACTIVE_CONCURRENCY:-8} + BULK_CONCURRENCY: ${BULK_CONCURRENCY:-1} + PARSE_CONCURRENCY: ${PARSE_CONCURRENCY:-2} + STREAM_PARSE_MB: ${STREAM_PARSE_MB:-8} + PER_HOST_CONN_CAP: ${PER_HOST_CONN_CAP:-10} + MAX_ACCOUNTS_PER_WORKER: ${MAX_ACCOUNTS_PER_WORKER:-400} + HOT_BUDGET_PER_WORKER: ${HOT_BUDGET_PER_WORKER:-50} + APP_VERSION: ${IMAGE_TAG:-local} + TZ: ${TZ:-UTC} + +services: + # One-shot: make the bind-mounted host folders writable by the unprivileged + # container users before anything tries to use them. Runs as root by design. + init: + image: alpine:3.20 + user: root + command: + - sh + - -c + - | + set -e + mkdir -p /mnt/blobs /mnt/backups /mnt/redis /mnt/meili + chown -R 10001:10001 /mnt/blobs # backend runs as uid 10001 + chown -R 999:999 /mnt/backups # pg_dump runs as postgres (999) + chown -R 999:999 /mnt/redis # redis runs as uid 999 + chmod 700 /mnt/backups + echo "[init] data directories ready" + volumes: + - ${DATA_ROOT}/blobs:/mnt/blobs + - ${DATA_ROOT}/backups:/mnt/backups + - ${DATA_ROOT}/redis:/mnt/redis + - ${DATA_ROOT}/meili:/mnt/meili + restart: "no" + logging: *logging + + postgres: + image: postgres:16 + environment: + POSTGRES_USER: ${POSTGRES_USER:-webmail} + POSTGRES_PASSWORD: ${POSTGRES_PASSWORD} + POSTGRES_DB: ${POSTGRES_DB:-webmail} + # Keeps the data directory one level down, so the bind mount can hold lost+found + # and NAS metadata files without initdb refusing to start. + PGDATA: /var/lib/postgresql/data/pgdata + TZ: ${TZ:-UTC} + command: + - postgres + - -c + - shared_buffers=${PG_SHARED_BUFFERS:-256MB} + - -c + - max_connections=${PG_MAX_CONNECTIONS:-100} + - -c + - work_mem=${PG_WORK_MEM:-8MB} + - -c + - maintenance_work_mem=${PG_MAINTENANCE_WORK_MEM:-128MB} + - -c + - effective_cache_size=${PG_EFFECTIVE_CACHE_SIZE:-768MB} + # Full page writes stay ON: NAS filesystems (btrfs/ext4) are not 8k-atomic. + volumes: + - ${DATA_ROOT}/postgres:/var/lib/postgresql/data + healthcheck: + test: ["CMD-SHELL", "pg_isready -U ${POSTGRES_USER:-webmail} -d ${POSTGRES_DB:-webmail}"] + interval: 10s + timeout: 5s + retries: 12 + start_period: 30s + restart: unless-stopped + mem_limit: ${PG_MEM_LIMIT:-768m} + stop_grace_period: 60s + logging: *logging + + redis: + image: redis:7-alpine + # Leases and the arq queues live here. AOF everysec means a NAS power cut + # costs at most a second of queue state instead of the whole queue. + command: + - redis-server + - --appendonly + - "yes" + - --appendfsync + - everysec + - --maxmemory + - ${REDIS_MAXMEMORY:-96mb} + - --maxmemory-policy + - noeviction + volumes: + - ${DATA_ROOT}/redis:/data + healthcheck: + test: ["CMD", "redis-cli", "ping"] + interval: 10s + timeout: 3s + retries: 10 + depends_on: + init: { condition: service_completed_successfully } + restart: unless-stopped + mem_limit: ${REDIS_MEM_LIMIT:-160m} + logging: *logging + + # Opt-in: `./deploy/webmail.sh up --meili` (or COMPOSE_PROFILES=meili). + # Costs ~700MB-1GB RSS; with SEARCH_BACKEND=pg the stack runs without it. + meilisearch: + image: getmeili/meilisearch:v1.8 + profiles: ["meili"] + environment: + MEILI_MASTER_KEY: ${MEILI_MASTER_KEY:-} + MEILI_ENV: production + MEILI_NO_ANALYTICS: "true" + TZ: ${TZ:-UTC} + volumes: + - ${DATA_ROOT}/meili:/meili_data + healthcheck: + test: ["CMD-SHELL", "wget -qO- http://127.0.0.1:7700/health | grep -q available"] + interval: 10s + timeout: 5s + retries: 12 + start_period: 20s + depends_on: + init: { condition: service_completed_successfully } + restart: unless-stopped + mem_limit: ${MEILI_MEM_LIMIT:-1g} + logging: *logging + + # One-shot schema upgrade: core Alembic head + every plugin's own branch. + # Everything else waits for this to exit 0, so a failed migration stops the + # roll instead of starting code against an old schema. + migrate: + <<: *backend-image + command: ["migrate"] + environment: + <<: *backend-env + ENABLED_PLUGINS: ${ENABLED_PLUGINS:-calendar contacts labels filters} + volumes: + - ${DATA_ROOT}/blobs:/data/blobs + depends_on: + postgres: { condition: service_healthy } + init: { condition: service_completed_successfully } + restart: "no" + logging: *logging + + api: + <<: *backend-image + command: ["api"] + environment: *backend-env + volumes: + - ${DATA_ROOT}/blobs:/data/blobs + healthcheck: + test: ["CMD-SHELL", "python -c \"import urllib.request,sys; sys.exit(0 if urllib.request.urlopen('http://127.0.0.1:8000/api/healthz', timeout=5).status==200 else 1)\""] + interval: 30s + timeout: 10s + retries: 5 + start_period: 30s + depends_on: + postgres: { condition: service_healthy } + redis: { condition: service_healthy } + migrate: { condition: service_completed_successfully } + restart: unless-stopped + mem_limit: ${API_MEM_LIMIT:-512m} + cpus: ${API_CPUS:-1.0} + logging: *logging + + worker-sync: + <<: *backend-image + command: ["worker-sync"] + environment: *backend-env + volumes: + - ${DATA_ROOT}/blobs:/data/blobs + # Persistent IMAP IDLE connections; the default 1024 is fine for a personal + # install but this costs nothing and matches the sync-engine v2 sizing. + ulimits: + nofile: { soft: 8192, hard: 8192 } + depends_on: + postgres: { condition: service_healthy } + redis: { condition: service_healthy } + migrate: { condition: service_completed_successfully } + restart: unless-stopped + mem_limit: ${WORKER_SYNC_MEM_LIMIT:-512m} + cpus: ${WORKER_CPUS:-1.5} + # Leases are released on SIGTERM (workers/run.py targets <10 s); give it room. + stop_grace_period: 30s + logging: *logging + + worker-jobs: + <<: *backend-image + command: ["worker-jobs"] + environment: *backend-env + volumes: + - ${DATA_ROOT}/blobs:/data/blobs + ulimits: + nofile: { soft: 8192, hard: 8192 } + depends_on: + postgres: { condition: service_healthy } + redis: { condition: service_healthy } + migrate: { condition: service_completed_successfully } + restart: unless-stopped + mem_limit: ${WORKER_JOBS_MEM_LIMIT:-640m} + cpus: ${WORKER_CPUS:-1.5} + stop_grace_period: 30s + logging: *logging + + web: + image: ${REGISTRY_PREFIX:-webmail}/web:${IMAGE_TAG:-local} + build: + context: .. + dockerfile: docker/Dockerfile.frontend + args: + APP_VERSION: ${IMAGE_TAG:-local} + GIT_SHA: ${GIT_SHA:-unknown} + environment: + TZ: ${TZ:-UTC} + ports: + # The ONLY published port. Bind to a specific NAS interface by setting + # WEB_BIND (e.g. 192.168.1.185) to keep it off other networks. + - "${WEB_BIND:-0.0.0.0}:${WEB_PORT:-8088}:80" + healthcheck: + test: ["CMD-SHELL", "wget -qO- http://127.0.0.1/healthz | grep -q ok"] + interval: 30s + timeout: 5s + retries: 5 + depends_on: + api: { condition: service_started } + restart: unless-stopped + mem_limit: ${WEB_MEM_LIMIT:-96m} + logging: *logging + + # Nightly logical backup of Postgres into ${DATA_ROOT}/backups, pruned to + # BACKUP_KEEP_DAYS. Point Hyper Backup / HBS3 at ${DATA_ROOT} and this plus the + # blob store is the whole recoverable state (the other half is MASTER_KEY — + # see docs/nas-install.md §"The three things that matter"). + backup: + image: postgres:16 + user: "999:999" + entrypoint: ["/bin/bash", "/usr/local/bin/backup.sh"] + environment: + PGHOST: postgres + PGUSER: ${POSTGRES_USER:-webmail} + PGPASSWORD: ${POSTGRES_PASSWORD} + PGDATABASE: ${POSTGRES_DB:-webmail} + BACKUP_DIR: /backups + BACKUP_AT: ${BACKUP_AT:-03:30} + BACKUP_KEEP_DAYS: ${BACKUP_KEEP_DAYS:-14} + TZ: ${TZ:-UTC} + volumes: + - ${DATA_ROOT}/backups:/backups + - ./backup.sh:/usr/local/bin/backup.sh:ro + depends_on: + postgres: { condition: service_healthy } + init: { condition: service_completed_successfully } + restart: unless-stopped + mem_limit: ${BACKUP_MEM_LIMIT:-256m} + logging: *logging + +networks: + default: + name: webmail-prod diff --git a/docker/entrypoint.sh b/docker/entrypoint.sh new file mode 100755 index 0000000..1809be1 --- /dev/null +++ b/docker/entrypoint.sh @@ -0,0 +1,62 @@ +#!/bin/sh +# One image, several roles. Called as: entrypoint [args...] +# +# api uvicorn, behind the `web` nginx container +# worker-sync lease-owned account sync supervisor (SPEC §6.3) +# worker-jobs arq consumer (interactive + bulk queues) +# worker-all both in one process (small installs; halves RAM use) +# migrate core Alembic upgrade + per-plugin install; exits 0 when done +# reindex rebuild the search index from Postgres +# plugins `python -m app.plugins.cli ...` passthrough +# shell /bin/sh (debugging) +# +# Any other argument is exec'd verbatim, so `docker compose run api python -c ...` +# still works. +set -eu + +: "${API_HOST:=0.0.0.0}" +: "${API_PORT:=8000}" +: "${API_WORKERS:=1}" +# Plugins whose migrations are applied on every `migrate` run. Space-separated so +# an operator can add a third-party plugin without editing the image. +: "${ENABLED_PLUGINS:=calendar contacts labels filters}" + +cmd="${1:-api}" +[ "$#" -gt 0 ] && shift || true + +run_migrations() { + echo "[entrypoint] core schema: alembic upgrade head" + alembic upgrade head + for plugin in $ENABLED_PLUGINS; do + echo "[entrypoint] plugin schema: $plugin" + python -m app.plugins.cli install "$plugin" + done + echo "[entrypoint] migrations complete" +} + +case "$cmd" in + api) + # --proxy-headers + --forwarded-allow-ips are what let the app see the + # real scheme through nginx/DSM. Without them every request looks like + # plain http and the session cookie loses its __Host- prefix and Secure + # flag (app/identity/cookies.py:22). The allow-list is the internal + # docker network only; this port is never published directly. + exec uvicorn app.main:create_app \ + --factory \ + --host "$API_HOST" \ + --port "$API_PORT" \ + --workers "$API_WORKERS" \ + --proxy-headers \ + --forwarded-allow-ips "${FORWARDED_ALLOW_IPS:-*}" \ + --no-server-header \ + "$@" + ;; + worker-sync) WORKER_ROLE=sync exec python -m workers.run "$@" ;; + worker-jobs) WORKER_ROLE=jobs exec python -m workers.run "$@" ;; + worker-all) WORKER_ROLE=all exec python -m workers.run "$@" ;; + migrate) run_migrations ;; + reindex) exec python scripts/reindex_search.py "$@" ;; + plugins) exec python -m app.plugins.cli "$@" ;; + shell) exec /bin/sh "$@" ;; + *) exec "$cmd" "$@" ;; +esac diff --git a/docker/nginx.conf b/docker/nginx.conf new file mode 100644 index 0000000..dd23a5a --- /dev/null +++ b/docker/nginx.conf @@ -0,0 +1,104 @@ +# Webmail front door: static SPA + reverse proxy to the API container. +# +# This file is dropped into /etc/nginx/conf.d/, i.e. it is already inside the +# `http` block — hence the bare `map` below. + +# Preserve the original scheme when something else terminates TLS in front of us +# (Synology's Login Portal reverse proxy, QNAP's, Caddy, Cloudflare Tunnel...). +# The API only emits Secure/__Host- cookies when it believes the request was +# https, so losing this header silently downgrades session cookies. +map $http_x_forwarded_proto $fwd_proto { + default $scheme; + https https; + http http; +} + +# Connection header for the WebSocket upgrade dance on /api/ws. +map $http_upgrade $connection_upgrade { + default upgrade; + '' close; +} + +upstream webmail_api { + server api:8000; + keepalive 16; +} + +server { + listen 80; + server_name _; + + root /usr/share/nginx/html; + index index.html; + + # Attachment uploads. Keep this comfortably above MAX_ATTACHMENT_MB (default + # 25) to leave room for multipart/base64 overhead. + client_max_body_size 64m; + client_body_timeout 300s; + + # The NAS is doing other jobs too — don't buffer big uploads to disk twice. + proxy_request_buffering off; + + gzip on; + gzip_vary on; + gzip_min_length 1024; + gzip_types text/plain text/css application/javascript application/json + image/svg+xml application/manifest+json; + + # Clickjacking / sniffing basics. The app's own CSP for mail HTML is applied + # server-side in app/indexer/sanitize.py; this is only the shell document. + add_header X-Content-Type-Options nosniff always; + add_header X-Frame-Options SAMEORIGIN always; + add_header Referrer-Policy no-referrer always; + + # ---- API ---------------------------------------------------------------- + location /api/ { + proxy_pass http://webmail_api; + proxy_http_version 1.1; + + proxy_set_header Host $host; + proxy_set_header X-Real-IP $remote_addr; + proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for; + proxy_set_header X-Forwarded-Proto $fwd_proto; + proxy_set_header X-Forwarded-Host $host; + + # WebSocket (/api/ws) and any long-poll: no upgrade means no live mail. + proxy_set_header Upgrade $http_upgrade; + proxy_set_header Connection $connection_upgrade; + + # Live updates hold the connection open; don't cut them at 60 s. + proxy_read_timeout 3600s; + proxy_send_timeout 3600s; + proxy_buffering off; + } + + # ---- static ------------------------------------------------------------- + # Vite emits content-hashed filenames into /assets, so they are immutable. + location /assets/ { + expires 1y; + add_header Cache-Control "public, immutable"; + try_files $uri =404; + } + + location /fonts/ { + expires 30d; + add_header Cache-Control "public"; + try_files $uri =404; + } + + location = /healthz { + access_log off; + return 200 "ok\n"; + add_header Content-Type text/plain; + } + + # SPA fallback: every non-file route renders index.html (react-router). + location / { + try_files $uri $uri/ /index.html; + } + + # index.html itself must never be cached, or a deploy won't be picked up. + location = /index.html { + add_header Cache-Control "no-store, must-revalidate"; + } +} diff --git a/docs/nas-install.md b/docs/nas-install.md new file mode 100644 index 0000000..f09cbf1 --- /dev/null +++ b/docs/nas-install.md @@ -0,0 +1,354 @@ +# Running Webmail on a Synology or QNAP NAS + +This is the operator runbook: how to install it, how to keep your mail safe from +anything Docker does to itself, and how to update without holding your breath. +It assumes the NAS is doing other work too, so the stack is deliberately modest: +**one published port, no host networking, memory caps on every container.** + +Everything here works on any Intel/AMD Docker host — the NAS parts are just the +specifics for DSM and QTS. + +--- + +## 1. What gets installed + +| Container | What it does | Published? | +|---|---|---| +| `web` | nginx: serves the UI, proxies `/api` (incl. the WebSocket) | **yes — the only one** | +| `api` | FastAPI app | no, internal network only | +| `worker-sync` | keeps IMAP connections open, syncs mail | no | +| `worker-jobs` | background jobs (send, backfill, purge, plugin jobs) | no | +| `postgres` | the database | no | +| `redis` | job queues + sync leases | no | +| `backup` | nightly verified database dump | no | +| `meilisearch` | optional better search (see §8) | no | +| `migrate`, `init` | run once at start, then exit | — | + +Plugins (calendar, contacts, labels, filters) are baked into the backend image +and their database schema is applied automatically on every start. + +**Requirements:** an Intel/AMD NAS with Docker (Synology *Container Manager*, +QNAP *Container Station*), ~2 GB RAM free, and 10 GB+ of disk for a typical +personal mailbox. Nothing is needed from the internet except your mail servers. + +--- + +## 2. Install + +### 2.1 Pick where the data lives + +This is the single most important decision. Use a **shared folder**, never a +Docker named volume: + +| NAS | Use | +|---|---| +| Synology | `/volume1/docker/webmail/data` | +| QNAP | `/share/Container/webmail/data` | + +Why: named volumes live inside Docker's internal storage, where Hyper Backup, +Snapshot Replication and HBS3 cannot see them, and where "delete the project" in +the NAS UI can take your mail with it. A shared folder is backed up by the tools +you already run, survives every container operation, and can be copied to +another machine wholesale. + +> **Never put this on an NFS or SMB mount.** Postgres on a network filesystem +> corrupts. `./deploy/webmail.sh doctor` checks this and refuses to be quiet +> about it. + +### 2.2 Get the code onto the NAS + +Enable SSH on the NAS (DSM: *Control Panel → Terminal & SNMP → Enable SSH*; +QTS: *Control Panel → Telnet/SSH*), then: + +```bash +ssh admin@your-nas +sudo -i +mkdir -p /volume1/docker/webmail && cd /volume1/docker/webmail +git clone /webmail.git src && cd src +``` + +No git on the NAS? Copy the folder over with File Station or `rsync` instead. + +### 2.3 Configure + +```bash +./deploy/webmail.sh init +``` + +This creates `docker/.env.prod` with a freshly generated `MASTER_KEY` and +database password. Then open it and set: + +```ini +DATA_ROOT=/volume1/docker/webmail/data # or /share/Container/webmail/data +APP_DOMAIN=mail.home.lan # your handles become you@this +TZ=Europe/Warsaw +WEB_PORT=8088 # avoid 5000/5001 (DSM), 8080 (QTS) +WEB_BIND=0.0.0.0 # or one NAS IP, e.g. 192.168.1.185 +``` + +> ### Copy `MASTER_KEY` into your password manager. Now. +> It encrypts every stored mail-account password. It exists **only** in this +> file. A perfect database backup without this key gives you your mail but not +> your credentials — you would have to re-enter every account password by hand. +> `docker/.env.prod` is `chmod 600` and git-ignored; treat it as a backup +> artefact in its own right. + +### 2.4 Start + +```bash +./deploy/webmail.sh doctor # sanity-checks config, disk, filesystem, ports +./deploy/webmail.sh up # builds the images on first run, then starts +``` + +First run builds both images. The frontend build wants ~2 GB of RAM: if your NAS +is small or you'd rather not tie it up, build on your PC instead — see §6.3. + +Open `http://your-nas:8088/`, create your account, and add a mail account +through the wizard. + +### 2.5 Synology Container Manager (UI instead of SSH) + +If you prefer the GUI: *Container Manager → Project → Create*, path +`/volume1/docker/webmail/src/docker`, and select `docker-compose.prod.yml`. +Container Manager reads a file named exactly `.env` from that folder, so run +`cp docker/.env.prod docker/.env` first (and re-copy it whenever you change +settings). The SSH route is still the better one — `webmail.sh` gives you the +backup, update, health-gate and rollback logic that the UI has no equivalent for. + +### 2.6 Starting automatically after a reboot + +Every service is `restart: unless-stopped`, so Docker brings the stack back on +boot by itself. On Synology, make sure Container Manager itself is set to start +on boot (it is by default). + +--- + +## 3. The three things that matter + +Containers are disposable. These are not: + +| # | What | Where | If you lose it | +|---|---|---|---| +| 1 | **Database** | `$DATA_ROOT/postgres` + nightly dumps in `$DATA_ROOT/backups` | Everything: accounts, threads, flags, calendar, contacts | +| 2 | **Blob store** | `$DATA_ROOT/blobs` | Mail bodies and attachments (re-downloadable from the servers, slowly) | +| 3 | **`MASTER_KEY`** | `docker/.env.prod` + your password manager | Every saved mail-account password, permanently | + +Everything else is derived and needs no backup: Redis holds queues and leases, +Meilisearch's index rebuilds from Postgres (§8), and images rebuild from source. + +### Backups + +Two layers, and you want both: + +**Nightly dumps (automatic).** The `backup` container writes a compressed +`pg_dump` to `$DATA_ROOT/backups` at `BACKUP_AT` (default 03:30) and keeps +`BACKUP_KEEP_DAYS` (14) of them. Every dump is verified with `pg_restore --list` +before it counts, so you never find out at restore time that it was truncated. + +**Off-box copies (you set this up, once).** Point your NAS backup tool at +`$DATA_ROOT`: + +- Synology: *Hyper Backup* → local/remote/cloud destination → include the + `docker/webmail` shared folder. Add *Snapshot Replication* on a Btrfs volume + for fast local rollback. +- QNAP: *HBS 3* → the `Container/webmail` folder. Add snapshots on QuTS hero. + +A snapshot of a *running* Postgres directory is crash-consistent — it usually +replays fine, but it is not a substitute for the dumps, which are verified and +restorable into a different Postgres version. Keep both. + +Also back up `docker/.env.prod` somewhere that is not the NAS. + +### Restore drill — do this once, before you need it + +```bash +./deploy/webmail.sh status # see available dumps +./deploy/webmail.sh restore webmail-20260914-033000.dump +``` + +It stops the app containers, restores in a single transaction (a failure leaves +the old database untouched), and starts everything back up. + +Full disaster recovery onto a new box: + +1. Install Docker, copy `$DATA_ROOT` back from your backup. +2. Copy the repo and `docker/.env.prod` — **with the original `MASTER_KEY`**. +3. `./deploy/webmail.sh up` + +That's it. If the blob store was lost but the database survived, mail bodies +refill from the mail servers as accounts re-sync. + +--- + +## 4. Updating + +```bash +git pull +./deploy/webmail.sh update +``` + +`update` does, in this order, stopping the moment anything fails: + +1. **Takes a verified database backup.** No backup, no update. +2. Builds (or pulls) the new images. A failure here changes nothing. +3. Runs migrations as a one-shot container — **before** any service restarts, so + a bad migration never leaves new code running against an old schema. +4. Rolls workers first, then API, then web. Sync leases transfer to the + replacement worker within ~90 s and message upserts are idempotent, so no + mail is lost or duplicated by a roll. +5. Health-gates for 120 s. **If the new version doesn't come up healthy it rolls + itself back automatically** and leaves you on the previous version. + +To go back manually: + +```bash +./deploy/webmail.sh rollback # previous version +./deploy/webmail.sh rollback v1.2.0 # a specific one +``` + +Rolling *code* back does not require rolling the *schema* back, as long as +migrations stay expand-then-contract (see `docs/migrations.md` discipline in +`deployment-setup-instructions.md` §9). For the rare breaking migration, restore +the dump that `update` took in step 1. + +### Building images elsewhere + +Building on the NAS is simple but slow and RAM-hungry. Two alternatives: + +```bash +# On your PC — push to your Gitea registry: +REGISTRY_PREFIX=192.168.1.185:8418/claude/webmail ./deploy/build-push.sh v1.0.0 +# then on the NAS, set REGISTRY_PREFIX + IMAGE_TAG in docker/.env.prod and: +./deploy/webmail.sh update v1.0.0 + +# Or with no registry at all — build to a file and carry it over: +./deploy/build-push.sh v1.0.0 --save +# copy webmail-v1.0.0.tar to the NAS, then: +docker load -i webmail-v1.0.0.tar && ./deploy/webmail.sh update v1.0.0 +``` + +--- + +## 5. Reaching it from outside the LAN + +### 5.1 Read this first + +**Anyone who can open the page can create an account on it.** Registration is +open by design (it's a personal server), so do not put this straight on the +public internet. Pick one: + +- **VPN** (Tailscale, WireGuard, or the NAS's own VPN Server) — simplest and + safest. The port stays LAN-only. +- **NAS reverse proxy with HTTPS**, restricted by firewall rules to the networks + you trust. + +### 5.2 HTTPS through the NAS reverse proxy + +DSM owns ports 80/443, which is why the stack publishes 8088. To put DSM's +certificate in front of it: + +*Control Panel → Login Portal → Advanced → Reverse Proxy → Create* + +| Field | Value | +|---|---| +| Source protocol / hostname / port | HTTPS / `mail.yourdomain` / 443 | +| Destination protocol / hostname / port | HTTP / `localhost` / `8088` | + +Then open **Custom Header → Create → WebSocket** and save. Without that, +`/api/ws` is blocked and the UI stops updating live. + +QNAP: *Network & Virtual Switch → Reverse Proxy* — same idea, and likewise +enable WebSocket support. + +Session cookies get the `Secure` and `__Host-` treatment automatically as soon +as the app sees an HTTPS request: nginx forwards `X-Forwarded-Proto`, and the +API is started with `--proxy-headers`. Nothing to configure. + +--- + +## 6. Resource usage + +Steady state for one or two mail accounts, `SEARCH_BACKEND=pg`: + +| | Typical | Cap (`docker/.env.prod`) | +|---|---|---| +| postgres | 150-300 MB | `PG_MEM_LIMIT=768m` | +| api | 150-250 MB | `API_MEM_LIMIT=512m` | +| worker-sync | 120-250 MB | `WORKER_SYNC_MEM_LIMIT=512m` | +| worker-jobs | 150-350 MB | `WORKER_JOBS_MEM_LIMIT=640m` | +| redis / web / backup | < 100 MB total | — | +| **total** | **~0.7-1.2 GB** | ~2.2 GB of limits | +| meilisearch (optional) | 300-800 MB | `MEILI_MEM_LIMIT=1g` | + +On a 2 GB NAS: stay on `SEARCH_BACKEND=pg` and drop `PG_SHARED_BUFFERS` to +`128MB`. Tight but workable. On 4 GB+ you have room for Meilisearch. + +Two NAS-specific realities worth knowing: + +- **Disk hibernation won't happen.** Active accounts hold IMAP IDLE connections + open and write as mail arrives. That is the feature working, not a bug. +- **Keep NTP on.** Two-factor codes fail on a drifting clock. + +--- + +## 7. Search: Postgres or Meilisearch + +`SEARCH_BACKEND=pg` (default) uses Postgres full-text search. No extra +container, no extra RAM, and it is perfectly good for a personal mailbox. + +`SEARCH_BACKEND=meili` adds typo tolerance, better ranking and stays fast on +large mailboxes, at the cost of a container and 300-800 MB. + +To switch: + +```bash +# in docker/.env.prod: SEARCH_BACKEND=meili and set MEILI_MASTER_KEY +./deploy/webmail.sh up # starts Meilisearch automatically +./deploy/webmail.sh reindex # build the index from existing mail +``` + +The index is **derived data** — every field it holds also lives in Postgres — so +it never needs backing up. If the Meilisearch volume is lost, or you switch +backends, or search ever looks stale, just rebuild it: + +```bash +./deploy/webmail.sh reindex # upsert everything (safe to re-run) +./deploy/webmail.sh reindex --purge # drop stale documents first +``` + +--- + +## 8. Troubleshooting + +| Symptom | Cause | Fix | +|---|---|---| +| `up` hangs on `postgres` | `DATA_ROOT` on NFS/SMB, or wrong permissions | `./deploy/webmail.sh doctor` | +| UI loads, no live updates | WebSocket not enabled in the NAS reverse proxy | §5.2 Custom Header → WebSocket | +| Login works, logs out on reload | HTTPS terminated upstream but `X-Forwarded-Proto` stripped | Use the NAS reverse proxy as documented; don't strip headers | +| "port is already allocated" | 8088 taken by another app | change `WEB_PORT` | +| Mail not syncing | worker stopped, or credentials rejected | `./deploy/webmail.sh logs worker-sync` | +| Search finds nothing after a restore | index not rebuilt | `./deploy/webmail.sh reindex --purge` | +| App won't start after editing `.env.prod` | malformed `MASTER_KEY` | `./deploy/webmail.sh doctor` | + +Logs: `./deploy/webmail.sh logs [service] -f`. All logs are capped at 10 MB ×3 +per container, so they can't fill the volume. + +--- + +## 9. Command reference + +``` +./deploy/webmail.sh init create data dirs + generate secrets +./deploy/webmail.sh doctor pre-flight checks +./deploy/webmail.sh up [--meili] start everything +./deploy/webmail.sh down stop (data untouched) +./deploy/webmail.sh status what's running, versions, disk, backups +./deploy/webmail.sh logs [svc] [-f] logs +./deploy/webmail.sh update [tag] backup -> migrate -> roll -> health-gate +./deploy/webmail.sh rollback [tag] back to the previous version +./deploy/webmail.sh backup verified dump, right now +./deploy/webmail.sh restore restore a dump (asks for confirmation) +./deploy/webmail.sh reindex [--purge] rebuild the search index +./deploy/webmail.sh psql database shell +./deploy/webmail.sh shell [svc] shell in a container +```