webmail-client/deployment-setup-instructions.md

Deployment & Release System — Agent Build Instructions

Companion to SPEC.md and DEV_ENVIRONMENT.md. This document tells you (the implementing agent) to build the complete versioning, CI/CD, and production-update system for this project. The human operator is not a DevOps specialist — your job is to make deployment a one-command, safe, reversible operation and to document the few manual steps clearly.

Everything lives inside the operator's own Gitea server (source, CI, and container registry). No GitHub, no external CI, no external registry.

---

0. What you are building (deliverables checklist)

Create all of the following, committed to the repo:

1. .gitea/workflows/ci.yml — test + build + push pipeline (runs on every push and PR).
2. .gitea/workflows/release.yml — on a version tag: test, build images, push to Gitea registry, deploy to production.
3. docker/Dockerfile.backend and docker/Dockerfile.frontend — production images (multi-stage, slim).
4. docker/docker-compose.prod.yml — the production stack, referencing images by pinned tag from the Gitea registry.
5. deploy/deploy.sh — the script that runs ON the production server to apply an update safely.
6. deploy/rollback.sh — one-command rollback to the previous version.
7. backend/app/api/health.py — /api/healthz and /api/version endpoints.
8. deploy/VERSION — plain-text current version, written by the release pipeline.
9. docs/deployment.md — a short operator runbook in plain language (how to release, how to roll back, what to do if it breaks). Write this for a non-expert.
10. docs/gitea-setup.md — the one-time manual setup the operator must do in Gitea's UI (documented step by step, see §7).

Do not require any tool the operator must learn. The whole system is: push a tag → it deploys. Run one script → it rolls back.

---

1. Versioning rules (implement, don't deviate)

• SemVer git tags are the single source of truth: v{MAJOR}.{MINOR}.{PATCH} (e.g. v1.4.2).
• The tag name becomes the image tag. Images are never tagged latest in production compose. latest may exist as a convenience alias but production always pins an explicit version.
• /api/version returns the running version (read from a build-time env APP_VERSION baked into the image). /api/healthz returns 200 only when DB + Redis are reachable.
• Every release is atomic (roll the whole tag) and reversible (redeploy the previous tag). Never hand-edit files on the production server to "patch" — always go through a tag.
• Tag creation is the human's explicit release action. Do NOT auto-tag from routine commits — routine commits run CI (test+build) but do not deploy. Only a v* tag deploys.

2. Images (docker/Dockerfile.backend, docker/Dockerfile.frontend)

• Multi-stage builds. Backend: build deps in a builder stage, copy only the venv + app into a slim python:3.11-slim runtime; run as a non-root user; ENV APP_VERSION set via build arg. Frontend: build with Node 20, serve the static bundle from nginx-alpine (or have the API serve it — pick API-served only if simpler for a single node; default to a separate web nginx image).
• Backend image entrypoint supports three roles via command: API (uvicorn), worker-jobs, worker-sync (matches SPEC §6.3 WORKER_ROLE). One image, three commands.
• Label images with the version and git SHA (org.opencontainers.image.version, .revision).
• Registry path: {GITEA_HOST}/{OWNER}/webmail/backend and .../web. Read GITEA_HOST/OWNER from CI variables, never hardcode.

3. CI pipeline (.gitea/workflows/ci.yml)

Trigger: push to any branch, and pull requests. Steps:

1. Spin up service containers (postgres:16, redis:7, meilisearch:v1.8, greenmail) — reuse docker/docker-compose.ci.yml from DEV_ENVIRONMENT.
2. Backend gates: ruff check, ruff format --check, mypy --strict app, pytest -q (unit + integration, coverage floors per SPEC §13).
3. Frontend gates: tsc --noEmit, eslint, vitest run.
4. Build both Docker images (validates the Dockerfiles) — but on plain branch pushes, do not push to the registry (build-only smoke).
5. Fail the whole workflow if any gate fails.

This gives every commit a green/red signal without deploying anything.

4. Release pipeline (.gitea/workflows/release.yml)

Trigger: push of a tag matching v*. Steps, in order, stopping on any failure:

1. Run the full CI gate set (§3) against the tagged commit. No shortcuts — a tag that fails tests must not deploy.
2. Build backend + web images with APP_VERSION={tag} and the git SHA.
3. Log in to the Gitea container registry (using a CI secret token) and push both images tagged {tag} and latest.
4. Deploy to production by connecting to the production host over SSH (key from a CI secret) and running deploy/deploy.sh {tag}. See §5.
5. On deploy success, write {tag} to deploy/VERSION and update /api/version expectation. On deploy failure, the pipeline must surface the error loudly and NOT leave a half-applied state (deploy.sh handles atomicity; see §5).

Keep the runner OFF the production box (run it on the dev/build machine or a spare fleet node) so a build can't starve the mail server.

5. The deploy script (deploy/deploy.sh) — safety is the whole point

Runs on the production host. Takes one argument: the target version tag. It MUST do, in this exact order:

1. Record current version (read running /api/version or deploy/VERSION) → save as the rollback target.
2. Snapshot the database BEFORE touching anything — trigger a pgBackRest backup (or a ZFS snapshot of tank/pg if pgBackRest isn't configured yet). Abort the deploy if the snapshot fails. This is non-negotiable: a bad migration must always be recoverable.
3. Pull the new images for {tag} from the Gitea registry. Abort if pull fails (nothing has changed yet — safe).
4. Run database migrations as a one-shot container: alembic upgrade head (this includes plugin migration branches per SPEC §11.1). Migrations MUST be backward-compatible / expand-then-contract so the brief mixed-version window is safe. Abort + alert if migration fails (DB snapshot from step 2 is the recovery path).
5. Roll services in dependency order: workers first (worker-sync, worker-jobs), then the API, then web — docker compose -f docker-compose.prod.yml up -d --wait with the new tag. --wait blocks on healthchecks.
6. Health-gate: poll /api/healthz for up to 60 s. If it doesn't go green, automatically invoke rollback.sh to the saved previous version and exit non-zero. Never leave production unhealthy.
7. On success: write {tag} to deploy/VERSION, log the old→new transition, done.

The script is idempotent and logs every step with timestamps. It reads all config (registry host, compose file path, healthcheck URL, ret(ry counts) from deploy/deploy.env, not hardcoded.

Because SPEC §6.3 sync workers are lease-owned and stateless, rolling them mid-operation is safe: leases transfer to the new worker within ~90 s, no mail lost, no duplicate rows (idempotent (mailbox_id, uid) upserts). Note this in the script comments so no future change breaks the assumption.

6. Rollback (deploy/rollback.sh)

One argument (target version) or, with no argument, the version recorded as "previous" by the last deploy. Steps: pull the old images (should be cached), roll services back to that tag, health-gate. Migration note: because migrations are expand-then-contract and backward-compatible, rolling the code back does not require rolling the schema back in the common case — document this clearly, and for the rare breaking migration, instruct the operator to restore the pre-deploy DB snapshot from step 5.2 (give the exact pgBackRest/ZFS restore command in docs/deployment.md).

7. One-time manual setup (docs/gitea-setup.md) — write this for a non-expert

The operator must do these once in Gitea's web UI; document each with exact click paths and screenshots-in-words:

1. Enable Gitea Actions for the repo (Settings → Actions).
2. Register a runner: provide the exact commands to run the act_runner container on the build machine, register it against the Gitea instance with a registration token, and label it (self-hosted, linux). Include the docker-compose snippet for the runner.
3. Enable the Gitea container registry (it's built in) and create a package-registry access token; store it as an Actions secret REGISTRY_TOKEN.
4. Add Actions secrets: REGISTRY_TOKEN, PROD_SSH_KEY (private key for the deploy user on the production host), PROD_HOST, PROD_USER. List exactly where in the UI (Settings → Actions → Secrets).
5. On the production host: create a low-privilege deploy user, add it to the docker group, install its public key, and docker login to the Gitea registry once (or bake it into deploy.sh via the token). Give copy-paste commands.
6. First deploy: how to cut the first release — git tag v1.0.0 && git push origin v1.0.0 — and what success looks like (/api/version returns it).

Keep this doc short, sequential, and copy-pasteable. Assume the reader will follow it literally.

8. Operator runbook (docs/deployment.md) — plain language

Cover, in a few short sections:

• How to release: "Tag and push. That's it." — git tag v1.4.3 && git push origin v1.4.3, then watch the Actions tab.
• How to see what's running in production: curl https://{host}/api/version.
• How to roll back: ssh {prod} 'cd /opt/webmail && ./deploy/rollback.sh'.
• If a deploy fails: it auto-rolls-back and production stays on the old version; where to read the logs; when to restore a DB snapshot (rare) with the exact command.
• What NOT to do: never edit files directly on the server, never use latest in the compose file, never skip the tag.

9. Migration discipline (encode as a rule)

Add a short docs/migrations.md: all Alembic migrations must be expand-then-contract — add new columns/tables in one release, backfill, switch code, drop old columns only in a later release once no running code needs them. This is what makes zero-downtime rolls and easy rollback possible. Add a CI check or PR-checklist item reminding of this; never generate a destructive migration (DROP/rename) in the same release that stops using the column.

10. Acceptance criteria

1. Pushing a normal commit runs CI (tests + image build) and does not deploy.
2. Pushing a v* tag whose tests pass builds images, pushes them to the Gitea registry, and deploys to production with no manual step beyond the tag push.
3. Pushing a v* tag whose tests fail produces no image and no deploy.
4. /api/version on production returns the deployed tag after a release.
5. A deploy whose new version fails its healthcheck auto-rolls-back and leaves production healthy on the prior version (test with a deliberately broken image).
6. rollback.sh returns production to the prior version in one command.
7. A DB snapshot exists (pgBackRest or ZFS) timestamped immediately before every migration — verify one is created on a test deploy.
8. Rolling sync workers during a deploy loses no mail and creates no duplicate message rows (ties to SPEC §6.3 acceptance tests).
9. docs/gitea-setup.md and docs/deployment.md are followable by a non-DevOps operator: every manual step has an exact command or click path.

11. Build order

1. Health/version endpoints (§1) + Dockerfiles (§2) + docker-compose.prod.yml.
2. ci.yml (get green tests + building images first).
3. Registry push + release.yml.
4. deploy.sh + rollback.sh + deploy.env (test against a staging target — the same compose on a different port on the dev box — before pointing at the real host).
5. The two docs (§7, §8) + migrations.md.
6. Prove all §10 acceptance criteria, including the deliberately-broken-image auto-rollback test.

Keep it boring and legible. The operator should never have to think about the mechanics again after the one-time Gitea setup.