webmail-client/deployment-setup-instructions.md

# Deployment & Release System — Agent Build Instructions

Companion to `SPEC.md` and `DEV_ENVIRONMENT.md`. This document tells you (the implementing agent) to **build the complete versioning, CI/CD, and production-update system** for this project. The human operator is not a DevOps specialist — your job is to make deployment a one-command, safe, reversible operation and to document the few manual steps clearly.

Everything lives inside the operator's own **Gitea** server (source, CI, and container registry). No GitHub, no external CI, no external registry.

---

## 0. What you are building (deliverables checklist)

Create all of the following, committed to the repo:

1. `.gitea/workflows/ci.yml` — test + build + push pipeline (runs on every push and PR).
2. `.gitea/workflows/release.yml` — on a version tag: test, build images, push to Gitea registry, deploy to production.
3. `docker/Dockerfile.backend` and `docker/Dockerfile.frontend` — production images (multi-stage, slim).
4. `docker/docker-compose.prod.yml` — the production stack, referencing images by **pinned tag** from the Gitea registry.
5. `deploy/deploy.sh` — the script that runs ON the production server to apply an update safely.
6. `deploy/rollback.sh` — one-command rollback to the previous version.
7. `backend/app/api/health.py` — `/api/healthz` and `/api/version` endpoints.
8. `deploy/VERSION` — plain-text current version, written by the release pipeline.
9. `docs/deployment.md` — a short **operator runbook** in plain language (how to release, how to roll back, what to do if it breaks). Write this for a non-expert.
10. `docs/gitea-setup.md` — the **one-time manual setup** the operator must do in Gitea's UI (documented step by step, see §7).

Do not require any tool the operator must learn. The whole system is: **push a tag → it deploys. Run one script → it rolls back.**

---

## 1. Versioning rules (implement, don't deviate)

- **SemVer git tags** are the single source of truth: `v{MAJOR}.{MINOR}.{PATCH}` (e.g. `v1.4.2`).
- The tag name becomes the image tag. Images are **never** tagged `latest` in production compose. `latest` may exist as a convenience alias but production always pins an explicit version.
- `/api/version` returns the running version (read from a build-time env `APP_VERSION` baked into the image). `/api/healthz` returns 200 only when DB + Redis are reachable.
- Every release is atomic (roll the whole tag) and reversible (redeploy the previous tag). Never hand-edit files on the production server to "patch" — always go through a tag.
- Tag creation is the human's explicit release action. Do NOT auto-tag from routine commits — routine commits run CI (test+build) but do not deploy. Only a `v*` tag deploys.

## 2. Images (`docker/Dockerfile.backend`, `docker/Dockerfile.frontend`)

- Multi-stage builds. Backend: build deps in a builder stage, copy only the venv + app into a slim `python:3.11-slim` runtime; run as a non-root user; `ENV APP_VERSION` set via build arg. Frontend: build with Node 20, serve the static bundle from nginx-alpine (or have the API serve it — pick API-served only if simpler for a single node; default to a separate `web` nginx image).
- Backend image entrypoint supports three roles via command: API (`uvicorn`), worker-jobs, worker-sync (matches SPEC §6.3 `WORKER_ROLE`). One image, three commands.
- Label images with the version and git SHA (`org.opencontainers.image.version`, `.revision`).
- Registry path: `{GITEA_HOST}/{OWNER}/webmail/backend` and `.../web`. Read `GITEA_HOST`/`OWNER` from CI variables, never hardcode.

## 3. CI pipeline (`.gitea/workflows/ci.yml`)

Trigger: push to any branch, and pull requests. Steps:
1. Spin up service containers (postgres:16, redis:7, meilisearch:v1.8, greenmail) — reuse `docker/docker-compose.ci.yml` from DEV_ENVIRONMENT.
2. Backend gates: `ruff check`, `ruff format --check`, `mypy --strict app`, `pytest -q` (unit + integration, coverage floors per SPEC §13).
3. Frontend gates: `tsc --noEmit`, `eslint`, `vitest run`.
4. Build both Docker images (validates the Dockerfiles) — but on plain branch pushes, **do not push** to the registry (build-only smoke).
5. Fail the whole workflow if any gate fails.

This gives every commit a green/red signal without deploying anything.

## 4. Release pipeline (`.gitea/workflows/release.yml`)

Trigger: push of a tag matching `v*`. Steps, in order, **stopping on any failure**:
1. Run the full CI gate set (§3) against the tagged commit. No shortcuts — a tag that fails tests must not deploy.
2. Build backend + `web` images with `APP_VERSION={tag}` and the git SHA.
3. Log in to the Gitea container registry (using a CI secret token) and push both images tagged `{tag}` and `latest`.
4. Deploy to production by connecting to the production host over SSH (key from a CI secret) and running `deploy/deploy.sh {tag}`. See §5.
5. On deploy success, write `{tag}` to `deploy/VERSION` and update `/api/version` expectation. On deploy failure, the pipeline must surface the error loudly and NOT leave a half-applied state (deploy.sh handles atomicity; see §5).

Keep the runner OFF the production box (run it on the dev/build machine or a spare fleet node) so a build can't starve the mail server.

## 5. The deploy script (`deploy/deploy.sh`) — safety is the whole point

Runs on the production host. Takes one argument: the target version tag. It MUST do, in this exact order:

1. **Record current version** (read running `/api/version` or `deploy/VERSION`) → save as the rollback target.
2. **Snapshot the database BEFORE touching anything** — trigger a `pgBackRest` backup (or a ZFS snapshot of `tank/pg` if pgBackRest isn't configured yet). Abort the deploy if the snapshot fails. This is non-negotiable: a bad migration must always be recoverable.
3. **Pull the new images** for `{tag}` from the Gitea registry. Abort if pull fails (nothing has changed yet — safe).
4. **Run database migrations** as a one-shot container: `alembic upgrade head` (this includes plugin migration branches per SPEC §11.1). Migrations MUST be backward-compatible / expand-then-contract so the brief mixed-version window is safe. Abort + alert if migration fails (DB snapshot from step 2 is the recovery path).
5. **Roll services in dependency order:** workers first (`worker-sync`, `worker-jobs`), then the API, then `web` — `docker compose -f docker-compose.prod.yml up -d --wait` with the new tag. `--wait` blocks on healthchecks.
6. **Health-gate:** poll `/api/healthz` for up to 60 s. If it doesn't go green, **automatically invoke `rollback.sh`** to the saved previous version and exit non-zero. Never leave production unhealthy.
7. On success: write `{tag}` to `deploy/VERSION`, log the old→new transition, done.

The script is idempotent and logs every step with timestamps. It reads all config (registry host, compose file path, healthcheck URL, ret(ry counts) from `deploy/deploy.env`, not hardcoded.

Because SPEC §6.3 sync workers are lease-owned and stateless, rolling them mid-operation is safe: leases transfer to the new worker within ~90 s, no mail lost, no duplicate rows (idempotent `(mailbox_id, uid)` upserts). Note this in the script comments so no future change breaks the assumption.

## 6. Rollback (`deploy/rollback.sh`)

One argument (target version) or, with no argument, the version recorded as "previous" by the last deploy. Steps: pull the old images (should be cached), roll services back to that tag, health-gate. **Migration note:** because migrations are expand-then-contract and backward-compatible, rolling the *code* back does not require rolling the *schema* back in the common case — document this clearly, and for the rare breaking migration, instruct the operator to restore the pre-deploy DB snapshot from step 5.2 (give the exact pgBackRest/ZFS restore command in `docs/deployment.md`).

## 7. One-time manual setup (`docs/gitea-setup.md`) — write this for a non-expert

The operator must do these once in Gitea's web UI; document each with exact click paths and screenshots-in-words:
1. **Enable Gitea Actions** for the repo (Settings → Actions).
2. **Register a runner:** provide the exact commands to run the `act_runner` container on the build machine, register it against the Gitea instance with a registration token, and label it (`self-hosted`, `linux`). Include the docker-compose snippet for the runner.
3. **Enable the Gitea container registry** (it's built in) and create a **package-registry access token**; store it as an Actions secret `REGISTRY_TOKEN`.
4. **Add Actions secrets:** `REGISTRY_TOKEN`, `PROD_SSH_KEY` (private key for the deploy user on the production host), `PROD_HOST`, `PROD_USER`. List exactly where in the UI (Settings → Actions → Secrets).
5. **On the production host:** create a low-privilege `deploy` user, add it to the `docker` group, install its public key, and `docker login` to the Gitea registry once (or bake it into deploy.sh via the token). Give copy-paste commands.
6. **First deploy:** how to cut the first release — `git tag v1.0.0 && git push origin v1.0.0` — and what success looks like (`/api/version` returns it).

Keep this doc short, sequential, and copy-pasteable. Assume the reader will follow it literally.

## 8. Operator runbook (`docs/deployment.md`) — plain language

Cover, in a few short sections:
- **How to release:** "Tag and push. That's it." — `git tag v1.4.3 && git push origin v1.4.3`, then watch the Actions tab.
- **How to see what's running in production:** `curl https://{host}/api/version`.
- **How to roll back:** `ssh {prod} 'cd /opt/webmail && ./deploy/rollback.sh'`.
- **If a deploy fails:** it auto-rolls-back and production stays on the old version; where to read the logs; when to restore a DB snapshot (rare) with the exact command.
- **What NOT to do:** never edit files directly on the server, never use `latest` in the compose file, never skip the tag.

## 9. Migration discipline (encode as a rule)

Add a short `docs/migrations.md`: all Alembic migrations must be **expand-then-contract** — add new columns/tables in one release, backfill, switch code, drop old columns only in a *later* release once no running code needs them. This is what makes zero-downtime rolls and easy rollback possible. Add a CI check or PR-checklist item reminding of this; never generate a destructive migration (DROP/rename) in the same release that stops using the column.

## 10. Acceptance criteria

1. Pushing a normal commit runs CI (tests + image build) and does **not** deploy.
2. Pushing a `v*` tag whose tests pass builds images, pushes them to the Gitea registry, and deploys to production with no manual step beyond the tag push.
3. Pushing a `v*` tag whose tests **fail** produces no image and no deploy.
4. `/api/version` on production returns the deployed tag after a release.
5. A deploy whose new version fails its healthcheck auto-rolls-back and leaves production healthy on the prior version (test with a deliberately broken image).
6. `rollback.sh` returns production to the prior version in one command.
7. A DB snapshot exists (pgBackRest or ZFS) timestamped immediately before every migration — verify one is created on a test deploy.
8. Rolling sync workers during a deploy loses no mail and creates no duplicate message rows (ties to SPEC §6.3 acceptance tests).
9. `docs/gitea-setup.md` and `docs/deployment.md` are followable by a non-DevOps operator: every manual step has an exact command or click path.

## 11. Build order

1. Health/version endpoints (§1) + Dockerfiles (§2) + `docker-compose.prod.yml`.
2. `ci.yml` (get green tests + building images first).
3. Registry push + `release.yml`.
4. `deploy.sh` + `rollback.sh` + `deploy.env` (test against a staging target — the same compose on a different port on the dev box — before pointing at the real host).
5. The two docs (§7, §8) + `migrations.md`.
6. Prove all §10 acceptance criteria, including the deliberately-broken-image auto-rollback test.

Keep it boring and legible. The operator should never have to think about the mechanics again after the one-time Gitea setup.