Deploy: fix the initdb race that made migrate time out
On a real Synology first run, `migrate` exited 1 with a bare asyncpg TimeoutError while the postgres log showed a perfectly healthy server. The healthcheck was `pg_isready` with no -h, so it checked the UNIX SOCKET. During first-time initialisation the postgres image runs a temporary server that listens on the socket only (listen_addresses=''), so pg_isready answered and the container was marked healthy while nothing was listening on TCP yet. Dependents then connected to postgres:5432, hung, and timed out. It only bites on the very first start of a fresh data directory, which is exactly the run an operator is doing when they have least idea what is wrong. - healthcheck uses `pg_isready -h 127.0.0.1 -p 5432`, so health means TCP is actually accepting. - `migrate` additionally waits for the port itself (DB_WAIT_SECONDS, default 180) before running alembic. It is a one-shot with no restart policy, so a few seconds of slow NAS disk should not fail the whole bring-up; the API and workers already self-heal through their restart policy. Verified with a first-time bring-up against a fresh data directory: migrate logs "database accepting connections", applies core + all four plugin branches, and the stack comes up serving. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01Nux3LCu4zP9nih4BRhPyHw
This commit is contained in:
+6
-1
@@ -136,7 +136,12 @@ services:
|
|||||||
volumes:
|
volumes:
|
||||||
- ${DATA_ROOT:-/NO-.env-FOUND-RUN-deploy_webmail.sh-init-FIRST}/postgres:/var/lib/postgresql/data
|
- ${DATA_ROOT:-/NO-.env-FOUND-RUN-deploy_webmail.sh-init-FIRST}/postgres:/var/lib/postgresql/data
|
||||||
healthcheck:
|
healthcheck:
|
||||||
test: ["CMD-SHELL", "pg_isready -U ${POSTGRES_USER:-webmail} -d ${POSTGRES_DB:-webmail}"]
|
# -h 127.0.0.1 forces a TCP check. Without it pg_isready uses the Unix
|
||||||
|
# socket, which the temporary server that the postgres image runs during
|
||||||
|
# first-time initdb DOES answer (it listens on the socket only) — so the
|
||||||
|
# container went healthy before TCP was up, and dependents connecting to
|
||||||
|
# postgres:5432 hung until they timed out.
|
||||||
|
test: ["CMD-SHELL", "pg_isready -h 127.0.0.1 -p 5432 -U ${POSTGRES_USER:-webmail} -d ${POSTGRES_DB:-webmail}"]
|
||||||
interval: 10s
|
interval: 10s
|
||||||
timeout: 5s
|
timeout: 5s
|
||||||
retries: 12
|
retries: 12
|
||||||
|
|||||||
@@ -24,7 +24,35 @@ set -eu
|
|||||||
cmd="${1:-api}"
|
cmd="${1:-api}"
|
||||||
[ "$#" -gt 0 ] && shift || true
|
[ "$#" -gt 0 ] && shift || true
|
||||||
|
|
||||||
|
# Belt and braces on top of the compose healthcheck: a NAS with slow disks can
|
||||||
|
# take a while over initdb, and `migrate` is a one-shot with no restart policy —
|
||||||
|
# if it starts a moment early it fails the whole bring-up.
|
||||||
|
wait_for_db() {
|
||||||
|
python - <<'PYEOF'
|
||||||
|
import os, socket, sys, time
|
||||||
|
from urllib.parse import urlparse
|
||||||
|
|
||||||
|
url = os.environ.get("DATABASE_URL", "")
|
||||||
|
parsed = urlparse(url.replace("+asyncpg", ""))
|
||||||
|
host = parsed.hostname or "postgres"
|
||||||
|
port = parsed.port or 5432
|
||||||
|
deadline = time.time() + float(os.environ.get("DB_WAIT_SECONDS", "180"))
|
||||||
|
|
||||||
|
while time.time() < deadline:
|
||||||
|
try:
|
||||||
|
with socket.create_connection((host, port), timeout=3):
|
||||||
|
print(f"[entrypoint] database accepting connections at {host}:{port}")
|
||||||
|
sys.exit(0)
|
||||||
|
except OSError:
|
||||||
|
time.sleep(2)
|
||||||
|
|
||||||
|
print(f"[entrypoint] ERROR: {host}:{port} never accepted a connection", file=sys.stderr)
|
||||||
|
sys.exit(1)
|
||||||
|
PYEOF
|
||||||
|
}
|
||||||
|
|
||||||
run_migrations() {
|
run_migrations() {
|
||||||
|
wait_for_db
|
||||||
echo "[entrypoint] core schema: alembic upgrade head"
|
echo "[entrypoint] core schema: alembic upgrade head"
|
||||||
alembic upgrade head
|
alembic upgrade head
|
||||||
for plugin in $ENABLED_PLUGINS; do
|
for plugin in $ENABLED_PLUGINS; do
|
||||||
|
|||||||
Reference in New Issue
Block a user