Deploy: LanguageTool spell checker in production, with optional n-grams

- docker-compose.yml: a `languagetool` service (erikvl87/languagetool:6.8),
  internal only, 1 GB heap, one CPU, a 28 GB memory limit that is room for the
  n-gram page cache (reclaimable, not memory in use), and a low-priority
  warm-up read of the n-gram files at start so first checks hit RAM even on
  mechanical disks. `init` creates $DATA_ROOT/languagetool-ngrams; the app
  containers get LANGUAGETOOL_URL (empty turns the check off).
- `webmail.sh ngrams [langs]` (default en de fr es): deploy/ngrams.py, run in
  the backend image, downloads LanguageTool's n-gram data with resume, checks
  free space first (unpacked is ~2x the zip: ~26 GB for all four), unpacks into
  a temp folder and moves it into place, then recreates the engine to load it.
  languagetool.org refuses Python's default User-Agent, so it sends its own.
- `doctor` reports whether the spell checker answers and which n-gram
  languages are installed.
- docs/nas-install.md and .env.prod.example describe all of it.

Verified locally: German n-grams (1.58 GB -> 3.06 GB) load, and catch what the
rules alone do not ("liegt ihm Schrank" -> im, "Der Patenten" -> Patienten).

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
This commit is contained in:
2026-10-05 07:44:36 +02:00
co-authored by Claude Opus 5.5
parent 1f2e539eb2
commit 139fff826e
5 changed files with 279 additions and 1 deletions
+48 -1
View File
@@ -69,6 +69,8 @@ x-backend-env: &backend-env
# DOES NOT — the controller answers 401, the plugin fails open, and the only
# visible symptom is "bayes": null on the stats endpoint.
RSPAMD_CONTROLLER_PASSWORD: ${RSPAMD_CONTROLLER_PASSWORD}
# Spelling and grammar check in the compose window. Empty turns it off.
LANGUAGETOOL_URL: ${LANGUAGETOOL_URL-http://languagetool:8010}
LOG_LEVEL: ${LOG_LEVEL:-INFO}
UNDO_SECONDS: ${UNDO_SECONDS:-10}
SESSION_TTL_DAYS: ${SESSION_TTL_DAYS:-30}
@@ -102,7 +104,8 @@ services:
- -c
- |
set -e
mkdir -p /mnt/postgres /mnt/blobs /mnt/backups /mnt/redis /mnt/meili /mnt/state
mkdir -p /mnt/postgres /mnt/blobs /mnt/backups /mnt/redis /mnt/meili /mnt/state \
/mnt/languagetool-ngrams
# Only recurse when ownership is actually wrong: `chown -R` over a large
# blob store on every start would be minutes of pointless disk churn.
[ "$$(stat -c %u /mnt/blobs)" = "10001" ] || chown -R 10001:10001 /mnt/blobs
@@ -262,6 +265,50 @@ services:
mem_limit: ${RSPAMD_MEM_LIMIT:-256m}
logging: *logging
# Spelling and grammar checking for the compose window. The app's API is its
# only client (POST /api/spellcheck); it is never published to the host.
#
# Optional n-gram data (`webmail.sh ngrams`: ~13 GB download, ~26 GB unpacked
# for en/de/fr/es) lets it catch real words used wrongly — "the affect was
# huge", "liegt ihm Schrank" — beyond its hand-written rules. It lives on disk
# and is read through the page cache, which Docker charges to this container:
# hence a memory limit far above the 1 GB Java heap, enough to keep all four
# languages cached. That room is file cache the kernel takes back whenever
# anything else needs memory, not memory in use. One CPU at most, so a burst
# of checking never slows sync, search or the API.
languagetool:
image: erikvl87/languagetool:6.8
# Read the n-gram files once at start (low priority, in the background), so
# even the first checks after a restart hit RAM instead of the disks.
command:
- bash
- -c
- |
if [ -d /ngrams ] && [ -n "$$(ls -A /ngrams 2>/dev/null)" ]; then
( nice -n 19 find /ngrams -type f -exec cat {} + > /dev/null 2>&1 & )
fi
exec bash start.sh
environment:
Java_Xms: 256m
Java_Xmx: ${LANGUAGETOOL_HEAP:-1g}
# Without the data the folder is empty and LanguageTool simply skips it.
langtool_languageModel: /ngrams
TZ: ${TZ:-UTC}
volumes:
- ${DATA_ROOT:-/NO-.env-FOUND-RUN-deploy_webmail.sh-init-FIRST}/languagetool-ngrams:/ngrams:ro
healthcheck:
test: ["CMD-SHELL", "wget -q -O /dev/null http://127.0.0.1:8010/v2/languages || exit 1"]
interval: 30s
timeout: 5s
retries: 5
start_period: 60s
depends_on:
init: { condition: service_completed_successfully }
restart: unless-stopped
mem_limit: ${LANGUAGETOOL_MEM_LIMIT:-28g}
cpus: ${LANGUAGETOOL_CPUS:-1.0}
logging: *logging
# Bayes tokens and fuzzy hashes. Separate from the app's redis on purpose
# (§C.2): this data is derived and retrainable, so it trades durability for
# room to grow — RDB snapshots every 5 min, and an eviction policy, where the