Deploy: LanguageTool spell checker in production, with optional n-grams

- docker-compose.yml: a `languagetool` service (erikvl87/languagetool:6.8),
  internal only, 1 GB heap, one CPU, a 28 GB memory limit that is room for the
  n-gram page cache (reclaimable, not memory in use), and a low-priority
  warm-up read of the n-gram files at start so first checks hit RAM even on
  mechanical disks. `init` creates $DATA_ROOT/languagetool-ngrams; the app
  containers get LANGUAGETOOL_URL (empty turns the check off).
- `webmail.sh ngrams [langs]` (default en de fr es): deploy/ngrams.py, run in
  the backend image, downloads LanguageTool's n-gram data with resume, checks
  free space first (unpacked is ~2x the zip: ~26 GB for all four), unpacks into
  a temp folder and moves it into place, then recreates the engine to load it.
  languagetool.org refuses Python's default User-Agent, so it sends its own.
- `doctor` reports whether the spell checker answers and which n-gram
  languages are installed.
- docs/nas-install.md and .env.prod.example describe all of it.

Verified locally: German n-grams (1.58 GB -> 3.06 GB) load, and catch what the
rules alone do not ("liegt ihm Schrank" -> im, "Der Patenten" -> Patienten).

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
This commit is contained in:
2026-10-05 07:44:36 +02:00
co-authored by Claude Opus 5.5
parent 1f2e539eb2
commit 139fff826e
5 changed files with 279 additions and 1 deletions
+33
View File
@@ -606,6 +606,38 @@ Its Bayes database lives in `$DATA_ROOT/rspamd-redis`. Like the search index,
that is derived data: it rides along in backups but never needs restoring, and
retraining rebuilds it.
### The spell checker (LanguageTool)
`languagetool` checks spelling and grammar in the compose window, in some 30
languages. Like the anti-spam engine nothing waits for it: when it is down the
compose window falls back to the browser's own spell checker. It is never
published to the host — the app's API is its only client.
It needs about **1.2 GB of RAM** (`LANGUAGETOOL_HEAP=1g` plus the JVM) and is
capped at one CPU core (`LANGUAGETOOL_CPUS`), so checking can never slow sync,
search or the API.
**Optional n-gram data** teaches it to catch real words used wrongly — "the
affect was huge", "Das Buch liegt ihm Schrank" — beyond its hand-written rules
(English has ~780 such word pairs, German ~70). Install it once:
```bash
./deploy/webmail.sh ngrams # en de fr es — or name languages: ngrams en de
```
That downloads about **13 GB** (English 8.4 GB, German/French/Spanish ~1.6 GB
each) into `$DATA_ROOT/languagetool-ngrams`, unpacks it to about **26 GB**, and
restarts the engine to load it. An interrupted download resumes when you run
the command again. There is no n-gram data for Polish; Polish text is checked
exactly as without it.
The data is read from disk on demand and cached by the kernel, so the memory
limit (`LANGUAGETOOL_MEM_LIMIT=28g`) is room for that cache, not memory in use.
At start the container reads the files once in the background, at low
priority, so even the first checks after a restart are served from RAM. On
mechanical disks that warm-up takes several minutes; an NVMe cache in front of
the volume makes it — and any later cache misses — fast.
Two NAS-specific realities worth knowing:
- **Disk hibernation won't happen.** Active accounts hold IMAP IDLE connections
@@ -839,6 +871,7 @@ directories are root-owned.
./deploy/webmail.sh backup verified dump, right now
./deploy/webmail.sh restore <dump> restore a dump (asks for confirmation)
./deploy/webmail.sh reindex [--purge] rebuild the search index
./deploy/webmail.sh ngrams [langs] spell-check n-gram data (default: en de fr es)
./deploy/webmail.sh psql database shell
./deploy/webmail.sh shell [svc] shell in a container
```