Deploy: LanguageTool spell checker in production, with optional n-grams
- docker-compose.yml: a `languagetool` service (erikvl87/languagetool:6.8),
internal only, 1 GB heap, one CPU, a 28 GB memory limit that is room for the
n-gram page cache (reclaimable, not memory in use), and a low-priority
warm-up read of the n-gram files at start so first checks hit RAM even on
mechanical disks. `init` creates $DATA_ROOT/languagetool-ngrams; the app
containers get LANGUAGETOOL_URL (empty turns the check off).
- `webmail.sh ngrams [langs]` (default en de fr es): deploy/ngrams.py, run in
the backend image, downloads LanguageTool's n-gram data with resume, checks
free space first (unpacked is ~2x the zip: ~26 GB for all four), unpacks into
a temp folder and moves it into place, then recreates the engine to load it.
languagetool.org refuses Python's default User-Agent, so it sends its own.
- `doctor` reports whether the spell checker answers and which n-gram
languages are installed.
- docs/nas-install.md and .env.prod.example describe all of it.
Verified locally: German n-grams (1.58 GB -> 3.06 GB) load, and catch what the
rules alone do not ("liegt ihm Schrank" -> im, "Der Patenten" -> Patienten).
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
This commit is contained in:
@@ -606,6 +606,38 @@ Its Bayes database lives in `$DATA_ROOT/rspamd-redis`. Like the search index,
|
||||
that is derived data: it rides along in backups but never needs restoring, and
|
||||
retraining rebuilds it.
|
||||
|
||||
### The spell checker (LanguageTool)
|
||||
|
||||
`languagetool` checks spelling and grammar in the compose window, in some 30
|
||||
languages. Like the anti-spam engine nothing waits for it: when it is down the
|
||||
compose window falls back to the browser's own spell checker. It is never
|
||||
published to the host — the app's API is its only client.
|
||||
|
||||
It needs about **1.2 GB of RAM** (`LANGUAGETOOL_HEAP=1g` plus the JVM) and is
|
||||
capped at one CPU core (`LANGUAGETOOL_CPUS`), so checking can never slow sync,
|
||||
search or the API.
|
||||
|
||||
**Optional n-gram data** teaches it to catch real words used wrongly — "the
|
||||
affect was huge", "Das Buch liegt ihm Schrank" — beyond its hand-written rules
|
||||
(English has ~780 such word pairs, German ~70). Install it once:
|
||||
|
||||
```bash
|
||||
./deploy/webmail.sh ngrams # en de fr es — or name languages: ngrams en de
|
||||
```
|
||||
|
||||
That downloads about **13 GB** (English 8.4 GB, German/French/Spanish ~1.6 GB
|
||||
each) into `$DATA_ROOT/languagetool-ngrams`, unpacks it to about **26 GB**, and
|
||||
restarts the engine to load it. An interrupted download resumes when you run
|
||||
the command again. There is no n-gram data for Polish; Polish text is checked
|
||||
exactly as without it.
|
||||
|
||||
The data is read from disk on demand and cached by the kernel, so the memory
|
||||
limit (`LANGUAGETOOL_MEM_LIMIT=28g`) is room for that cache, not memory in use.
|
||||
At start the container reads the files once in the background, at low
|
||||
priority, so even the first checks after a restart are served from RAM. On
|
||||
mechanical disks that warm-up takes several minutes; an NVMe cache in front of
|
||||
the volume makes it — and any later cache misses — fast.
|
||||
|
||||
Two NAS-specific realities worth knowing:
|
||||
|
||||
- **Disk hibernation won't happen.** Active accounts hold IMAP IDLE connections
|
||||
@@ -839,6 +871,7 @@ directories are root-owned.
|
||||
./deploy/webmail.sh backup verified dump, right now
|
||||
./deploy/webmail.sh restore <dump> restore a dump (asks for confirmation)
|
||||
./deploy/webmail.sh reindex [--purge] rebuild the search index
|
||||
./deploy/webmail.sh ngrams [langs] spell-check n-gram data (default: en de fr es)
|
||||
./deploy/webmail.sh psql database shell
|
||||
./deploy/webmail.sh shell [svc] shell in a container
|
||||
```
|
||||
|
||||
Reference in New Issue
Block a user