Operations & recovery
doctor, health, diagnose and run: finding what is broken, why, and bringing it back. Plus machine-readable output and running wor from a program.
The recovery story
When something is down, three commands answer three different questions, then a fourth fixes it. They share guarantees: read-only, never prompt for sudo, every probe has a timeout, exit code 0 or 1 so cron and monitors can use them, and they take no workspace lock (an incident tool must not be the thing left waiting).
| Command | Question | Scope | Takes lock |
|---|---|---|---|
wor doctor | Are runtimes and tools installed and the workspace set up? | The machine | yes (it can refresh the default vhost) |
wor info | What is this service? (no judgement) | One service | yes |
wor health | Are my services actually serving? | Every enabled service | no |
wor diagnose | Why is this one broken, and how do I fix it? | One host or service | no |
wor run | Make everything enabled be running. | Every enabled service | yes |
wor health
The "the site is down but I don't know which one" sweep, for example after a reboot. For every enabled service it checks the process and port layer, then fires one real HTTP request through the web server (the first registered host, dialling 127.0.0.1 with a Host header). The HTTP request is the point: permission, vhost and proxy problems never kill a process, so "pool accepting connections" does not mean the site is reachable. That lesson came from a real incident on 2026-07-06 where a pool was healthy while nginx returned 403 and 502.
Example outputs on this page come from real runs of wor on macOS. Machine-specific values (tool versions, disk sizes, dates, file names) have been replaced with typical ones and long paths shortened, so your numbers will differ. On Linux, paths and a few platform-specific checks differ.
$ wor health
WOR Health (2 enabled services)
--------------------------------------------
Disk Usage : 38.0G / 100.0G (38%)
--------------------------------------------
[ok] com-example/myapp
Status : Online (served by web server)
Runtime: static
[ok] cert myapp.example.com: valid for 1000 more day(s)
[ok] https://myapp.example.com -> 200
[warn] default/web
Status : Online (served by web server)
Runtime: static
[i] no host registered -- HTTP not probed
--------------------------------------------
Healthy : 1
Warning : 1
Failed : 0Here com-example/myapp is healthy (certificate valid, a real HTTPS request returned 200). default/web is a warning because no host name is registered for it, so there was nothing to probe. Exit code 0.
Layout
A machine summary, then one card per service.
- Header: Host CPU (percent across all cores, from two samples of
/proc/stat), Host Memory (MemTotal minus MemAvailable), Disk Usage (the filesystem holdingWOR_HOME). - Card: Status; Runtime (with the php-fpm worker count for dedicated pools); CPU (100% = one full core, like
top); Memory (RSS and percent of total RAM); Uptime (PM2 only); an HTTP line✓|⚠|✗ <url> -> <code>; and acertline for a service whose first host has a certificate (days left, or why the last sync failed). Lines with no data are hidden rather than showing "-". - Resource sources: PM2 from
pm2 jlistmonit; systemd fromCPUUsageNSecandMemoryCurrentdeltas; a php pool by summing every worker in/procwhose title isphp-fpm: pool <name>. The whole report waits for a single ~200 to 250 ms sample. The/procreader is Linux-only, so macOS sees only PM2's values.
Three tiers
| Tier | Meaning | Exit code |
|---|---|---|
| healthy | Process, port and HTTP all fine. | 0 |
| warning | HTTP 404 (normal for many APIs with no / page; false FAILs make people stop trusting the command), no host to probe, pool drift from .wor/php.ini, or a certificate that is expiring, expired or failed its last sync. Shown visibly. | 0 |
| failed | Broken process, or HTTP 4xx, 5xx, refused or timeout. | 1 |
It ends with a healthy, warning and failed tally and a wor diagnose <target> suggestion per failed service. This used to be wor diagnose --all; it was split out on 2026-07-06 because "diagnose" means analysing a patient you have already identified, while this command finds who is sick.
wor diagnose
For one down or misbehaving host or domain/service. The goal is to cut time to recovery: the admin learns the cause and the fix command from one run. It never changes anything. No auto-fix, no --fix: while production is down, the admin decides.
It follows the request path, outside in
The first failure along the chain is the most likely root cause. A layer that cannot be checked because an earlier one failed is marked SKIP. Rows print live as they are checked; with everything down the probes together can take 10 to 15 seconds, and a blank screen is worse.
| # | Layer | Checks |
|---|---|---|
| 1 | Config | Service exists and is enabled; entry point exists (and is executable for go); docroot; runtime installed. |
| 2 | DNS / hosts | Host resolves and points at this machine (127.0.0.1 for local domains); the /etc/hosts entry is present. |
| 3 | Web server | Master running; this host's vhost present and enabled; config test (run unelevated, reported as "not verified" when it would need root); certificate files and expiry read with crypto/x509 (no openssl): FAIL when expired, WARN under 14 days. |
| 4 | Process | PM2: daemon alive, status, restart count and uptime (many restarts in 10 minutes means crash loop). The special case "machine just rebooted, PM2 empty" gives the direct verdict to run wor run. systemd: ActiveState, SubState, Result (oom-kill, start-limit-hit), NRestarts, ExecMainStatus. php-fpm: master running, pool file, socket exists and is accessible to the web server user, php-fpm -t, plus a php settings drift check. static: skipped. |
| 5 | Port | Is anything listening on the service port, and does the PID match this service? A port stolen by another process (the classic EADDRINUSE) shows the thief's name and PID. |
| 6 | HTTP, two layers | Direct to 127.0.0.1:<port> (bypassing the web server), then through the web server with the real Host header (so it cannot wander off to a CDN or proxy). |
| 7 | Files and disk | Can the web server user traverse to the docroot (Debian family)? Disk at or above 95% on the filesystem holding WOR_HOME is a loud warning. |
| 8 | Logs | PM2 error log, journalctl -u, the pool's error log, the nginx or Apache error log. Known patterns are translated into a cause and a fix. |
A real run
A node service that is registered but whose process manager is not running, on a name that does not resolve. Note that the root cause is the one thing to fix first, and the DNS finding is kept as a second possibility rather than being mixed into it:
$ wor diagnose api.example.com
WOR Diagnose
------------
Target : com-example/api
Host : api.example.com [ssl: none]
Runtime: node v20.11.1
Server : nginx (nginx/1.24.0)
Checks
------------------------------------------------
[PASS] config enabled (node, port 3100), entry app.js found
[FAIL] dns api.example.com does not resolve: lookup api.example.com: no such host
[PASS] nginx running, vhost ok, config test ok
[PASS] ssl none on record (plain http)
[FAIL] process pm2 daemon unreachable: pm2 jlist: exit status 1
[SKIP] port (process not running)
[FAIL] http-app direct 127.0.0.1:3100 -> connection refused
[PASS] http-host via api.example.com :80 -> 200
[SKIP] files reachability only auto-checked on Debian/Ubuntu
[WARN] disk filesystem holding WOR_HOME is 96% full
[SKIP] logs no logs readable (may need root, or nothing logged yet)
Summary
------------------------------------------------
Status: FAILED (3 fail, 1 warn)
Root cause:
pm2 daemon is unreachable
Cascade (same problem, seen from other layers):
- http-app: direct 127.0.0.1:3100 -> connection refused
Suggested fix (run yourself -- wor diagnose never changes anything)
------------------------------------------------
1. Fix the root cause -- pm2 daemon is unreachable:
wor run
wor service logs com-example/api
2. If it persists -- DNS does not resolve api.example.com:
check the DNS A/AAAA record for api.example.com at your DNS provider
3. Verify:
wor diagnose com-example/api
The HTTP decider
| Direct | Via host | Conclusion |
|---|---|---|
| OK | OK | The service is fine. The problem may be off-machine (real DNS, firewall, CDN). |
| OK | FAIL | Web server, vhost or SSL. |
| FAIL | - | The app itself is dead: see the process layer and logs. |
Through the host, 502 means upstream dead, 504 app hung or slow, 403 permission, 404 wrong docroot or route. php and static have no port, so only the host probe applies.
Log patterns it recognises
| Pattern | Cause |
|---|---|
EADDRINUSE | Port collision. Find the thief or change the port. |
MODULE_NOT_FOUND / Cannot find module | Dependencies not installed (npm install). |
permission denied | File or socket permissions, with the right setfacl or chmod for the context. |
Out of memory / oom-kill | Not enough RAM: check memory, reduce workers. |
ENOENT | Missing file or path: check the entry point and .env. |
SSL_ERROR / certificate | Certificate problem: see wor ssl .... |
The table is a plain Go slice; adding a pattern means adding a row, with no plugin system.
The verdict: one root cause, not a pile of FAILs
The synthesis is what makes diagnose worth more than running systemctl, pm2, nginx -t and curl separately. Every FAIL produces a cause with a kind (proc, perm, port, tls, ...) and a static confidence (high, medium, low).
- Causes of the same kind from different layers merge (HTTP 403 plus a files-blocked check are one permission problem). The higher confidence wins the wording.
- A matching log pattern confirms an existing cause, raising it to high, or adds the reason to the process cause ("app crashes on start, Cannot find module"). A lone pattern with no confirming layer is low.
- Ranked by confidence, ties broken by layer order along the request path. The report names at most two "other possibilities" as a hedge against mis-ranking.
- Cascade lists only FAILs proven to be the same problem by kind merging. It never guesses a causal link: a wrong claim is worse than no claim.
- Log evidence from the nginx or Apache error log discards lines older than one hour (a line whose timestamp cannot be parsed is kept), so lines from before a config fix cannot hijack the root cause. A log line naming a
/domains/path outside the currentWOR_HOMEconcludes "a config from an old installation is still active" rather than a generic permission issue. - Special cases: machine just rebooted with PM2 empty gives
wor run; a service that died shortly after the latest deploy giveswor rollback <target>; an expired certificate points atwor ssl; every layer passing says plainly that the machine looks fine and the problem is probably external.
An illustrative run with log evidence
Constructed to show the Evidence section, which appears when a log pattern is found:
$ wor diagnose api.example.com
WOR Diagnose
------------
Target : example.com/api
Host : api.example.com [ssl: letsencrypt]
Runtime: node v20.11.1
Server : nginx (nginx/1.24.0)
Checks
------------------------------------------------
[PASS] config enabled (node, port 3100), entry app.js found
[PASS] dns api.example.com -> 127.0.0.1 (this machine)
[PASS] nginx running, vhost ok, config test ok
[PASS] ssl letsencrypt cert valid (45d left)
[FAIL] process pm2 status: errored (15 restarts)
[SKIP] port (process not running)
[FAIL] http-app direct 127.0.0.1:3100 -> connection refused
[FAIL] http-host via api.example.com :443 -> 502
[WARN] logs 1 known error pattern(s) in pm2 error log -- see evidence below
Summary
------------------------------------------------
Status: FAILED (3 fail, 1 warn)
Root cause:
app crashes on start (pm2 gave up restarting it) -- Node.js dependency missing (Cannot find module)
Cascade (same problem, seen from other layers):
- http-app: direct 127.0.0.1:3100 -> connection refused
- http-host: via api.example.com :443 -> 502
Evidence
------------------------------------------------
pm2 error log:
Error: Cannot find module 'express'
Suggested fix (run yourself -- wor diagnose never changes anything)
------------------------------------------------
1. Fix the root cause -- app crashes on start (...) -- Node.js dependency missing:
wor service logs example.com/api
wor service restart example.com/api
cd <WOR_HOME>/domains/example.com/api && npm install && wor service restart example.com/api
2. Verify:
wor diagnose example.com/api
Summary sits after Checks on purpose: the verdict ends up at the bottom, next to the prompt, which is where the eye lands first. Evidence is grouped by source, duplicate lines are collapsed as (xN) (compared with numbers stripped, since timestamps and pids differ on every repeat), truncated at about 160 characters, three lines per source. The fix section is a numbered runbook: step 1 fixes the root cause, later steps are the remaining independent causes by rank, and it always closes with a Verify step. A final "Also worth checking" list holds loose suggestions (a certificate nearing expiry, a flapping process, a rollback hint).
wor doctor versus the others
doctor answers "is the machine and runtime installed and ready?", health answers "are the services still serving?". Check lists and the Security section are on the command reference.
wor run as the recovery command
After a reboot, a crashed process manager or a stopped web server, wor run brings every enabled service back, starting only what is not running. See the command reference for the exact order. If PM2 has never been set up to start at boot, it offers to register that (showing the command first), so node services come back after the next reboot too.
Machine-readable output: --json
For a program reading wor rather than a person. WOR HCP, the web control panel that manages a wor host, used to screen-scrape wor doctor by splitting on the ✓ ⚠ ✗ glyphs, which breaks quietly the day someone rewords a check. Reformatting a health report must not be a breaking change.
wor version --json
wor doctor --json
wor health --json
wor ssl status <host|domain/service> --json
The contract
- An allowlist of four.
--jsonanywhere else is a usage error that names the four, never a silently ignored flag. Every published payload is a shape that must keep working, so each command converted is a permanent obligation; mutating commands give a caller an exit code and live progress, and interactive ones have no output shape at all. A fifth is added when a real screen needs it, one command at a time. "schema": 1in every document. wor and its readers are upgraded independently. The number is bumped when an existing field changes meaning or disappears, never when a field is added (a reader ignores a field it does not know).- stdout carries exactly one JSON document, always, including on every kind of failure: busy lock, uninitialised workspace, refusal under sudo,
--jsonon the wrong command, or an error from the command. Failure prints{"schema":1,"error":"..."}. The humanERROR:line stays on stderr. Without this rule the first thing every reader writes is a guess about whether empty stdout means "healthy and silent" or "wor died". - Exit codes are unchanged.
doctor --jsonandhealth --jsonstill exit 1 on a real failure, and repeat the verdict as"failed"in the document.
doctor returns an ordered environment list plus checks, each with section, level (ok, warn, fail), the printed message and any notes (the indented paths and [INFO] fix lines under a check). health returns host figures, one entry per service with its own level, and a summary. Figures that were not measured are absent fields, never zeros.
Running wor from a program: --non-interactive
Many commands ask questions. When nobody can answer, wor stops rather than guessing:
- End of input is cancel, never a default. If stdin ends before a line is typed:
ERROR: no answer to "<question>" (stdin closed) -- cancelled.and exit 1. Piped answers (printf 'y\n' | wor ...) still work until they run out; Enter in a terminal still takes the default. --non-interactive(anywhere on the line) orWOR_NONINTERACTIVE=1makes the first question end the command at once, naming it, without reading stdin. Pass the flags that answer it instead:--yes,--add-hosts,--redirect, ...- sudo runs as
sudo -n, so an action needing a password fails immediately instead of hanging on a terminal that is not there, and wor's own "wor needs to run ... with sudo?" question is not asked.-ngrants nothing; what sudo allows is still sudo's decision.
Two kinds of question behave differently, on purpose:
| Kind | Behaviour | Why |
|---|---|---|
| The sudo question | Declined, so the action fails through its normal error path. | It is asked deep inside privileged operations whose own error paths restore what they changed (the vhost snapshot, the pool rollback). An abort would jump past them. |
Optional follow-ups after the work is done ("Deploy now?" after source clone or rollback, the php-fpm restart offer, "create your first website?" at the end of setup) | Skipped, and the command still succeeds. | Aborting would report finished work as failed or stop halfway. |
Backups and rollback safety
- Source:
wor source backupzips tobackups/<domain>/source/, filtered through the root.gitignore.source cloneandrollbackback up automatically first. - Database:
wor database backupwrites gzip dumps per day. Backup only; there is no restore command, deliberately. - Remember what a gitignored backup omits. The zip skips anything
.gitignoreexcludes, which commonly includes.envand.wor. That is whysource cloneasks about both before replacing a tree. - Removing a domain asks about backups, logs and web data in that order and deletes nothing until the last answer.
Using the exit codes from cron
# every 5 minutes: exit 1 when something is FAILED, 0 when healthy or only warning
*/5 * * * * /usr/local/bin/wor health --non-interactive --json > /var/tmp/wor-health.json || mailx -s "wor health failed" [email protected] < /var/tmp/wor-health.json
This is an illustration of how the guarantees compose, not a feature wor ships: health takes no lock, never prompts for sudo, has bounded probes, and writes one JSON document to stdout.