WOR Hostdocumentation v1.0.5 Home Downloads

Operations & recovery

doctor, health, diagnose and run: finding what is broken, why, and bringing it back. Plus machine-readable output and running wor from a program.

The recovery story

When something is down, three commands answer three different questions, then a fourth fixes it. They share guarantees: read-only, never prompt for sudo, every probe has a timeout, exit code 0 or 1 so cron and monitors can use them, and they take no workspace lock (an incident tool must not be the thing left waiting).

doctoris the machine ready?
→
healthwhich services are broken?
→
diagnose <target>why, and what to run
→
runbring everything back
CommandQuestionScopeTakes lock
wor doctorAre runtimes and tools installed and the workspace set up?The machineyes (it can refresh the default vhost)
wor infoWhat is this service? (no judgement)One serviceyes
wor healthAre my services actually serving?Every enabled serviceno
wor diagnoseWhy is this one broken, and how do I fix it?One host or serviceno
wor runMake everything enabled be running.Every enabled serviceyes

wor health

The "the site is down but I don't know which one" sweep, for example after a reboot. For every enabled service it checks the process and port layer, then fires one real HTTP request through the web server (the first registered host, dialling 127.0.0.1 with a Host header). The HTTP request is the point: permission, vhost and proxy problems never kill a process, so "pool accepting connections" does not mean the site is reachable. That lesson came from a real incident on 2026-07-06 where a pool was healthy while nginx returned 403 and 502.

Example outputs on this page come from real runs of wor on macOS. Machine-specific values (tool versions, disk sizes, dates, file names) have been replaced with typical ones and long paths shortened, so your numbers will differ. On Linux, paths and a few platform-specific checks differ.

$ wor health
WOR Health (2 enabled services)
--------------------------------------------
Disk Usage  : 38.0G / 100.0G (38%)
--------------------------------------------

[ok] com-example/myapp
    Status : Online (served by web server)
    Runtime: static
    [ok] cert myapp.example.com: valid for 1000 more day(s)
    [ok] https://myapp.example.com -> 200

[warn] default/web
    Status : Online (served by web server)
    Runtime: static
    [i] no host registered -- HTTP not probed

--------------------------------------------
Healthy : 1
Warning : 1
Failed  : 0

Here com-example/myapp is healthy (certificate valid, a real HTTPS request returned 200). default/web is a warning because no host name is registered for it, so there was nothing to probe. Exit code 0.

Layout

A machine summary, then one card per service.

Three tiers

TierMeaningExit code
healthyProcess, port and HTTP all fine.0
warningHTTP 404 (normal for many APIs with no / page; false FAILs make people stop trusting the command), no host to probe, pool drift from .wor/php.ini, or a certificate that is expiring, expired or failed its last sync. Shown visibly.0
failedBroken process, or HTTP 4xx, 5xx, refused or timeout.1

It ends with a healthy, warning and failed tally and a wor diagnose <target> suggestion per failed service. This used to be wor diagnose --all; it was split out on 2026-07-06 because "diagnose" means analysing a patient you have already identified, while this command finds who is sick.

wor diagnose

For one down or misbehaving host or domain/service. The goal is to cut time to recovery: the admin learns the cause and the fix command from one run. It never changes anything. No auto-fix, no --fix: while production is down, the admin decides.

It follows the request path, outside in

The first failure along the chain is the most likely root cause. A layer that cannot be checked because an earlier one failed is marked SKIP. Rows print live as they are checked; with everything down the probes together can take 10 to 15 seconds, and a blank screen is worse.

#LayerChecks
1ConfigService exists and is enabled; entry point exists (and is executable for go); docroot; runtime installed.
2DNS / hostsHost resolves and points at this machine (127.0.0.1 for local domains); the /etc/hosts entry is present.
3Web serverMaster running; this host's vhost present and enabled; config test (run unelevated, reported as "not verified" when it would need root); certificate files and expiry read with crypto/x509 (no openssl): FAIL when expired, WARN under 14 days.
4ProcessPM2: daemon alive, status, restart count and uptime (many restarts in 10 minutes means crash loop). The special case "machine just rebooted, PM2 empty" gives the direct verdict to run wor run. systemd: ActiveState, SubState, Result (oom-kill, start-limit-hit), NRestarts, ExecMainStatus. php-fpm: master running, pool file, socket exists and is accessible to the web server user, php-fpm -t, plus a php settings drift check. static: skipped.
5PortIs anything listening on the service port, and does the PID match this service? A port stolen by another process (the classic EADDRINUSE) shows the thief's name and PID.
6HTTP, two layersDirect to 127.0.0.1:<port> (bypassing the web server), then through the web server with the real Host header (so it cannot wander off to a CDN or proxy).
7Files and diskCan the web server user traverse to the docroot (Debian family)? Disk at or above 95% on the filesystem holding WOR_HOME is a loud warning.
8LogsPM2 error log, journalctl -u, the pool's error log, the nginx or Apache error log. Known patterns are translated into a cause and a fix.

A real run

A node service that is registered but whose process manager is not running, on a name that does not resolve. Note that the root cause is the one thing to fix first, and the DNS finding is kept as a second possibility rather than being mixed into it:

$ wor diagnose api.example.com
WOR Diagnose
------------
Target : com-example/api
Host   : api.example.com  [ssl: none]
Runtime: node v20.11.1
Server : nginx (nginx/1.24.0)

Checks
------------------------------------------------
[PASS] config    enabled (node, port 3100), entry app.js found
[FAIL] dns       api.example.com does not resolve: lookup api.example.com: no such host
[PASS] nginx     running, vhost ok, config test ok
[PASS] ssl       none on record (plain http)
[FAIL] process   pm2 daemon unreachable: pm2 jlist: exit status 1
[SKIP] port      (process not running)
[FAIL] http-app  direct 127.0.0.1:3100 -> connection refused
[PASS] http-host via api.example.com :80 -> 200
[SKIP] files     reachability only auto-checked on Debian/Ubuntu
[WARN] disk      filesystem holding WOR_HOME is 96% full
[SKIP] logs      no logs readable (may need root, or nothing logged yet)

Summary
------------------------------------------------
Status: FAILED (3 fail, 1 warn)

Root cause:
  pm2 daemon is unreachable

Cascade (same problem, seen from other layers):
  - http-app: direct 127.0.0.1:3100 -> connection refused

Suggested fix (run yourself -- wor diagnose never changes anything)
------------------------------------------------
1. Fix the root cause -- pm2 daemon is unreachable:
     wor run
     wor service logs com-example/api
2. If it persists -- DNS does not resolve api.example.com:
     check the DNS A/AAAA record for api.example.com at your DNS provider
3. Verify:
     wor diagnose com-example/api

The HTTP decider

DirectVia hostConclusion
OKOKThe service is fine. The problem may be off-machine (real DNS, firewall, CDN).
OKFAILWeb server, vhost or SSL.
FAIL-The app itself is dead: see the process layer and logs.

Through the host, 502 means upstream dead, 504 app hung or slow, 403 permission, 404 wrong docroot or route. php and static have no port, so only the host probe applies.

Log patterns it recognises

PatternCause
EADDRINUSEPort collision. Find the thief or change the port.
MODULE_NOT_FOUND / Cannot find moduleDependencies not installed (npm install).
permission deniedFile or socket permissions, with the right setfacl or chmod for the context.
Out of memory / oom-killNot enough RAM: check memory, reduce workers.
ENOENTMissing file or path: check the entry point and .env.
SSL_ERROR / certificateCertificate problem: see wor ssl ....

The table is a plain Go slice; adding a pattern means adding a row, with no plugin system.

The verdict: one root cause, not a pile of FAILs

The synthesis is what makes diagnose worth more than running systemctl, pm2, nginx -t and curl separately. Every FAIL produces a cause with a kind (proc, perm, port, tls, ...) and a static confidence (high, medium, low).

An illustrative run with log evidence

Constructed to show the Evidence section, which appears when a log pattern is found:

$ wor diagnose api.example.com

WOR Diagnose
------------
Target : example.com/api
Host   : api.example.com  [ssl: letsencrypt]
Runtime: node v20.11.1
Server : nginx (nginx/1.24.0)

Checks
------------------------------------------------
[PASS] config    enabled (node, port 3100), entry app.js found
[PASS] dns       api.example.com -> 127.0.0.1 (this machine)
[PASS] nginx     running, vhost ok, config test ok
[PASS] ssl       letsencrypt cert valid (45d left)
[FAIL] process   pm2 status: errored (15 restarts)
[SKIP] port      (process not running)
[FAIL] http-app  direct 127.0.0.1:3100 -> connection refused
[FAIL] http-host via api.example.com :443 -> 502
[WARN] logs      1 known error pattern(s) in pm2 error log -- see evidence below

Summary
------------------------------------------------
Status: FAILED (3 fail, 1 warn)

Root cause:
  app crashes on start (pm2 gave up restarting it) -- Node.js dependency missing (Cannot find module)

Cascade (same problem, seen from other layers):
  - http-app: direct 127.0.0.1:3100 -> connection refused
  - http-host: via api.example.com :443 -> 502

Evidence
------------------------------------------------
  pm2 error log:
    Error: Cannot find module 'express'

Suggested fix (run yourself -- wor diagnose never changes anything)
------------------------------------------------
1. Fix the root cause -- app crashes on start (...) -- Node.js dependency missing:
     wor service logs example.com/api
     wor service restart example.com/api
     cd <WOR_HOME>/domains/example.com/api && npm install && wor service restart example.com/api
2. Verify:
     wor diagnose example.com/api

Summary sits after Checks on purpose: the verdict ends up at the bottom, next to the prompt, which is where the eye lands first. Evidence is grouped by source, duplicate lines are collapsed as (xN) (compared with numbers stripped, since timestamps and pids differ on every repeat), truncated at about 160 characters, three lines per source. The fix section is a numbered runbook: step 1 fixes the root cause, later steps are the remaining independent causes by rank, and it always closes with a Verify step. A final "Also worth checking" list holds loose suggestions (a certificate nearing expiry, a flapping process, a rollback hint).

wor doctor versus the others

doctor answers "is the machine and runtime installed and ready?", health answers "are the services still serving?". Check lists and the Security section are on the command reference.

wor run as the recovery command

After a reboot, a crashed process manager or a stopped web server, wor run brings every enabled service back, starting only what is not running. See the command reference for the exact order. If PM2 has never been set up to start at boot, it offers to register that (showing the command first), so node services come back after the next reboot too.

Machine-readable output: --json

For a program reading wor rather than a person. WOR HCP, the web control panel that manages a wor host, used to screen-scrape wor doctor by splitting on the ✓ ⚠ ✗ glyphs, which breaks quietly the day someone rewords a check. Reformatting a health report must not be a breaking change.

wor version --json
wor doctor  --json
wor health  --json
wor ssl status <host|domain/service> --json

The contract

doctor returns an ordered environment list plus checks, each with section, level (ok, warn, fail), the printed message and any notes (the indented paths and [INFO] fix lines under a check). health returns host figures, one entry per service with its own level, and a summary. Figures that were not measured are absent fields, never zeros.

Running wor from a program: --non-interactive

Many commands ask questions. When nobody can answer, wor stops rather than guessing:

Two kinds of question behave differently, on purpose:

KindBehaviourWhy
The sudo questionDeclined, so the action fails through its normal error path.It is asked deep inside privileged operations whose own error paths restore what they changed (the vhost snapshot, the pool rollback). An abort would jump past them.
Optional follow-ups after the work is done ("Deploy now?" after source clone or rollback, the php-fpm restart offer, "create your first website?" at the end of setup)Skipped, and the command still succeeds.Aborting would report finished work as failed or stop halfway.

Backups and rollback safety

Using the exit codes from cron

# every 5 minutes: exit 1 when something is FAILED, 0 when healthy or only warning
*/5 * * * *  /usr/local/bin/wor health --non-interactive --json > /var/tmp/wor-health.json || mailx -s "wor health failed" [email protected] < /var/tmp/wor-health.json

This is an illustration of how the guarantees compose, not a feature wor ships: health takes no lock, never prompts for sudo, has bounded probes, and writes one JSON document to stdout.