Browse the docs
Self-hosting
Monitoring and doctor
Watch a self-hosted Crunu server on four levels: the health endpoint, the doctor check, audit log integrity, and alerts sent to webhooks and email.
Last updated
On this page
A self-hosted server needs watching on four levels:
| What | How | How often |
|---|---|---|
| The server responds | /v2/health in an external uptime monitor | every minute |
| The server is set up correctly | node scripts/doctor.mjs | after every deploy and once a day |
| The audit log is intact | Audit log, then Verify integrity | once a week and on any suspicion |
| Suspicious events | Alerts and their notification channels | as they happen |
Health endpoint
https://vault.company.example/v2/health answers 200 with {"ok":true,"service":"crunu",…}. Add it to any external uptime monitor, such as UptimeRobot, Better Stack or Cloudflare health checks: expect status 200 with "ok":true in the body every minute, and alert after two failures in a row.
While the server is down, machines with the agent keep working with the variables they already have until their lease ends, up to 24 hours. To find out why the server is down, read the platform's log:
| Platform | Log |
|---|---|
| Your own server | docker compose -f deploy/docker/compose.yml logs --tail 200 server |
| Cloudflare | wrangler tail, or the Worker's Logs |
| Vercel | vercel logs, or the project's Logs |
| Cloud Run | the service's Logs in the console |
doctor
node scripts/doctor.mjs https://vault.company.exampleEach check passes, warns, fails or is skipped. The exit code is 1 if any check fails, so you can run doctor from cron or a scheduled CI pipeline and get notified. --json prints results for programs.
The operator token
Migration state, database latency and the last cleanup are available only with a separate token. Set STATUS_TOKEN in the server's .env, at least 32 characters and different from BOOTSTRAP_TOKEN and CRON_SECRET, deploy again, and run doctor with the same value:
STATUS_TOKEN='<value from .env>' node scripts/doctor.mjs https://vault.company.exampleWithout the token, the metrics check is skipped. The same endpoint, GET /v2/status?format=prometheus with the header Authorization: Bearer <STATUS_TOKEN>, can be scraped by Prometheus or Grafana. It reports build, database latency, migrations, cleanup, sessions, machines on lease and the alert queue, never secret values or email addresses.
What doctor checks
| Check | What a warning or failure means | What to do |
|---|---|---|
| Health | the server doesn't answer or answers with an error. Slower than 1.5 s is a warning | read the platform log |
| Build version | the server runs a different build than expected | deploy the right version |
| TLS | no HTTPS, a rejected certificate, or one that expires soon: under 21 days warns, under 7 days fails | renew it. Caddy and the platforms do it automatically, so check DNS and ports |
| Security headers | HSTS, Content-Security-Policy, frame blocking, X-Content-Type-Options or Referrer-Policy is missing | something in front of the server strips them. Configure it to leave them alone |
| API caching | a proxy may store API responses | configure it not to cache /v2/* |
| API compression | a proxy compresses API responses although the server asks it not to | turn off compression for /v2/*, see Cloudflare Workers |
| Server clock | the clock differs from yours by over 30 s. Over 2 minutes fails, and sign-in starts to fail | turn on time sync on the server |
| Agent release | the agent release settings are wrong or the files are unreachable | check the release settings |
| Metrics | with the token: unapplied migrations, unknown migrations, a slow database, cleanup that stalls or fails, failed alert deliveries, or alerts that couldn't be recorded | unapplied: read the server log. Unknown: expected after a rollback. Cleanup: check the platform's cron |
| Unknown-address key | with the token: this instance can't read the shared key it uses for unknown email addresses | see Rotate the server key |
| Client address | with the token: the server sees the wrong client address or none, so per-address limits and network rules don't work as intended | set TRUST_PROXY to match your proxies |
Audit log integrity
Each organization's audit log is a hash chain: every record contains the hash of the previous one. Changing, deleting or inserting a record in the middle shows up as a break when the hashes are recomputed. Owners and admins check it in Audit log with Verify integrity. See Audit log.
A chain can't show by itself that its latest records were cut off. Once a week, write down the number and hash of the latest record outside the server, in a ticket or a note. If a later check shows a lower last number, or a different hash for the same number, the log was altered. That calls for an investigation.
Alerts
The Alerts section collects events a person should look at. Repeats of the same event fold into one alert with a counter. They include sign-in from a new device, a reused enrollment code, a machine signing in from a new address, a replayed CI token, a series of access denials, an expired lease, a recovery in progress, expiring certificates, keys or file copies not delivered, and network rule events. See Alerts for what each one means and what to do.
Send alerts to a webhook, signed with HMAC, which works with Slack, Teams or PagerDuty through their incoming webhooks, or to email. Email needs RESEND_API_KEY and ALERT_EMAIL_FROM on the server. Each channel has a severity threshold, and a test button sends a test message. Secret values never appear in alerts, because the server doesn't know them.
Cleanup
The server regularly deletes expired sessions, one-time challenges, codes and share links, and trims the audit log to its retention period. On your own server it runs every minute, on Cloudflare every 5 minutes, on Vercel on the --cron schedule, and on Cloud Run only along with requests. When cleanup doesn't run, security doesn't suffer, because deadlines are checked on every request. The database grows and the log isn't trimmed, and some state-based alerts are delayed.