States and alert reasons

The complete list, in one place. Every table below is generated from the same types the server dispatches on, so a state or reason cannot ship without appearing here.

Monitor states

Read the uptime column carefully. There are three answers, not two. A state can count as available, count as downtime, or sit outside the calculation entirely. That third category is the one people are surprised by: blocked is not scored either way, because charging a job for its dependency's outage would be wrong and quietly counting it as available would hide the outage.

State	Meaning	Uptime	Alerts
new	Created and has never reported. A heartbeat monitor stays here until its first ping.	not measured	None. A monitor that has never run has nothing to be late for.
up	Reporting on schedule, or responding as expected.	available	A recovery alert when arriving here from down.
late	Past its expected time but still inside the grace window.	available	None. Grace exists so a job that runs two minutes late does not page.
warn	The run completed with problems: an exit code mapped to warn, a duration regression, changed output, a stale upstream.	available	Never pages and never opens an incident. Delivery follows `warn_delivery`.
blocked	An upstream this monitor depends on is down, so its own failure has no cause of its own.	not measured	None. The upstream alert names the blocked dependents instead.
down	Missed its window past grace, failed explicitly, or failed a check.	downtime	Pages, opens an incident, repeats on `repeat_interval_seconds` if set.
paused	Deliberately not being watched. Pings are still accepted and recorded.	not measured	None.

Alert reasons

Every value details.reason can take on a webhook, and the line each one produces in an email or a chat message. Severity here is what the reason does to the monitor, not how alarming it sounds. Two of them read like warnings and are not: persistent_warnings is what a warning that never stops becomes, and run_pileup goes down whatever your overlap policy says.

Takes the monitor down (15)

Reason	What happened	What to do
no_ping	A heartbeat monitor did not ping inside its period plus grace.	Check the job actually ran. If it ran and the ping failed, the curl is missing --retry or the host could not reach the internet.
fail_ping	The job explicitly reported failure by pinging /fail.	Read the log excerpt on the run. The job knew it failed and said so.
http_error	An uptime check got a status outside the expected range.	The status code is on the check result. Widen `expected_status_min`/`max` if the code is legitimate.
timeout	An uptime check exceeded `timeout_ms`.	Raise the timeout if the endpoint is legitimately slow, or fix the endpoint. Consider `degraded_threshold_ms` to catch the slowdown before it becomes a timeout.
connection_error	The host refused the connection, DNS failed, or TLS did not negotiate.	Usually DNS or a firewall. The error text on the check result names which.
keyword_mismatch	The response was healthy by status code but `expect_text` was absent or `forbid_text` was present.	A 200 that renders an error page is exactly what this catches. Read the body.
ssl_expiring	The certificate expires inside the warning window.	Renew it. Automated renewal that silently stopped is the common cause.
metric_missing	A metric rule is enabled and the ping carried no value for that metric.	The job stopped reporting the number. That is usually a code path exiting before it measured anything.
metric_below_min	A reported metric fell below the rule floor.	The classic case is a backup reporting rows=0. The job succeeded and produced nothing.
metric_above_max	A reported metric rose above the rule ceiling.	Queue depth and error counts are the usual sources.
metric_below_baseline	A metric fell further below its learned median than `baseline_deviation_pct` allows.	Nothing crossed a fixed threshold. The number is simply far below what this job normally reports at this time.
persistent_warnings	Consecutive warns crossed `warn_escalate_after`. A warning that never stops is a failure nobody saw.	Treat it as the underlying warn reason, which has been true for several runs now.
run_pileup	Three or more runs of the same job were in flight at once. This goes down whatever `overlap_policy` says.	Runs are stacking faster than they finish. Stop the schedule before the host runs out of memory.
expected_shortfall	The run declared how much work it had and finished with less of it done. `details.expected` holds the declared value, `details.value` the reported one.	The run itself is the witness here, so this is rarely a false alarm. Read the excerpt for the batch that stopped.
expected_mismatch	Declared and reported differ by more than the tolerance, in either direction.	Same as a shortfall, but check for double-processing as well as for work skipped.

Warns and never pages (11)

Reason	What happened	What to do
exit_code	The run exited with a code an exit-code rule maps to warn.	The label on the rule says what the code means. Change the rule if the code should page instead.
stale_upstream	The run succeeded, but an upstream it depends on has not succeeded inside the freshness window.	The job read old data. Fix the upstream, then re-run this one.
overlapping_runs	A new run started while the previous one was still open, and `overlap_policy` is warn or alert.	The job is taking longer than its interval. Lengthen the interval or make the job exit when a copy is already running.
duration_regression	The run took longer than its baseline by more than `duration_regression_pct`.	Compare against the duration chart on the monitor page. A step change points at a deploy; a slope points at data growth.
output_changed	The log excerpt differs from the last successful run, ignoring timestamps, counts and ids.	A job whose output changed shape without failing. Read the diff on the run.
completion_budget	The job ran fewer times over the budget window than `completion_budget_pct` requires.	Nothing is down. Something is skipping runs. Check the scheduler, not the job.
expected_surplus	The run reported more than it declared. Usually the counting step undercounted, not the work.	Compare the counting query against the processing query. They are looking at different sets.
expected_missing_actual	The run declared work, finished, and never reported what it did.	The declaration reached us and the reading did not. Check that the terminal ping sends the paired metric, without the `expected_` prefix.
expected_missing	A run that always declares stopped declaring. Sent only when the metric is marked required.	The counting step probably failed or was removed. The run may still be doing the work; nobody can tell any more.
expected_zero_suspicious	The run declared zero on a fleet configured to treat zero as suspicious.	A counting query that errors and returns 0 makes a run with nothing to do look identical to one that failed. Check the count, not the job.
expected_value_anomalous	The declaration itself is unlike what this instance normally declares. Compared against the instance own history, never against other instances.	A quiet day is a legitimate reason. If it is not quiet, the count step is reading the wrong thing.

Recovery (1)

Reason	What happened	What to do
recovered	The monitor returned to up. Sent on `monitor.up`.	Nothing to do.

Webhook events

The value of the event field and the X-PulseWatch-Event header. Treat the set as open: new events are added additively, and the payload version stays at 1.

Event	Sent when
monitor.down	The monitor entered down. Carries the reason and, on a heartbeat, when the ping was expected.
monitor.up	The monitor recovered. `details.reason` is `recovered`.
monitor.ssl_expiring	The certificate on an uptime monitor is inside the expiry warning window.
monitor.error_new	A stack-trace fingerprint never seen on this monitor appeared. Can fire while already down.
monitor.error_persisting	The same fingerprint failed on N consecutive runs, where N is `escalate_after_consecutive`.
monitor.warn	A run completed with problems. The monitor is still available and no incident opens.
monitor.persistent_warnings	Consecutive warns crossed `warn_escalate_after`. This one does open an incident.
monitor.warn_digest	The batched roll-up of warnings for one monitor, sent on the digest schedule rather than per run.
fleet.breached	A fleet crossed its alert threshold. One delivery for the whole fleet. Branch on `details.reason` to tell missing instances from a rule breach or a blown completion budget.
fleet.recovered	A fleet dropped back below its threshold. Names the instances that returned.
monitor.completion_budget	A monitor ran fewer times over its budget window than the budget allows. Never sent alongside a down alert for the same period.

A fleet delivery has no monitor object. It carries a fleet object and a details.reason of missing_instances, rule_breach or completion_budget. Branch on that, not on details.missing, which is 0 for the last two: nothing is missing in either case.

Fleet instance states

A fleet counts missing and nothing else. The other four states exist so that a machine registered this morning, or one decommissioned three weeks ago, cannot breach a fleet.

State	Meaning
learning	Registered but has not yet reported on schedule. Never counts as missing, so a machine added this morning cannot breach a fleet this afternoon.
reporting	Reported inside its expected window.
quiet	Silent, and outside its active window. A laptop switched off at 17:00 is quiet, not missing. Out of the denominator and never alerts, because being off during hours you told us it would be off is not evidence of anything.
late	Past the window, inside grace.
missing	Past window and grace. This is the only state the alert modes count.
dormant	Silent for longer than `dormant_after_days`. Still listed, but out of the denominator: a fleet of 200 with 40 abandoned machines could otherwise never read better than 80%, and a number that cannot reach full is a number people stop reading.
retired	Silent for longer than `retire_after_days`. Out of the denominator entirely, because a decommissioned machine should not drag a fleet down forever.

Scheduled run slots

A completion budget divides the window into the times a job was scheduled to run and resolves each one. These are the five answers. The one worth knowing is late: a budget asks whether the work happened, not whether it was punctual, so a late run counts toward the rate. Punctuality is what the grace window and duration regression are for.

Status	Meaning
pending	In the future, or inside its grace window and still open. Not yet judged.
satisfied	A ping arrived within grace.
late	A ping arrived after grace but before the next slot. Counts as run, not as on time. A completion budget asks whether the work happened, so this counts toward it.
missed	Grace elapsed with no ping at all.
excluded	Maintenance, paused or blocked covered this slot. Counted in neither the numerator nor the denominator.

Reading a state transition

Alerts name both states. This is the sequence for a nightly job that skipped files for three runs, then stopped running:

text

03:00  ping /exit/100   up    -> warn     exit_code        (digest, no page)
04:00  ping /exit/100   warn  -> warn     exit_code        (digest, no page)
05:00  ping /exit/100   warn  -> down     persistent_warnings   PAGES, opens an incident
06:05  no ping          down  -> down     no_ping          repeat interval, if set
09:00  ping             down  -> up       recovered        resolves the incident

Three warns did not page. The fourth event did, because warn_escalate_after defaults to 3 and a warning that never stops is a failure nobody looked at.

How alerting works for when an alert fires rather than what it says.
Troubleshooting by symptom when you have the behaviour and need the cause.
Webhooks for the full payload these values appear in.
Glossary for the terms used across these pages.