How alerting works

A worked timeline

A nightly job with a 5 minute delay before alerting, hourly repeats, and new-error alerts on. It fails at 03:00 and recovers at 06:00.

03:00  the job fails and sends /fail
       monitor goes DOWN immediately, incident opens, clock starts
       no alert yet: you asked to wait 5 minutes

03:05  delay elapsed, still down
       DOWN ALERT sent to every channel on the monitor
       (this is the alert the incident is measured against)

04:05  still down, one hour since the last alert
       REPEAT ALERT sent

05:00  a run fails with a DIFFERENT stack trace than before
       NEW ERROR ALERT sent, even though the monitor is already down
       this is deliberate: a second failure mode during an outage is news

05:05  still down
       REPEAT ALERT sent

06:00  a successful ping arrives
       monitor goes UP, RECOVERY ALERT sent
       incident resolves with duration 3h 0m, measured 03:00 -> 06:00

What triggers what

"State unchanged" means the alert is about something that happened inside an existing state. A new-error alert does not move a monitor to down; the thing that made it fail already did that.

Delay before alerting

The monitor goes down the moment it fails. The alert waits for the delay you set. If the monitor recovers inside that window, nothing is sent at all: no down alert, and therefore no recovery alert either. That is the point of the setting, and it is why a short blip is completely silent rather than a matched pair of messages.

Repeat alerts while still down

Off by default. When set, PulseWatch re-sends the down alert on that interval for as long as the monitor stays down, measured from the last alert rather than from the failure. The minimum accepted interval is 15 minutes; anything shorter is treated as off, so a misconfiguration cannot turn one outage into a flood.

New error types during an outage

If your failing pings carry stderr, PulseWatch groups stack traces by fingerprint. The first time a fingerprint appears it raises its own alert, even while the monitor is already down. A job that was failing on a timeout and starts failing on a null dereference is new information, and waiting for recovery to surface it would be worse. These alerts are capped at one per monitor per minute, and several new fingerprints arriving together are batched into a single message.

Escalation

Set a consecutive-run threshold and the same error fingerprint failing that many runs in a row sends one escalation. Once per crossing, not once per run, so a job stuck failing all night pages you once rather than forty times.

Muting

Muting is per error group, from the Errors tab. A muted group raises no new-error and no escalation alerts. It does not touch the monitor's own up and down alerting, which keeps working exactly as before. Use it for a known-noisy error you have decided to live with. Optional auto-unmute after a set number of days.

Runbooks

A runbook is what you would tell someone if they called you at 03:12: what the job does, where it runs, how to check whether it really failed, how to safely rerun it, who to escalate to. PulseWatch appends it to the alert itself, after the facts, so the alert is the answer rather than the start of a search through a wiki.

Write one per monitor under Edit → Runbook, or set an account-wide default in Settings that every monitor without its own falls back to. Markdown: headings, lists, inline code and http(s) links. You can interpolate the alert's own facts with {{monitor_name}}, {{reason}}, {{exit_code}}, {{error_message}}, {{metric_values}}, {{last_ping_at}}, {{state}} and {{dashboard_url}}. A variable the alert cannot fill renders as empty rather than leaving a placeholder in the message.

Chat channels get the first 800 characters and a link to the rest; email gets 4,000; webhooks receive details.runbook_url and details.runbook_excerpt. Runbooks are attached to down, escalated-warning, error and completion-budget alerts, not to recoveries, since nobody needs rerun instructions for a job that just came back.

Acknowledge and snooze

Acknowledging stops repeat alerts and escalations for an outage. It never stops the recovery: knowing it came back is the one message worth interrupting for, and it never stops a different failure on the same monitor. An acknowledgement covers the problem you read, not every problem that follows it, so a new error fingerprint or a different reason still alerts.

Snoozing is broader: every alert for that monitor is held until it expires, including the recovery. Durations are 1 hour, 3 hours, or until 08:00 in the monitor's own timezone, so a 03:12 alert snoozed “until morning” waits about five hours, not twenty-nine. When it expires and the monitor is still down you get one message saying so, then normal alerting.

Both are available from the alert itself: inline buttons in Telegram, links in email, Slack and Discord, and ack_url / snooze_url in the webhook payload. Those links work without signing in, which is the point, so they are treated as secrets: 32 random bytes, stored only as a hash, scoped to one monitor and one action, single use, and dead after 24 hours or when the incident resolves. Everything that happens appears in the incident timeline with the channel it came from.

Maintenance windows

During a window covering that monitor, alerts are held at delivery time. State still changes, history is still recorded, and incidents still open and close. Only the notification is suppressed, so the dashboard remains truthful about what happened while you were deploying. Windows can apply to every monitor or to a chosen set.

Recovery and incidents

A recovery alert is sent once, when a monitor returns to up, and only if a down alert was actually delivered for that outage.

An incident opens when a monitor goes down and resolves when that monitor returns to up. Its duration is measured from the first failure to the recovery, not from when the alert was sent, so a delay before alerting does not shorten the recorded outage. If a monitor recovers and fails again within five minutes, the existing incident reopens rather than a second one being created, so a flapping job produces one incident instead of a dozen.

More on grouping and stack traces in reporting errors.