Incidents, runbooks, acknowledge and maintenance

Incidents

When a monitor goes down, an incident opens. It records when the outage started, what the reason was, every alert sent about it, and when it resolved. When the monitor recovers, the incident closes.

• A monitor that recovers and fails again within five minutes rejoins the same incident rather than opening a second. A flapping job produces one incident, not forty.
• Consecutive warnings crossing warn_escalate_after open an incident too, with reason persistent_warnings. Ordinary warns never do.
• An incident can be published to your status page. Publishing is a separate decision from opening, so an internal outage stays internal unless you say otherwise.

Incidents need Pro or above. On Free the state and history still record; there is simply no incident object grouping them.

Runbooks: the instructions arrive with the alert

A runbook is markdown attached to a monitor and delivered inside every alert about it. Nobody has to find the wiki, and nobody has to be the person who knows.

1. Open the monitor and find the Runbook field. It starts with a template of headings; an untouched template is stored as empty so alerts do not carry five bare headings.
2. Write the steps. Numbered commands, not prose. The reader is half awake.
   markdown

   ## Nightly backup failed
   
   1. Check disk on db-01: `df -h /var/backups`
   2. If full, prune: `find /var/backups -mtime +14 -delete`
   3. Re-run: `sudo -u postgres /opt/backup.sh`
   
   Escalate to the data team if it fails twice.
   Last ping was {{last_ping_at}} and the reason is {{reason}}.
3. Save, then use Send test alert.
4. Confirm it worked: the test alert contains the rendered runbook with the variables filled in.
   text

   🔴 Nightly backup is down
   reason: no_ping
   
   Nightly backup failed
   1. Check disk on db-01: df -h /var/backups
   2. If full, prune: find /var/backups -mtime +14 -delete
   3. Re-run: sudo -u postgres /opt/backup.sh
   Escalate to the data team if it fails twice.
   Last ping was 03:00 yesterday and the reason is no_ping.
   
   https://pulsewatcher.vercel.app/app/monitors/<id>

Runbooks are trimmed to fit the channel: 800 characters in a chat message or webhook, 4,000 in an email. Write the first three steps as the ones that matter, because those are the ones that always arrive.

These variables interpolate. Anything else is left alone:

{{monitor_name}}{{state}}{{reason}}{{last_ping_at}}{{exit_code}}{{metric_values}}{{error_message}}{{dashboard_url}}

Set the account default first. One paragraph naming who to contact and where the dashboard is covers every monitor you have not written a specific runbook for, which is most of them.

Acknowledge and snooze from the alert

Every down alert carries two links. They work without logging in, which is the point: someone on a phone at 3am can stop the repeats in one tap.

• Acknowledge means somebody has seen it. Repeats stop. The monitor is still down and the incident is still open, because acknowledging is not fixing.
• Snooze silences the monitor for a while. State and history keep recording.

An acknowledgement is narrow on purpose. It never holds a recovery, and it never holds an alert whose reason or error fingerprint differs from the one that was acknowledged. If a second, different thing breaks on the same monitor, you hear about it.

Treat these links as secrets. Anyone holding one can act without a session, which is why they expire in a day and work once.

Maintenance windows

A weekly recurring period during which notifications are suppressed. Use it for the Sunday 02:00 patch window that takes half the estate down on purpose.

1. Open Settings and find Maintenance windows.
2. Add a window: the days, the start time, the length, and the timezone.
3. Scope it. A window can cover the whole account, or only the monitors and fleets you select. Scoping is usually right: patching the database servers should not silence the payment API.
4. Confirm it worked: during the window, a monitor that goes down still shows as down on the dashboard, still records the state change, and sends nothing. The suppressed send is written to the alert history with the reason, so a silent window is auditable rather than invisible.

Suppression covers notification only. Uptime, incidents and completion budgets all treat a maintenance period as excluded rather than as healthy, so a window cannot be used to make a bad month look good. Maintenance windows need Pro or above.

Change correlation

Record your deploys and config changes, and PulseWatch shows the ones that happened shortly before an incident. It reports timing. It does not claim cause, and the line saying so is not optional:

Deployed api v4.18.2
2026-08-11 14:52, 8 minutes before this incident
https://github.com/acme/api/releases/tag/v4.18.2
(timing only, not necessarily the cause)

The default window is 2 hours, adjustable up to 24, and at most 5 candidates are shown. A list of everything that happened in the last day would correlate with everything and mean nothing.

The other direction is available too: given a change, which incidents followed it. That is the more useful question after a release.

Variations

What can go wrong

Related

• How alerting works for what happens before an incident opens.
• Reporting errors so the incident carries the stack trace rather than only a reason.
• Status pages to publish an incident to the people affected by it.
• States and reasons for which reasons open an incident and which never do.