Exit codes and the warn state

Reporting the code

Three ways, all equivalent. The path form is the one to reach for.

Valid codes are 0 to 255. A malformed value is dropped rather than failing the ping, so /exit/ with nothing after it is treated as a plain ping. If your rule seems to be ignored, check the code actually arrived.

Setting it up

1. Create a heartbeat monitor and copy its ping URL. Nothing here needs a paid plan; the defaults below apply to every account.
2. Change the job to report its code. The important detail is the separator: ; runs the curl whatever happened, && runs it only on success, which means a failing job sends nothing and shows up as silence.
   bash

   # The whole change: report the code instead of only reporting success.
   # Note the ; not &&. With &&, a failing job sends nothing at all.
   0 3 * * * /opt/backup.sh; curl -fsS -m 10 --retry 5 \
     "https://pulsewatcher.up.railway.app/ping/<token>/exit/$?"
3. Let it run once. On the monitor page the run now reads exit 0 (ok) rather than just success.
4. Open the Exit codes card on the monitor page. This is a card of its own, below the metric rules, not a field inside the monitor form.
5. Add a rule: from and to (both 100 for a single code), an outcome, and a label. The label is what the alert says, so write the meaning rather than repeating the number.
6. Confirm it worked: send a test ping with that code and watch the run. It should show your label and the outcome you chose.
   bash

   curl -fsS "https://pulsewatcher.up.railway.app/ping/<token>/exit/100"
   Rules are cached for up to a minute on the ping path, so an edit can take one more ping to take effect.

Worked example: a backup that skips locked files

The job exits 100 when it completed but had to skip something.

#!/usr/bin/env bash
set -uo pipefail

skipped=0
for f in /data/*.sql; do
  if ! restore "$f"; then
    skipped=$((skipped + 1))
  fi
done

if [ "$skipped" -gt 0 ]; then
  echo "skipped $skipped locked files" >&2
  exit 100          # completed, with problems
fi
exit 0

With the default rules and no configuration at all, 100 is already warn with the label completed with issues. Adding a rule of your own changes the label to something specific:

from 100  to 100  outcome warn  label "skipped locked files"

The monitor stays up, no incident opens, nobody is paged, and the next morning the digest says:

Subject: Nightly restore: 3 warnings in the last 24 hours

3 in the last 24 hours. Nothing here took the monitor down.

03:00  exit 100 (skipped locked files): warn
04:00  exit 100 (skipped locked files): warn
05:00  exit 100 (skipped locked files): warn

The defaults you already have

Every account starts with a set of rules covering the conventions. You only add a rule where your job means something different.

The 100 to 119 range is the recommended convention. Changing exit 1 to exit 100 in a script is the cheapest way to start using the warn state, and it needs no configuration at all.

What each outcome does

Which rule wins

Rules can live on a monitor, on a fleet, or on the account as a default. When several match a code, the most specific one wins:

1. A rule scoped to this monitor beats an account default.
2. Among equally scoped rules, the narrower range wins.

That is why the broad 129 to 165 killed by signal rule and the specific 137 SIGKILL, likely out of memory rule coexist without either knowing about the other. Exit 137 gets the OOM label; exit 140 gets the general one.

A non-zero code on a plain success ping is treated as authoritative, and the code wins. A wrapper that pings unconditionally would otherwise file every failed run as a success.

How warnings are delivered

A warn never pages. What varies is when you hear about it.

A warning that never stops becomes an alert

warn_escalate_after defaults to 3 and takes a value from 1 to 100. After that many consecutive warns the monitor goes down with reason persistent_warnings, and that one does page and does open an incident.

03:00  exit 100   up   -> warn   digest, no page
04:00  exit 100   warn -> warn   digest, no page
05:00  exit 100   warn -> down   PAGES, opens an incident
                                 reason: persistent_warnings

Without escalation, warn would be a way to make a real failure permanently invisible. A backup that has skipped files every night for three weeks is not having a good month.

Every field

Exit-code rules are available on every plan, including Free. warn_delivery and warn_escalate_after are fields on the monitor, documented with the rest of the monitor configuration.

Variations

What can go wrong

Related

• Metric rules when the job exits 0 and the number it produced is wrong. An exit code says how the run ended; a metric says what it did.
• Reporting errors to send the stack trace along with the failure.
• States and reasons for exactly what warn does to uptime and incidents.
• Fleets to declare one set of exit-code rules for hundreds of machines.