Metrics, thresholds and baselines
A job can report numbers with its ping: rows written, files skipped, queue depth, seconds spent. Rules turn those numbers into an outcome. This catches the failure no exit code can see, where the backup runs, exits 0, and writes an empty file every night for a month.
Sending a number
Add them to the query string, or POST a JSON object. Names are lowercase, 1 to 32 characters, letters, digits and underscore. Up to 10 metrics per ping. A malformed name or a non-numeric value is dropped rather than failing the ping, because a monitoring call must never be the reason a job reports failure.
PING_URL="https://pulsewatcher.up.railway.app/ping/<token>" rows=$(psql -tAc "select count(*) from imported") curl -fsS -m 10 --retry 5 "$PING_URL?rows=$rows&errors=0"
Setting up your first rule
- Create a heartbeat monitor and put its ping URL in the job. Nothing else is needed yet.
- Change the ping to carry the number the job cares about. For a backup that is usually the row count.bash
0 3 * * * rows=$(pg_dump mydb | tee dump.sql | wc -l); \ curl -fsS -m 10 --retry 5 \ "https://pulsewatcher.up.railway.app/ping/<token>?rows=$rows"
- Let it run at least once. Open the monitor page: the metric now appears on the run and in the chart above it.
- In the Metric rules card, add a rule. The name field suggests metrics this monitor has already reported, so you are picking from real names rather than remembering one.
- Set a floor. For a backup, a number well below the smallest legitimate night, not just below zero.
- Confirm it worked: send a ping that breaches it.The monitor goes down within a few seconds and the alert names the metric and the bound:bash
curl -fsS "https://pulsewatcher.up.railway.app/ping/<token>?rows=0"
Send a healthy value to recover.text🔴 Nightly backup is down reason: metric_below_min rows = 0 (min 1000) Last ping: 03:00:12 https://pulsewatcher.vercel.app/app/monitors/<id>
Fixed bounds
min and max are the whole feature for most jobs. Set either or both. A rule with neither and no baseline does nothing.
The rule cap is per owner: 2 on Free, 5 on Pro and Business.
When a fixed bound is the wrong tool
A floor has to be a number somebody chose, and it goes stale. Pick it low enough to never false-alarm and it stops catching anything; pick it tight and it fires every time the business has a quiet week.
A baseline rule says "far below what this job normally reports" instead. PulseWatch takes the median of recent runs and breaches when a value falls further below it than baseline_deviation_pct allows. Median, not mean, so one catastrophic night does not drag the expectation down to meet itself.
🔴 Nightly backup is down reason: metric_below_baseline rows = 11,204, 78% below the median for a Tuesday (51,880) https://pulsewatcher.vercel.app/app/monitors/<id>
Buckets: comparing like with like
Most jobs have a shape. A job that processes a tenth as much on Sunday is not broken on Sunday, and a baseline that pools every run together will either alert every weekend or learn an expectation so wide it catches nothing. A bucket says which past runs count as comparable.
| Bucket | Min samples | Compares against | Choose it when |
|---|---|---|---|
| none | 7 | All recent runs together | Right for a job whose numbers do not care what day it is. |
| day_of_week | 4 | The same weekday | Right when Monday genuinely differs from Sunday. Roughly 8 weeks of that weekday. |
| weekday_weekend | 5 | Weekdays against weekdays, weekends against weekends | The cheap version of day_of_week. Right when the only real split is working days. |
| hour_of_day | 5 | The same hour | Right for a job that runs many times a day with a daily shape. Expensive in samples on a once-daily job. |
Each bucket looks back over 8 comparable periods. For day_of_week that is roughly 8 weeks of that one weekday. Below the minimum sample count the rule reports as learning and does not fire, so a new rule cannot alert on two data points.
none keeps a higher minimum of 7 on purpose. Rules that already ran with it should not start alerting on thinner data than before.
Overrides for the days that are genuinely different
A bucket handles a recurring shape. An override handles a known exception: month end, Sundays, a maintenance day. Up to 5 per rule.
# Rule: rows min 1000 # Override: Sundays, min 50 # # Sunday's run is a delta, not a full load. Without the override you either # alert every Sunday or you lower the floor to 50 all week and stop catching # the empty dump on a Tuesday.
Rules on a fleet
A metric rule can belong to a fleet instead of a monitor. It is then declared once and evaluated against each instance separately, so a floor of rows >= 1000 is checked against every machine's own number rather than against a fleet total. The editor is the same card, on the fleet page.
The fleet page also rolls each metric up across instances, showing the min, median, p95 and max, and naming the instances furthest from the median. Outliers are measured against the median so a few broken machines cannot move the comparison to meet themselves.
Every field
| Field | Type | Default | Range | Effect | Plan |
|---|---|---|---|---|---|
| metric | text | (required) | 1 to 32 chars, a-z 0-9 _ | Which number the rule watches. | all |
| min | number | null | any | Floor. Breach when the value is below it. | all |
| max | number | null | any | Ceiling. Breach when the value is above it. | all |
| baseline_deviation_pct | number | null | any | Breach when the value is this far below the learned median. | Pro+ |
| baseline_bucket | enum | none | none, day_of_week, weekday_weekend, hour_of_day | Which past runs the median is taken from. | Pro+ |
| baseline_min_samples | int | per bucket | 1 or more | How many observations before the baseline may fire. | Pro+ |
| baseline_timezone | text | resolved on save | IANA name | Which clock day and hour boundaries are read in. | Pro+ |
| severity | enum | down | warn, down | Whether a breach takes the monitor down or only warns. | all |
| enabled | boolean | true | A disabled rule is stored and skipped. | all |
Variations
A backup that must not be empty
rows, min 1000, severity down. The simplest useful rule there is.
A queue that must not build up
queue_depth, max 5000, severity down. Report the depth at the end of every run.
A job with a weekly shape
rows, baseline deviation 60%, bucket day_of_week. No fixed number to maintain, and Sunday is compared with other Sundays.
A count that is interesting but not urgent
skipped, max 0, severity warn. Arrives in the daily digest instead of paging.
Two rules on one metric
A wide fixed floor as a hard backstop, plus a baseline rule for the subtle drop. The floor catches zero immediately; the baseline catches the slow decay from 50,000 to 11,000.
What can go wrong
The rule never fires
Usually the name. It must match exactly, and names are lowercased on arrival. Open a recent ping and read the metrics it actually carried.
The monitor went down with reason metric_missing
An enabled rule found no value for its metric on a ping. That is deliberate: a job that stops reporting its row count has usually stopped doing the work that produces one. If the metric is genuinely optional, disable the rule rather than ignoring the alert.
The baseline has been learning for weeks
The bucket is too fine for how often the job runs. hour_of_day on a once-daily job collects one sample per day for that hour. Widen the bucket.
Numbers are missing from some pings
Report the metric on every run, including runs where it is zero. Gaps slow learning and make metric_missing ambiguous.
A metric was silently dropped
Names outside a-z 0-9 _, longer than 32 characters, non-numeric values, and anything past the tenth metric on a ping are discarded. The ping still succeeds.
Related
- Exit codes for how the run ended. A metric says what it produced. Most jobs want both.
- Duration when the number you care about is how long it took. That is measured for you, with no metric to report.
- Pinging API for the full ping surface, including log excerpts.
- Troubleshooting when a rule has never fired and you expected it to.