PulseWatchPulsewatch

Pinging API

Every heartbeat monitor gets a unique ping URL. Hit it at the end of each successful run, and optionally attach metrics and log output. That is how you catch the failure uptime tools miss: the cron that exits 0 having done nothing.

Fastest start: let the monitor create itself

You do not have to create the monitor first. Take the auto-provision key from Settings, pick any name for the job, and ping it. The monitor appears on the first call and every later ping with that name goes to it.

bash
# nothing to set up first: this creates "nightly-backup" and pings it
curl -fsS -m 10 --retry 5 "https://pulsewatcher.up.railway.app/ping/<your-key>/nightly-backup"

# /fail and /start work the same way
curl -fsS "https://pulsewatcher.up.railway.app/ping/<your-key>/nightly-backup/fail"

# so do metrics, from the very first call
curl -fsS "https://pulsewatcher.up.railway.app/ping/<your-key>/nightly-backup?rows=1400"
  • Names use lowercase letters, numbers, dash and underscore, up to 64 characters. fail and start are reserved, because /ping/<key>/fail would be ambiguous.
  • A new monitor starts in learning. It records pings but never alerts. After three pings PulseWatch takes the median gap between them, snaps it to a sensible period, adds a grace window of half that, and switches to normal operation. The inferred schedule is shown on the monitor and you can edit it at any time.
  • The key respects your plan's monitor limit, and is rate limited to 20 new monitors an hour.

What the key can and cannot do

The auto-provision key is per account and is weaker than an API key on purpose. Anyone holding it can create monitors in your account and send false pings, which is noise. It cannot read your monitors, pings, metrics or anything else, and it cannot delete or change existing monitors. Rotate it from Settings if it leaks; the old key stops working immediately. Turn the whole feature off with one toggle.

Base URL & endpoints

Base URL: https://pulsewatcher.up.railway.app/ping/<token>. The token is a UUID unique to one monitor: treat it as a secret. Every endpoint accepts GET, POST or HEAD; use POST when you want to send a JSON body or a log excerpt.

GET|POST|HEAD /ping/<token>success: resets the clock; may carry metrics + a log excerpt
GET|POST|HEAD /ping/<token>/failexplicit failure: the monitor goes down immediately
GET|POST|HEAD /ping/<token>/startrun began: enables run durations, max-duration timeouts and overlap detection

A ping returns 200 OK on success and 404 for an unknown token. Ping endpoints are rate-limited per token and per source IP; a well-behaved --retry client never hits the limit.

Success & failure semantics

  • A success ping resets the monitor's clock. Miss the next one past the grace window and the monitor goes down.
  • A /fail ping takes the monitor down right away: wire it into your script's error path.
  • A /start ping marks the run as in-flight so PulseWatch can measure duration, alert if a run overruns its max duration, and notice when the next run begins before this one finished. It never resets the heartbeat clock: starting is not finishing.
  • Recovery is automatic: the next compliant success ping clears the incident and sends an up alert.

Exit codes, and the middle ground between up and down

Real jobs have a middle: a backup that completed but skipped two files, an import that finished with three malformed rows. Paging on those trains people to ignore alerts. Ignoring them entirely means the slow degradation is invisible until it becomes an outage.

Report your exit code and PulseWatch decides which it was. Three forms: use whichever is nearest:

bash
curl -fsS "https://pulsewatcher.up.railway.app/ping/<token>?exit=$?"          # on any ping
curl -fsS "https://pulsewatcher.up.railway.app/ping/<token>/exit/$?"        # explicit path
curl -fsS -X POST -H 'content-type: application/json' \
     -d '{"exit_code": 100}' "https://pulsewatcher.up.railway.app/ping/<token>"

Exit 100–119 means “completed with issues”

That range produces a warning rather than an alert. A warning never counts as downtime, never opens an incident and never pages you: it clears on the next clean run. If your script currently does exit 1 on a partial failure, exit 100 is a one-character change that stops it waking anybody up.

bash
if [ "$skipped" -gt 0 ]; then
  echo "skipped $skipped files"
  exit 100          # done, but degraded
fi

What the other codes mean

  • 0: success.
  • 1–63: down, the conventional “it failed” range.
  • 64–78: down, labelled with the standard sysexits meanings, so the alert says “configuration error” rather than “exit 78”.
  • 124: down, timeout (GNU timeout).
  • 137: down, SIGKILL, likely out of memory. Worth its own line because it is common and people lose an hour to it.
  • 143: down, SIGTERM.

A non-zero code on the plain success URL is a contradiction, and the code wins: the usual cause is a wrapper that pings unconditionally, which would otherwise report every failed run as a success. Ranges are configurable per monitor or per account.

Warnings arrive as one summary a day by default, not a message per warning. Change that per monitor under Alerting, or turn them off entirely.

Overlapping runs: the failure that looks healthy

A job scheduled every five minutes that starts taking seven does not look broken. The pings keep arriving on time and the monitor stays green: right up until the host runs out of memory, or two copies of the job write to the same file.

PulseWatch can catch this, but only if your job sends /start. Without it there is no record that a run is open, so there is nothing to overlap. This is the main reason to send it:

bash
curl -fsS -m 10 "https://pulsewatcher.up.railway.app/ping/<token>/start"   # at the top of your script
./your-job.sh
curl -fsS -m 10 "https://pulsewatcher.up.railway.app/ping/<token>?exit=$?"  # at the end

What happens when runs collide

When a /start arrives while an earlier run is still open, the monitor's overlap policy decides. Set it per monitor under Overlapping runs.

warn (default)monitor goes to warn: no incident, no page, clears on the next clean run
alertmonitor goes down and alerts, like any other failure
ignorestate untouched, but the overlap is still recorded

Three at once always alerts

Three or more concurrent runs take the monitor down regardless of the policy. Two runs is a job that ran long once. Three means the first overlap never cleared and the next is already stacking on top. That is a pile-up in progress, and warning quietly about it is the wrong kind of quiet.

A /start never resets the heartbeat clock, even when an overlap takes the monitor down: otherwise it would hide the run that is about to not finish. Runs abandoned by a killed process are closed automatically, so one crash does not report a pile-up forever.

Reliability: retry from the client

A ping lost to a transient network blip is a false alarm waiting to happen. Let curl retry, and cap how long it can hang around:

bash
curl -fsS -m 10 --retry 5 "https://pulsewatcher.up.railway.app/ping/<token>"

-f fail on HTTP errors · -s silent · -S still show errors · -m 10 10s cap · --retry 5 retry transient failures.

Metrics: numbers on your pings

Add numeric query parameters to a success ping and PulseWatch stores them as telemetry. Row counts, error counts, bytes written, queue depth: anything you can measure. Run duration arrives free as duration_ms whenever the job sent a /start ping first, and can be ruled on like any other metric. Then create rules (Monitor → Metrics): a floor (rows ≥ 1000), a ceiling (errors ≤ 5), or a baseline (Pro: value dropped more than N% below the median of the last 14 runs. That is a statistical floor, not anomaly detection). A configured metric that goes missing counts as a violation too. The ping still counts as received, because the job did run, but the monitor goes down on the violation until a compliant ping clears it.

bash
# the canonical case: the job "succeeds" but processed nothing
psql -c "COPY (…) TO STDOUT" | import-rows
curl -fsS --retry 3 "https://pulsewatcher.up.railway.app/ping/<token>?rows=1400"

JSON body works too (query params win on conflict)

bash
curl -fsS -X POST "https://pulsewatcher.up.railway.app/ping/<token>" \
  -H "Content-Type: application/json" \
  -d '{"rows": 1400, "errors": 2, "duration_s": 41.5}'
  • Names match [a-z0-9_], up to 32 chars (lowercased on intake).
  • Finite numbers only; up to 10 metrics per ping. Invalid metrics are silently dropped: a bad metric never fails the ping.
  • Rule counts per monitor: 2 on Free, 5 on Pro/Business. Baselines are Pro+.

Schedule-aware baselines

Here is the case that breaks a naive baseline. A job imports about 1400 rows every weekday and about 5 on Sunday, because almost nothing happens at the weekend. Lump all runs together and the median sits near 1400, so every Sunday looks like a catastrophic drop and fires an alert. After a few weekends people stop reading the alerts, which is worse than not having them.

Set Compare against on the rule to group history before taking the median:

All recent runsThe original behaviour. One median over everything.
Same weekdaySunday is compared with past Sundays. Fixes the case above.
Weekday vs weekendTwo buckets, Mon–Fri and Sat/Sun.
Same hour of dayFor jobs that run several times a day with a daily shape.

Buckets use the monitor's timezone when it has one (cron monitors do), otherwise the timezone recorded on the rule. The lookback is the last 8 occurrences within the bucket, so "same weekday" looks back about 8 weeks rather than 8 runs.

It will not alert until it has seen enough

A bucket stays quiet until it has enough observations: 7 for all-recent-runs, 4 for same-weekday, and 5 for the other two. Below that the rule is skipped and the monitor page shows "learning: N of M samples" with the median so far, so you can watch it form. A bucket whose median is 0 is also skipped, because a percentage below zero means nothing.

Exceptions you declare yourself

When you already know the pattern, say so instead of making PulseWatch infer it. An exception scoped to a day, an hour range, or both replaces the rule's thresholds for pings inside it:

text
Rule:      rows, baseline −30%, compare against same weekday
Exception: On Sun, floor is 1

Sunday   -> the exception applies, so 5 rows passes
Tuesday  -> the rule applies, so 245 against a 1400 median alerts

The most specific match wins: day plus hour beats day, which beats hour, which beats a blanket exception. Up to 5 per rule. Baselines, buckets and exceptions are Pro features; floors and ceilings work on every plan.

Alerts say which baseline they used, for example "245 is 82% below the Tuesday median of 1380 (8 samples)", so you can judge it without opening the dashboard.

Log excerpts

POST a non-JSON body and PulseWatch keeps the first part as a run-log excerpt (Free 4 KB, Pro 16 KB, Business 64 KB). Anything past the cap is read and discarded, and the excerpt is flagged as truncated. ANSI colour codes and control characters are stripped, and the text is scrubbed for secrets: passwords, tokens, API keys, JWTs, private keys and credentials inside connection strings become [REDACTED:…] before storage. Excerpts show up in execution history and in down alerts. Scrubbing is best-effort: do not deliberately print secrets or personal data in ping bodies.

bash
# capture your job's output as a log excerpt (shown in alerts + history)
OUTPUT="$(./backup.sh 2>&1)"
curl -fsS --data-binary @- "https://pulsewatcher.up.railway.app/ping/<token>" <<< "$OUTPUT"

Full pattern

Start, then either success or fail. Shell is the common case; the same three calls from application code look like this:

PING_URL="https://pulsewatcher.up.railway.app/ping/<token>"

curl -fsS -m 10 --retry 5 "$PING_URL/start"   # run began (optional)
./job.sh || { curl -fsS -m 10 "$PING_URL/fail"; exit 1; }
curl -fsS -m 10 --retry 5 "$PING_URL"          # success

Metrics from application code

PING_URL="https://pulsewatcher.up.railway.app/ping/<token>"
rows=$(psql -tAc "select count(*) from imported")

curl -fsS -m 10 --retry 5 "$PING_URL?rows=$rows&errors=0"

Scheduler & CI examples

# ping only if the job exits 0; -m caps the ping itself at 10s
0 3 * * * /opt/backup.sh && curl -fsS -m 10 --retry 5 "https://pulsewatcher.up.railway.app/ping/<token>"

Timezone gotchas

  • Plain-interval monitors are timezone-agnostic: they only care about the gap between pings.
  • For cron-scheduled monitors (Pro), set the monitor's timezone to match the box that runs the job, or an expected time will look "late" by your UTC offset.
  • cron on the server and PulseWatch's schedule are independent: a host in UTC firing at 03:00 needs the monitor set to UTC, not your local zone.
  • Daylight-saving shifts move wall-clock cron times; give a grace window wide enough to absorb the jump.