PulseWatchPulsewatch

Troubleshooting by symptom

Organised by what you observed, not by which feature is involved, because when something is wrong you know the symptom and not yet the cause. Causes are listed in the order they turn out to be responsible.

Start here

Most heartbeat problems are one question: did the request arrive? Answer it first.

bash
# Does a ping reach us at all? Run this on the machine that runs the job.
curl -fsS -m 10 -w '\nHTTP %{http_code} in %{time_total}s\n' \
  https://pulsewatcher.up.railway.app/ping/<token>

# Expected: HTTP 200. Anything else is the answer.
#   000  -> the request never completed: DNS, firewall or proxy
#   404  -> wrong token, or the monitor was deleted
#   429  -> rate limited: 60 pings/minute per token

The state looks wrong

My job ran successfully, but PulseWatch says it is down

1The ping never left the machine

How to tell: Look at the monitor page. If Last ping predates the run, nothing arrived. Run the curl by hand on the same host.

Fix: Usually egress: a firewall, a proxy that needs configuring, or DNS. Add --retry 5 so a single dropped packet does not cost you an alert.

2The ping ran only on success and the shell short-circuited

How to tell: A line shaped job.sh && curl ... sends nothing when the job exits non-zero, which reads as silence rather than as failure.

Fix: Report both outcomes. Use /exit/$? so the exit code decides, rather than making success the only thing you report.

3cron has a different PATH and curl is not on it

How to tell: Check the cron log for a "command not found" line.

Fix: Use the absolute path, usually /usr/bin/curl.

4The job takes longer than the grace window

How to tell: The ping arrives, but after the monitor already went down, so you get an alert and then a recovery a few minutes later.

Fix: Widen grace to cover the job's worst normal run, or ping /start at the beginning so the run is visible while it is still going.

The monitor has been in `new` for hours and never went down

1That is correct behaviour

How to tell: A monitor that has never received a ping has nothing to be late for.

Fix: Send one ping. The clock starts at the first one. If you want the monitor to alert before it has ever run, there is nothing to alert on yet, so create it as part of the deploy that also installs the job.

The uptime percentage does not match what I counted

1Three states are excluded from the calculation, not two

How to tell: blocked, paused and new are in neither the numerator nor the denominator. Only down is downtime.

Fix: See the state reference. A dependent that was blocked for six hours did not have six hours of downtime and did not have six hours of uptime either.

2Warns are counted as available

How to tell: A run that completed with problems still ran.

Fix: If a warn should count against you, map that exit code to down instead.

Monitors paused themselves

1A plan downgrade took you below the monitor cap

How to tell: The oldest active monitors are paused until the count fits, newest left running. Nothing is deleted.

Fix: Upgrade and they resume automatically, or delete monitors you no longer need and unpause the ones you want.

Alerts

A monitor is down and I received nothing

1No verified channel is attached

How to tell: An unverified channel is skipped silently by design, so a typo in an address cannot send your alerts to a stranger.

Fix: Verify the channel, then attach it to the monitor.

2The channel is not included in your plan

How to tell: Free accounts have email only. Telegram and webhooks need Pro or above.

Fix: Attach an email channel, or upgrade.

3A maintenance window is active

How to tell: The state still changed and the history still recorded. Suppressed sends are written to the alert history, so a silent window is auditable.

Fix: Check the window covers the time you expected an alert.

4The monitor is snoozed or the incident was acknowledged

How to tell: Both stop notifications without changing state.

Fix: Clear the snooze on the monitor page.

5The upstream took the blame

How to tell: If this monitor depends on one that is down, it went blocked, and the upstream's alert names it instead. That is one alert for one problem, working as intended.

Fix: Nothing. Read the upstream alert.

One broken thing produced a dozen alerts

1The chain is not declared

How to tell: Every job downstream of the broken one failed on its own account.

Fix: Declare the dependencies. Dependents go blocked and stay quiet, and the upstream alert names them.

2Many machines running the same job are separate monitors

How to tell: Two hundred monitors means two hundred alerts.

Fix: Use a fleet: one alert for the fleet, naming the machines that stopped.

3The repeat interval is short

How to tell: The same alert arriving over and over is the repeat, not new failures.

Fix: Raise the repeat interval, or acknowledge the incident to stop the repeats.

I got an alert and nothing was wrong

1Grace is tighter than the job's real variance

How to tell: The alert is followed by a recovery within a few minutes, repeatedly. The advisor flags this pattern.

Fix: Widen grace to cover the slowest normal run, not the average one.

2An uptime check hit a 200 that renders an error page

How to tell: Reason keyword_mismatch means the status was fine and the body was not.

Fix: That is the check doing its job. Read the body.

3The job is genuinely fine and its exit code is not

How to tell: Reason exit_code with a code you consider harmless.

Fix: Add an exit-code rule mapping that code to warn or ignore.

Warnings never reach me

1Warnings are batched by default

How to tell: warn_delivery defaults to digest_daily, so a warn arrives in the daily summary rather than immediately.

Fix: Set it to immediate if the warn is worth interrupting someone for. Consider whether it is.

2A quiet day sends nothing

How to tell: A digest with no warnings in it is not sent.

Fix: Nothing. That is the point.

Rules that do not fire

A metric rule has never fired

1The metric name does not match what the job sends

How to tell: Open a recent ping and read the metrics it carried. The name must match exactly, including case.

Fix: Rename the rule, or the metric. The rule editor suggests names already seen.

2The rule is disabled

How to tell: A disabled rule is stored and skipped.

Fix: Enable it.

3Nothing has crossed the bound

How to tell: The advisor tells a rule watching something healthy from a rule that has never matched anything, using the last time it fired.

Fix: Confirm the bound is where you meant it. A floor of 0 catches nothing.

A baseline rule is still learning

1Not enough comparable runs yet

How to tell: A bucketed baseline compares like with like, so an hour_of_day bucket on a daily job gets one sample a day for that hour.

Fix: Either wait, or widen the bucket. A daily job rarely needs anything finer than day_of_week.

2The job does not report the metric on every run

How to tell: Gaps in the samples slow learning down.

Fix: Report the number on every run, including the runs where it is zero.

My exit-code rule is being ignored

1A more specific rule matched first

How to tell: A rule covering exactly one code beats a rule covering a range that contains it.

Fix: Narrow or remove the competing rule.

2The code never arrived

How to tell: A malformed code is dropped rather than failing the ping, so /exit/ with an empty value looks like a plain success. Valid codes are 0 to 255.

Fix: Use /exit/$? immediately after the command, before anything else can overwrite it.

3Rules are cached briefly

How to tell: An edit can take up to a minute to apply to the ping path.

Fix: Wait a minute and send another ping.

Duration is never recorded

1Nothing pings /start

How to tell: Without a start there is no interval to measure.

Fix: Ping /start before the work and the plain URL after it. This is also what makes overlap detection and max-duration timeouts possible.

Fleets

A fleet says instances are missing and they are running

1The instance key changed

How to tell: Keys are case preserving and compared exactly. A host that started reporting as EDGE-07 after previously sending edge-07 is a new instance, and the old one goes quiet.

Fix: Pin the key. Use a stable identifier rather than something derived at boot, and retire the stale entry.

2The machine was decommissioned

How to tell: A retired instance leaves the denominator; a merely silent one does not.

Fix: Set a retirement window so departed machines drop out on their own.

A fleet filled up with instances I do not recognise

1Auto-register accepted every key that arrived

How to tell: Something is generating keys per run, per container or per boot, rather than per machine.

Fix: Fix the key at the source, then turn auto-register off so unknown keys are rejected instead of joining.

A completion budget is set and never says anything

1Instances are too new to judge

How to tell: An instance needs at least five scheduled runs behind it before a percentage means anything. Below that it is skipped rather than failed.

Fix: Wait for the history to build.

2Late runs count as runs

How to tell: A budget asks whether the work happened, not whether it was punctual. A machine that is always forty minutes late is at 100%.

Fix: Use duration regression or a tighter grace window for punctuality. They are different questions.

Delivery and integrations

My webhook signature check fails

1The body was parsed before it was verified

How to tell: The HMAC covers the raw bytes. Re-serialising the JSON changes whitespace and key order, and the digest no longer matches.

Fix: Capture the raw body first, verify, then parse. Express needs express.raw() on that route.

2Comparing with ===

How to tell: It works, and it leaks timing.

Fix: Use a constant-time comparison. The webhook page has a verified snippet in eight languages.

My webhook handler has no branch for an event it received

1The event set is open

How to tell: New events are added additively and the payload version stays at 1. Fleet deliveries have a fleet object and no monitor object.

Fix: Branch on details.reason, and default unknown events to logging rather than throwing. The full list is on the reference page.

2Branching on details.missing for a fleet

How to tell: A rule breach and a blown completion budget both report missing: 0, because nothing is missing in either. Every machine reported.

Fix: Branch on details.reason instead.

The REST API returns 401, 403 or 429

1401: the key is wrong or missing

How to tell: The header is Authorization: Bearer <key>.

Fix: Re-copy the key. It is shown once at creation.

2403: writing on a plan without write access

How to tell: Reads work on every plan. Writes need Pro or above.

Fix: Upgrade, or drive the change from the dashboard.

3429: 60 requests per minute per key

How to tell: The response body names the limit.

Fix: Back off and retry. When provisioning many monitors in a loop, add a small delay between calls.

A monitor is not on my status page

1It is not marked for publication

How to tell: Only monitors you explicitly publish appear.

Fix: Turn on the status page flag for that monitor.

2The page itself is not enabled

How to tell: A status page needs a slug set on the account.

Fix: Set the slug in settings. It must be 3 to 40 characters.

Data

Older runs and pings have disappeared

1Retention

How to tell: 7 days on Free, 30 on Pro, 90 on Business. Pruning is automatic.

Fix: Export what you need to keep, or upgrade. Rolled-up daily uptime and latency are kept for a year regardless.

My log excerpt is cut off

1The per-plan body cap

How to tell: 4 KiB on Free, 16 KiB on Pro, 64 KiB on Business.

Fix: Send the tail rather than the whole log. The last 50 lines contain the stack trace; the first 5,000 contain the banner.

Failures are not being grouped into an error group

1No parseable trace in the body

How to tell: Grouping needs a stack trace it can parse. A one-line message has no frames to fingerprint.

Fix: Send stderr as the ping body on failure. See reporting errors for the redirect that captures it.

2Error grouping is not on your plan

How to tell: It needs Pro or above.

Fix: Upgrade, or read the excerpt on the run directly.

Still stuck

The states and reasons reference explains any value you see on an alert. The configuration advisor finds monitoring that will not do its job before it fails to do it. On Pro and above, the in-app assistant answers questions about your own configuration.