PulseWatchPulsewatch
FAQDashboard

When things arrive wrong

Agents run on machines you do not control, on networks that drop, with clocks that drift. This page is what happens to every malformed, missing, duplicated and out-of-order case, what you see when it happens, and whether it needs you.

One rule underneath all of these: a bad value never fails a ping. A ping is accepted, queued and answered OK; anything unusable in it is dropped and the rest is kept. Your job never fails because a metric was misspelled. The only non-200 worth retrying is a 5xx, which means the ping did not land anywhere.

Nothing arrives

No ping at all, inside the active window

What happens
After the reporting period plus its grace the instance goes late, then missing. The fleet counts it and, once the fleet alert mode is satisfied, sends one alert naming it alongside anything else that is missing.
What you see
The instance shows late, then missing, on the fleet page.
What to do
This is the check working. Look at the machine.

No ping at all, outside the active window

What happens
Nothing. Quiet is not late. With the default silent behaviour the instance is not counted as missing and nothing is recorded.
What you see
The instance shows quiet.
What to do
Nothing, unless the machine was meant to be on. Set that instance outside-window behaviour to warn or alert if it is the exception.

An instance that has never reported since it was created

What happens
It starts as learning, not reporting. A machine that has never been seen cannot be late, because there is nothing to be late against.
What you see
The instance shows learning until its first ping.
What to do
If it stays there, the agent has never run or is pinging the wrong token. Check the agent logs for a 404.

Half a run arrives

A start with no end

What happens
The run stays open and the declaration is held on the instance with a deadline: the fleet maximum run duration, or a period plus its grace when no maximum is set. When the deadline passes, the declaration is recorded as expected_missing_actual and cleared, so the next start never inherits a promise from a finished run.
What you see
captured
{
  "instance_key":         "edge-h",
  "last_outcome":         null,
  "open_run_expected":    { "expected_rows": 500 },
  "open_run_deadline_at": "2026-08-19 22:17:49+00"
}
What to do
The run started and never finished. This is the case the whole feature exists for; read the agent logs on that machine.

An end with no start

What happens
Accepted and recorded. The run has no measured beginning, so duration_ms stays null rather than being invented from a start that never happened. Both timestamps on the run are the ping time.
What you see
A run row with no duration.
What to do
Nothing, if your agent does not send starts. If it does, the start is being dropped, which also costs you the declaration.

Declared, and no reading ever arrives

What happens
Whatever the on-missing-actual setting says. The default is alert, because a run that announced work and never reported on it is the headline case.
What you see
captured
{
  "reason":   "expected_missing_actual",
  "expected": 100,
  "actual":   null,
  "detail":   "declared 100 rows expected, run never reported an actual value",
  "severity": "down"
}
What to do
The declaration reached us and the reading did not. Check that the terminal ping sends the paired metric, without the expected_ prefix.

A reading with no declaration

What happens
Nothing, normally: an actual with no declaration is not evaluated, because inventing a comparison is how a feature starts alerting on jobs that never opted in. If the metric is marked required, a run that STOPPED declaring is itself reported.
What you see
captured, with required set
{
  "reason":   "expected_missing",
  "expected": null,
  "actual":   12,
  "detail":   "reported 12 for rows without declaring expected_rows first",
  "severity": "warn"
}
What to do
The counting step was removed or failed. The run may still be doing the work; nobody can tell any more.

The numbers are strange

Both expected and actual are zero

What happens
Your choice, and the choice matters. Nothing to do and that is fine is the default. Do not compare skips the run. Unusual here raises expected_zero_suspicious as a warning, which is the right setting for a job that always has something to do.
What you see
captured, with suspicious set
{
  "reason":   "expected_zero_suspicious",
  "expected": 0,
  "actual":   0,
  "detail":   "declared 0 rows expected, which may mean the counting step failed",
  "severity": "warn"
}
What to do
A counting query that errors and returns 0 makes a run with nothing to do look identical to one that failed. Check the count, not the job.

The declaration is unlike this instance history

What happens
If a deviation percentage is set, the declared value is compared against this instance own past declarations, bucketed by weekday, using the median. Never against other instances. It needs at least seven comparable runs first, and it warns rather than pages.
What you see
captured
{
  "declared":  4,
  "median":    782.5,
  "samples":   8,
  "anomalous": true,
  "detail":    "declared 4 rows expected, this instance normally declares
                around 782.5 on Tuesdays. The count step may have failed."
}
What to do
A quiet day is a legitimate reason. If it was not quiet, the counting step is reading the wrong thing.

Far below what was declared

What happens
Every rule on the metric is evaluated and the most serious break decides, so a lenient rule can never hide a strict one. A break still inside its grace count still counts towards the streak.
What you see
captured
{
  "reason":   "expected_shortfall",
  "expected": 780,
  "actual":   88,
  "ratio":    0.11282051282051282,
  "detail":   "declared 780, processed 88, 692 short",
  "severity": "down"
}
What to do
The run itself is the witness, so this is rarely a false alarm. Read the excerpt for the batch that stopped.

Far above what was declared

What happens
Only reported if you configured a rule for it. The default shortfall rule ignores overruns, because processing more than declared is usually a re-run or a race rather than a loss.
What you see
expected_surplus, at minor severity by default.
What to do
Compare the counting query against the processing query. They are looking at different sets.

A negative declared value

What happens
Refused. A negative count is not an amount of work, it is a broken counting step, so it never reaches the rules and never produces a completion ratio. It is reported as an implausible declaration, at warn.
What you see
expected_value_anomalous, naming the negative number.
What to do
Your counting query is returning a negative. It is almost always a subtraction that went the wrong way.

Values arrive malformed

All four of these were sent to a live fleet. In every case the ping was accepted and the usable parts were kept:

SentStoredWhy
?expected_rows=lots&rows=12{"rows":12}Not a number. Dropped; the reading is kept.
?expected_rows=1e999&rows=5{"rows":5}Not finite. Same treatment.
{"rows": 40,excerpt onlyNot JSON, so it is treated as output and kept as the excerpt.
30 metrics, 30 declarations10 and 10Ten per ping, per bucket. Metrics a rule watches are kept first.

A JSON body over the size limit

What happens
The body is not stored, but its numeric fields are still promoted to metrics. A 20 KB payload against an 8 KiB limit kept rows: 40 and discarded the blob.
What you see
The payload column is empty and the metrics are present. The reason is logged.
What to do
Send the numbers you want rules on as query parameters, and keep the payload for context.

An excerpt over the plan cap

What happens
Cut on a byte budget, never rejected. A trailing partial UTF-8 sequence is stripped rather than stored as a replacement character. The cap is 4 KiB on Free, 16 KiB on Pro and 64 KiB on Business, and the ping body itself is bounded at 64 KiB before any of that.
What you see
The excerpt ends mid-line.
What to do
Send the last few hundred lines rather than the whole log. Nobody reads past the first screen anyway.

A metric name using the reserved prefix as a reading

What happens
It is stored as a declaration, not a reading, because that is what the prefix means. Sending ?expected_rows=88 as your result creates a declaration of 88 with no reading, not a comparison.
What you see
expected_missing_actual on a run you thought reported.
What to do
The result goes under the name without the prefix. expected_rows is the promise; rows is what happened.

Runs arrive twice, or in the wrong order

A retry with the same run_id

What happens
Counted once. The run history has always deduplicated on the run id; the instance state now does too, so a retry no longer adds to a warn streak or a grace count for a run that only happened once.
What you see
captured, three identical pings
{
  "instance_key":      "edge-retry",
  "consecutive_warns": 1,
  "expected_breaks":   { "expected_rows": 1 },
  "run_rows":          1
}
What to do
Nothing, but a fleet replaying steadily means an agent timing out on every run. That is worth a look even though nothing here is wrong.

Pings arriving out of order

What happens
Within one batch they are applied in the order they were received, not the order they arrived, so a machine that failed and then recovered is not left looking broken. Across batches the queue is drained in order.
What you see
Nothing unusual.
What to do
Send a run_id. It is what binds a start and an end together when the network reorders them.

A start while a run is already open

What happens
The earlier run is closed as abandoned with the reason superseded by a later start, and the new one opens. It is recorded rather than overwritten: overwriting loses the earlier start, and the next close then measures a duration from the wrong beginning.
What you see
An abandoned run in the history and an overlap recorded.
What to do
Two copies of the job are running, or the previous one crashed without pinging. Both are worth knowing about.

Clock skew on the client machine

What happens
Nothing at all. The arrival time is stamped by the server when the ping lands, never taken from the client, so a machine whose clock is a day out is timed correctly.
What you see
Nothing.
What to do
Nothing here. Fix the clock for the sake of the machine own logs.

Runs that do not end

A run past the maximum duration

What happens
The abandoned-run sweep closes it. For a fleet that means clearing the open-run marker on the instance, because a fleet run does not exist as a row until it completes. Leaving the marker set is the harm worth preventing: the next terminal ping would measure its duration from a start belonging to a run that died hours ago.
What you see
The open run disappears and the next run measures itself correctly.
What to do
Set max duration to something a healthy run never reaches. Without it the fallback is a period plus its grace.

An exit code with no output

What happens
The exit code decides the outcome on its own. Exit code rules map ranges to outcomes, and a code with no matching rule is a failure.
What you see
A failed run with an exit code and no excerpt.
What to do
Send stderr on the failure path. An exit code tells you it broke; the excerpt tells you where.

Output with an exit code of zero

What happens
Kept. A success ping with a text body stores that body as the excerpt, which is how a run that warns about something without failing keeps its evidence.
What you see
A successful run with an excerpt.
What to do
Nothing. This is worth doing deliberately for jobs that skip work quietly.

Numbers on a failure ping

What happens
Use /exit/<code> rather than /fail if the run has partial numbers to report. Readings are taken from /exit and from a plain success ping; /fail records the failure and the excerpt but not the metrics.
What you see
A run that used /fail has no metrics even though the query string carried some.
What to do
Change /fail to /exit/1. You keep the failure and you gain the numbers, which is what tells you it did 97 of 412 rather than none of them.

Everything fails at once, or only one thing does

The distinction the fleet is built around. One alert either way, and the shape of it tells you which you are looking at.

One instance

What happens
Named on its own in the alert, at a severity scaled to how small a share of the fleet it is.
What you see
minor severity, one instance named.
What to do
Look at that machine.

Many instances, one cause

What happens
Grouped into one finding per cause, naming the worst first. Worst is the size of the gap, not the length of the streak, so the branch that lost 1,150 rows leads the branch that lost 419 even if the second has been failing longer.
What you see
captured
expected_rows shortfall on 3 instances
3 of 4 instances: pharmacy-23, pharmacy-04, pharmacy-11

pharmacy-23: declared 2,140, processed 990, 1,150 short, 2 in a row
pharmacy-04: declared 863, processed 402, 461 short, 1 in a row
pharmacy-11: declared 780, processed 361, 419 short, 2 in a row
What to do
One cause across several machines is usually yours: a release, a central server, a schema change. Start there rather than at any one branch.

Most of the fleet at once

What happens
Severity rises with blast radius, and if the affected machines share a version, region or label, the alert says so. A storm cap limits alerts per fleet per hour; suppressed ones are collapsed rather than discarded, and the next alert says what happened in between.
What you see
major or critical, with a cohort line if there is one.
What to do
If you received fewer alerts than you expected during an incident, the storm cap is usually why.

Related