Expected metrics, end to end

The problem this solves

A branch till uploads 863 rows one night and 41 the next, and both are correct: one is a Saturday in Lyon and the other is a Tuesday in a village. No threshold is right for both, so a metric rule that says rows > 500 either screams at the village or ignores the city.

An expected metric asks nobody to know the number. The run counts its own work, says so, does the work, and says what it did. The comparison is between those two, which makes it correct for every branch without being configured per branch.

$ curl "$PW/pharmacy-11/start?run_id=n1&expected_rows=780"
OK
$ curl "$PW/pharmacy-11?run_id=n1&rows=88"
OK

# exit code 0. finished on time. reported a number.
# 692 rows were never touched.

1. The fleet

Create the fleet on the Fleets page. The schedule is the reporting period, not a cron: every till reports on its own five-hour cycle rather than on the hour together.

Period: 5 hours

How often each machine is expected to report. It is per instance, so a till that starts its cycle at 04:00 is judged against 09:00, not against its neighbours.

Grace: 30 minutes

A till on a shop wifi will not be punctual. Grace is the slack before late means anything, and 10% of the period is a reasonable starting point.

Timezone: Europe/Paris

Only matters once there is an active window or a weekday bucket, and both come later on this page. Set it now so neither is wrong when it arrives.

Maximum run duration: 90 minutes

A sync that has been running for 90 minutes is not slow, it is stuck. This also becomes the deadline for an unanswered declaration, which matters in step 5.

Duration regression: 50%

Separate from the above, and softer. Every till is compared with its own recent days, and a run half again past that machine's usual is reported as a warning. It never pages: a slow branch is not a dead branch.

2. Hours the shop is open

A till is switched off at closing. Without saying so, every branch is late by midnight and the fleet pages you about a chain that is working perfectly.

Set the window to 20:00 to 23:00, Monday to Saturday: the sync runs after close, and Sunday the shops do not open. Leave the outside-window behaviour on silent, which is the default and the right answer for machines that are genuinely off. Quiet is not late.

The two other settings exist for the exceptions. warn records a silent machine without paging, for a branch you expect to be on. alert ignores the window entirely, for the one always-on server sitting inside a fleet of tills. Both can be set per instance, so the exception does not have to become the rule.

3. What the run reports, and what it promises

Two kinds of number, and the difference is the whole feature.

The pairing is the name. expected_rows is compared against rows, and the expected_ prefix is reserved: a metric rule can never be created with it, and a declaration is never stored as an ordinary reading.

For this fleet, declare expected_rows. Report rows and stock_updates. The first pair answers did it do the work; the second is an ordinary metric for a rule if you want one.

4. The rules

A definition with no rules is recorded and never compared, so at least one is required. Six kinds, and the threshold means something different in each:

A ratio floor is between 0 and 1. Entering 99 for 99% asks for ninety-nine times the declared work and fails every run forever, so the editor refuses it.

This fleet gets two rules on one metric, which is the reason rules are a list:

At least 0.98 of declared, warns, grace 1 run, only above 20

A branch that leaves 2% behind twice running has a problem worth looking at during the day. The grace stops one flaky night reporting anything at all, and the minimum stops a night with four prescriptions tripping a percentage.

Not short by more than 50, fails the run, only above 20

Fifty rows lost is not a rounding error on any branch. No grace: this one is worth knowing about the first time.

Every rule is evaluated and the most serious break decides, so the lenient rule cannot hide the strict one.

5. The agent

Four calls. The count before the work, the result after it, and a failure path that sends what went wrong.

#!/usr/bin/env bash
set -uo pipefail

PW="https://pulsewatcher.up.railway.app/ping/pwf_<your-token>"
INSTANCE="$(hostname)"          # the instance key. one per till.
RUN_ID="$(date +%s)-$$"         # ties the start and the end to one run

# 1. COUNT FIRST. This is the declaration, and it is the number the
#    comparison is against. If this query fails and returns 0, see the
#    "declares zero" note below -- that case is handled, not ignored.
ROWS=$(psql -tAc "select count(*) from sales_queue where synced_at is null")

# 2. OPEN THE RUN and declare. --retry matters: a dropped start means
#    no duration and no declaration for this run.
curl -fsS -m 10 --retry 5 \
  "$PW/$INSTANCE/start?run_id=$RUN_ID&expected_rows=$ROWS"

# 3. THE WORK. stderr is kept so a failure can send it.
if OUT=$(./sync.py --batch "$ROWS" 2>&1); then
    DONE=$(echo "$OUT" | grep -oP 'uploaded=\K\d+')
    STOCK=$(echo "$OUT" | grep -oP 'stock=\K\d+')

    # 4a. SUCCESS. The reading, plus a JSON body for anything a rule
    #     does not need but a person reading the run might want.
    curl -fsS -m 10 --retry 5 -X POST \
      -H 'content-type: application/json' \
      -d "{\"rows\":$DONE,\"stock_updates\":$STOCK,\"branch\":\"$INSTANCE\"}" \
      "$PW/$INSTANCE?run_id=$RUN_ID"
else
    # 4b. FAILURE. The exit code becomes the outcome, the body becomes
    #     the excerpt, and any reading taken before the failure still
    #     counts -- a run that did 97 of 412 and then died says so.
    CODE=$?
    curl -fsS -m 10 -X POST -H 'content-type: text/plain' \
      --data "$OUT" \
      "$PW/$INSTANCE/exit/$CODE?run_id=$RUN_ID&rows=${DONE:-0}"
    exit $CODE
fi

On the wire, and the responses are what the server actually returns:

$ curl -sS -w " [HTTP %{http_code}]" \
    "$PW/pharmacy-04/start?run_id=doc-n1-04&expected_rows=863"
OK [HTTP 200]

$ curl -sS -X POST -H 'content-type: application/json' \
    --data '{"rows":863,"stock_updates":211,"branch":"lyon-part-dieu"}' \
    -w " [HTTP %{http_code}]" "$PW/pharmacy-04?run_id=doc-n1-04"
OK [HTTP 200]

A ping is always 200 and always OK. It is accepted and queued, then applied by the worker; the endpoint never makes your agent wait for evaluation and never fails your job because a metric was malformed. The one exception is a 5xx, which means the ping did not land anywhere and retrying is right.

6. What arrives

The declaration and the reading are stored apart, so one is never mistaken for the other. This is the queue immediately after the two calls above:

instance_key   kind      metrics                          expected_metrics
pharmacy-04    start     null                             {"expected_rows": 863}
pharmacy-04    success   {"rows":863,"stock_updates":211} null

A moment later the worker has applied them. Both halves land on the same run row, bound by the same run_id, and the completion ratio is derived from them:

{
  "started_at": "2026-08-19T20:47:07.298+00:00",
  "ended_at":   "2026-08-19T20:47:07.299+00:00",
  "outcome":    "success",
  "duration_ms": 1,
  "metrics": {
    "rows": 863,
    "stock_updates": 216,
    "completion_ratio_rows": 1
  },
  "expected_metrics": { "expected_rows": 863 }
}

completion_ratio_rows is an ordinary metric from that point on. It gets a chart, a history, and a baseline rule if you want one, which is how "this branch normally completes 99% and today it did 40%" becomes a question you can ask.

7. A healthy night

Four branches, four different sizes of night, nothing to report:

instance      outcome   rows    declared  ratio  breaks
pharmacy-04   success     863       863    1.0   {}
pharmacy-07   success      41        41    1.0   {}
pharmacy-11   success     780       780    1.0   {}
pharmacy-23   success    2 140     2 140   1.0   {}

A 41-row branch and a 2,140-row branch are both at 1.0. That is the property no threshold has: the same configuration is correct for both.

8. Three nights that are not healthy

A branch that never ran

No ping at all. Expected metrics have nothing to say here and neither do metric rules; this is the fleet reporting check, and after the period plus the grace the instance goes late, then missing. Inside the active window it is an alert, outside it the machine is off and nothing fires.

A branch that ran and did a fraction of the work

pharmacy-11 declared 780 and processed 88. The run exited zero and reported a number, so nothing else in the product has an opinion about it:

{
  "instance_key": "pharmacy-11",
  "last_outcome": "fail",
  "last_metrics": {
    "rows": 88,
    "stock_updates": 19,
    "completion_ratio_rows": 0.1128
  },
  "expected_breaks": { "expected_rows": 1 },
  "last_expected": {
    "metric":   "expected_rows",
    "paired":   "rows",
    "reason":   "expected_shortfall",
    "expected": 780,
    "actual":   88,
    "ratio":    0.11282051282051282,
    "detail":   "declared 780, processed 88, 692 short",
    "severity": "down"
  }
}

Both rules broke. The strict one decided, because the most serious break always does: had the lenient rule decided, its grace of one run would have absorbed this and the night would have been recorded as a success.

A bad release, one cohort short

Agent 2.4.1 goes out and three branches drop to roughly 46% of what they declare. Three machines, three different sizes, one cause, and one alert:

[MAJOR] Fleet Pharmacy POS sync: expected_rows shortfall on 3 instances

🔴 Pharmacy POS sync: instances reporting problems

expected_rows shortfall on 3 instances
3 of 4 instances: pharmacy-23, pharmacy-04, pharmacy-11

pharmacy-23: declared 2,140, processed 990, 1,150 short, 2 in a row
pharmacy-04: declared 863, processed 402, 461 short, 1 in a row
pharmacy-11: declared 780, processed 361, 419 short, 2 in a row

expected_rows declared zero on 1 instance
1 of 4 instances: pharmacy-07

pharmacy-07: declared 0 rows expected, which may mean the counting step failed, 1 in a row

Grouped by cause, and worst first within a cause. Worst means the size of the gap, not the length of the streak: pharmacy-23 leads on 1,150 rows even though pharmacy-11 has been failing exactly as long. A branch that lost 1,150 rows is the one to open first.

9. The night the numbers agree and the work did not happen

The counting query errors and returns 0. Declared is 0, processed is 0, every rule passes, and the night reports a clean success with nothing done. No comparison can catch this, because both sides of it are wrong in the same direction.

Two settings cover it. The first is what a declaration of zero means for your fleet: for a pharmacy chain, a branch with genuinely nothing to sync is unusual, so set it to suspicious rather than the default:

{
  "instance_key": "pharmacy-07",
  "last_outcome": "warn",
  "last_expected": {
    "reason":   "expected_zero_suspicious",
    "expected": 0,
    "actual":   0,
    "detail":   "declared 0 rows expected, which may mean the counting step failed",
    "severity": "warn"
  }
}

The second catches the version of it that is not zero. Set flag a declaration this far from normal to 60%, and the declared value is checked against this branch own history of declarations, bucketed by weekday:

{
  "metric":    "expected_rows",
  "declared":  4,
  "median":    782.5,
  "samples":   8,
  "anomalous": true,
  "detail":    "declared 4 rows expected, this instance normally declares
                around 782.5 on Tuesdays. The count step may have failed."
}

Never against other branches: a village declaring 40 and a city declaring 4,000 are both right, and comparing them would make the village permanently anomalous. It needs seven comparable nights before it will say anything, and it warns rather than pages, because a genuinely quiet Tuesday is a legitimate reason for a low count and a check that pages on those is a check somebody turns off.

10. The run that promised and went quiet

A till declares 500 rows at 20:15 and is never heard from again. The instance is still inside its reporting period, its last metrics look normal, and nothing about a missing comparison resembles a failure. Only the declaration knows that work was promised.

{
  "instance_key":        "edge-h",
  "last_outcome":        null,
  "open_run_expected":   { "expected_rows": 500 },
  "open_run_deadline_at": "2026-08-19 22:17:49+00"
}

The deadline is the fleet maximum run duration, 90 minutes here, or a period plus its grace when no maximum is set. When it passes, the declaration is recorded as unanswered and cleared, so the next start never inherits a promise from a run that is already over, and the finding joins the next fleet alert.

Where to look

Fleet → Metrics

The editor, and the distribution of completion ratios across every branch. That chart answers one question: a wall of amber means every machine fell short, which is usually something on your side; two amber bars mean two machines fell short, which is usually something on theirs.

An instance → Recent runs

A declared-versus-done column per run, so you can see the night it started.

Fleet → Metrics → Slowest machines

Each instance's typical run against the fleet's, ranked by ratio. This finds the branch that has always been slow, which comparing a machine with its own past never will.

Fleet → Alerts

What was sent, when, and what was suppressed by the storm cap.

Related

• When things arrive wrong for every malformed, duplicated, out-of-order and missing case, with what happens to it.
• Fleets for what a fleet is and every configuration field.
• Running a fleet for active windows, alert severity, cohorts and bulk actions.
• Metrics and baselines for ordinary metric rules and the baselines a completion ratio can use.