Fleets: one monitor for many machines

When a fleet is the right shape

Use a fleet when the machines are interchangeable: the same job, the same schedule, the same meaning of failure. A POS terminal in every shop. A sync agent on every branch server. A scraper on every worker node.

Use separate monitors when the jobs differ. An extract, a transform and a load are three different jobs that happen to run on one host, and they want three monitors and a dependency chain, not a fleet.

Setting one up

1. Fleets are on Business, with an instance cap of 500. Downgrading pauses your fleets rather than deleting them: instances and history survive, and they resume on an upgrade or a complimentary plan.
2. In the dashboard, open Fleets and choose New fleet. Give it a name. The slug fills itself in from the name and stops following it once you edit it directly, so your ping URLs do not change under you.
3. Set the schedule: how often each machine reports, and how late it may be. Grace should cover the slowest normal run, not the average one.
4. Set Expected instances if you know the number. Without it the denominator is however many machines have registered, which means a machine that never came online at all is invisible rather than missing.
5. Save. The fleet page shows one ping URL for the whole fleet, ending in a placeholder for the machine name.
6. Put that URL in the job on every machine, substituting a stable identifier for the placeholder. Each machine registers itself on its first ping.
   bash

   # Every machine runs the same line. Each substitutes its own name.
   # $(hostname) works; so does a site id, a serial, or anything stable.
   0 * * * * /opt/agent/run.sh && \
     curl -fsS -m 10 --retry 5 \
     https://pulsewatcher.up.railway.app/ping/pwf_<token>/$(hostname)
7. Confirm it worked: within a minute the fleet page lists each machine that has pinged. New arrivals sit in learning until they have reported on schedule once, and a machine in learning can never breach a fleet. When they move to reporting, the fleet is live.

Spreading a large fleet across the minute

0 * * * * on ten thousand machines means ten thousand pings in the same second. Ingestion is built for that and holds up: 10,000 instances inside a 60 second window measured p50 154ms, p99 276ms, no dropped pings and a queue that drained in 14 seconds. The part that does not hold up is the network in between. If those machines sit behind one NAT gateway, the burst reaches us as thousands of new connections from a single source IP, and that is the pattern edges and firewalls treat as an attack. We have measured connections being refused at that shape.

Above roughly 200 machines, give each one a fixed offset instead. Note that this is not a random sleep. A random delay each hour would make the gap between one machine's pings vary between 55 and 65 minutes, and a fleet with a tight grace would start alerting about machines that are working. An offset derived from the machine's own name is stable, so every interval stays exactly one hour and the fleet still arrives spread out.

# Above 200 machines, spread the arrivals. Same line everywhere:
# each host derives its own offset from its own name, so machine A always
# pings at +266s and machine B always at +38s.
0 * * * * sleep $(( $(hostname | cksum | cut -d' ' -f1) % 300 )); \
  /opt/agent/run.sh && \
  curl -fsS -m 10 --retry 5 \
  https://pulsewatcher.up.railway.app/ping/pwf_<token>/$(hostname)

Machines that are meant to be off

A desktop switched off at 17:00 is not an incident. The same silence from a payment gateway at 03:00 is. By default a fleet is active every hour of every day, so every machine is expected to report around the clock. Give it active hours and time only counts toward lateness inside those hours.

That last sentence is the whole feature, and it is worth being precise about. Suppose the machines ping hourly and work 09:00 to 17:00. One reports at 16:55 and is switched off at 17:00. At 09:01 the next morning it has been silent for sixteen hours, which against an hourly schedule would be very missing indeed. But only six of those minutes were inside its window: five on Monday, one on Tuesday. It is on time, and so is the rest of the fleet, so nothing fires and nothing needs suppressing.

A machine that stops during the working day is caught normally. If it last reported at 10:00 and it is now 11:10, that is seventy minutes of window time, and it is missing. It stays missing when the window closes at 17:00 rather than turning into “off, as expected”, because it broke while it was supposed to be working and that does not stop being true at five o'clock.

Individual machines override any of these. The case it exists for is the one always-on server sitting inside a fleet of desktops: give that instance alert and it is judged on the wall clock while everything around it keeps office hours.

Timezones are per instance, so one set of hours across three countries is three different windows, which is usually what people mean. Daylight saving is handled: a window states local hours and keeps them across the change.

Instance keys

The last path segment is the instance key. It must start with a letter or digit and may contain letters, digits, dot, underscore, colon and hyphen, up to 128 characters. Case is preserved and compared exactly, so EDGE-07 and edge-07 are two different machines.

Pick something stable. A key derived from a container id or a boot-time random value creates a new instance on every restart, and the fleet fills with machines that ping once and go quiet forever. If that has already happened, fix the key at the source and turn auto_register off so unknown keys are rejected rather than joining.

Reporting outcomes, not just presence

A bare ping means the run succeeded. Instances can also report failure or an exit code, exactly as a monitor does, and the fleet evaluates the result per machine.

# report the exit code so the fleet can classify the run
0 * * * * /opt/agent/run.sh; \
  curl -fsS -m 10 --retry 5 \
  "https://pulsewatcher.up.railway.app/ping/pwf_<token>/$(hostname)/exit/$?"

# or report failure explicitly, with stderr as the body
0 * * * * /opt/agent/run.sh 2>/tmp/agent.err || \
  curl -fsS -m 10 --retry 5 --data-binary @/tmp/agent.err \
  https://pulsewatcher.up.railway.app/ping/pwf_<token>/$(hostname)/fail

const base = 'https://pulsewatcher.up.railway.app/ping/pwf_<token>/' + os.hostname()

try {
  const rows = await syncStock()
  // metrics ride along on the success ping
  await fetch(`${base}?rows=${rows}`)
} catch (err) {
  await fetch(`${base}/fail`, { method: 'POST', body: String(err.stack ?? err) })
}

What counts as a fleet failure

Only missing instances count. A machine in learning, late or retired does not, for reasons set out in the state reference.

One alert, naming the machines

This is the point of the feature. Three shops going quiet produces one message that says which three, not three messages and not a message that says only "3 instances".

🔴 Pharmacy POS agents: 3 of 200 instances missing

Not reporting: shop-014, shop-088, shop-141

Expected every 1h, 5m grace. Last seen 1h 12m ago.

https://pulsewatcher.vercel.app/app/fleets/<id>

Up to 12 machines are named individually. Past that the alert names 12 and adds a count, because a message listing 180 hostnames is a message nobody reads.

Configuration is declared once and evaluated per machine

A fleet carries the same configuration surface a monitor does. Each setting is a fleet default, applied to every instance and evaluated against that instance's own data.

Two defaults deliberately differ from a monitor's. Overlap starts at ignore and new-error alerting starts off, because turning either on across a fleet that already exists would light up every machine at once for behaviour that has been normal all along.

Rules on a fleet

1. Open the fleet page. The Metric rules and Exit codes cards at the bottom are the same editors the monitor page uses.
2. Add a rule. The metric name field suggests names the fleet has actually seen, taken from the latest values every instance reported.
3. Save. The rule applies to every instance from its next ping. There is no per-machine copy to keep in sync.
4. Confirm it worked: the fleet page shows a rollup across instances for each metric, with the min, median, p95 and max, and it names the instances furthest from the median. Outliers are measured against the median rather than the mean, so a handful of broken machines cannot drag the threshold out to meet themselves.

A breach on many machines is aggregated the same way an outage is: grouped by cause, one message, the machines named. Twelve instances exiting 100 is one line saying exit 100 on 12 instances followed by which twelve.

Completion budgets ask each machine, not the average

A fleet can pass "is anyone missing" while quietly failing "did each machine run as often as it was supposed to". An instance reporting six days in seven is never absent long enough to be called missing, and loses a seventh of its work every week.

Set a budget on the fleet and every instance is judged against it separately. Never averaged: 199 machines at 100% and one at 40% averages to 99.7% and reads as healthy while that machine drops three runs in five.

📉 Pharmacy POS agents: 1 of 200 instances below budget

1 of 200 instances ran less than 95% of their scheduled times over
30 days. Worst: shop-088 at 40%.

shop-088: 40% (60 of 100 scheduled runs missed)

https://pulsewatcher.vercel.app/app/fleets/<id>

• An instance needs at least 5 scheduled runs behind it before a percentage means anything. Below that it is skipped, not failed.
• The window starts when the instance first reported, so a machine added yesterday is not measured against a month it was not present for.
• A late run still counts as a run. A budget asks whether the work happened. Punctuality is what grace and duration regression are for.
• The window is 1 to 90 days, defaulting to 30. It stops at 90 because fleet slots are computed rather than stored, and a year across 500 machines is millions of calculations to answer one question.
• Retired instances are out of the estate and are not judged.

Silencing one machine

A single instance can be snoozed from the instance grid on the fleet page without silencing the fleet. Use it for the shop that is closed for refurbishment while the other 199 keep reporting. Maintenance windows can also be scoped to a fleet, which suppresses notifications while state and history keep recording.

Variations

What can go wrong

{
  "version": "1",
  "event": "fleet.breached",
  "sent_at": "2026-08-12T09:00:00.000Z",
  "fleet": { "id": "…", "name": "Pharmacy POS agents", "slug": "pharmacy-pos" },
  "details": {
    "reason": "missing_instances",
    "missing": 3,
    "total": 200,
    "missing_pct": 1.5,
    "missing_instances": ["shop-014", "shop-088", "shop-141"],
    "returned_instances": [],
    "detail": "3 of 200 instances missing"
  },
  "dashboard_url": "https://pulsewatcher.vercel.app/app/fleets/…"
}

Every field

Every fleet is on Pro or above, so nothing in this table has a per-plan difference beyond the instance cap. The monitor configuration a fleet also carries is documented on the monitor page with the same defaults, except the two noted above.

Related

• Monitors when the jobs are different from each other. A fleet is for the same job repeated.
• Dependencies when jobs run in order. A fleet and a chain solve different problems and compose fine.
• Duration and completion budgets for the machine that runs but not often enough.
• Troubleshooting fleets when instances are reported missing and you can see them running.