Running a fleet

Machines that switch off at night

A fleet of laptops or shop terminals is silent every evening. Without saying so, every one of them is late by morning and the fleet pages you at 02:00 about a fleet that is working perfectly.

An active window says when a machine is expected to be running. Lateness is then measured in window time, not wall-clock time: a machine that goes quiet at 18:00 has used no lateness by 09:00 the next morning, because none of those hours counted.

Active days      Mon Tue Wed Thu Fri
Active hours     08:00 to 18:00
Timezone         Europe/Paris
Outside hours    silent

Outside those hours has three settings:

silent

The default. Outside its hours a machine is quiet, and quiet is not late. Nothing is recorded and nothing fires.

warn

Outside its hours a silent machine is recorded as a warning. Visible in the fleet, never an incident, never a page.

alert

The window is ignored entirely. Use it for the one always-on server inside a fleet of desktops.

Windows are set on the fleet and can be overridden per instance, which is the point: one fleet can hold shops in three timezones and the server in the back office. An instance with its own timezone uses it; otherwise it inherits the fleet's.

Holidays

Dates declared inactive on your account are removed from window time everywhere, so a fleet does not page you on Christmas Day for machines nobody switched on. They apply to fleets with a window and to fleets without one.

If you do not know the hours

PulseWatch watches each instance's own reporting history and suggests a window when the pattern is clear enough. Suggestions are never applied on their own. You see the proposed hours and the evidence, and accept or dismiss them.

Keeping one bad release from sending a thousand alerts

A fleet sends one alert, naming the machines. That is the whole feature. Several things then shape whether it is sent at all and what it says.

Severity

Every alert carries a severity, and severity rises with blast radius. Twelve of ten thousand machines missing is not the same event as nine thousand, and the subject line says which one it is.

Escalation only ever raises a severity, never lowers one, so an override you set to say "this always matters" is never quietly downgraded because only a few machines were affected this time.

Cohorts

When the failing machines have something in common, the alert says so. This is the difference between an alert you have to investigate and one that has already told you the answer:

[MAJOR] Fleet client-agent: 412 of 9,847 instances missing

All 412 are reporting version 2.4.1.

Cohorts are found across whatever the agents report: version, region, client id, or any label you send. A concentration is only named when it is genuinely concentrated, so a fleet that is simply all on one version does not get told that every time.

Drops that never cross a threshold

A fleet that normally reports at 98% and slides to 71% over a week clears every absolute floor anybody would set. Rate-of-change alerting compares against the fleet's own recent history instead, using the median of previous days so a single bad day cannot hide the next one.

The storm cap

A fleet flapping in and out of breach could page you every pass. A cap limits how many alerts one fleet sends per hour. Suppressed alerts are collapsed, never discarded: the state change is recorded, and the next alert that does go out says what happened in between.

Warnings, and who hears about them

A warning never opens an incident, never counts as downtime and never escalates. What it does is get delivered, or not, according to the most specific setting that applies:

per-reason on the instance     "on this machine, only exit-code warnings"
the instance                   "this machine is noisy, keep it quiet"
per-reason on the fleet        "across the fleet, only duration warnings"
the fleet                      "warnings from this fleet go to a daily digest"
the account                    your default for everything

Each level can be off, digest or immediate. The instance page shows the resolved answer and which level decided it, because "off" on its own leaves you hunting through four levels to find which one turned it off.

Acting on thousands of instances at once

Filter the instance list, check the count, then apply an action to everything that matches. The count on screen is the count acted on: the list, the saved view and the bulk action all use the same filter.

Retire

Takes instances out of the fleet count. A retired machine is not missing, it is gone. Reversible.

Unretire

Brings them back as learning, not reporting: a machine that has been away has to earn its place again before it can be called late.

Snooze

Silences them for a number of hours. They keep reporting and keep their history.

Add tag / Remove tag

Labels you apply, as opposed to labels the agent reports. Filterable and usable as a bulk target.

Before it runs you get the filter as a sentence rather than a number, because "retire 4,182 instances" is not something anybody can check and "retire 4,182 instances that are dormant and tagged pilot" is.

Filtering by what agents report

Filter on the labels instances send, as field=value pairs:

version=2.4.1 region=eu-west

Matches are exact. Substring matching would look helpful and select a set nobody pictured, since 2.4 matches both 2.4.1 and 12.4.0, and on a bulk retire that difference is machines.

Saved views and export

A filter you use often can be saved and reapplied. The same filter drives CSV export, so what you download is what you were looking at. Export streams the whole result, not the page you can see.

Sending data with a ping

Any ping can carry a JSON body. Numeric fields become metrics automatically, so a rule can target them exactly as it targets a query parameter, and this is the preferred way to send structured data.

curl -fsS -m 10 -X POST \
  -H 'Content-Type: application/json' \
  -d '{"version":"2.4.1","region":"eu-west","records_uploaded":1432}' \
  https://pulsewatcher.up.railway.app/ping/<fleet_token>/$(hostname)

Some keys are promoted to their own indexed columns, because they are what people filter a whole fleet by:

version   hostname   client_id   region   os   agent_build   records_uploaded

Everything else is kept as free-form JSON. Bodies are capped, 8KB by default. An oversized body is rejected, not truncated: half a JSON document is invalid data that looks like data, and a rule reading a truncated field would compare against a number nobody sent.

Log excerpts without drowning in them

Ten thousand machines storing their output every cycle is untenable, and ten thousand identical "backup complete" lines are not evidence of anything. So:

• Failing and warning runs always have their output stored.
• Successful runs store a configurable percentage, 0 by default, or only when the output differs from that machine's last stored success.
• A fleet-wide cap evicts oldest first, so the ceiling is a ceiling rather than a suggestion.

Sampling picks the same machines each cycle rather than a random subset. A random tenth gives you a different tenth every time, so no machine has a history and the one you want to compare against yesterday never has yesterday.

Reading the workspace

A fleet has its own workspace with Overview, Instances, Metrics, Errors, Incidents, Alerts and Settings. One time-range control at the top applies to every chart, so two charts never describe different windows.

Reporting rate

The headline. Share of the fleet reporting over time, drawn against its own history with the alert threshold marked, which is where a bad release becomes obvious.

Cycle heatmap

Hour of day across, day down. A quiet block with straight edges that repeats every day is a schedule, not an outage. A gap that ignores the grid is the one to look at.

Duration profile

Median and p95 as a band. The band widening matters more than either line moving: it means a subset has started struggling while the majority holds the average steady.

Metric distributions

Every instance's latest value for a metric, as a histogram. Two humps mean two populations, usually a version, a region, or a subset that is quietly broken. A median alone hides that completely.

Top error groups

Ranked by how many machines are affected, not how many times it happened. One machine in a crash loop is noisy; four hundred machines failing once is a release.

Freshness

When the fleet was last evaluated and whether pings are waiting. This is how you tell "nothing is wrong" from "we stopped looking".

Limits, and where each one came from

Fleet settings shows the ceilings actually in force and names the level that set each one, so a number that differs from the published plan limits explains itself instead of becoming a support ticket:

Instances        2000, raised on your account
Metric rules     25, from your plan
History kept     90 days, set for this fleet
Payload size     8KB a run
Alerts           at most 6 an hour

Limits layer fleet first, then account, then plan, and the most specific one wins.

Retention

Fleet history is pruned in tiers, ordered by size against usefulness:

1. Log excerpts first. Largest thing on a run and the least reusable.
2. Payload bodies next. The promoted fields survive, so "which version was this machine on in March" still answers.
3. Run rows last, and by then daily per-instance summaries have captured what they contributed.

Those daily summaries are kept far longer than the runs behind them, so a machine keeps a history after the individual rows are gone.

When something looks wrong

Instances are reported missing and I can see them running

Check the freshness strip first: if evaluation is stale or pings are queued, the states are old rather than wrong. Then check the instance name. Case is preserved, so HOST01 and host01 are two machines and one of them has stopped reporting.

The fleet pages me every night

The machines are off and nothing has said so. Set an active window, and set outside hours to silent.

A machine is off but the fleet still expects it

Retire it. Retired is not missing. If it comes back it returns as learning and has to report normally again before it can be called late.

One noisy machine keeps warning

Set warning delivery to off on that instance. The fleet keeps its own setting, and the instance page will show that the instance is the level that turned it off.

I got one alert and expected several

That is the design. A fleet sends one alert naming the machines. If you got fewer than expected over an hour, check whether the storm cap collapsed them; the next alert says what happened in between.

The completion budget shows nothing

The fleet has more successful runs in the window than can be read in one page load, so a percentage would describe part of the fleet while looking like all of it. The alert still uses the whole window.

A new instance appeared that I did not expect

Any name that pings the fleet URL registers itself. That is usually a test run or a new machine imaged from a template that kept a hostname. Retire it, or turn off auto-registration on the fleet.

Deleting a fleet

Fleet settings has a delete that removes the fleet and everything belonging to it: every instance, every run and its stored output, rollups, incidents, error groups and saved views. There is no undo and no recovery window.

Because of that, you type the fleet's name to confirm. The name is checked by the database, not only by the page, so it is a real gate rather than a dialog. Before you confirm you are shown the counts of what is about to go.

A large fleet takes a while, and the page reports progress while it works. Leave it open. If you close it part way the fleet is already hidden and stops being evaluated, and the rest is cleaned up in the background, so you will not find a half-deleted fleet still sending alerts.

If you only want to stop paying attention to a fleet, pause it instead, or retire the instances you no longer run. Both keep the history.

Related

• Fleets for what a fleet is, creating one, and every configuration field.
• Metrics and rules for the rules that run on the numbers instances report.
• Error grouping for how failing output is fingerprinted into groups.
• Troubleshooting for problems that are not fleet-specific.