Run duration, overlap and completion budgets
Four questions about a job that "did it succeed" cannot answer: how long did it take, is it still running, did runs stack on top of each other, and did it run as often as it was supposed to. The last one is the quietest failure in monitoring. A nightly job that runs four nights in seven never triggers a single alert, because on the four nights it runs it succeeds, and on the three it does not, nothing happens at all.
Measuring a run
Ping /start when the work begins and the plain URL when it ends. The interval between them is the duration. This one change also enables max-duration timeouts and overlap detection, which have nothing to measure without it.
# /start before the work, the plain URL after it. # Without the /start there is no interval to measure. 0 2 * * * curl -fsS -m 10 https://pulsewatcher.up.railway.app/ping/<token>/start && \ /opt/etl.sh; \ curl -fsS -m 10 --retry 5 "https://pulsewatcher.up.railway.app/ping/<token>/exit/$?"
const base = 'https://pulsewatcher.up.railway.app/ping/<token>'
await fetch(`${base}/start`)
try {
await runEtl()
await fetch(base)
} catch (err) {
await fetch(`${base}/fail`, { method: 'POST', body: String(err.stack ?? err) })
}Confirm it worked: the monitor page shows a duration on the run and starts drawing the duration chart. Two or three runs in, the chart is worth reading.
A run that never finishes
Set max_duration_seconds and a run still open past it takes the monitor down, with no need to wait for the next scheduled ping. A job that hangs on a network read at 02:05 is caught at 02:35 rather than at 02:00 tomorrow.
Set it above the slowest legitimate run, not the average one. This is a hard timeout, not a judgement about speed.
A run that is getting slower
duration_regression_pct compares this run against the median of comparable past runs. It warns, and never pages: a job that took three times as long and finished is not an outage, and paging for it at 3am would be wrong.
🟠 Nightly ETL completed with issues reason: duration_regression Took 41m 12s. Baseline for a Tuesday is 8m 30s (384% over). https://pulsewatcher.vercel.app/app/monitors/<id>
Buckets work exactly as they do for metric baselines. A job that legitimately takes an hour on Sunday and four minutes on Tuesday wants day_of_week, or it will warn every Sunday forever.
Forecasting
Where the duration chart shows a consistent trend, the monitor page projects when the runtime will cross your maximum. A job growing 4% a week is fine today and will breach a one-hour timeout in eleven weeks, which is a thing worth knowing in week one rather than week eleven.
This is read-only. There is nothing to configure and no alert attached to it. A projection appears when there are at least 15 samples, the trend explains at least half the variance, and the crossing lands within 180 days. Otherwise nothing is shown, on the grounds that a forecast from noisy data is worse than no forecast.
Runs stacking up
If a run is still open when the next one starts, the job is taking longer than its interval. overlap_policy decides what that means:
| Policy | Meaning |
|---|---|
| ignore | Concurrent runs are normal for this job. The fleet default, because fleet agents are often long-running and legitimately overlap. |
| warn | The monitor default. Recorded and delivered on the warn schedule, never pages. |
| alert | Takes the monitor down. Right when two copies of this job running at once corrupts something. |
Three or more runs in flight at once is different, and goes down whatever the policy says. Two overlapping runs is a job that is a bit slow. Three is a job that is falling behind faster than it finishes, and the host is minutes from running out of memory.
🔴 Nightly ETL is down reason: run_pileup 3 runs of this job are in flight; the oldest started 22 minutes ago. https://pulsewatcher.vercel.app/app/monitors/<id>
Completion budgets
Everything above asks about runs that happened. A budget asks about the ones that did not. Set completion_budget_pct and PulseWatch divides the window into the times the job was scheduled to run, resolves each one, and tells you when the share that actually ran falls below your number.
🟠 Nightly ETL ran less often than its budget reason: completion_budget Ran 24 of 30 scheduled times over 30 days (80%, budget 95%). Missed: Aug 2, Aug 5, Aug 6, Aug 11, Aug 19, Aug 25. https://pulsewatcher.vercel.app/app/monitors/<id>
- Open the monitor and set a budget percentage and a window. 95% over 30 days is a reasonable start for a nightly job: it tolerates one bad night a month and not two.
- Leave it. A budget is a statement about a month, and it is evaluated on a slow cadence rather than on every ping.
- Confirm it is running: the monitor page shows the completion rate for the window, and the individual slots that were missed.
Two rules make the number honest. A late run counts as a run, because the budget asks whether the work happened and not whether it was punctual. Slots covered by maintenance, by a pause, or by a blocked upstream are excluded from both halves of the fraction, because the job was never expected to run then.
Below 5 scheduled runs no percentage is reported. Two runs out of three is 67% and means nothing.
Budgets on a fleet
A fleet budget is judged per instance and never averaged. 199 machines at 100% and one at 40% averages to 99.7%, which reads as healthy while that one machine drops three runs in five. The alert names the machines below budget rather than reporting a fleet-wide figure. The window there is capped at 90 days rather than 365, because fleet slots are computed rather than stored.
Output that changed shape
output_change_warn compares the log excerpt with the last successful run and warns when it differs. Timestamps, counts and ids are ignored, so a line reading wrote 48,210 rows in 4.1s does not warn every night for being a different number. A new line appearing, or an expected one vanishing, does.
Every field
| Field | Type | Default | Range | Effect |
|---|---|---|---|---|
| max_duration_seconds | int | null | 1 to 86,400 | A run still open after this long goes down. Needs a /start ping. |
| duration_regression_pct | number | null | 1 to 10,000 | Warn when a run takes this much longer than its baseline. The UI preset is 200, meaning three times as long. |
| duration_baseline_bucket | enum | none | none, day_of_week, weekday_weekend, hour_of_day | Which past runs the duration baseline compares against. |
| overlap_policy | enum | warn on a monitor, ignore on a fleet | ignore, warn, alert | What a run starting while another is open means. |
| completion_budget_pct | number | null | 0.1 to 100 | The share of scheduled runs that must actually have run. |
| budget_window_days | int | 30 | 1 to 365 on a monitor, 1 to 90 on a fleet | The period the budget is measured over. |
| output_change_warn | boolean | false | Warn when the log excerpt differs in shape from the last successful run. |
All of these are available on every plan. What differs by plan is how much history is kept to compute them against: 7 days on Free, 30 on Pro, 90 on Business.
Variations
A nightly ETL that must not hang
/start plus max_duration_seconds of 5400. If it is still going 90 minutes in, something is stuck.
A job whose runtime is creeping up
duration_regression_pct of 200 with the day_of_week bucket. Warns rather than pages, and the forecast on the monitor page tells you how long you have.
A job that must never run twice
overlap_policy of alert. Two concurrent copies writing the same table is worth waking someone for.
A weekly report that quietly stops
A completion budget of 90% over 90 days. Weekly jobs are where skipped runs hide longest, because eight weeks of history is only eight data points.
A scraper whose output should look the same every day
output_change_warn on. When the site changes its markup, the job still succeeds and the output stops looking like the output.
What can go wrong
No duration is ever recorded
Nothing is pinging /start. Without it there is no interval, and max duration and overlap detection have nothing to work with either.
A crashed run leaves a run open forever
A /start with no ending ping stays open. Set max_duration_seconds so it is closed and reported rather than sitting there. Report failure on the error path as well as success on the happy path.
Regression warnings every weekend
The baseline is pooling every run together. Set a bucket so weekends are compared with weekends.
The completion budget says nothing
Either there are fewer than 5 scheduled runs in the window, or the job really is running: a run that arrives 40 minutes late still counts. Punctuality is what grace and duration regression measure.
Output change warns constantly
The excerpt contains something that varies and is not a timestamp, a count or an id. Trim what you send to the part whose shape is meant to be stable.
Related
- Monitors for grace, which decides how late is late. Grace and a completion budget answer different questions about the same lateness.
- Metrics when the number that matters is what the job produced rather than how long it took.
- Fleets to apply all of this across many machines at once.
- States and reasons for what each slot status means and which reasons page.