Alerting¶

Which conditions should page somebody, and which are normal operation. A reconciliation system produces a large amount of routine activity that resembles a fault, so the distinction is set out explicitly.

The conditions below are written out as Prometheus rules in examples/prometheus, along with a scrape configuration and a promtool test that asserts each one fires on the series the agent exports.

A host that stops reconciling is the failure to catch¶

The other alerts here fire on a reported problem. This one fires on the absence of a report, which makes it the one most easily omitted.

It has two distinct shapes, and a fleet needs both.

The agent has exited. The endpoint stops answering and the scrape fails.

up{job="datum"} == 0

Immediate, needs no threshold, and needs no assumption about the reconciliation interval. This is the main practical argument for serving metrics from the agent rather than writing a file, because a file left behind by a dead agent keeps scraping successfully.

The agent is alive and not converging. The process is up, the endpoint answers, and passes are failing or wedged.

time() - datum_pass_last_success_timestamp_seconds > 7200

up does not cover this case, since the agent is running and answering scrapes. The threshold is a multiple of the reconciliation interval, not an absolute figure. Two hours is four missed passes at the default of thirty minutes, which tolerates a transient failure and a slow pass while still catching a genuine stall.

This relies on metrics being absolute timestamps and not elapsed times, and it is the only staleness signal available to a fleet using the textfile mechanism instead of the endpoint.

Both alerts need pairing with whatever already detects an unreachable machine, since a host that is entirely down produces up == 0 for every job on it and is not specifically a Datum problem.

Alerts worth having¶

The last four are different in kind from the ones above them, because they fire on a control instead of a failure. A host refusing revisions is behaving as specified, and the reason to page on it is that the host has stopped receiving changes, which has the same effect as an agent that has stopped running.

The two mode="none" and baseline_present == 0 conditions are better treated as an inventory query than an alert on most fleets, because they are steady states, not events. Both indicate a control that is switched off, so they are listed here even though they are steady states.

A commit that fails to resolve trips the bad-revision alert on every host the change matched. Group the alert by revision rather than by host, or one bad commit produces one page per host.

Alerts not worth having¶

Individual resource changes. A resource being updated is Datum working. Alerting on changed passes means alerting on every deployment, and the signal is drowned within a day.

A single failed pass. Transient failures happen, a package manager lock being the common cause, and the next pass retries. Alerting on a sustained failure rather than an instantaneous one is what the staleness and failing conditions above already do.

Drift itself, on an enforcing host. Drift is ordinary input that the next pass corrects. A host that drifts and converges is behaving correctly, and the interesting case is a resource that drifts repeatedly, which is a different question Datum does not yet answer.

Drift on an observing host, by default. A host in observe mode is expected to report drift, since nothing is correcting it. During adoption the useful signal is whether drift is growing, so alerting on its presence produces a permanent alarm.

What to put on a dashboard¶

The fleet view answers how many hosts are in each state, how far behind the slowest host is, and how many hosts are on a last-known-good revision instead of the newest.

hosts by state            converged 486  degraded 9  failed 3  awaiting-reboot 2
worst revision lag        4h 12m
hosts behind newest       11
passes failing            3

The per-host view is where somebody lands after an alert, and it needs the host's state, its revision and last-known-good, the timestamps of its last attempt and last success, and its resource counts by state. That is exactly the status model, which is why status is designed as a model and not as console output.

What these alerts cannot detect¶

Every alert here fires on what a host reports about itself, so none of them detect a host that has been compromised and is reporting healthy. A machine under an attacker's control can emit a perfect set of converged metrics indefinitely.

Observability catches hosts that are broken, behind, or silent. It does not establish that a host is in the state it claims, and the threat model is explicit that reporting is not attestation. Closing that would need something the host cannot forge, which is a different problem from monitoring and is recorded as an open question rather than solved here.