Configuration validation¶

Writing a syntactically invalid sshd_config and restarting sshd locks everybody out of a machine. Most applications can check their own configuration, and Datum uses that check before the change goes live.

Validating configuration content is a separate thing from checking that a repository is well formed, which is manifest validation and happens at a different stage. This page always means the former.

Validation sits inside apply¶

The five phases apply to every resource. Configuration validation applies only to resources whose content some application can check, which is a minority, so it is a step within apply rather than a sixth phase.

Observe  ->  Diff  ->  Plan  ->  Apply  ->  Verify
                                 │
                                 ├── stage
                                 ├── validate
                                 └── activate

It sits inside apply because it operates on content that has been staged and not yet activated, which is a state that only exists during apply and only for one resource at a time.

Staging already makes this possible¶

The File provider already writes to a temporary file in the target directory and renames it into place, because a rename is atomic and a partial write is not.

That existing mechanism is what validation needs. Between the write and the rename there is a complete copy of the proposed content on the same filesystem, at a path the validator can be pointed at, while the live file is untouched.

write staged content      /etc/ssh/.datum.tmp.a91f
set ownership and mode
validate                  sshd -t -f /etc/ssh/.datum.tmp.a91f
rename into place         /etc/ssh/sshd_config

Validation failing means the rename never happens, the temporary file is removed, and the live configuration is as it was. A failed validation changes nothing on the host.

Validators are referenced by name¶

A validator is referenced by name.

datum: v1alpha1
type: File

name: sshd-config

desired:
  path: /etc/ssh/sshd_config
  owner: root
  group: root
  mode: "0600"
  source: files/sshd_config
  validate: sshd

sshd names a validator definition that a provider supplies. The repository does not say what command to run, what arguments to pass, or where the binary is.

A field holding a command line is arbitrary root execution driven by repository content, which ADR-0011 rules out. A validator is the most likely route by which that capability would have arrived without being noticed.

Nothing is validated implicitly. A File at /etc/nginx/nginx.conf does not acquire nginx -t from its path. Behaviour inferred from a path needs internal knowledge to predict, which the understandable principle rules out, and opting in is one field.

Validators Datum expects to ship¶

The nginx row is the general problem, not an nginx quirk. An application whose configuration is one file validates cleanly in isolation, and an application whose configuration is a directory of includes does not, because validating one fragment says nothing about the whole.

Configurations spanning several files¶

This is the part the design does not fully solve.

A validator pointed at one staged fragment of a multi-file configuration either ignores the rest, which makes the check meaningless, or reads the live copies of the rest, which validates a combination that will never exist because the other fragments are about to change too.

The position taken is to validate the assembled result after writing and before activating.

write every changed file in the group
validate the application configuration as a whole
reload or restart only if validation passed

A service that is not reloaded keeps running the configuration it already loaded, so a failed validation leaves files on disk that are wrong and a service that is unaffected. The host is then one whose configuration will be wrong after its next restart, and the service is still serving in the meantime.

When a validator is missing¶

A named validator whose binary is not installed on the host is a failure, not a skip.

Proceeding without it would mean validation stopping on the hosts where the tooling is absent, and nothing saying so. A repository asking for validation gets validation or gets an error.

The usual cause is a missing dependency, and the fix is an ordering edge to the package providing the validator, which the manifest can already express.

Validators run the binary that is installed¶

A validator invokes the application's own tooling as it exists on that host, so the content is checked against the version that is going to consume it rather than against whatever version the repository was written for.

This is what makes validation useful. A directive accepted by nginx 1.24 and removed in 1.26 is caught on the hosts running 1.26 and nowhere else, which is the set of hosts it affects. Checking in CI against a fixed version does not reproduce it.

The same property is a hazard on a fleet that is not uniform. A change validated successfully on every host running one version and rejected on the hosts running another produces a partial rollout, where some hosts converged and some are failed, and the repository content is identical across all of them.

web-001   nginx 1.24   converged
web-002   nginx 1.24   converged
web-003   nginx 1.26   failed      validation rejected an unknown directive

That outcome is correct, and it is where somebody has to decide what the fleet should run. Diagnosing it depends on the failure report naming the validator and the host rather than reporting a generic validation error, which is why validator output is captured.

Version differences between hosts are expressible the same way any other difference is, through a layer whose matcher narrows to the hosts running each version. That costs a declared label per host and it is the only mechanism that keeps the difference visible in the repository.

Validators must not have side effects¶

A validator reads the staged content and reports whether it is acceptable. It does not write, does not restart anything, and does not touch the live configuration.

This is a requirement on the validator definitions Datum ships rather than something Datum can enforce, since it cannot inspect what a third-party binary does. A validator that modified state would break the guarantee that a failed apply changed nothing, which is what validation before the rename is for.

Privileges and output¶

Validators run as the agent does, which is root. A configuration file at mode 0600 owned by root cannot be read by anything less. This is also why validator definitions are provider-supplied code rather than repository content.

Validator output is captured and reported on failure, and the output is treated as potentially sensitive. A validator that echoes the offending line of a configuration file will echo whatever that line contains, which for a file holding a credential means the credential lands in the failure report.

Output from a validator on a resource marked sensitive is suppressed, and only the fact of failure and the validator's exit status are reported. For everything else the output is included because diagnosing a syntax error without it is guesswork.

Validation against verification¶

The two are separate mechanisms answering separate questions.

Validation is optional, narrow, and prevents a class of failure. Verification is universal, mandatory, and detects failures of every class including the ones validation cannot anticipate.

A configuration can validate and still be wrong, because sshd -t checks syntax and not whether the resulting policy is sensible, so validation passing is never taken as evidence that the change was correct. Verification is what establishes that the host reached the state that was asked for, and convergence is defined by it.

The combinations are as follows.

In all three the resource is failed, dependents are blocked, independent resources continue, and the next pass observes and tries again. Datum attempts no recovery of its own, since there is no rollback, and a recovery path for the validation case alone would cover the least damaging failure of the three.

Verification can be stronger than field comparison¶

Verification re-reads a resource and confirms it holds the state that was asked for, which for most resources means comparing the same fields the diff compared. A provider is permitted to check more than that where it knows something the field set does not express.

File[sshd-config]     fields match, and the staged content validated
Service[sshd]         active, enabled, and the unit reports no failed reload
Package[nginx]        installed at the requested version, and the package database is consistent

A Service provider confirming that a unit is active, not merely that systemd accepted the start request, is the clearest example. The field says state: running and the useful check is whether the process stayed up, which are different questions on any service that exits shortly after starting, and how long a provider should wait before deciding is undecided.

The limit on this is that a stronger check has to be a check on the same thing. A provider may read more of the host to establish that the resource reached its declared state, and it may not extend verification into a claim about whether the resulting system works, which is the subject of the next section.

Verification is not a health check¶

Verification establishes that a host reached the state the repository described. It does not establish that the software on that host is functioning, and that line is where Datum's responsibility ends rather than a gap to be filled later.

Datum verifies      nginx is installed, its configuration matches, its unit is active
Datum does not      the site returns 200, certificates are valid, latency is acceptable

An HTTP check would be the obvious addition and it is declined. A resource whose verified state depends on a response from a network service is a resource whose convergence depends on something outside the host, so a downstream outage would report as configuration drift, and a pass would fail for a reason no change to the repository could fix.

The schedule is the deeper problem. Datum runs periodically and reports what it found during a pass, and a health check is only useful continuously, so one on a reconciliation schedule reports that a service was responding at some point in the last interval. That is not monitoring, and building it would produce something that looks like monitoring closely enough to be relied on.

Where Datum's responsibility ends¶

The boundary is stated here so that the rest of the site can rely on it.

The first two rows are the whole of Datum's claim. Everything below them is a different discipline with different tooling, different data retention and a different response time, and the reason metrics are exposed over HTTP is to make Datum a source of data for that tooling rather than a substitute for it.

The third row is the one with a catch. Datum cannot report that its own agent has stopped, since a stopped agent reports nothing, so the signal that a host is no longer being managed has to come from something outside it noticing an absence. That is staleness alerting, and it is the one part of monitoring a Datum deployment cannot do without.