Testing¶

Datum changes operating systems as root, so a test suite that only proves the code compiles and the YAML parses proves very little. This page records what is tested where, what a supported distribution has been through, and what blocks a release.

The levels¶

The level a test belongs at is decided by what it needs, not by what it is about. A test of precedence ordering needs no host and is a unit test even though precedence is a fleet concept. A test that apt reports an installed version correctly needs apt and a package database, so it is an integration test even though it covers two lines of code.

Unit tests cover everything above the provider boundary, which is most of Datum. Document parsing, matcher evaluation, layer merging, precedence, conflict detection, graph construction, cycle detection, plan ordering and action selection are all pure functions from repository content and observed state to a plan, and all of them can be tested without a host existing.

That is a deliberate consequence of the architecture. Resolution does not read the host, so the half of Datum most likely to contain subtle logic errors is the half that needs no infrastructure to test.

Integration tests cover one provider against real tooling. The apt provider against a real apt, the File provider against a real filesystem, the systemd provider against a real systemd. Mocks are not used at this level, because the differences that matter between distributions are in command output formats and failure modes, which is exactly what a mock gets wrong.

System tests run complete passes against a host and assert on the outcome, the plan, and the state afterwards. This is where idempotence, drift correction, failure containment and verification are tested.

End-to-end tests involve a Git remote and more than one host, and cover the things that only exist at that scale, such as a commit reaching hosts, revision lag appearing in metrics, and a bad commit leaving hosts on their last known good.

What containers can and cannot test¶

Containers are cheap, fast, and genuinely sufficient for a large part of the matrix. They are also insufficient in ways that matter for exactly the resource types Datum manages.

The dividing line is whether the resource touches the kernel or the init system. Everything that is filesystem or package manager work is faithfully testable in a container, and everything that is kernel or PID 1 work is not.

Which runtime for what¶

systemd-nspawn is the useful middle. It gives a real systemd as PID 1 with a real dbus, which a plain container does not, at a fraction of a VM's cost, and it covers most Service behaviour including enablement across a container restart.

It does not cover reboots of the host kernel or anything in /proc/sys, so it narrows the VM set without removing it.

The matrix¶

Two axes, and they are not multiplied out in full.

Rocky Linux and AlmaLinux are represented by one RHEL-family row instead of three, because they differ from RHEL in branding and ID, not in package or service behaviour. If that assumption turns out to be wrong for a specific provider, that provider's tests gain the extra rows and the whole suite does not.

Distributions are tested against their actual released images. A minimal official image for each version, pulled and not built, because the differences that matter are in what the distribution ships by default, and a locally-built image tests the build instead of the distribution.

Architectures¶

amd64 and arm64. Both are ordinary targets now, and arm64 is where assumptions about paths, package naming and available versions tend to break.

Other architectures are not tested and therefore not supported. Running a full matrix on more would cost more than the coverage returns until somebody needs one, at which point that architecture gains rows instead of being assumed to work.

Testing specific behaviours¶

Idempotence is tested by running a pass twice and asserting the second plan is empty. That is the observable definition, not a proxy for it, and every provider gets the test for every action it supports. A provider that passes on the first run and produces a non-empty second plan has a normalisation bug, and this test is the only reliable way to find it.

Artificial drift is introduced by changing the host directly and running another pass. Delete the package, rewrite the file, stop the service, change the mode, then assert the plan contains exactly the expected action and that the pass converges. Drift is ordinary input, so a drift test is a normal pass against a modified host and nothing special.

Partial failures are produced by making one resource fail on purpose, usually by making a path unwritable, and asserting three things. The failing resource is failed, resources that require it are blocked, and unrelated resources still converge. That last assertion is the one that catches an over-eager abort.

Interrupted reconciliation is tested by killing the agent mid-pass and running another pass. The assertion is that the second pass converges, which is the property that matters, because there is no partially-applied plan to resume and the next pass observes the host and plans from what it finds. Killing the agent at different points, including between a staged write and its rename, is tested explicitly.

Agent crash recovery is the same test with the agent restarted by its supervisor instead of the harness, and it additionally asserts that the recorded last-known-good revision survived and that no lock or temporary file blocks the next pass. Leftover .datum.tmp files in target directories are the expected failure here.

An unavailable Git remote is tested by blocking access and asserting the host keeps reconciling its last known good, reports the failure to fetch, and continues correcting drift. The inverse test matters as much, and it asserts that the host advances once the remote returns.

Malformed desired state covers every row of the manifest validation table, and each asserts the same two things, which are that the error names the offending document and that the host was not touched. This suite is cheap, runs in containers, and protects the property that a bad commit costs nothing.

Package upgrades and repository failures need a controlled package repository, not the distribution's real one, so the suite hosts its own repository with two versions of a test package. That makes version pinning, upgrades, downgrades and a repository returning errors or timing out all testable without depending on the internet or on what upstream happens to be serving.

Reboots are tested in VMs by applying a change that requires one, asserting the host reports awaiting-reboot, rebooting, and asserting the state afterwards. The important assertion is that Datum did not reboot the machine itself.

Network configuration is tested in VMs with out-of-band access, so that a change which breaks networking does not disconnect the harness. A VM with a serial console or a second interface reserved for the harness makes a test that breaks connectivity survivable, which is not possible when the only access is the thing under test.

Privileged filesystem operations are tested in throwaway VMs or containers, never on a developer's machine or a shared runner. The path resolution tests deliberately create hostile conditions, including symlinks pointing at sensitive files and directories swapped mid-operation, and those tests are the ones that damage a host if the code under test is wrong.

Unsupported operations are tested by asserting the right refusal. A Service resource on Alpine with no OpenRC provider must produce skip with a reason and a degraded host, not a failure and not a silent success, and that assertion is as important as the ones covering things that work.

Provider conformance¶

Every provider passes a common conformance suite before it counts as existing.

The suite is written once against the provider interface and parameterised by provider, so adding a package provider means running the existing suite rather than writing new tests. It asserts the behaviour every provider owes regardless of what it manages.

That suite is what makes the provider boundary real and not aspirational. A second package provider passing the same tests as the first is the actual evidence that apt assumptions did not leak upward, which is why the implementation order puts a second package provider early.

What "supported" requires¶

A support matrix entry needs all of the following, on that distribution at that version, against its released image.

The provider passes the full conformance suite, exercising every action the resource type defines, and the idempotence test passes. Verification is shown to catch an introduced discrepancy, failures produce reported errors rather than partial changes, and any field the provider cannot observe is documented on the resource type page.

Anything short of that is not a support entry. Work in progress, a provider that mostly works, and a distribution somebody tried by hand all read as unsupported, which is the point of writing the criteria down before there is pressure to bend them.

What runs when¶

The split is by cost and by likelihood of catching something. A pull request runs everything fast enough to keep a contributor waiting for, which is most of the logic and one distribution end to end. The VM matrix runs nightly because it is slow and because the failures it finds are usually distribution-specific instead of introduced by the change under review.

Release gating is the full matrix passing. A release with a failing VM row ships a provider that does not work on a distribution the matrix claims, and the matrix is the only thing a user has to go on.

Keeping the matrix affordable¶

Six distributions, ten versions, two architectures and four test levels multiplies into something nobody will wait for and nothing will pay for.

Three things keep it bounded. Most tests are container tests, which are cheap enough that the distribution axis barely matters. VM tests run only for resources that need a kernel or an init system, which is a small subset. And the full matrix runs nightly and at release, not per commit, so the expensive axis is crossed on a schedule instead of on every push.