Testing¶
Datum changes operating systems as root. Its tests cover both the logic that decides what to change and providers running against real system tools. This page describes the checks that exist today and the coverage still to build. The support matrix records which providers and distributions are supported.
Running checks locally¶
From the repository root, with the Go version specified in go.mod:
make lint # Go formatting, vet and tests
make test-build # rebuild with two versions and check the binary reports each one
make test runs just go test ./.... For a focused change, run the affected package first, then
run the full suite before committing:
The ordinary Go suite runs on Linux and macOS. Tests that apply Linux state skip on other operating
systems; a successful macOS run therefore does not establish Linux provider coverage. With Docker
running, make test-linux runs the ordinary suite in a Linux container.
For documentation changes, run make docs-check. It installs the Python documentation toolchain
in .venv, builds the site in strict mode, and uses a Node container to parse Mermaid diagrams.
See working on the documentation for preview and build instructions.
Integration targets¶
These targets require Docker. Run the target for the provider or subsystem you changed. The
integration build tag enables tests that install packages, change accounts or modify system
state, so use the container targets instead of running the tagged suite on your workstation.
| Command | What it exercises |
|---|---|
make test-apt |
The apt provider against Debian's real package tools |
make test-dnf |
The dnf provider against Fedora 41's real package tools |
make test-user |
User and group operations against a real account database |
make test-sysctl |
Runtime kernel parameters and persistence files in a privileged container |
make test-systemd |
The systemd provider and agent service in a container booted with systemd as PID 1 |
make test-git |
Git operations and signature verification against real signed repositories |
make test-source |
Revision selection and fallback against a real signed repository |
make test-package |
Building and installing the Debian and RPM packages, then checking the installed files |
make test-prometheus |
Parsing the example Prometheus configuration and testing the alert rules with promtool |
The Debian suites use the golang:1.25 image, and the dnf suite uses fedora:41. Package installation
checks use debian:trixie and fedora:41. These are the images selected by the Makefile, not a claim
that every Debian or Fedora release has been tested. Package suites need access to their
repositories to install dependencies.
Container coverage and its limits¶
A container provides real package tools, account databases and filesystem operations. The systemd
suite additionally boots systemd as PID 1 with cgroup access; it is a privileged Docker container,
not an ordinary container running systemctl without a service manager.
The sysctl suite changes net.ipv4.ip_forward in the container's own network namespace. It checks
writes to procfs, invalid-value rejection and persistence-file contents. This provides useful
kernel interaction coverage, but does not establish that arbitrary sysctls are isolated from the
host or that persisted values survive a reboot. New sysctl tests must choose parameters with the
same care.
There is currently no VM test matrix. Host reboots, kernel upgrades, recovery after power loss and network changes that can disconnect a host need separate coverage. A container test cannot establish those behaviours.
What CI runs¶
Pull requests and pushes to main run the same Go workflow:
- Formatting,
go vetand the ordinary Go suite on Linux. - The ordinary Go suite on macOS.
- The build-version regression test and cross-compilation for Linux
amd64andarm64. - Debian and RPM package installation checks and Prometheus rule tests.
- All seven integration targets from
test-aptthroughtest-sourcein the table above.
The All checks job collects the Go workflow results for branch protection. A separate
documentation workflow parses Mermaid diagrams and runs a clean, strict site build. Successful
pushes to main also deploy the documentation.
Cross-compiling for both architectures checks that the binaries can be built. It does not run the provider suites on both architectures; integration tests use the runner's container architecture.
Tags starting with v trigger the release workflow. It reruns formatting, vet, the Go tests and all
seven integration suites, builds binaries and packages for both Linux architectures, installs and
checks packages for the runner's architecture, and checks the native binary's version. It then
signs checksums and publishes the release artefacts. There is no nightly workflow or additional VM
release gate.
The workflow files in .github/workflows/ and the root Makefile are the executable definitions of
these checks. Update this page when their coverage changes.
Writing useful regression tests¶
Start with the user-visible failure and make the test fail before changing the implementation. Parsing and resolution tests should assert the resolved result or the error returned for bad input. Filesystem tests should inspect the resulting files and check that unrelated paths remain untouched. Provider integration tests should assert real observed state after applying a change.
For providers, cover convergence with a second pass, deliberately introduced drift, failures and verification mismatches. Test unsupported operations as explicit refusals. Follow the provider safety rules when constructing hostile filesystem cases.
internal/provider/providertest supplies an in-memory provider for engine tests. It is not a shared
conformance suite against every real provider. A reusable conformance suite remains future work;
its useful requirements include observation without mutation, idempotence, truthful failure
reporting and explicit treatment of fields a provider cannot observe.
Keeping the matrix affordable¶
Broader distribution, architecture and VM coverage is planned, not currently running. Add rows in response to provider behaviour that needs testing, and keep fast parsing, resolution and engine tests independent of that infrastructure. Use containers where real userspace tools are enough; reserve VMs for behaviour that needs a host kernel, reboot or full machine lifecycle.
A future matrix should test named distribution versions and both supported architectures before extending the support claims. Nightly and release VM runs are possible ways to control cost, but neither is implemented.
Open question
Where the VM matrix runs is undecided. Hosted runners with nested virtualisation, dedicated hardware and disposable cloud instances have different cost and maintenance requirements. The choice will determine how frequently the broader matrix can run.