Volume 4 — Cross-Platform System Management — contents
Volume 4 — Cross-Platform System Management
  1. Chapter 1 — Why a Contract Layer
  2. Chapter 2 — The 52 Cross-OS Contracts
  3. Chapter 3 — featurebase and the Per-OS Implementation Pattern
  4. Chapter 4 — RHEL Implementation
  5. Chapter 5 — Ubuntu / Debian Implementation
  6. Chapter 6 — macOS Implementation
  7. Chapter 7 — Windows Implementation
  8. Chapter 8 — UEFI Firmware Implementation
  9. Chapter 9 — Spec-Driven Development for System Administration
  10. Chapter 10 — Building Cross-OS Test Parity
Other volumes
References / The Code Craft AI books / Volume 4 — Cross-Platform System Management

Chapter 10 — Building Cross-OS Test Parity

Nine chapters walked the contracts, the implementations, and the conventions. This chapter is the volume's payoff: how the discipline turns into a green-across-OSes test campaign at scale. Thousands of tests; five OSes; one uniform shape; predictable outcomes.

The volume's earlier chapters established the machinery. This chapter walks the operational shape — what an SDET's day-to-day work looks like on a project that's adopted the sysman tier. The chapter is the volume's closer; by the end, a reader should know how to set up, run, and triage a cross-OS test campaign against the framework.

The reference numbers

The codecraftai-sysman-* packages collectively ship:

  • 5 reference implementations (RHEL, Ubuntu, macOS, Windows, UEFI).
  • ~245 manager classes (the per-OS package counts vary; the rough total is 5 × ~50 features).
  • ~270 test files combined (RHEL 53 + Ubuntu 41 + macOS 45 + Windows 45 + UEFI 175 — UEFI's count is disproportionate because of the 3-path dispatch matrix).
  • Several thousand individual test invocations — parametrised tests in pytest's sense, one test file containing many test functions.

The volume's outline mentioned "3,656 / 3,656 testbase tests passing." That count is test invocations, not test files. A single test file might have 30 test functions, each parametrised across 5–10 input combinations, totalling hundreds of individual assert calls.

The discipline of getting that count to pass everywhere across five OSes is what this chapter walks.

The setup

A cross-OS test campaign typically requires:

One CI runner per OS

The framework's tests aren't fully cross-host (you can't run a Windows test against a Linux runner without a Windows host somewhere). A typical setup:

  • One Linux runner for RHEL and Ubuntu tests.
  • One macOS runner for macOS tests.
  • One Windows runner for Windows tests.
  • The Linux runner usually doubles as the UEFI test runner, using override_host_os (Convention 10) to exercise the Windows and macOS dispatch paths.

Volume 2 chapter 3's Pod model applies — each runner is a Pod; the test campaign is more Pods, not more parallel tests within a Pod.

Pinned tool versions

dnf 4.x produces different output than dnf 5.x. winget 1.0 differs from 1.5. The test discipline pins tool versions per OS (the per-OS spec documents the supported range). When a CI runner gets an unexpected tool version, the parser tests catch it before the broader test suite runs.

The Aspects defaults

Per Volume 5, every sysman manager runs commands through an ICommandRunner that takes Aspects. Test code typically uses defaults, but slow OSes (macOS waiting for softwareupdate, Windows waiting for service start) need longer timeouts:

SLOW_ASPECTS = AspectsCmd(
    completion_timeout=300,    # 5 minutes for slow operations
    completion_interval=15,
    action_pattern=ActionPattern.DO_UNTIL_SUCCESS,
)

result = service_mgr.start("nginx", aspects=SLOW_ASPECTS)

Tests that exercise long-running operations override the defaults. Tests that exercise quick operations use the short defaults and fail fast on hangs.

A worked test scenario

A test that exercises the same feature on three OSes:

import pytest

from ccai.sysman.linux.redhat.packagemanagement import PackageManager as RhelPM
from ccai.sysman.linux.ubuntu.packagemanagement import UbuntuPackageManager
from ccai.sysman.macos.packagemanagement import MacosPackageManager

import os

OS = os.environ.get("TARGET_OS", "linux-redhat").lower()


@pytest.fixture
def package_mgr(runner):
    if OS == "linux-redhat":
        return RhelPM(runner=runner, config=None)
    elif OS == "linux-ubuntu":
        return UbuntuPackageManager(runner=runner, config=None)
    elif OS == "macos":
        return MacosPackageManager(runner=runner, config=brew_config())
    else:
        pytest.skip(f"Unsupported OS for this test: {OS}")


def test_install_nginx_idempotent(package_mgr):
    """
        Verify nginx installs idempotently across all supported OSes.
    """
    # Initial install — may already be present.
    first = package_mgr.install("nginx")
    assert first.status in (
        # contract-specified outcomes
        type(first.status).SUCCESS,
        type(first.status).SKIPPED,
    )

    # Second install — must be SKIPPED per contract.
    second = package_mgr.install("nginx")
    assert second.status == type(second.status).SKIPPED

Three things to notice:

  1. Per-OS manager selection. The fixture picks the right manager class based on TARGET_OS. The test body doesn't care which.
  2. Test against the contract. The assertions check status in (SUCCESS, SKIPPED) — the contract's vocabulary, not the OS's.
  3. type(first.status) — the status enum is per-OS (PackageActionStatus for RHEL, UbuntuPackageActionStatus for Ubuntu, etc.), but they all have the same value names. The test uses type(...) to look up the enum dynamically.

The test runs unchanged on three CI runners. Each runner sets TARGET_OS differently. The test's assertions are the contract's promise.

What gets tested in a contract suite

A representative test suite for one feature (e.g., PackageManager) covers:

Command-construction tests

For each write verb, assert the constructed argv matches the expected pattern:

def test_install_constructs_dnf_argv():
    config = FeatureConfig(dry_run=True)
    mgr = RhelPM(runner=mock_runner, config=config)

    result = mgr.install("nginx", version="1.21.6")

    assert result.command == ["dnf", "install", "-y", "nginx-1.21.6"]
    assert mock_runner.last_call is None  # dry-run; runner not invoked

Multiplied across every write verb across every feature across every OS, this is hundreds of tests. Each is small and fast (no real subprocess calls).

Idempotency tests

For each write verb that mutates state, verify a second call returns SKIPPED:

def test_install_skipped_when_already_installed(monkeypatch):
    mock_runner.queue_query_response(installed_version="1.21.6")
    mgr = RhelPM(runner=mock_runner, config=...)

    result = mgr.install("nginx", version="1.21.6")

    assert result.status == PackageActionStatus.SKIPPED
    assert mock_runner.last_install_call is None  # no real install attempted

The test mocks the query response (saying nginx 1.21.6 is installed), then asserts that calling install doesn't result in a runner call. The idempotency check passed.

Dry-run tests

For each write verb, verify dry_run=True skips the runner:

def test_install_dry_run(mock_runner):
    config = FeatureConfig(dry_run=True)
    mgr = RhelPM(runner=mock_runner, config=config)

    result = mgr.install("nginx")

    assert result.status == PackageActionStatus.DRY_RUN
    assert mock_runner.calls == []  # no calls made
    assert result.command == ["dnf", "install", "-y", "nginx"]

Elevation tests

Verify writes elevate (when elevated=True) and reads don't:

def test_install_elevates(mock_runner):
    config = FeatureConfig(elevated=True)
    mgr = RhelPM(runner=mock_runner, config=config)

    mgr.install("nginx")

    last_call = mock_runner.calls[-1]
    assert last_call.elevated is True


def test_query_does_not_elevate(mock_runner):
    config = FeatureConfig(elevated=True)  # caller asks for elevation
    mgr = RhelPM(runner=mock_runner, config=config)

    mgr.query("nginx")

    last_call = mock_runner.calls[-1]
    assert last_call.elevated is False  # but reads ignore it

Output-parsing tests

Verify query results parse correctly from representative output:

def test_list_installed_parses_dnf_output(mock_runner):
    mock_runner.queue_response(stdout=DNF_LIST_OUTPUT_FIXTURE)
    mgr = RhelPM(runner=mock_runner, config=...)

    packages = mgr.list_installed()

    assert len(packages) == 247
    nginx = next(p for p in packages if p.name == "nginx")
    assert nginx.version == "1.21.6"

Fixtures (DNF_LIST_OUTPUT_FIXTURE) are real captured output from a representative system. They're checked into the test suite.

Error-handling tests

Verify the manager handles failures correctly:

def test_install_failed_when_package_doesnt_exist(mock_runner):
    mock_runner.queue_response(exit_code=1, stderr="No match for argument: bogus")
    mgr = RhelPM(runner=mock_runner, config=...)

    result = mgr.install("bogus-package")

    assert result.status == PackageActionStatus.FAILED
    assert result.exit_code == 1
    assert "No match" in result.stderr

How the volume of tests is manageable

A reader doing the math sees the test count is large. ~50 features × 8 minimum-required tests per feature × 5 OSes = 2000 tests minimum. Real packages have more (each verb has command-construction + idempotency + dry-run + elevation + output-parsing tests).

Three things make this manageable:

The conventions are enforced by tests indirectly

A manager that follows the conventions passes the tests by construction. A manager that violates a convention fails a test. Engineers don't write per-convention tests; they write per-feature tests, and the conventions surface in the patterns.

Mock runners run fast

A test that uses a mock runner doesn't invoke dnf or apt-get or Get-Service. The test runs in milliseconds. A package's full unit-test suite typically completes in under a minute even at 500+ tests.

Test parameters are reusable

Many tests are parametrised across feature variants. A single test function plus pytest's parametrize runs the same logic against five different verbs, ten different input combinations, three different status outcomes — one test source becomes dozens of test invocations.

Integration tests against real OSes

The unit tests use mock runners. For integration testing — actually invoking dnf / apt-get / brew / winget — the framework's discipline is:

  • Run integration tests separately from unit tests.
  • Run them periodically (nightly), not per-PR.
  • Use real CI runners (RHEL VM, Ubuntu VM, macOS host, Windows VM).
  • Treat integration-test failures as infrastructure issues — the test logic is unit-tested; the integration test verifies the OS hasn't changed underneath.

This is consistent with Volume 1 chapter 1's discipline: unit tests verify the framework's logic; integration tests verify the system under test's behaviour. Both are needed; they have different cadences and different failure implications.

Triaging a cross-OS failure

A failure pattern an SDET runs into: "the test passes on RHEL and Ubuntu but fails on macOS." Triage steps:

Step 1 — Read the failure record

The failed test's *ActionResult carries:

  • status — FAILED.
  • command — the argv that ran (Convention 2).
  • stdout / stderr — what the tool produced.
  • exit_code — the tool's exit.
  • detail — a stable token if Convention 5 applies.

A common failure: macOS's result.detail == "unsupported_arbitrary_capsule_file_macos". The test was trying to apply a UEFI capsule via apply_capsule_file; macOS doesn't support arbitrary capsule files. The test should skip on macOS, not fail.

Step 2 — Check Convention 5 tokens

A failure with a stable unsupported_* token usually indicates the test should skip on that OS:

result = capsule_mgr.apply_capsule_file(path)
if result.status == ActionStatus.FAILED and result.detail.startswith("unsupported_"):
    pytest.skip(f"Feature unsupported on this OS: {result.detail}")
assert result.status == ActionStatus.SUCCESS

The test code becomes more sophisticated — it checks for the stable token and skips gracefully. Multiplied across the test suite, this discipline keeps the test count high (comprehensive coverage) without forcing every test to pass on every OS.

Step 3 — Check Convention 6 (Windows elevation)

If the failure is on Windows with result.elevation_blocked == True, the test was running under unelevated PowerShell. Either:

  • Run the test under elevated PowerShell.
  • Skip the test when not elevated.
result = mgr.install("nginx")
if getattr(result, "elevation_blocked", False):
    pytest.skip("This test requires elevated PowerShell.")

Step 4 — Check macOS elevation inversion

If the failure is on macOS with brew errors mentioning "running as root," the test wasn't configured with brew_user. Convention 11 requires the operator to set this.

config = MacosPackageManagerConfig(
    brew_user=os.environ.get("BREW_USER", "ci-runner"),
)
mgr = MacosPackageManager(runner=runner, config=config)

Step 5 — Check the per-OS spec for known limitations

Each per-OS spec's "Implementation Drift" section documents acknowledged divergences. A test failing because of a documented divergence is a test bug (didn't account for the divergence), not a manager bug.

What this discipline produces

The volume's earlier chapters described a discipline. This chapter shows what the discipline produces:

Predictable test failures

Failures fall into three buckets:

  1. Real bugs in the manager — fix the manager.
  2. Tool / OS-version drift — fix the parser or update the supported-version range.
  3. Convention misuse — fix the test (skip on unsupported, configure brew_user, etc.).

There's no fourth bucket of "I have no idea why this failed." The framework's structured output (Convention 2, Convention 5, Convention 6) gives the failed test enough information to triage.

Reproducible cross-OS test runs

Same test code, same expected outcomes (modulo OS-specific skips), green across all supported OSes. An SDET who writes a feature test against the contract gets cross-OS coverage for free — the contract's verbs, status vocabulary, and conventions take care of the per-OS variation.

Maintainable scale

50 features × 5 OSes is a lot of code. The discipline makes it maintainable:

  • The contracts document the cross-OS API.
  • The conventions document the cross-OS shape.
  • The tests verify the conventions are followed.
  • Code review catches convention violations.

A new engineer onboarding to the sysman tier reads the conventions, picks a feature, follows the patterns. No deep tribal knowledge required.

End of Volume 4

The volume started by asking why a contract layer exists (chapter 1) and ended by walking a green-across-OSes test campaign (this chapter).

The five reference implementations (RHEL, Ubuntu, macOS, Windows, UEFI) demonstrate the discipline across genuinely different operating systems. Each implementation has its quirks (Homebrew elevation, Windows PowerShell, UEFI single- tier dispatch), and the contract layer absorbs them without forcing test code to know about them.

The chapters that come next are in other volumes. Volume 2's distributed automation invokes sysman feature execution through bridge tasks. Volume 3's orchestration triggers sysman campaigns. Volume 5's interop layer is the wire that sysman managers use. Volume 6's landscape-and-topology is the environment sysman operates against.

Volume 4's contribution to the series is the cross-OS discipline — the proof that genuinely different operating systems can be automated through a single contract layer when the layer is honest about divergences (Convention 5, Convention 11) and uniform in shape (Conventions 2, 4, 6, 7). Where Volume 5 said "talk to remote systems consistently," Volume 4 says "ask remote systems to do things consistently."

View this page as Markdown