Chapter 9 — Building Tests Around Landscape Concepts (Single-Host Edition)

A test framework that knows about the landscape becomes a distributed test framework. A test framework that does not know about the landscape stays a unit test framework. The hard part is the third option: a framework that is aware the landscape exists, ships a contract for the landscape to implement, but stays usable on a single host without a landscape at all. This chapter is about how testbase walks that line.

The previous eight chapters of Volume 1 walked testbase as a single-host test framework. Tests are functions, fixtures are originators, results stream to a JSOS file, mocks stand in where real systems would be too expensive. Everything in those chapters works on an SDET's laptop with no special infrastructure.

This chapter is the bridge. It carries the contract that a distributed test framework must implement to extend testbase — the one Volume 2 walks in detail. But it carries it from inside the single-host frame: what does a single-host test look like that would, if the landscape were available, exercise multi-host behaviour? And how does testbase let a single-host author write that test without the landscape machinery being a hard dependency?

Vocabulary: Landscape, Topology, Runtime, Pod

Before going further, four words need precise meanings. Readers who have used pytest fixtures or unittest.mock will recognise none of these as testing-framework vocabulary — they belong to the automation vocabulary, and conflating them with each other is the most common mistake the rest of this chapter exists to prevent.

Landscape answers the question what resources exist and how are they physically connected? A Landscape file enumerates the devices, services, and other resources the suite has access to — their addresses, their protocols, their credentials, the network they live on. It is the physical inventory + wiring.

Topology answers the question how are those resources logically organised, and what roles do they play together? A Topology file says things like "Players A, B, C form a Home Sound System" or "Nodes A, B, C are cluster members and Node D is a spare." It is the functional grouping + role assignment.

The two are distinct. The same Landscape can underpin many Topologies (one fleet of devices, organised differently for different test campaigns). The same Topology pattern can apply to many Landscapes (the cluster-with-spare shape, applied to different fleets). In codecraftai-configuration's loader, they live in separate MergeMaps — LANDSCAPE_CONFIGURATION_MAP and TOPOLOGY_CONFIGURATION_MAP — exactly because they are independent inputs.

Runtime answers the question how should the suite execute? A Runtime is a behavioural / diagnostic profile — Normal, Debug, Diagnostics, UPNP Diagnostics, Bluetooth Diagnostics. Runtimes influence how loud the logging is, what extra state to capture, what protocol traces to enable. They do not influence what resources exist (that is the Landscape) or how those resources are organised (that is the Topology).

Pod is the runtime instance produced by combining a Landscape with a Topology. A Pod is what a test job actually runs against — a concrete set of resources organised concretely. Pods are the unit of horizontal scaling at the CI layer (more on that below).

There is no codecraftai-topology package. Topology is a configuration category, not a package. The shape of a topology file beyond a small canonical convention (a clusters section with name / group / nodes / spares keys consumed by LandscapeOperationalLayer._create_clusters in codecraftai-landscaping) is left to the user to implement. The framework loads the file; the user decides what's in it.

A single-host landscape is the special case where the "landscape" is one machine — possibly the SDET's laptop, possibly a dedicated CI runner — and the structured description collapses to "this host." The interesting question is: how does a test express that it needs landscape concepts without requiring the landscape package?

The contract: testbase's fire_* hooks

The contract sits on TestJob's five fire_* lifecycle hooks (Chapter 2):

• fire_attach_to_framework(tseq) — register integration couplings with the test framework.
• fire_attach_to_environment(tseq) — establish connectivity to the environment.
• fire_activate_configuration() — activate the run's configuration profile.
• fire_activate_integration() — bring up cross-host integration.
• fire_activate_operations() — start side processes that other tests will rely on.

On a single-host run using TestJob directly, the last three are no-op stubs (codecraftai-testbase/source/packages/ccai/testbase/testjob.py:399–409) — they only log a debug line and return. Volume 2's InteropLibraryTestJob (codecraftai-interop/source/packages/ccai/interop/testing/testjob.py:22) overrides each of them to drive landscape activation:

• fire_activate_configuration() calls landscape.activate_configuration() (line 114).
• fire_activate_integration() calls landscape.activate_integration() (line 124).
• fire_activate_operations() calls landscape.activate_operations() (line 134).

The hooks are the contract. A single-host test never notices they exist; a distributed test relies on them; both use the same TestJob lifecycle.

A new class sits in the chain — but not on these hooks. Between TestJob and InteropLibraryTestJob there is now an intermediate base class, IntegratedTestJob (codecraftai-integration-runtime/source/packages/ccai/integrationruntime/integratedtestjob.py), introduced by the integration-runtime package. It is worth naming here so the chain is not surprising in Volume 2, but it is important to be precise about what it adds: IntegratedTestJob is about result recording, not landscape behaviour. It overrides exactly three provider hooks — _create_output_context(), _create_run_metadata(), and _create_recorder() — to route results into the activated app's presence folder, stamp them with CI/build metadata from RuntimeVariables, and render the HTML summary. It does not touch the five fire_* landscape hooks; those stay as no-op stubs all the way down to InteropLibraryTestJob, which is still where landscape activation is wired. (See Chapter 5 for the recorder/output-context seams and Volume 2 for the full inheritance cascade.) At the time of writing, the interop tier's re-parent onto IntegratedTestJob is still in flight: InteropLibraryTestJob currently extends TestJob directly (codecraftai-interop/source/packages/ccai/interop/testing/testjob.py:22), so a distributed run does not yet inherit the integration result-recording behaviour through this chain.

What lives at the boundary

Three concepts straddle the single-host / distributed boundary. A single-host author writes tests against them in shapes that work on either side.

1. Couplings with establish_connectivity

The IntegrationCoupling base class (codecraftai-testbase/source/packages/ccai/testbase/coupling/integrationcoupling.py:512) exposes a method:

def establish_connectivity(self,
                           allow_missing_devices: bool = False) -> None:
    """
        Bring the coupling online.

        :param allow_missing_devices: When True, the coupling reports
            but does not raise when one or more declared devices are
            unreachable.  Used in single-host smoke runs where the
            full device set is not present.
    """
    ...

The allow_missing_devices parameter is the boundary mechanism. On a distributed run with all devices present, the default False is correct: a missing device is a hard failure. On a single-host run where the test author wants to verify the coupling's graceful-degradation path, allow_missing_devices=True lets the coupling come online with a partial device set.

This is the Aspects pattern from Volume 0 chapter 4 once more: the behaviour parameter is a parameter, not a decorator. The single-host test author and the distributed test author write the same coupling code — they pass different values for the parameter.

2. Mock-yielding originators

A test that needs landscape-shaped resources but runs on a single host can attach a mock originator (Chapter 7):

@testbase.factories.resource_factory(scope=ResourceLifespan.Test)
def device_pool_mock():
    """
        Yields a Test-scoped mock of the device pool for single-host
        tests.
    """
    pool = Mock(spec=DevicePool, mock_type=MockType.Instance)
    pool.list_devices = SideEffect(lambda: [])
    pool.acquire = SideEffect(lambda name: Mock(spec=Device,
                                                 mock_type=MockType.Instance))
    yield pool
    return

A test attaches device_pool_mock exactly as it would attach the real device pool. On a single-host run, the mock is what the test sees; on a distributed run, swap the originator for the real one (device_pool_real) and the test runs unchanged.

The decision of which originator to attach is best made at the package level — a single_host_factories.py module that exports mock originators for unit tests, a distributed_factories.py module that exports real originators for the integration suite.

3. Landscape-conditional test markers

A test that only makes sense with a real landscape can be marked:

@testbase.skip_if(lambda: not is_distributed_landscape())
def test_replication_lag_is_under_threshold():
    """
        Verify replication lag across the cluster.
    """
    ...

The is_distributed_landscape callable runs at discovery time (per Chapter 2's discovery / collection split). On a single host the callable returns False; the test is skipped before the landscape package is even imported. On a distributed landscape it returns True and the test runs.

The trick: the callable cannot import the landscape package at module top-level. A safe shape is:

def is_distributed_landscape() -> bool:
    """
        Return True if the running framework has a distributed
        landscape attached.  Imports landscape lazily to avoid
        single-host import failures.
    """
    try:
        from ccai.interop.testing.testjob import InteropLibraryTestJob
    except ImportError:
        rtnval = False
        return rtnval

    rtnval = isinstance(testbase.current_job, InteropLibraryTestJob)
    return rtnval

The try / except ImportError here is not swallowing an exception in the Volume 0 chapter 3 sense — it is a sentinel import-time check. The function returns False if the distributed framework is not installed; the function returns False if the running job is not a distributed job; in both cases the test is skipped. No landscape package is required to evaluate the predicate.

The single-host test that would also run distributed

A test author who wants their test to run on either side writes it to depend on the coupling not the landscape:

@testbase.attach.parameter(some_billing_service, identifier="svc")
def test_invoice_round_trips(svc):
    """
        Verify an invoice round-trips through the billing service.

        Runs identically on single-host (with svc as a coupling-mock)
        and distributed (with svc as a coupling to the real billing
        service).
    """
    invoice = svc.create_invoice(amount=Decimal("100.00"))
    looked_up = svc.get_invoice(invoice.id)
    rtnval = (looked_up.amount == invoice.amount)
    assert rtnval
    return

The originator some_billing_service decides what svc actually is. The test does not know. The test depends on the coupling's interface — Volume 0 chapter 4's contracts.

This is the shape that maximises reuse. The same test runs as a unit test (mock coupling), as a single-host integration test (real billing service on the same host), and as a distributed test (real billing service on another host) — by choosing the originator appropriately.

Landscape-aware fire_* on the distributed side

Just so the contract is concrete: here is what InteropLibraryTestJob's overrides look like (forward reference to Volume 2):

# codecraftai-interop/source/packages/ccai/interop/testing/testjob.py:88

def fire_attach_to_environment(self, tseq) -> None:
    """
        Attach the sequencer to the environment, providing the
        landscape as the connection vehicle.
    """
    landscape = self._get_landscape()
    tseq.attach_to_environment(landscape=landscape)
    return

The landscape gets passed in as a kwarg to the sequencer's attach_to_environment. The sequencer accepts the landscape via **kwargs; on a single-host run no landscape is passed and the sequencer's environment attachment runs without one. On a distributed run the landscape is the vehicle through which couplings discover devices, fetch credentials, and schedule cross-host activity.

The single-host author does not write the override. The single-host author writes tests; testbase's fire_* stubs fire as no-ops; the run completes. The distributed author inherits InteropLibraryTestJob and gets the overrides for free.

A pragmatic split: singlehost/ and distributed/

In the portfolio, large test suites split their test roots:

<repo>/source/testroots/<vendor>/tests/
                           ├── singlehost/      # mock coupling, fast
                           ├── distributed/     # real coupling, slow
                           └── ...

A run picks one root or the other. The singlehost/ root has its own factory module that yields mocks; the distributed/ root has its own factory module that yields real coupling instances. The tests themselves may share between the two roots — but it is more common, and clearer, to keep them separate so a reader knows immediately which side they are on.

The case study in Chapter 10 is the singlehost/ root of codecraftai-agent-kit.

Determinism: single actor, multiple observers

The biggest single thing distributed automation testing requires that single-host application testing does not is enforced single-actor-per-environment. This is not a framework feature in the implementation sense — there is no lock somewhere preventing two tests from running. It is a design property the framework commits to and the test author honours.

The argument:

When a test interacts with a machine in a distributed environment, the effect of the interaction takes time to propagate to the other machines. The test must wait — usually by polling or by subscribing to an event — until the effect is observable everywhere it needs to be observed. If a second actor can cause effects in the same environment during that wait window, the order in which effects appear is not controlled by the test. Two effects that arrive in different orders on different runs produce different states; the test is no longer deterministic.

The conclusion: in any given environment, one actor at a time.

The framework's resolution to "but I want concurrent observation of the system under test" is tandem testing: the test is the single actor, and one or more validators attach to the test scope as observers. The validator base class lives in codecraftai-testbase/source/packages/ccai/testbase/tandem/validators.py and inherits ValidatorCoupling. The framework calls validator.attach_to_test(testscope, suffix), the validator registers its own TestResult alongside the test's, and when the test runs the validator runs in tandem — watching, polling, producing its own pass/fail outcomes recorded in the JSOS stream (Chapter 5) the same way the test's are.

A test with three validators attached is one test with four recorded outcomes — the test plus three observers. No test runs concurrently with another in the same environment.

Where parallelism lives

Parallelism in distributed automation is at the job level, not the test level. The unit of parallelism is the Pod.

A CI orchestration that wants more horizontal scale provisions more Pods, not more parallel tests within a Pod. Pytest's pytest-xdist (parallel test execution within a process) is exactly the wrong model for a distributed automation suite.

The CI hand-off — verified contracts

The Landscape and Topology files are not just configuration. They are contracts between the CI system and the test job, and the test job verifies both before it runs the first test.

• The Landscape hand-off says: "Here are the resources allotted to you." The test job verifies each resource's contact information is correct — every resource declared in the Landscape can be reached at its declared address with its declared protocol using its declared credentials. Failure means the CI system is wrong about what it gave you.
• The Topology hand-off says: "Here is the initial configuration of those resources." The test job verifies the resources are configured as the topology indicates — if the topology says "cluster of A, B, C; spare D," verification confirms the cluster membership and spare assignment. Failure means the CI system is wrong about how the resources were configured.

The framework supports this through the LandscapeConfigurationLayer.validate_topology() override hook and the validate_topology flag on LandscapeActivationParams (default True). The framework ships a stub validator; users override it with the verification logic appropriate to their topology schema.

Verification runs before the first test fires. A test against an unverified environment cannot be deterministic — verification is a precondition for the single-actor model above. Volume 6 walks the verification mechanics in detail; this chapter establishes the principle.

Inside a Landscape file: services, devices, Features

The Landscape is not just a flat list of devices. It has two top-level sections that together enable the verification contract above:

• services — declares the infrastructure services the devices use. A serial-concentrator service offering serial connections on an IP and port; a power controller offering per-port power control; a credential vault. Each service has a name, an address, and a type.
• devices — declares each device with its primary access (the deviceType and the fields that configure it: host, credentials, platform) and its Features (additional capabilities provided by the services above).

A real device declaration shape:

- deviceType: network/ssh         # primary access type
  host: 192.168.1.30              # primary access address
  platform: linux
  credentials:
      - casey-node                # composable list of named credentials
  features:
      isolation: true                          # boolean capability flag
      serial: serial-concentrator: 1           # service-reference Feature
      power: LPC934: 1                         # service-reference Feature
  skip: false

Three layers in one declaration:

1. Primary access — deviceType: network/ssh plus host and credentials. The deviceType selects the Coordinator that attaches the primary ProtocolExtension; SSH access here is not a Feature, it's the deviceType.
2. Features — additional capabilities beyond the primary access. Two shapes:
   • Boolean capability flags — isolation: true. Pure data; no Agent, no service. Test code or framework can query "does this device have isolation?"
   • Service-reference Features — serial: serial-concentrator:1, power: LPC934:1. Names a service from the services section + a port. At runtime the framework resolves the service, instantiates the matching Agent against the service's address, and exposes a ProtocolExtension on the device (device.serial, device.power).
3. Declarative metadata — platform, skip, credentials.

Coordinators, Agents, ProtocolExtensions, Features

Four words that need precise definitions, because casual usage of "extension" is ambiguous in the portfolio:

• Agents speak protocols. An SSH Agent speaks SSH; a UPnP Agent speaks UPnP; a Serial Agent speaks the serial wire protocol; a Power Agent speaks the power-control protocol. The Agent is the protocol-speaking object.
• Coordinators manage devices and attach Agents to devices. A Coordinator owns a class of devices (e.g., the SSH Coordinator owns devices reachable over SSH) and attaches the right Agent to each Device it manages.
• ProtocolExtensions connect a specific protocol Agent to a Device. A ProtocolExtension is the binding object that wraps the Agent and exposes its protocol surface on the Device. This is the user-facing extension point — a user adding a new protocol writes a ProtocolExtension that wraps their Agent.
• Features are declared on devices in the Landscape and reference services declared in the same Landscape. A Feature declaration is configuration data on the Device's features tag that says "I have capability X, here's how to access it."

The full layering for a service-reference Feature:

Landscape file
├── services section
│     └── serial-concentrator: type=serial-concentrator, address=10.0.0.50
└── devices section
      └── device-1
            └── features tag: { serial: serial-concentrator: 1 }
                                ↓ at runtime
              device-1 instance
                ├── managed by Serial Coordinator
                └── .serial → ProtocolExtension
                                └── wraps → Serial Agent
                                              └── connects to serial-concentrator port 1

This also makes Landscape verification (the CI hand-off check above) concrete. The verification walks every Device's Features, resolves each Feature's named service in the services section, instantiates the Agent against the service's address, and confirms it can connect. A broken Feature → service reference, an unreachable service, a deviceType whose Coordinator can't reach host — these are the failures verification catches before the first test fires.

The customisation mechanism: subclass Landscape and Topology

Per the meta-thesis from Volume 0 chapter 1, the framework supplies the skeleton and patterns; the organisation customises. For the environment model, the concrete customisation mechanism is subclassing Landscape and Topology.

The framework supplies:

• The base Landscape class — knows how to parse the services / devices / features shape and dispatch the built-in Features it ships support for.
• The base Topology class — knows how to parse the canonical clusters shape (name / group / nodes / spares) consumed by LandscapeOperationalLayer._create_clusters.
• Verification stub hooks (validate_topology() and friends) and the activation hooks driven by TestJob.fire_*.

The user supplies — by subclassing:

• Recognition and handling of project-specific Feature names (anything beyond serial / power / isolation) — by subclassing Landscape.
• Recognition and handling of project-specific Topology sections (anything beyond clusters) — by subclassing Topology.
• The verification logic for what those custom shapes mean — by overriding validate_topology() (and friends) in the subclasses.

The subclasses get plugged into a run by being returned from the user's TestJob subclass's template methods (chapter 2's "TestJob as customisation entry point" cascade).

This is the concrete mechanism behind every "left to the user" note in the books: open schemas + subclassed handlers, plumbed in via the user's TestJob subclass. Schema openness is the what; subclassing is the how; TestJob is the where.

Composability across categories

A property of codecraftai-configuration worth establishing here because the rest of Volume 1's tests depend on understanding it: every configuration category is composable.

The six categories Volume 3 walks — Specifications, Landscapes, Topologies, Runtimes, Credentials, Data Profiles — all use the same MergeMap overlay model. A category-level configuration the test job sees is the merged result of one or more named source files at that category. Each category's resolver (resolve_landscape_configuration, resolve_topology_configuration, etc.) takes a list of names and a list of files, loads each in order, and overlays them. Later names override earlier ones where keys collide.

Examples:

• Landscape: Checkout Resources + Environment Resources — the resources that came from a pool checkout, merged with the fixed environment resources (servers, databases, persistent infrastructure).
• Topology: combining a cluster-membership topology with a gateway-routing topology to form the effective topology for a particular test campaign.
• Runtime: UPNP Diagnostics + Bluetooth Diagnostics — diagnostic profiles stack to form one effective runtime.
• Credentials: lab-wide credentials merged with experiment-specific overrides.

This composability is the property that makes the same suite runnable on a developer laptop (light Runtime, mock Landscape) and on a nightly full-sweep run (full Runtime, real Landscape + Topology) without rewriting the tests. Volume 3 chapter 14 walks the discipline at orchestration scale; this chapter just needs the principle named, because Chapter 10's case study is itself a worked example: the same agent-kit suite runs at TESTBASE_MAX_EXAMPLES_CAP=10 (a Runtime knob from Chapter 4) without changing a single test file.

The landscape as a stable contract

The reason this design works is that landscape is a contract, not an implementation. The contract is:

• A TestJob subclass overrides the five fire_* hooks.
• The overrides accept the landscape via a template method (_get_landscape()) the subclass implements.
• The sequencer's environment / framework attachment accepts the landscape via **kwargs.

InteropLibraryTestJob (Volume 2) is one implementation of the contract. A different implementation — a research-grade landscape that lives in a future package — would override the same hooks differently and the same tests would run. The fire_* hooks are the interface; the landscape is the implementation strategy.

This is Volume 0 chapter 4 again: protocols at the top, code beneath. The single-host author depends on the protocol; the distributed author implements the protocol; neither writes code that depends on the other side.

What this chapter set up

By the end of this chapter the reader should be able to:

• Identify which fire_* hooks fire as no-ops on single-host and which become non-trivial on distributed.
• Decide whether a given test should mock the coupling, use a real local coupling, or skip on single-host.
• Write a test whose factory choice (mock vs. real) decides whether the test runs in single-host or distributed mode.
• Read a _get_landscape() template method (in Volume 2) and understand how it bridges to the single-host runner.

That fluency is the single-host half of the contract. Volume 2 carries the distributed half. The two volumes share the same TestJob lifecycle, the same coupling base class, the same JSOS streaming format, and the same agents CLI output shape. Only the hooks' bodies differ.

What's next

• Chapter 10 — Case study. The codecraftai-agent-kit single-host suite at TESTBASE_MAX_EXAMPLES_CAP=10, finishing in 49 minutes wall time. A working demonstration of every chapter that came before.