Chapter 3 — The Pod Model — Where Parallelism Actually Lives

Tests within a Pod run sequentially because the Pod is a shared environment. Jobs across independent Pods run in parallel because Pods are independent environments. The unit of horizontal scaling in distributed automation is the Pod — not the test, not the host, not the runner.

The previous chapter established the single-actor determinism commitment: within any environment, one actor at a time. An honest reader's next question is: if I can't parallelise tests, how does my CI run faster?

The answer is the Pod model.

What a Pod is

A Pod is the runtime instance produced by combining a Landscape (the physical inventory and wiring — Volume 6 chapter 1) with a Topology (the logical organisation and roles — Volume 6 chapters 5 and 6). A Pod is a concrete, interoperating set of resources that a job runs against.

Volume 1 chapter 9 introduced the vocabulary; Volume 6 walks the schema in detail. This chapter establishes what a Pod is for: a Pod is the unit of horizontal CI parallelism.

A Pod has the following properties:

1. Internal interoperability. Within a Pod, the resources can talk to each other — the Landscape and Topology describe their relationships and the framework activates them.
2. External isolation. Pods do not share resources. A device in Pod A is not in Pod B. A service in Pod A is not in Pod B. An effect caused in Pod A cannot propagate to Pod B.
3. Single-actor execution. A Pod has one test job running against it at a time (chapter 2's commitment); inside that job, tests run sequentially.

The third property is what makes the second property load-bearing. Pods don't share resources and each Pod is single-actor → different Pods cannot affect each other → jobs across Pods can run in parallel without violating determinism.

The parallelism table

The table is the framework's complete answer to the parallelism question. Three rows, two no's, one yes. The yes is at the Pod level; the no's are at every finer granularity.

How Pods get provisioned

A Pod is realised at the orchestration layer. Volume 3 chapter 4 walks the Specification configuration category — the declarative manifest that tells CI tooling "I need this set of resources to form an automation Pod." CI tooling reads the Specification, performs the checkout-or-create, and emits the Landscape and Topology files for the next stage. Those files are the Pod's static description; the runtime instantiation into a running Pod happens when a DistributedTestJob activates the Landscape and Topology (chapter 6 of this volume).

The orchestration-side flow:

Specification file (declares "I need a Pod of 5 hosts")
        ↓ CI tooling
   checkout / create resources
        ↓
   emits Landscape file + Topology file
        ↓ orchestration triggers DistributedTestJob
        ↓
   InteropLibraryTestJob.fire_activate_configuration()
   InteropLibraryTestJob.fire_activate_integration()
   InteropLibraryTestJob.fire_activate_operations()
        ↓
   Pod is now realised — devices online, services reachable,
   topology verified
        ↓
   tests run sequentially against this Pod
        ↓
   Pod is torn down (or returned to the pool)

A "10-fold parallel CI" in this model means ten Specifications, ten Pods, ten DistributedTestJobs running concurrently. Each Specification produces one Pod; each Pod runs one job; the ten jobs do not affect each other because the ten Pods are independent.

A worked example — three Pods running in parallel

Suppose a CI run wants to verify a sysman feature against three OS variants: RHEL, Ubuntu, macOS. Each variant needs a 5-host Pod (one controller + four DUTs).

Three Specifications declare three Pods:

# Specification A — RHEL Pod
pod:
    name: rhel-feature-test-pod
    resources:
        - controller-rhel-1
        - dut-rhel-1
        - dut-rhel-2
        - dut-rhel-3
        - dut-rhel-4

# Specification B — Ubuntu Pod
pod:
    name: ubuntu-feature-test-pod
    resources:
        - controller-ubuntu-1
        - dut-ubuntu-1
        # ... etc.

# Specification C — macOS Pod
pod:
    name: macos-feature-test-pod
    resources:
        - controller-macos-1
        - dut-macos-1
        # ... etc.

CI runs all three Specifications in parallel. Each emits its own Landscape and Topology. Three DistributedTestJob processes run concurrently — one per Pod. Inside each job, the sysman feature tests run sequentially.

Three Pods. Three jobs in parallel. Within each job, tests run one at a time. No tests-in-parallel anywhere — but three-jobs-in-parallel.

If the sysman feature suite for one OS takes thirty minutes sequentially, the cross-OS run finishes in thirty minutes total (parallel across Pods), not ninety (serial across Pods). Three-fold horizontal scaling at the Pod level; zero test-level parallelism within a Pod.

What this means for CI scaling

The framework's stance on horizontal scaling is uniform:

To go faster, provision more Pods. Do not parallelise tests within a Pod.

This is a stance that surprises engineers coming from application-test traditions where pytest-xdist's --workers=N option is the default scaling answer. The framework's answer is different because the problem is different — application tests don't share state across processes, so parallel pytest workers don't race; distributed automation tests do share state (the Pod's resources), and parallel workers within a Pod would race.

The stance has consequences:

• Specifications become first-class infrastructure. Volume 3 chapter 4 walks the Specification category as the unit of horizontal scale. A team that wants their CI to finish twice as fast doesn't double their pytest workers; they double their Specifications.
• Pod isolation must be real. A Pod that secretly shares a service with another Pod (a database, a license server, a shared queue) is not really a Pod. The framework's Specification-to-Landscape pipeline in Volume 3 chapter 4 is pessimistic about shared services for this reason.
• Pod size becomes a tuning parameter. A 5-host Pod finishes a sysman suite in 30 minutes; a 10-host Pod finishes it in 18 minutes (more parallelism within the suite at the test design level, not at the framework level). The framework's job is to be honest about the trade-off; a larger Pod amortises Pod-setup cost over more tests but is more expensive to provision.

What this rules out

The Pod model rules out a few patterns that look attractive but break the determinism commitment:

"Parallel tests with shared fixtures"

# Wrong — two tests racing on shared fixture state.
@pytest.mark.parametrize("scenario", scenarios)
@pytest.mark.parallel(workers=4)  # never use this
def test_scenario(scenario):
    ...

If scenarios exercises shared fixture state in a real Pod, the four workers race. The right answer in distributed automation is to provision one Pod per scenario — four Pods, four jobs, parallel jobs, sequential tests within each job.

"Background actors" running between tests

# Wrong — the background actor is a second actor in the Pod.
def setup_module():
    threading.Thread(target=monitor_drift, daemon=True).start()

A daemon thread inside the test process can be an observer (if it only watches and records). But if the daemon ever acts — sends a heartbeat, restarts a service — it becomes a second actor in the Pod and breaks determinism. The right shape for ongoing observation is a tandem validator (chapter 5).

"Cross-Pod dependencies" in a single job

# Wrong — one test reaches into a different Pod's service.
def test_cross_pod():
    pod_a_service = LandscapeSingleton().get_pod("pod-a").service("X")
    pod_b_service = LandscapeSingleton().get_pod("pod-b").service("X")
    # ... acts on both

A test acting on two Pods is acting on what should be a single larger Pod. Either combine them into one Pod (and accept the serial execution) or split the test (and accept the cross-Pod sequencing concern, which is an orchestration-level problem — Volume 3 chapter 22's flow control covers it).

What the next chapter does

Chapter 4 walks the TestJob inheritance chain in detail. The chain is the customisation cascade (Volume 0 chapter 1's meta-thesis applied here): TestJob (testbase) → InteropLibraryTestJob (interop) → DistributedTestJob (testplus). Each layer adds one responsibility. The chain is how a project plugs in its custom Landscape and Topology subclasses.

Chapter 5 then walks the tandem-validator implementation in depth — the actual codecraftai-testbase/.../tandem/validators.py machinery, with the real attach_to_test signature and the JSOS recording flow.

By the end of those two chapters, a reader will have the full picture of how the determinism model from chapter 2 is held together by the chain of customisation.