Chapter 8 — The DistributedTestSequencer

The base TestSequencer walks tests in declaration order; the DistributedTestSequencer does the same. This chapter walks what the distributed sequencer adds — landscape awareness on the attach_to_* methods — and what it deliberately does not add: parallelism.

The base TestSequencer from Volume 1 chapter 2 is the test-execution loop. It walks the discovered test tree, acquires resource factories, runs each test in declaration order, records results. It is the heart of testbase.

The DistributedTestSequencer is the distributed test job's sequencer. A reader's natural assumption — that the distributed sequencer parallelises tests across hosts or workers — is the wrong one. This chapter explains why, walks what the distributed sequencer actually adds, and connects the design to chapter 2's determinism commitment.

Class declaration

File: codecraftai-testplus/source/packages/ccai/testplus/distributedtestsequencer.py:33

class DistributedTestSequencer(TestSequencer):

    def __init__(self,
                 jobtitle: str,
                 root: str,
                 includes: Sequence[str],
                 excludes: Sequence[str],
                 metafilters: List[MetaFilter],
                 job_seed: int = None):
        super().__init__(
            jobtitle=jobtitle,
            root=root,
            includes=includes,
            excludes=excludes,
            metafilters=metafilters,
            job_seed=job_seed,
        )

        self._landscape = None
        return

Two things to notice:

1. The class extends TestSequencer directly. It does not override the test-execution loop. The base sequencer's execute_tests() (in codecraftai-testbase/source/packages/ccai/testbase/sequencing/testsequencer.py:412) walks tests in declaration order, one at a time, exactly as the single-host version does.
2. The constructor adds one new instance variable: _landscape = None. This is the slot for the landscape the distributed sequencer will receive on attach_to_environment / attach_to_framework. It is initialised to None per the Volume 0 chapter 2 discipline (every instance variable initialised in __init__).

What the distributed sequencer adds: landscape-aware coupling attachment

# Lines 57–70

def attach_to_framework(self, landscape=None):
    """
        Attach the framework's integrations to the landscape.
    """
    self._landscape = landscape

    integ_coupling_table = self._integration_couplings
    for integ_key, integ_coupling in integ_coupling_table.items():
        integ_coupling.attach_to_framework(landscape)

    return

# Lines 72–106

def attach_to_environment(self, landscape=None):
    """
        Attach the framework's integrations to the environment.
    """
    self._landscape = landscape

    # Walk the integration couplings; let each one attach to the environment
    # given the landscape now available to them.
    integ_coupling_table = self._integration_couplings
    for integ_key, integ_coupling in integ_coupling_table.items():
        integ_coupling.attach_to_environment(landscape)

    # ... additional landscape-aware setup ...

    return

The two methods accept a landscape parameter, store it on the sequencer, and propagate it to every registered integration coupling. This is the only substantive difference between DistributedTestSequencer and base TestSequencer.

The semantic difference: in the base sequencer, integration couplings attach without a landscape (they get None or are never called for landscape-using methods). In the distributed sequencer, integration couplings receive the landscape and can use it to discover devices, fetch credentials, register tandem validators against specific hosts, etc.

The flow that delivers the landscape:

DistributedTestJob (testplus)
    └── _create_sequencer() returns a DistributedTestSequencer
        ↑
        │ called by TestJob.execute()
        │
InteropLibraryTestJob (interop)
    └── fire_attach_to_environment(tseq):
            landscape = self._get_landscape()
            tseq.attach_to_environment(landscape=landscape)  ← landed here

The sequencer never constructs a landscape. It receives one from the test job's fire_* overrides. The separation is chapter 4's customisation cascade in action: the test job chooses which landscape (via _get_landscape()); the sequencer just wires the landscape through.

What the distributed sequencer does NOT add: parallelism

The framing question: if the distributed sequencer doesn't parallelise tests, why does it exist?

The answer matters because it's the framework's clearest statement of the determinism model from chapter 2. The distributed sequencer:

• Does not parallelise tests within a Pod.
• Does not have a concept of "workers" or "concurrent test execution."
• Does not contain a thread pool, an asyncio event loop, or any parallelism primitive for test execution.

It exists to be landscape-aware, not landscape-parallel. The landscape gives integration couplings the information they need to operate against multiple hosts; the test-execution loop still runs one test at a time because the determinism commitment requires it.

A reader looking for parallelism in the source will find it in two places:

1. Inside the wire layer (Volume 5). When a test calls into multiple hosts, the SSH agent (chapter 10 of this volume) may issue concurrent commands across hosts. This is intra-test concurrency — the test, as the single actor, distributes its own actions. The Pod's other actors are still excluded.
2. Across Pods, via orchestration (Volume 3). Multiple Specifications produce multiple Pods; multiple DistributedTestJob processes run concurrently against independent Pods. This is the Pod-level parallelism of chapter 3.

Neither of these is in the sequencer. The sequencer's job is serial test execution within a Pod.

What a distributed run looks like through the sequencer

Walking the path from DistributedTestJob.__enter__ to the first test:

1. __enter__ calls begin(). The job constructs the recorder.
2. begin() calls _create_sequencer() — returns a DistributedTestSequencer.
3. The job fires fire_attach_to_environment(tseq). The InteropLibraryTestJob override fetches the landscape via _get_landscape() and passes it to tseq.attach_to_environment(landscape=landscape).
4. The sequencer's attach_to_environment stores the landscape, walks every integration coupling, and calls integ_coupling.attach_to_environment(landscape). Each coupling now has the landscape and can do its work.
5. The job fires fire_attach_to_framework(tseq). Same shape — sequencer propagates the landscape to couplings' attach_to_framework.
6. The job fires the three fire_activate_* hooks. Each delegates to the landscape (chapter 6's three layers). After these complete, the Pod is operationally live.
7. The job fires fire_diagnostic_capture_pre_testrun. The sequencer captures pre-test diagnostic state.
8. The sequencer enters its Traversal phase. For each test in declaration order:
   • The recorder receives a test_started event.
   • The test runs. Tandem validators (chapter 5) attached to the test scope run in tandem.
   • The test returns; tandem validators finalise.
   • The recorder receives test_finished and per-validator events.
9. The sequencer's traversal ends. The job fires fire_diagnostic_capture_post_testrun.
10. __exit__ calls finalize(). The recorder writes summary and consolidated documents. The job tears down.

Step 8 is sequential. N tests in the suite mean N iterations of the traversal loop, one at a time. The framework does not expose a knob to make step 8 concurrent.

Why this design holds up at scale

Three properties make the serial-within-Pod design hold up even on large suites:

Test bodies dominate framework overhead

Volume 1 chapter 10 measured the runner's per-test overhead at single-digit milliseconds. A test body that takes 5 seconds spends 99.98% of its time in the body, not the framework. Adding parallelism to the framework's loop would optimise the 0.02%; the cost (broken determinism) outweighs the benefit.

Test concurrency lives inside the test

A test that needs to drive ten hosts in parallel does so inside the test body, by issuing SSH commands concurrently through the SSH agent (chapter 10). The test, as the single actor, distributes its own actions. The other actors (the ten hosts) are not framework actors; they are the system under test responding to the actor's commands.

Pod fan-out is the right granularity for horizontal scale

When a campaign needs more parallelism, the answer is more Pods. Volume 3 chapter 4's Specifications are how Pods are provisioned; chapter 3 of this volume walked why Pod-level parallelism is the only safe parallelism level.

So: the sequencer stays serial; the test body gets intra-test concurrency through the wire; Pod fan-out provides horizontal scale. Three layers, each appropriate.

Differences from the base sequencer summarised

The table is short because the difference is small. The distributed sequencer is the base sequencer with a landscape slot and two methods that take a landscape parameter. Everything else — including the determinism-preserving serial execution — is inherited verbatim.

Implications for custom sequencers

A project that wants a custom sequencer subclasses DistributedTestSequencer (or, for non-distributed but landscape-aware scenarios, TestSequencer). The custom sequencer plugs into a DistributedTestJob subclass via the _create_sequencer() template method (chapter 4).

Reasons to subclass:

• Custom integration handling. The default attach_to_* walks every coupling and calls its method; a subclass can intercept to add ordering, filtering, parallelism of the attachment (different from parallelism of test execution), or extra logging.
• Custom diagnostic capture. The base capture is delegated to the recorder; a subclass can add per-host snapshots taken at phase boundaries.
• Custom traversal logging. Adding rich per-test pre-flight checks at the sequencer level (rather than fixture level) when those checks span couplings.

What not to do in a custom sequencer subclass: introduce test-level parallelism. The framework's contract with the test author is that tests run serially within a Pod. A sequencer that violates the contract violates chapter 2's commitment and breaks every downstream assumption.

What the next chapter does

Chapter 9 walks job_seed and reproducibility. Volume 1 chapter 4 introduced the per-test seed (SHA-256(job_seed + ":" + test_name)) for property-based tests; chapter 9 of this volume walks how a single seed survives the trip across machines in a distributed run, and why deterministic input generation across N hosts requires exactly the discipline the sequencer's serial execution provides.