Chapter 4 — The TestJob Inheritance Chain — The Customisation Cascade

Four classes, each adding one responsibility, plus three naming-only fallbacks. The chain is the framework's customisation entry point — the meta-thesis from Volume 0 chapter 1 (skeleton + patterns) made concrete. This chapter walks the chain from base to distributed and shows how a project plugs in its custom Landscape and Topology.

The TestJob inheritance chain is the single most important piece of code in distributed automation. It is the place where:

• The single-host substrate from Volume 1 meets the distributed extension Volume 2 walks.
• The framework's lifecycle hooks become non-trivial.
• The user's customisations (custom Landscape, custom Topology, custom Validators) plug in.

The chain is small. Seven classes, three of them naming-only. The non-trivial work happens at four layers, and each layer adds exactly one responsibility.

The seven classes

The diagram below is the intended chain. One edge — IntegratedTestJob → InteropLibraryTestJob — has not yet landed (see "In-flight migration" immediately after): in today's source InteropLibraryTestJob extends TestJob directly, so as built it and IntegratedTestJob are siblings, both extending TestJob.

TestJob (testbase)                       ← single-host base + stdout recorder + standalone seams
   │
   ├── DefaultTestJob (testbase)         ← naming fallback
   │
   └── IntegratedTestJob (integration-runtime)   ← integration result recording
          │
          ├── DefaultIntegratedTestJob (integration-runtime)  ← naming fallback
          │
          └── InteropLibraryTestJob (interop)    ← landscape activation
                 │
                 └── DistributedTestJob (testplus)   ← distributed sequencer
                        │
                        └── DefaultDistributedTestJob (testplus)  ← naming fallback

The non-trivial-work layers are TestJob, IntegratedTestJob, InteropLibraryTestJob, and DistributedTestJob. The other three are naming-only fallbacks for entry points that need a default job_type=.

In-flight migration. IntegratedTestJob is newly built and TestJob already exposes the three provider hooks it overrides. But the re-parenting of interop's InteropLibraryTestJob onto IntegratedTestJob has not yet landed: in the interop source today, InteropLibraryTestJob still extends TestJob directly (codecraftai-interop/source/packages/ccai/interop/testing/testjob.py:22 reads class InteropLibraryTestJob(TestJob)). The diagram above is the intended chain; the section on InteropLibraryTestJob notes where the source currently diverges. This is the same kind of partially-landed refactor the c785e9b narrative below describes — reported honestly rather than overclaimed.

Layer 1 — TestJob (the near-standalone single-host base)

File: codecraftai-testbase/source/packages/ccai/testbase/testjob.py:45

TestJob is the foundation. Volume 1 chapter 2 walked it in detail; this chapter recaps what's relevant for distributed extension.

testbase is now near-standalone — it no longer depends on codecraftai-configuration, codecraftai-runtime, or codecraftai-this-app. A plain TestJob runs without activation: no presence folder, no RuntimeVariables, no this-app. It reads the handful of core env knobs it needs (AM_DEBUG_, AM_DIAGNOSTICS_, AM_TESTROOT, AM_OUTPUT_DIRECTORY, AM_RUN_ID, AM_TESTRUN_TIMEOUT) directly from os.environ via the tiny, dependency-free ccai.testbase.environment module, and records its run summary to stdout. Everything heavier — the presence-folder results layout, JSOS-to-disk, the HTML summary, CI/build metadata — is the responsibility of IntegratedTestJob in the next layer.

The class establishes:

• The constructor with logger, testroot, includes/excludes, metafilters, job_seed, and the test module.
• __enter__ / __exit__ driving begin() / finalize().
• The five fire_* lifecycle hooks — all stubs on single-host:
  • fire_attach_to_environment(tseq) — line 379.
  • fire_attach_to_framework(tseq) — line 389.
  • fire_activate_configuration() — line 399.
  • fire_activate_integration() — line 403.
  • fire_activate_operations() — line 407.
• The provider/template methods that customisation plugs into:
  • _create_sequencer() (line 575) — returns a TestSequencer (the base) by default; subclasses override to return a DistributedTestSequencer or other variant.
  • _create_recorder(...) (line 564) — returns a StdoutResultRecorder by default. It prints the run summary to stdout and reuses the JSOS/JSON serialisation (so testrun_results_stream.jsos / testrun_summary.json are still written when an output dir is used, and still consumed by the testbase agents CLI), but it skips the HTML rendering. Subclasses that want disk-backed JSOS plus a rendered HTML summary override this to return a JsonResultRecorder.
  • _create_output_context() (line 583) — returns a SimpleOutputContext (an IOutputContext) by default, rooting artefacts under AM_OUTPUT_DIRECTORY or <cwd>/.output.
  • _create_run_metadata() (line 595) — returns a StandaloneRunMetadata (an IRunMetadata) by default: auto-generated UUID run_id, "(unknown)" build / pipeline / job, title from the job's name.

The five fire_* hooks are the contract. They run at specific phase boundaries during a test run (Volume 1 chapter 2 walks the order). On a single-host run they are no-ops; on distributed, the subclasses below give them bodies.

The hooks are lifecycle hooks, not template methods. They run during the lifecycle; they do not return values that the framework consumes. The provider methods (_create_sequencer, _create_recorder, _create_output_context, _create_run_metadata, and the abstract _get_landscape introduced two layers down) are the value-returning customisation points. _create_run_metadata() / _create_output_context() are invoked once in the constructor __init__ (lines 95–96) so the run's identity and artefact roots are fixed before begin() or any test executes.

What TestJob does NOT have

A point worth being explicit about because the next layer adds it: TestJob has no concept of a landscape. There is no _get_landscape() method, no landscape parameter, no import of codecraftai-landscaping from anywhere in testbase. This is intentional — codecraftai-testbase is the foundation tier (Volume 0); codecraftai-landscaping is upstream of it. The dependency arrow runs landscaping → interop → testbase, not the other way. Volume 0 chapter 4 walks the protocol-based discipline that makes this possible.

The landscape concept enters two layers down, in InteropLibraryTestJob — but does so as an abstract hook the user (or a downstream package) implements, not as a direct import. The result: testbase stays landscape-free and the chain remains importable on a host that doesn't have landscaping installed.

Layer 2 — IntegratedTestJob (integration result recording)

File: codecraftai-integration-runtime/source/packages/ccai/integrationruntime/integratedtestjob.py:51

IntegratedTestJob is the layer that re-supplies the heavyweight reporting the near-standalone TestJob deliberately leaves out. It lives in the new codecraftai-integration-runtime package (deps: testbase + this-app + extension) and adds exactly one responsibility: integration result recording. It does so by overriding the three provider hooks TestJob exposes — and nothing else.

Single responsibility. IntegratedTestJob does not override the five fire_* landscape hooks. Those were not moved here; they stay in InteropLibraryTestJob (Layer 3). IntegratedTestJob's only job is to swap the standalone stdout/.output result plumbing for activation-backed presence-folder plumbing.

The three provider overrides

def _create_output_context(self) -> IOutputContext:
    inst = RuntimeOutputContext()
    return inst

def _create_run_metadata(self) -> IRunMetadata:
    run_title = "Test Run"
    if self.name is not None and len(self.name) > 0:
        run_title = self.name
    inst = RuntimeRunMetadata(title=run_title)
    return inst

def _create_recorder(self, runid, start, render_info, build_info,
                     pipeline_info, job_info) -> ResultRecorder:
    inst = JsonResultRecorder(runid=runid, start=start, render_info=render_info,
                              apod=self._apod, build_info=build_info,
                              pipeline_info=pipeline_info, job_info=job_info)
    return inst

Each override replaces a standalone default with its integration counterpart:

• _create_output_context() → RuntimeOutputContext (codecraftai-integration-runtime/source/packages/ccai/integrationruntime/runtimeoutputcontext.py) — delegates to ccai.thisapp.paths so results, diagnostics, and per-testcase by-products land under the activated app's well-known presence folder. It is activation-gated: it raises SemanticError unless the process was activated as a job (this-app activation sets AM_JOB_TYPE / AM_PRESENCE_DIR / AM_RESULT_PATH_FOR_TESTS). This replaces TestJob's SimpleOutputContext, which roots under <cwd>/.output.
• _create_run_metadata() → RuntimeRunMetadata (codecraftai-integration-runtime/source/packages/ccai/integrationruntime/runtimerunmetadata.py) — sources build / pipeline / job metadata from the resolved RuntimeVariables (the real CI/runtime context) instead of the "(unknown)" placeholders StandaloneRunMetadata supplies.
• _create_recorder() → JsonResultRecorder — JSOS streaming to disk plus the rendered HTML summary, in place of TestJob's StdoutResultRecorder.

Activation is a precondition, not a hook

IntegratedTestJob assumes the process was already activated as a test-run job before begin() resolves the presence-folder paths. The activation itself — establish_app_persona + resolve_runtime_variables + activate_profile_job(JobType.TestRun) — lives in ccai.integrationruntime.initialize, and integration runs reach the job through ccai.integrationruntime.entrypoints.integrated_test_entrypoint, which activates and then delegates to testbase's standalone generic_test_entrypoint. This keeps activation out of the class hierarchy: IntegratedTestJob consumes the activated context (via RuntimeOutputContext / RuntimeRunMetadata); it does not perform activation.

DefaultIntegratedTestJob (line 91) is the naming-only fallback for this layer — same role as DefaultTestJob and DefaultDistributedTestJob.

Layer 3 — InteropLibraryTestJob (the landscape-aware hinge)

File: codecraftai-interop/source/packages/ccai/interop/testing/testjob.py:22

InteropLibraryTestJob is the hinge of the chain. It adds exactly one responsibility: landscape activation on the five fire_* hooks. It does so without taking a hard dependency on codecraftai-landscaping — by introducing an abstract template method _get_landscape() that subclasses must override.

In-flight: the parent is moving. In the intended chain, InteropLibraryTestJob extends IntegratedTestJob (Layer 2) so a distributed run inherits both integration result recording and landscape activation. That re-parenting has not yet landed. In the interop source today the class still extends TestJob directly — see the declaration below. Until the migration completes, an interop/testplus run does not automatically pick up IntegratedTestJob's presence-folder recording from the class hierarchy; it gets the standalone stdout plumbing from TestJob unless the entry point arranges otherwise. This is reported as-built, not as-intended.

The class declaration

# codecraftai-interop/source/packages/ccai/interop/testing/testjob.py:22
# As built today — still extends TestJob.  Intended parent: IntegratedTestJob.

class InteropLibraryTestJob(TestJob):
    """
        Interop-tier test-job base class.

        Extends :class:`ccai.testbase.testjob.TestJob` (single-host) with
        the **landscape-activation lifecycle** required by any test job that
        runs against a real environment described through the landscape model
        — devices, services, coordinators, control points, integration
        couplings.
    """

The abstract _get_landscape() template method

# Lines 60–86

def _get_landscape(self) -> Any:
    """
        Returns the landscape object used by the activation /
        attachment hooks.

        Concrete subclasses must override this method to return the
        landscape instance their environment is built from
        (typically ``LandscapeSingleton()`` from
        ``ccai.landscaping.wellknown``).

        :raises NotImplementedError: When called on a class that does
                                     not override the hook.
    """
    err_msg_lines = [
        "InteropLibraryTestJob subclasses must override _get_landscape().",
        "    SUBCLASS: {}".format(type(self).__name__),
    ]
    err_msg = "\n".join(err_msg_lines)
    raise NotImplementedError(err_msg)

The body raises NotImplementedError with a descriptive message naming the subclass that failed to override. This is the Volume 0 chapter 3 discipline: errors should never pass silently. An InteropLibraryTestJob subclass that forgot _get_landscape() fails loudly on first attempt to activate.

The five fire_* overrides

# Line 88 — fire_attach_to_environment

def fire_attach_to_environment(self, tseq: TestSequencer) -> None:
    """
        Attach the sequencer to the environment using the landscape.
    """
    landscape = self._get_landscape()
    tseq.attach_to_environment(landscape=landscape)
    return

# Line 101 — fire_attach_to_framework

def fire_attach_to_framework(self, tseq: TestSequencer) -> None:
    """
        Attach the sequencer to the framework using the landscape.
    """
    landscape = self._get_landscape()
    tseq.attach_to_framework(landscape=landscape)
    return

# Line 114 — fire_activate_configuration

def fire_activate_configuration(self) -> None:
    """
        Activate the landscape's configuration layer.
    """
    landscape = self._get_landscape()
    landscape.activate_configuration()
    return

# Line 124 — fire_activate_integration

def fire_activate_integration(self) -> None:
    """
        Activate the landscape's integration layer.
    """
    landscape = self._get_landscape()
    landscape.activate_integration()
    return

# Line 134 — fire_activate_operations

def fire_activate_operations(self) -> None:
    """
        Activate the landscape's operational layer.
    """
    landscape = self._get_landscape()
    landscape.activate_operations()
    return

The pattern is identical for every hook: fetch the landscape via _get_landscape(), delegate to the right method on the landscape. The landscape is fetched fresh each time because LandscapeSingleton() (the typical implementation) is a singleton — the cost is one dict lookup, not a re-instantiation.

The c785e9b commit — what this layer was extracted from

The introduction of InteropLibraryTestJob happened in commit c785e9b ("Add InteropLibraryTestJob — interop-tier base class for landscape-aware test jobs"). Before that commit, DistributedTestJob was a standalone class that duplicated ~600 lines of TestJob verbatim with the five fire_* overrides bolted on. The duplication had three problems:

1. The duplication itself. Two copies of the lifecycle logic meant any change to TestJob had to be made twice.
2. The dependency direction. DistributedTestJob was in codecraftai-testplus, which is downstream of codecraftai-testbase. The duplicated code in DistributedTestJob re-implemented TestJob semantics in a downstream package — a clear inheritance smell.
3. The hook implementation. The five fire_* overrides in the original DistributedTestJob had hard imports of LandscapeSingleton. This made the class un-importable on a host without codecraftai-landscaping installed — an ergonomic failure at the package boundary.

The c785e9b refactor introduced InteropLibraryTestJob as the intermediate layer that:

• Inherits cleanly from TestJob (no duplication).
• Lives in codecraftai-interop (a tier above testbase but below landscaping).
• Defines _get_landscape() as abstract, deferring the landscape import to the concrete subclass.

Result: DistributedTestJob collapsed from ~615 lines to ~90.

The IntegratedTestJob layer (Layer 2) is the next step of the same factoring discipline: rather than baking presence-folder recording, RuntimeVariables metadata, and HTML rendering into TestJob (which would re-couple the foundation tier to this-app and RuntimeVariables), that behaviour was lifted into a separate codecraftai-integration-runtime package and slotted in between TestJob and InteropLibraryTestJob. As noted above, the final wiring — re-parenting InteropLibraryTestJob onto IntegratedTestJob — is the in-flight step; until it lands, interop's class still extends TestJob, the same way DistributedTestJob once extended TestJob before c785e9b.

Layer 4 — DistributedTestJob (the distributed plug-in)

File: codecraftai-testplus/source/packages/ccai/testplus/distributedtestjob.py:24

DistributedTestJob is the layer that concretes the abstract _get_landscape() and adds the distributed sequencer. It is ~90 lines, almost all of which is the two override methods.

_get_landscape() — concrete

# Lines 49–57

def _get_landscape(self) -> Any:
    """
        Returns the singleton :class:`Landscape` instance for the
        distributed test environment.

        :returns: The active :class:`Landscape`.
    """
    landscape = LandscapeSingleton()
    return landscape

The concrete _get_landscape returns LandscapeSingleton() from ccai.landscaping.wellknown. The singleton is factory-overridable (a custom Landscape subclass registered through the factory protocol replaces the default — Volume 0 chapter 7 walks the factory model), so a project's custom landscape plugs in here without modifying DistributedTestJob itself.

The customisation pattern: the project subclasses Landscape, registers the subclass through the factory protocol, LandscapeSingleton() returns the subclass, the rest of the chain is unchanged.

_create_sequencer() — distributed

# Lines 59–81

def _create_sequencer(self) -> DistributedTestSequencer:
    """
        Hook used by :meth:`TestJob.execute` to construct the test
        sequencer for this job.  Returns a
        :class:`DistributedTestSequencer` so the sequencer's
        ``attach_to_environment`` / ``attach_to_framework`` methods
        accept the landscape supplied by
        :class:`InteropLibraryTestJob`.
    """
    inst = DistributedTestSequencer(
        self.title,
        self._testroot,
        includes=self.includes,
        excludes=self.excludes,
        metafilters=self.metafilters,
        job_seed=self.job_seed,
    )
    return inst

_create_sequencer returns a DistributedTestSequencer (chapter 8 of this volume walks it in detail). The sequencer is the test-execution loop; the distributed variant differs from the base in that it accepts a landscape parameter on attach_to_environment() / attach_to_framework() — which is where InteropLibraryTestJob's overrides land the landscape.

A note on the self.title bug

Line 74 references self.title, but the TestJob base class defines the attribute as self._title. There is no title property defined anywhere in the chain. At runtime, this raises AttributeError.

This is a real bug in the source. The book reports it faithfully because pretending the chain is bug-free would mislead the reader. The fix is straightforward — change self.title to self._title (or expose a @property title on TestJob). An SDET reading this chapter and running into the same issue should know they're not imagining it.

The bug also illustrates a benefit of the layered chain: the fix is a one-line change in one file. Before the c785e9b refactor, the same bug — if it existed — could have been in the duplicated copy and gone uncaught.

Layers 5, 6, and 7 — the naming-only fallbacks

DefaultTestJob (in testbase, line 613), DefaultIntegratedTestJob (in integration-runtime, line 91), and DefaultDistributedTestJob (in testplus, lines 84–90) are trivial:

class DefaultDistributedTestJob(DistributedTestJob):
    """
        The DefaultDistributedTestJob is utilized as a job container when
        a job was not specified.
    """
    name = "Test Job"
    description = "Unspecified test job."

They exist so that the entry points (generic_test_entrypoint, integrated_test_entrypoint, distributed_test_entrypoint) have a reasonable default job_type= value. A user with no custom job-type subclass uses the default; a user with a custom subclass passes job_type=YourCustomJob.

The customisation cascade

Volume 0 chapter 1's meta-thesis: framework supplies the skeleton and patterns; the organisation customises and refines. The chain above is the concrete realisation for distributed testing.

A project building a custom distributed test scenario:

1. Pick the right starting layer.

   • Single-host, no-activation customisation (stdout recorder, .output artefacts) → subclass TestJob.
   • Single-host customisation that needs activation-backed presence-folder recording, RuntimeVariables metadata, and an HTML summary → subclass IntegratedTestJob (and run via integrated_test_entrypoint so activation happens first).
   • Landscape-aware single-pod customisation that doesn't need a custom sequencer → subclass InteropLibraryTestJob and implement _get_landscape().
   • Distributed multi-host customisation → subclass DistributedTestJob (and override _create_sequencer() if you need a custom sequencer).

2. Override the provider methods you need.

   • _get_landscape() to return your Landscape subclass (typically via LandscapeSingleton() after registering your subclass through the factory).
   • _create_sequencer() for a custom sequencer.
   • _create_recorder() for a custom recorder (stdout vs. JSOS
     • HTML is itself decided here — TestJob defaults to StdoutResultRecorder, IntegratedTestJob to JsonResultRecorder).
   • _create_output_context() / _create_run_metadata() for a custom artefact root or run-identity source.
   • The five fire_* hooks if you need extra activation behaviour beyond what landscape activation provides (rare — usually the layered base classes suffice).

3. Hand the subclass to the entry point.

   distributed_test_entrypoint(job_type=YourCustomJob)
   

The entry point is codecraftai-testplus/source/packages/ccai/testplus/entrypoints.py:38 for distributed jobs, codecraftai-integration-runtime/source/packages/ccai/integrationruntime/entrypoints.py (integrated_test_entrypoint) for activated integration jobs, and codecraftai-testbase/source/packages/ccai/testbase/entrypoints.py:135 (generic_test_entrypoint) for standalone single-host. Real scaffolding examples live in codecraftai-testbase/source/tests/ccai_tests/testbase/integration/test_custom_job_type_integration.py (CustomTestJob, FailingCustomTestJob).

Not in the chain — OrchestrationJob

A point of vocabulary worth re-emphasising (Volume 3 chapter 1 also makes it): OrchestrationJob (in codecraftai-landscaping/source/packages/ccai/landscaping/orchestrationjob.py:23) is not a TestJob subclass. It inherits from ContextUser.

OrchestrationJob is for automation tasks that need a landscape but are not tests — environment provisioning, inventory discovery, compliance audits. It carries job metadata (name, title, description, owner, requires_landscape, requires_topology, requires_runtime, requires_credentials as class attributes) and wraps the lifecycle in a begin() → execute() → end() contract that is not the fire_* lifecycle from TestJob.

The two hierarchies meet at the bridge tasks of Volume 3 chapter 25 — taskplus's DistributedTestJobTask invokes a DistributedTestJob; OrchestrationJob is invoked by taskplus directly without passing through the test machinery.

When this volume says "the test job," it means DistributedTestJob or its subclasses. When Volume 3 says "the orchestration," it means a taskplus workflow that may or may not invoke a test job. The two meet at the bridge; they don't share a hierarchy.

What the next chapter does

Chapter 5 walks the tandem-validator implementation in detail: the Validator base class, attach_to_test, the JSOS recording flow, the wait_for_it integration from Volume 0 chapter 5. It is the implementation half of chapter 2's single-actor + multi-observer rationale.