Chapter 2 — testbase Architecture

A test framework's architecture is the answer to a single question: what happens, in what order, when you run the suite? The interesting part of the answer is never the test functions themselves — those are just the leaves. The interesting part is the path the runner takes from "here is a test root" to "here are the results." This chapter walks that path.

codecraftai-testbase is not a small framework. Counting only its own source tree it is around 40,000 lines of Python; the test suite that exercises it is 146 test files (codecraftai-testbase/source/packages/ccai/testbase/, codecraftai-testbase/source/tests/). But the architecture is small in the sense that matters: the number of concepts a reader has to hold in mind to follow a run is fewer than ten.

This chapter introduces those concepts and walks a single non-distributed run from CLI invocation to result file. Later chapters revisit the same path looking at a single concept in depth — the resource factory model in Chapter 3, the property runner in Chapter 4, the JSOS recorder in Chapter 5.

It is worth stating up front what testbase is not coupled to. The framework's core run path is near-standalone: it depends only on codecraftai-arcli, codecraftai-foundations (which consolidated the former errors, interfaces and waiting packages), and codecraftai-xmodules. It does not depend on codecraftai-configuration, codecraftai-runtime, or codecraftai-this-app. The handful of core environment knobs it needs (AM_TESTROOT, AM_OUTPUT_DIRECTORY, AM_RUN_ID, AM_TESTRUN_TIMEOUT, the AM_DEBUG_*/AM_DIAGNOSTICS_* flags) are read directly from os.environ through a tiny dependency-free module, ccai.testbase.environment. Everything richer — the RuntimeVariables descriptor set, the well-known presence-folder paths, CI/build metadata — is the integration layer's job, reached through injected seams rather than imports. (The source core is fully decoupled; the testbase CLI/bootstrap still imports the runtime/this-app machinery for activation, a clean-up that is in-flight at the time of writing.)

The five concepts

A reader who finishes this chapter should be able to define each of the following in their own words. Everything else in the framework is built from them.

1. TestRef — a lazy, metadata-rich reference to a single test function.
2. TestCollector — the discovery component that builds a tree of TestRef objects from a test root.
3. TestSequencer — the component that walks the discovered tree through five phases, applying graph operations.
4. TestJob — the job-spanning runner that owns the lifecycle: enter → fire hooks → run → finalize → exit.
5. ResultRecorder — the recorder that captures the run. Plain testbase defaults to StdoutResultRecorder, which echoes the summary to stdout and writes the machine-readable JSOS/JSON artefacts; the HTML-rendering JsonResultRecorder is the integration runtime's choice (covered below).

Around these five there are couplings (integration, configuration, validator), the property runner, the agents CLI, and the mocking module — but those are extensions of the core, not the core.

A test as code

Before the architecture, a single example, so that the rest of the chapter can refer to a concrete shape. A testbase test is a plain Python function:

# tests/example/test_arithmetic.py
import testbase

def test_addition_is_commutative():
    """
        Verify that integer addition is commutative.
    """
    a = 2
    b = 3
    rtnval = (a + b) == (b + a)
    assert rtnval
    return

No subclass. No @pytest.mark.unit. No fixtures imported by default. The function name starts with test_, the file lives under a test root, and the framework finds it. The explicit-return discipline from Volume 0 chapter 1 is in play: the function ends with return even though the function returns nothing, because future-you may add a return value, and the breakpoint anchor is worth the line.

That is the leaf. Now the path that gets us to the leaf.

Discovery: TestRef and TestCollector

The discovery process answers a single question: given a test root path, what test functions exist under it?

TestRef (codecraftai-testbase/source/packages/ccai/testbase/testref.py:25–65) is the data class. Each TestRef wraps a FunctionType and stores:

• The function reference itself (testfunc: FunctionType).
• A list of monikers — string identifiers used to filter tests by name pattern at discovery time.
• A list of pivots — keyed parametrisation values that vary across instances of the same test function.
• An async flag — whether the function is async def.

TestCollector (codecraftai-testbase/source/packages/ccai/testbase/testcollector.py:60–231) is the orchestrator. Given a test root, includes/excludes, and metafilters, the collector:

1. Walks the file system under the test root.
2. Imports each test module.
3. Calls collect_test_references() (codecraftai-testbase/source/packages/ccai/testbase/queries.py:29–94) on each imported module. That function uses inspect.getmembers(mod, inspect.isfunction) to find functions whose names match the test prefix and wraps each in a TestRef.
4. Applies the includes/excludes/metafilters to prune the references that don't match.
5. Builds a TestGroup tree — a hierarchical structure mirroring the package layout, with TestRef objects at the leaves.

The output of discovery is a TestGroup tree. Nothing has run yet; nothing has been imported except the test modules themselves.

Sequencing: TestSequencer's five phases

Once discovery has produced a tree, the sequencer drives it through five phases. The phases are an enum:

# codecraftai-testbase/source/packages/ccai/testbase/sequencing/testsequencer.py:106–150

class TEST_SEQUENCER_PHASES(IntEnum):
    Initial = 0
    Discovery = 1
    Collection = 2
    Graph = 3
    Traversal = 4

Each phase has a job:

• Initial — sequencer constructed; no work done.
• Discovery — the TestCollector walks the test root. At the end of this phase the tree of TestRef objects exists but no resources have been resolved.
• Collection — the sequencer collects originators (the resource factories from Chapter 3) and integrations needed to satisfy the tests' declared dependencies.
• Graph — the sequencer builds the resource dependency graph and computes the order in which resources must be acquired and released. This is where cyclic dependencies surface.
• Traversal — the sequencer walks the graph, executing tests in dependency-respecting order.

The sequencer's phases are not the same as the TestJob's fire_* hooks. The phases describe how the sequencer thinks about the suite; the hooks describe what events the job broadcasts to the integrations. A given fire_* hook can fire at a particular phase boundary (the docstrings call out which ones — fire_diagnostic_capture_pre_testrun's docstring at testjob.py:487–492 documents its position).

TestJob: the lifecycle owner — and the customisation entry point

TestJob (codecraftai-testbase/source/packages/ccai/testbase/testjob.py:45) is the singleton-like object that owns the run. It has a constructor (line 68) that stores the logger, the test root, the includes/excludes, and — crucially — obtains its run-level metadata and output directories through injected seams rather than reaching into a runtime/configuration layer. The constructor calls two provider hooks: _create_run_metadata() returns an IRunMetadata (line 95) and _create_output_context() returns an IOutputContext (line 96). It is used as a context manager:

with TestJob(logger=logger, testroot="tests/", ...) as job:
    job.execute()

__enter__ (codecraftai-testbase/source/packages/ccai/testbase/testjob.py:109) calls begin(). __exit__ (codecraftai-testbase/source/packages/ccai/testbase/testjob.py:113) calls finalize(). The run itself happens in execute() (line 157), invoked between enter and exit. Across the three methods the job:

1. begin() (line 135): resolves the results directory from the output context (self._output_context.get_test_results_dir()) and computes the artefact filenames under it. Note: begin() does not construct the recorder or the sequencer — those are created lazily inside execute().
2. fire_attach_to_framework(tseq) (line 389): asks the sequencer to attach integrations to the framework — register originators, wire up couplings.
3. fire_attach_to_environment(tseq) (line 379): asks the sequencer to attach integrations to the environment. On a single-host run the integrations may have nothing to attach to; on a distributed run this hook drives device wake-up.
4. fire_diagnostic_capture_pre_testrun(tseq, level) (line 463): captures pre-test diagnostic state — host details, env vars, package versions.
5. The recorder is constructed (inside execute(), line 256) via _create_recorder(...) and entered as a context manager.
6. The run itself: the sequencer's traversal phase walks the tree, executing tests and streaming results.
7. fire_diagnostic_capture_post_testrun(tseq, level) (line 411): captures post-test diagnostic state for failure triage.
8. recorder.finalize(): in the finally block of execute() (line 326), the recorder writes its artefacts and the run summary is rendered to the log.
9. finalize() (line 521): the TestJob-level finalize is a no-op on the base class — there are no integrations to tear down on a single-host standalone run. Subclasses override it when they own teardown.

The fire_* hooks are the broadcast points. Two of them — the attach hooks — do real work even on plain testbase (they delegate to the sequencer). The other three landscape hooks — fire_activate_configuration() (line 399), fire_activate_integration() (line 403), and fire_activate_operations() (line 407) — are stubs on the base class: each logs a debug line and returns. In fact execute() does not even call the activation hooks; they exist purely as override points for the landscape-activating subclass.

Where landscape activation lives. The five landscape fire_* hooks (the two fire_attach_* plus the three fire_activate_*) have bodies in Volume 2's InteropLibraryTestJob (in codecraftai-interop), not here. On plain testbase there is no landscape to activate, so the activation hooks stay stubs. Importantly, IntegratedTestJob in the integration runtime does not override these five hooks either — its job is result recording, not landscape activation. The order is the contract; the interop subclass fills the stubs in. A reader who understands this lifecycle understands the distributed runner's lifecycle too — the order doesn't change, just the hooks' bodies.

TestJob is the customisation entry point

TestJob is not just the lifecycle owner — it is the place where the user customises the framework. Per the meta-thesis from Volume 0 chapter 1 ("framework supplies the skeleton and patterns; the organisation customises"), TestJob is the specific, concrete plug-in point.

The customisation works by subclassing. The user picks a starting layer and overrides the template methods they need. There is a four-deep inheritance chain, each layer adding one concern:

• TestJob (single-host base, this chapter; in codecraftai-testbase). Standalone: stdout recorder, injected IRunMetadata/IOutputContext seams, the five landscape fire_* hooks as stubs. Override _create_sequencer(), _create_recorder(), _create_output_context(), _create_run_metadata(), or any hook for single-host customisation.
• IntegratedTestJob (in codecraftai-integration-runtime; ccai.integrationruntime.integratedtestjob). Adds integration result recording: it overrides exactly three provider hooks — _create_output_context() to a presence-folder-backed RuntimeOutputContext, _create_run_metadata() to a RuntimeVariables-backed RuntimeRunMetadata, and _create_recorder() to the HTML-rendering JsonResultRecorder. It does not touch the five landscape fire_* hooks.
• InteropLibraryTestJob (Volume 2; in codecraftai-interop). Adds landscape activation: it fills in the five fire_* hooks and declares the abstract _get_landscape(). Override _get_landscape() to return your project's Landscape subclass; the framework picks up the custom Features and topology shapes through that landscape.
• DistributedTestJob (Volume 2; in codecraftai-testplus). Adds the distributed sequencer and a concrete _get_landscape() for distributed multi-host scenarios.

In-flight note. The chain above is the as-built design, but the re-parenting is not fully landed: IntegratedTestJob is built and in place, yet InteropLibraryTestJob still extends TestJob directly in the interop source today (codecraftai-interop/source/packages/ccai/interop/testing/testjob.py:22), not IntegratedTestJob. Slotting interop onto IntegratedTestJob is the remaining step. When it lands, an interop run will pick up the presence-folder output context and HTML recorder for free; until then interop runs still get the testbase defaults for those.

The framework's entry points accept the user's subclass:

# Single-host
generic_test_entrypoint(job_type=YourCustomJob)

# Distributed
distributed_test_entrypoint(job_type=YourCustomDistributedJob)

Real scaffolding examples live in codecraftai-testbase/source/tests/ccai_tests/testbase/integration/test_custom_job_type_integration.py (CustomTestJob, FailingCustomTestJob). An SDET writing a custom job type follows that pattern: subclass at the right layer, override the template methods, plug into the entry point.

The construction cascade that follows from a custom TestJob:

User-defined TestJob subclass             ← CUSTOMISATION ENTRY POINT
    │
    ├─ creates → TestSequencer (custom or default)
    │              │
    │              ├─ creates → Landscape (user's subclass — Vol 6)
    │              ├─ creates → Topology  (user's subclass — Vol 6)
    │              └─ orchestrates the lifecycle
    │
    └─ owns the lifecycle (the five fire_* hooks above)

Everything Vols 2 and 6 cover about Landscape and Topology subclassing hangs off this cascade. The user's TestJob subclass is the integration point; the rest is what the template methods return.

The recorder pipeline

The base TestJob records to stdout by default. _create_recorder() (codecraftai-testbase/source/packages/ccai/testbase/testjob.py:564) returns a StdoutResultRecorder (codecraftai-testbase/source/packages/ccai/testbase/results/recorders/stdoutresultrecorder.py:36), not a JsonResultRecorder. The recorder is constructed inside execute() (not begin()) and entered as a context manager; every test event the sequencer fires (test started, test finished, error, skip) is forwarded to it.

StdoutResultRecorder is a thin subclass of JsonResultRecorder that reuses the JSOS/JSON serialisation but skips the HTML rendering: its initialize_report() is a no-op (no template to publish) and its finalize() echoes the run summary to stdout once. So a standalone testbase run prints its summary and — when an output directory is in play — still writes the machine-readable artefacts that the testbase agents CLI consumes:

• testrun_summary.json — written at finalize. High-level pass/fail counts, duration, environment fingerprint. This is what a green/red dashboard reads.
• testrun_results_stream.jsos — written as tests finish. Each test event becomes a JSON record; records are separated by an ASCII record-separator character. A consumer can tail this file and parse incrementally. Chapter 5 covers the format in depth.
• testrun_results.json — written at finalize. The consolidated final document containing the same information as the stream file but reorganised for batch consumption.

What plain testbase does not produce is the rendered HTML summary. That, along with the presence-folder results layout and the CI/build/pipeline metadata, is the responsibility of IntegratedTestJob in codecraftai-integration-runtime, which overrides _create_recorder() to return the full JsonResultRecorder (codecraftai-testbase/source/packages/ccai/testbase/results/recorders/jsonresultrecorder.py:90).

Standalone vs. integrated. The split is deliberate. Plain testbase runs without activation — no presence folder, no RuntimeVariables, no this-app — and records to stdout plus the JSOS/JSON files under whatever output directory the SimpleOutputContext resolves (AM_OUTPUT_DIRECTORY or <cwd>/.output). The well-known presence-folder results directory, the HTML summary, and the build/pipeline/job metadata are layered on by IntegratedTestJob via its three overridden provider hooks (_create_output_context, _create_run_metadata, _create_recorder).

Couplings: where integrations live

A coupling is a piece of code that lives between testbase and some other system. Three families exist:

• IntegrationCoupling — the most common. An integration coupling represents "testbase needs to know about that external thing." Examples: a coupling to a credentials store, a coupling to a device pool, a coupling to a dataprofile registry. The base class is at codecraftai-testbase/source/packages/ccai/testbase/coupling/integrationcoupling.py, and the public method establish_connectivity(allow_missing_devices=False) (line 512) is the place where the coupling decides whether it can run with whatever it has.
• ConfigurationCoupling — represents "testbase needs to consume a configuration source" — runtime files, landscape files, etc.
• ValidatorCoupling — represents "testbase needs to validate something against a schema, a policy, or an evolving system state." Validators are the framework's observer mechanism: one or more validators attach to a test scope via attach_to_test(testscope, suffix) and run in tandem with the test, producing their own outcomes recorded alongside the test's result. The base class lives in codecraftai-testbase/source/packages/ccai/testbase/tandem/validators.py. This is the substrate for tandem testing — the framework's answer to "how do you run a test deterministically against a distributed system without forbidding observation." Volume 2 covers the tandem model in depth; the single-host text in Chapter 9 of this volume carries the bridge.

Couplings are registered as integrations during the Collection phase, attached to the framework during fire_attach_to_framework, and connected during fire_attach_to_environment. Chapter 3 covers the coupling lifecycle in detail when discussing how resource factories are scoped.

Volume 2 extends the coupling model with landscape-aware couplings. Volume 3 extends it further with credential-store couplings that fetch from external secret managers. The model is the same in all three volumes — only the kinds of things being coupled change.

Decorators: what's in the public API

A reader skimming a testbase test file will see a small set of decorators. They divide cleanly:

• Discovery / classification — none. Tests are plain functions; no @testbase.test decorator exists. The name prefix is the contract.
• Resource attachment — @testbase.attach.parameter(...) attaches a parameter (a fixture, in pytest vocabulary) to a specific test (codecraftai-testbase/source/packages/ccai/testbase/declaration/attachments.py).
• Resource origination — @integration_factory(), @configuration_factory(), and @resource_factory() register the source of a resource (codecraftai-testbase/source/packages/ccai/testbase/declaration/registrations.py:45–93). These are usually written in framework or extension modules, not test files.
• Property-based testing — @given(...) is the Hypothesis-style decorator declaring strategies for a test's parameters (codecraftai-testbase/source/packages/ccai/testbase/properties/decorators.py:23–97). Chapter 4 covers it.
• Skip / conditional — @skip_if(...), @only_if(...), and the metafilter family. These are evaluated at discovery time and may prune the TestRef from the tree before the Graph phase.

The set is intentionally small. Per the Volume 0 chapter 2 discipline on decorators, testbase decorators are either metadata-attaching (@given, @skip_if) or registration (@integration_factory). None of them modify behaviour silently. None of them inject a retry or timeout — those are parameters on the test or on the fixture, per the Aspects pattern from Volume 0 chapter 4.

Async support

Tests can be async def. The TestRef carries an async flag, and the property runner / sequencer call into asyncio for those tests. The handling is uniform — async tests live in the same tree, get the same lifecycle, produce the same JSOS records. The framework does not present "async tests" as a separate category to the test author.

Where the test root lives

Throughout testbase the test root is the directory that contains test packages. The convention across the portfolio is:

<repo>/source/testroots/ccai/tests/
                                 ├── singlehost/
                                 ├── distributed/
                                 └── ...

codecraftai-agent-kit's test root is the basis for Chapter 10's case study. codecraftai-testbase's own test root is at codecraftai-testbase/source/tests/ and demonstrates testbase testing testbase — a useful self-application that catches framework regressions.

A run, end to end

Bringing the pieces together: what happens when an SDET runs

testbase run --testroot tests/ --include "test_arithmetic*"

(rough form — the actual CLI uses arcli; Chapter 8 walks the syntax):

1. The arcli entry point parses the command and constructs a TestJob (its constructor resolves the IRunMetadata and IOutputContext seams).
2. The job's __enter__ runs begin(), which resolves the results directory from the output context and computes the artefact filenames. execute() then drives the rest.
3. execute() creates the sequencer and enters the Discovery phase: walks tests/, imports modules, calls collect_test_references(), applies filters, builds the TestGroup tree.
4. The sequencer enters Collection: collects originators and integrations.
5. The job fires fire_attach_to_framework and fire_attach_to_environment. Couplings register and establish connectivity.
6. The sequencer enters Graph: builds the resource dependency graph, plans acquisition order.
7. The job fires fire_diagnostic_capture_pre_testrun. (On the base TestJob the three fire_activate_* hooks are stubs and are not called from execute() at all; a landscape-activating subclass is where they do work.)
8. The recorder is created via _create_recorder() — StdoutResultRecorder on plain testbase — and entered.
9. The sequencer enters Traversal: for each test in dependency-respecting order, the recorder receives a "started" event, the test runs, the recorder receives a "finished" event, and a record is written to the JSOS stream.
10. The job fires fire_diagnostic_capture_post_testrun.
11. In the finally block, recorder.finalize() writes the summary and consolidated documents and echoes the summary to stdout; the run summary is rendered to the log. __exit__ then calls TestJob.finalize(), a no-op on the standalone base.

Each numbered step has a chapter reference somewhere in this volume. The next chapter goes deep on step 4 and 6 — the resource dependency graph, the way originators register, and the way @testbase.attach.parameter lines up with it all.

What's next

• Chapter 3 — Resource factories and dependency injection. How originators register, how @testbase.attach.parameter and @integration_factory differ, how the dependency graph is built.
• Chapter 4 — Property-based testing with @given. The TESTBASE_MAX_EXAMPLES_CAP ceiling, the SHA-256(job_seed + ":" + test_name) per-test seed, the distinction between a strategy's max_examples and the cap.
• Chapter 5 — Streaming JSOS results. The format, the three output files, the consumer patterns.