# Chapter 1 — Why `unittest` and `pytest` Fall Short for Automation > *Application unit tests verify a function in isolation. Integration > tests verify two collaborators in isolation. Automation tests > verify a system in motion — devices waking, credentials being > fetched, configurations being merged, jobs being scheduled across > hosts. The tools that ship with Python were never designed for > the third thing. This chapter is about what's missing, and why > patching `pytest` to fill the gap eventually fights the framework > rather than helping it.* `unittest` and `pytest` are excellent tools. Across millions of Python codebases they verify the correctness of functions, classes, modules, and small graphs of objects. The `codecraftai-*` portfolio uses both — `pytest` runs on the framework's *own* unit tests, `unittest` is the foundation of `unittest.mock`'s spec-aware mocking, and the discipline of writing testable functions started in those communities long before automation engineering picked it up. This volume is not a polemic against pytest. It is an explanation of why a particular kind of test — the kind that runs against a *system* rather than a *function* — needs a different framework. That framework is `codecraftai-testbase`, and the rest of Volume 1 walks through it. This chapter establishes *why testbase exists* by listing the constraints automation testing places on a runner, then asking which of those constraints `pytest` was designed to satisfy. The answer turns out to be: not many. That's not a criticism of pytest. It's a statement about scope. ## The shape of an automation test An automation test, in this volume's sense, is a test where: 1. The system under test is **not in the test process**. The system under test is a device, a fleet of devices, a service running on another machine, or a constellation of services coordinated by an orchestration layer. 2. The test must **acquire and release real resources** — SSH sessions, credentials, license tokens, devices that other tests in the suite may also need, network namespaces. 3. The test must **survive failures it can't reproduce locally**. A flaky network on host 7 of 12 is part of the production environment; the framework has to tell you what happened on host 7 with enough detail that you don't need to reproduce. 4. The test must **stream results** while it runs, not write them all at the end. A four-hour suite that crashes in hour three should still tell you what passed in hours one and two. 5. The test must **be machine-readable** — by CI dashboards, replay tools, AI agents extracting failure patterns. Stdout and a green-or-red exit code don't carry enough information. 6. The test must **describe its environment** — which devices it needs, which credentials, which configuration profile — so that a runner on another machine can reproduce or refuse the run. The first three of these are what most engineers mean when they say "automation testing is hard." The last three are what most engineers don't notice until they try to scale a test suite past about five hundred tests across more than one host. Automation test frameworks live or die on the last three. ## What `pytest` does well It is worth being precise about pytest's strengths before explaining why testbase exists, because dismissing pytest is a mistake too — many of testbase's design choices are pytest's choices made differently for a different problem. Pytest does the following extremely well: - **Test discovery is implicit.** Functions whose names start with `test_` in files whose names match `test_*.py` are tests. No registration ceremony. - **Fixtures compose cleanly.** A fixture can depend on other fixtures, and pytest works out the dependency order. - **Parametrisation is first-class.** `@pytest.mark.parametrize` generates test variants by inspection of the test function's parameters. - **Plugins are first-class.** Reporting, parallelism, randomisation, coverage, hypothesis, xdist — the plugin ecosystem is the framework's strongest feature. - **Failure reports are diagnostic.** Pytest's assertion rewriting gives you the values of the operands of a failed assertion without having to write `assertEqual(a, b, "...")`. `codecraftai-testbase` keeps the first three properties. Tests are plain functions whose names start with `test_`, found by inspecting modules under a test root (see Chapter 2). Fixtures compose; they are just expressed differently. Parametrisation exists in two forms — explicit data-driven (akin to pytest's `parametrize`) and property-based (akin to Hypothesis). What testbase *doesn't* keep is the assumption that the framework runs in the same process as the system under test. ## What automation tests need that pytest doesn't ship There are seven gaps. The remainder of Volume 1 is, in effect, an extended explanation of how testbase fills them. ### 1. A lifecycle wider than the test function A pytest test has fixtures with `function`, `class`, `module`, `session` scope. That's enough when the system under test lives inside the Python process. When the system under test is *outside* the Python process, you need lifecycle hooks for things that don't map onto fixtures: - **Attaching to the framework**: registering integration couplings, binding the test runner to a results pipeline. - **Attaching to the environment**: connecting to devices, opening SSH sessions, fetching credentials. - **Activating configuration**: telling the landscape that this particular run wants this particular profile. - **Activating integration**: bringing online the cross-host collaboration the suite depends on. - **Activating operations**: starting the side processes (log shippers, metric agents) that other tests will rely on. In `codecraftai-testbase`, these are five `fire_*` hooks on `TestJob`: `fire_attach_to_framework`, `fire_attach_to_environment`, `fire_activate_configuration`, `fire_activate_integration`, `fire_activate_operations` (file: `codecraftai-testbase/source/packages/ccai/testbase/testjob.py:379–408`). On a single-host run all five are no-op stubs — they exist because Volume 2's `InteropLibraryTestJob` (in `codecraftai-interop`, `source/packages/ccai/interop/testing/testjob.py:22`) overrides them to drive landscape activation. The hooks are *the* place where testbase's single-host runner and Volume 2's distributed runner meet. A pytest fixture cannot model "I need the entire test session to have woken up the lab's twelve devices and pre-fetched their credentials before the first test starts, and put them to sleep after the last test ends" without contortions. A `fire_*` hook on a job-spanning singleton can. ### 2. Tests that declare what they need, not what they import A pytest test imports its fixtures. An automation test cannot always do that, because the resource it needs may not be present on the host the test author is editing on. In testbase the resource declaration is a separate decoration: ```python @testbase.attach.parameter(some_device, identifier="dev") def test_device_responds(dev): """ Verify that the lab device responds to a probe. """ response = dev.probe() rtnval = response.is_ok() assert rtnval return ``` The `@testbase.attach.parameter(source, identifier=...)` form (`codecraftai-testbase/source/packages/ccai/testbase/declaration/attachments.py`) uses an originator to produce the parameter, with constraints on lifespan and assigned scope. The originator can come from a *different package* than the test, registered via the namespace package extension model from Volume 0 chapter 7. A test author on a laptop without the device declares the dependency without having to import a module that would fail at import time. ### 3. Results that stream, not aggregate Pytest writes its summary at the end. Plugins exist to write JUnit XML or other formats, but the framework's mental model is "run all the tests, then report." An automation suite that takes hours to run cannot afford to lose its history when the runner crashes. Testbase produces three result artefacts during a run: - `testsummary.json` — the high-level summary of pass/fail counts. - `testresults_stream.jsos` — a stream file written *as tests finish*, one record per event, prefixed by an ASCII record separator character (`codecraftai-testbase/source/packages/ccai/testbase/results/recorders/jsonresultrecorder.py:90–183`). - `testresults.json` — the consolidated final document, written at the end. The stream file is the one that matters for hours-long runs. A CI dashboard, a local tail process, or an AI agent reading `failures` (Chapter 6) can begin extracting failures *while the suite is running*. A crashed run still leaves a partial stream that tells you what passed and what hadn't started. Chapter 5 walks the JSOS format in detail. ### 4. Property-based testing with a CI fast path Hypothesis is excellent at finding edge cases and shrinking them. But Hypothesis defaults — generating examples until either the test passes a budget or finds a failure — are calibrated for running locally with attention. A 3,700-test suite that triggers Hypothesis's default `max_examples=100` on every property test will run for hours without finding anything new. Testbase ships an environment-variable cap: `TESTBASE_MAX_EXAMPLES_CAP` (`codecraftai-testbase/source/packages/ccai/testbase/properties/propertyrunner.py:47–79`) caps the per-test `max_examples` regardless of what the test declared. CI runs at `TESTBASE_MAX_EXAMPLES_CAP=10` to keep the fast path under an hour; full-sweep runs leave the cap unset. The cap is a runtime knob, not a test-author knob — exactly the right shape for a build-engineer tuning the suite without editing hundreds of test files. Chapter 4 is the full treatment. ### 5. A machine-readable surface Modern CI dashboards extract test summaries by parsing stdout. That works until the dashboard wants to do something more ambitious — diff the failing tests against last week's, group failures by which device they touched, drive an AI agent that decides whether a flake is real. Stdout doesn't survive that. Testbase ships an `agents` CLI subcommand (`codecraftai-testbase/source/packages/ccai/testbase/cli/cmdtree/agents/__init__.py:18–28`) with three groups: `summary`, `testresult`, `config`. Each returns structured data — extracted from the JSOS stream by a parser shared across the subcommands (`codecraftai-testbase/source/packages/ccai/testbase/cli/cmdtree/agents/jsosparser.py`) — that an agent or dashboard can consume directly. The agent-readable surface is *the same* surface a human reads; Chapter 6 walks the contract. ### 6. A landscape boundary The most subtle gap. An automation test runs in an environment — a "landscape" in the portfolio's vocabulary. The landscape includes devices, services, credentials, network reachability. The test framework needs to be aware that the landscape *exists*, without dragging the entire landscape implementation into the unit-test process. Testbase solves this with a *contract*: `TestJob` exposes the five `fire_*` hooks as no-op stubs. `InteropLibraryTestJob` (Volume 2) overrides them to drive landscape activation. A single-host test using `TestJob` directly never needs the landscape; a distributed test using `DistributedTestJob` (also Volume 2) does. The contract sits exactly at the boundary, so that single-host test authors don't pay landscape's import-time cost and distributed test authors get landscape hooks where the lifecycle actually fires. Chapter 9 carries the single-host half of this contract; Volume 2 carries the distributed half. ### 7. Single actor, multiple observers — tandem testing, not parallel The least-obvious gap, and the one that most cleanly distinguishes automation testing from application testing. In a distributed environment where the machines under test interact with each other, a test is only deterministic when *one actor* causes effects in the environment at a time. When the controller (the test) interacts with a machine, the test must wait for the effect to propagate to all the other machines in the environment before checking that the effect took hold. Often there are multiple checks — a poll loop watching for a state, or an event subscription waiting for a state change. If more than one actor is allowed to cause effects concurrently, the ordering of events stops being deterministic, and the test stops being deterministic with it. This is *not* a parallelism problem the framework can solve by adding locks. It is a determinism property of the system under test. Application testing avoids it by running the system under test inside the test process, where the test is the only controller; automation testing has to enforce it explicitly, because the system under test is *outside* the process and nothing prevents another actor from reaching it. Pytest's plugin ecosystem includes `pytest-xdist` for parallel test execution. That plugin is exactly the wrong tool for an automation suite where the tests share an environment. Two parallel tests against the same fleet *will* race on shared state and the suite *will* turn flaky. Testbase enforces single-actor execution by treating the test itself as the actor and providing a separate mechanism for *observers*: - The **test** is the single actor. - **Observers** are `Validator` instances (subclassing `ValidatorCoupling`) that the framework attaches to the test scope via `attach_to_test(testscope, suffix)`. They watch for effects, run *in tandem* with the test, and produce their own per-observer outcomes recorded alongside the test's result. This is **tandem testing**: one actor + N attached observers. The mental model is a *tandem bicycle*: one driver pedals and steers; one or more observers ride along watching the world go by. The driver is the only one steering — that's the single-actor commitment — but the observers are full participants in the ride, with their own perspective and their own report at the end. File: `codecraftai-testbase/source/packages/ccai/testbase/tandem/validators.py`. Where parallelism *does* live: at the **job level**, across independent **Pods**. A Pod is a set of devices configured to interoperate with each other (Volume 6 covers Pod formation in detail). Within one Pod, jobs run sequentially because a Pod is a shared environment. Across Pods, jobs run in parallel because Pods are independent — effects in one Pod cannot propagate to another. A CI orchestration that wants more horizontal scale provisions more Pods, not more parallel tests within a Pod. Volume 3 covers the orchestration side; Volume 2 covers the distributed test side; the two volumes meet at the Pod boundary. ## Where `unittest` still lives in the portfolio There is a second, quieter reason this chapter is not a polemic against the standard library: testbase does not try to replace `unittest` everywhere. It replaces it *almost* everywhere — and the handful of packages it deliberately leaves on `unittest` are the ones that make this honesty possible. Earlier in the portfolio's life, the set of packages that had to stay on plain `unittest` was large, because testbase dragged in a chain of dependencies — configuration, the runtime foundation, this-app — that many low-level packages could not import without a cycle. That chain is being severed. testbase is becoming **near-standalone**: its intended dependencies are exactly `codecraftai-arcli`, `codecraftai-foundations` (which consolidated `codecraftai-errors`, `codecraftai-interfaces` and `codecraftai-waiting`), and `codecraftai-xmodules` — no configuration tier, no runtime foundation, no this-app. The *core* runner already lives on that floor: `TestJob` reads its handful of environment knobs directly via the dependency-free `ccai.testbase.environment` module, and it resolves output and run metadata through injected seams (`IOutputContext` / `IRunMetadata`) rather than importing the runtime. The presence-folder results layout, the JSOS-to-disk-with-HTML summary, and the CI/build metadata that used to be testbase's business are now the job of `IntegratedTestJob` in `codecraftai-integration-runtime`; plain testbase records to stdout and runs without activation. Chapter 2 and Volume 2 walk that seam. This is partly in-flight. The runner core is off the old chain, but testbase's *CLI and bootstrap* (`ccai.testbase.initialize`, the `cli/` command tree, the result server) still import `ccai.runtime.*` and `ccai.thisapp.*`, and `pyproject.toml` still lists configuration, runtime, and this-app among its dependencies. Finishing that decoupling — and trimming the dependency list to the five-package floor — is the remaining step; the architecture below describes the floor the package is converging on. The practical consequence is that testbase can now be used to test **almost every other package in the portfolio** — the device abstractions, the interop layer, the orchestration tiers, the REST clients. A package no longer has to choose `unittest` just because it sits below the framework's old dependency chain. What remains on `unittest` is a small, **permanent** exclusion set: - `codecraftai-foundations` (formerly the separate errors, interfaces and waiting packages) - `codecraftai-xmodules` - `codecraftai-arcli` - `codecraftai-extension` The rationale is a single rule with one product decision attached. The first five are testbase's *own* dependencies — a framework cannot test the packages it is built on without a bootstrap cycle, so those stay on `unittest` by necessity. (testbase depends on `arcli` for its own CLI, which is why `arcli` is on this list even though it is otherwise a general-purpose toolkit.) The sixth, `codecraftai-extension`, is the product decision: it underpins the namespace-package extensibility model that `IntegratedTestJob` registers through (the same model Volume 0 chapter 7 introduced), and keeping it on `unittest` avoids a circularity between the extension machinery and the framework that consumes it. Everything outside that set has shifted — or can shift — to testbase. The split is not "framework versus standard library as a matter of taste"; it is a precise dependency-floor boundary, and the list above is the floor. That shift is no longer hypothetical. Across the portfolio's unittest-bearing packages — the `configuration` and `this-app` foundation tiers, the `credentials` and `networking` device layers, the `integration-runtime` reporting package, the `timetravel-pod` tracer, the `storage-database` and `dataprofiles` libraries, and the `sysman-common` tier — the tests have been converted to testbase at the source level. Each package's tests now live as plain `def test_*` functions under `source/testroots/ccai/tests/…` (discovered by the runner via `--includes`) where they had been `unittest.TestCase` methods under `source/tests/`. The conversion is one-to-one: every original test method becomes exactly one test function; `setUp` / `tearDown` state is replicated per test through module-level helper functions (testbase tests have no class lifecycle); `unittest.mock` is retained as-is; duplicate method names across sibling `TestCase` classes are disambiguated rather than collided; and `IsolatedAsyncioTestCase` tests are wrapped with `asyncio.run`. The migration keeps both suites in parallel for safety — the legacy `unittest` files remain authoritative, and the merge gate keeps running them, until each converted suite is verified green under the testbase runner and the package's CI is cut over to `testbase testing run`. (Two of the converted tiers, `this-app` and `configuration`, are a special case: testbase still declares them among its own dependencies, so they cannot take a testbase test-dependency — and thus cannot run under the testbase runner — until testbase finishes shedding that chain. Their test *sources* are converted; their cutover waits on the decoupling described above.) ## What this volume builds Volume 1's ten chapters walk testbase from the outside in: - Chapter 2 covers the architecture — the pieces that make a run go. - Chapter 3 covers resource factories — how tests acquire and release the things they need. - Chapter 4 covers `@given` and the property-based testing model with the CI cap. - Chapter 5 covers JSOS streaming results. - Chapter 6 covers the `agents` machine-readable CLI. - Chapter 7 covers testbase's own mocking primitives. - Chapter 8 covers `arcli`, the declarative CLI toolkit underneath every `codecraftai-*` CLI. - Chapter 9 carries the single-host half of the landscape contract — the public API Volume 2 will extend. - Chapter 10 is a case study: the 3,700-test single-host suite in `codecraftai-agent-kit`, with `TESTBASE_MAX_EXAMPLES_CAP=10`, finishing in a CI window short enough that engineers run it voluntarily. By the end, a senior SDET should be able to read a testbase test file and know exactly which framework concept each line is using. That fluency is the prerequisite for Volume 2 (the distributed extension) and Volume 3 (orchestration-level invocation) to make sense. ## A note on continuity Every chapter in Volume 1 assumes the reader has read Volume 0 — or knows the foundation tier well enough to skip it. When a chapter says "the explicit-`return` discipline" or "the `SemanticError` rule" or "the Aspects pattern," it is referring to material from Volume 0 chapters 1–4. Volume 0 anchored those ideas; Volume 1 uses them. When a chapter forward-references the distributed extension, it will say so explicitly: "Volume 2 chapter N covers the distributed-half of this contract." The single-host text never relies on the distributed text — but it carefully marks the seams, because Volume 2 is the volume that walks across them. ## What we have not written This volume *is not* a tutorial on Python testing. It assumes you can write a passing pytest test, you understand parametrisation, you have used a fixture. The volume's job is to show you how an automation-grade test framework is different — and why those differences are *necessary*, not preferences. This volume *is not* a comparison shop. We do not survey `Robot Framework`, `behave`, `pytest-bdd`, or any other Python testing framework. Those tools have their reasons; testbase has ours. The volume's job is to show why testbase is the shape it is, so that a reader who needs to *extend* it knows where the seams are. This volume *does not* cover distributed test execution. That is Volume 2. Where the contract that distributed extends sits inside testbase, this volume marks the contract. Where the *implementation* lives, Volume 2 picks up. ## What we have committed to By the end of Volume 1 you will: - Know the five `fire_*` lifecycle hooks of `TestJob` and what each does on a single-host run. - Know the resource-factory model and how originators register with the injection registry from Volume 0 chapter 7. - Know the JSOS streaming format and how the `agents` CLI consumes it. - Know `arcli`'s `Command` / `ArgField` / `OptionField` model and the validator-and-parser hardening pattern from PR #13. - Be able to read a testbase test file and identify, line by line, which framework concept each construct represents. That fluency is what Volume 2 builds on. And Volume 3. And parts of Volumes 6, 7, and 8. This is the substrate. The next chapter walks the architecture itself.