Chapter 10 — Case Study: A 3,700-Test Suite That Runs on One Host

Chapters 1 through 9 walked the framework one concept at a time. This chapter watches the framework run. The subject is the codecraftai-agent-kit single-host suite — 292 test files, more than 3,700 test invocations, finishing under an hour with TESTBASE_MAX_EXAMPLES_CAP=10. Every concept from the previous chapters lands in the same run.

A book chapter on test framework architecture is only as honest as the test suite it can point at. This chapter points at one: codecraftai-agent-kit, the test root at source/testroots/ccai/tests/singlehost/, used as the reference single-host suite for the portfolio.

The suite predates this book. It was written by the same SDETs who wrote testbase, against the framework testbase itself. It is the place in the portfolio where the framework's design assumptions get exercised at scale — where "thousands of tests" stops being an aspirational claim and becomes a measurement.

The shape of the suite

codecraftai-agent-kit/source/testroots/ccai/tests/singlehost/
├── agentkit/
│   ├── test_costtracker_properties.py
│   ├── test_costtracker_observability.py
│   ├── test_invocation_log.py
│   ├── test_memory_management.py
│   └── ... (dozens more)
├── tooling/
│   ├── test_tool_registry.py
│   ├── test_tool_dispatch.py
│   └── ...
├── observability/
│   └── ...
└── ...

By the numbers (counted at the time of writing this volume):

• Test files: 292 (across the singlehost/ root).
• def test_ declarations: 308.
• Test invocations: more than 3,700 (a single property- based test with max_examples=10 becomes 10 invocations; with max_examples=50 it becomes 50; the total across the suite is the sum).
• Wall time at TESTBASE_MAX_EXAMPLES_CAP=10: ~49 minutes on a single CI runner.
• Wall time at full sweep (cap unset): several hours.

The 3,700-invocation count is the one that matters for the case study: the suite is worth running because it covers that many distinct executions; the cap makes it runnable in a CI window.

Why this suite is the right case study

codecraftai-agent-kit is the agent SDK from Volume 7 — the Python toolkit for building production AI agents on top of the foundation tier. The single-host test root exercises:

• Cost tracking (token counting, budgeting).
• Tool dispatch (mapping tool calls to handlers).
• Memory management (long-conversation truncation).
• Observability hooks (events, traces).
• Configuration loading (Volume 0 chapter 9 in practice).
• Error vocabulary (Volume 0 chapter 3 — the agent raises typed errors the host application can catch).

Every chapter of Volume 1 lands somewhere in this suite. Property-based testing (Chapter 4) is heavily used in the cost-tracker tests. The JSOS stream (Chapter 5) is the suite's output. The agents CLI (Chapter 6) is the way the suite's results are read. Mocks (Chapter 7) stand in for the model-call boundary. arcli (Chapter 8) drives the test runner. The landscape boundary (Chapter 9) is explicitly not invoked — the suite runs on one host with mock couplings.

Run anatomy: the wall-clock breakdown

A representative full run, instrumented with phase markers in the JSOS stream, breaks down approximately as:

The numbers are not surprising — the property-based tests dominate. At cap=10, each property test runs 10 iterations; at the full Hypothesis-default of 100 the same tests would run 10× longer, pushing the suite north of 7 hours.

This is the budget Chapter 4 promised. The cap is the knob.

Property-based tests in the wild

A representative cost-tracker property test:

# codecraftai-agent-kit/source/testroots/ccai/tests/singlehost/
# agentkit/test_costtracker_properties.py

import testbase
from testbase.strategies import integers
from ccai.agentkit.costtracker import CostTracker

@testbase.given(
    input_tokens=integers(min_value=0, max_value=10**6),
    output_tokens=integers(min_value=0, max_value=10**6),
    constraints={"max_examples": 50},
)
def test_total_cost_is_input_plus_output_for_any_token_counts(
    input_tokens: int,
    output_tokens: int
) -> None:
    """
        Verify that total cost equals input cost plus output cost
        across the full range of token counts.
    """
    tracker = CostTracker(model="claude-opus-4-7")
    tracker.record_input(input_tokens)
    tracker.record_output(output_tokens)

    expected = (
        tracker.input_unit_cost * input_tokens
        + tracker.output_unit_cost * output_tokens
    )
    actual = tracker.total_cost()
    rtnval = (actual == expected)
    assert rtnval
    return

Three observations:

1. The strategy is integers(0, 10**6) for both tokens — covers realistic and adversarial inputs.
2. max_examples=50 is what the test author wrote. At TESTBASE_MAX_EXAMPLES_CAP=10 the runner runs 10. At TESTBASE_MAX_EXAMPLES_CAP=200 the runner runs 50 (the cap is an upper bound, not a multiplier).
3. The test follows Volume 0 chapter 2's discipline: typed parameters, explicit return, rtnval named intermediate, no compound return statements.

A run with cap=10 generates 10 (input_tokens, output_tokens) pairs from the SHA-256-derived per-test seed (Chapter 4). The same job seed reproduces the same 10 pairs. A failure is reproducible end-to-end without re-running the suite.

Mocking the model-call boundary

The agent SDK's central concern is talking to language models. None of the single-host tests actually call a model. Instead, the test root's factory module yields a mock:

# codecraftai-agent-kit/source/packages/ccai/agentkit/factories.py

@testbase.factories.resource_factory(scope=ResourceLifespan.Test)
def mock_anthropic_client():
    """
        Yields a Test-scoped mock of the Anthropic client for
        single-host tests.
    """
    client = Mock(spec=AnthropicClient, mock_type=MockType.Instance)
    client.messages.create = SideEffect(
        lambda **kwargs: build_canned_response(**kwargs)
    )
    yield client
    return

The build_canned_response function returns a deterministic response shape for any prompt. The test does not depend on the language model's actual output — it depends on the shape of the response, which the SDK then parses.

Tests of the parser attach mock_anthropic_client; tests of the cost tracker attach a different mock that yields specific token counts; tests of the tool dispatcher mock the tool callable. Each mock is local to its test domain; none of them pretends to be the full model.

What the suite proves about the framework

A test suite of this size, finishing in under an hour, is a specific claim about the framework: that the framework's overhead per test is tiny. Working back from the wall times:

• Property-based: 38 min / ~3,000 invocations ≈ 760 ms per invocation — most of which is the test body itself, not the runner.
• Example-based: 9 min / ~700 tests ≈ 770 ms per test — similar.

The runner overhead per invocation is on the order of single-digit milliseconds. This is the cost of one JSOS record write, one resource-factory generator step, one sequencer dispatch. The test framework's plumbing is cheap; the test bodies themselves are what take the time.

This is by design. Chapter 5's commitment to flushing after every JSOS record was a per-record cost the framework deliberately accepts; the case study shows that even at 3,700 invocations the cost is in the noise.

Reading the run with the agents CLI

After the suite finishes, the standard inspection pattern:

# Did anything fail?
testbase agents summary extract --results results/20260430-153012/
# Output: {"passed": 3680, "failed": 18, "errored": 4, "skipped": 10, ...}

# What failed?
testbase agents testresult failures --results results/20260430-153012/
# Output: {"failures": [...18 entries...]}

# How was the run configured?
testbase agents config extract --config-file results/20260430-153012/startup-configuration.json
# Output: {"testroot": "...", "max_examples_cap": 10, "job_seed": "...", ...}

The three commands are the three audiences from Chapter 6. A CI dashboard reads the summary; an engineer (or AI agent) reads the failures; a build engineer auditing a flaky run reads the config extract to learn the seed and the cap that were in effect.

A specific use of the config-extract output: the build engineer can re-run the suite with the same job seed — making property-based runs reproducible across machines. The seed is in the JSON; pasting it into the next run's invocation reproduces every per-test seed via the SHA-256 derivation (Chapter 4).

What the cap does to coverage

The cap reduces the number of generated examples per property test. Does it reduce coverage?

In the strict sense, yes — fewer inputs are tested. In the practical sense, less than expected, for two reasons:

1. The strategies are designed to find edge cases quickly. integers(0, 10**6) doesn't generate uniform random integers; it biases toward boundary values (0, 1, the maximum) and around the midpoint. Most failures surface in the first few examples regardless of cap.
2. The job seed varies across runs. A property test with cap=10 sees different inputs in tomorrow's CI run than in today's, because the job seed is fresh. Over many runs the suite covers a much wider input space than any single run.

The trade-off is "depth in any single run" vs. "breadth across many runs." The portfolio chooses breadth: cap=10 in CI on every PR; full-sweep cap-unset in nightly. An adversarial input that surfaces in nightly's run gets captured (with its job seed) and replayed in regression tests; the next day's CI runs include it as a deterministic example.

This pattern — "find adversarial inputs at full sweep, lock them as regression tests" — is the loop that makes property-based testing economical. The cap makes the loop fast.

What the case study does not show

A few things this case study deliberately doesn't demonstrate, because they belong to other volumes:

• Real model calls. Volume 7 (the agent SDK volume) walks tests that actually call a language model. Those tests run as a separate (smaller, slower) integration suite. Volume 1's case study is the unit-and-property suite that runs on every PR.
• Distributed execution. Volume 2's distributed-test case study runs on a multi-host landscape. The single- host suite here is the substrate; the distributed suite extends it.
• CI integration. Volume 3 walks the orchestration layer that invokes a testbase suite from a taskplus workflow. The case study here treats testbase as invoked directly; Volume 3 wraps it.

This volume's job is to make the framework legible. The case study is a demonstration of legibility at scale.

The shape of a successful run

What the engineer sees on a green run, abbreviated:

$ testbase run --testroot source/testroots/ccai/tests/singlehost/ \
              --max-examples-cap 10

[2026-04-30 15:30:12] Phase: Discovery (started)
[2026-04-30 15:30:24] Phase: Discovery (finished, 292 modules, 308 tests)
[2026-04-30 15:30:25] Phase: Collection (started)
[2026-04-30 15:30:28] Phase: Collection (finished, 47 originators)
[2026-04-30 15:30:28] Phase: Graph (started)
[2026-04-30 15:30:29] Phase: Graph (finished, 354 nodes, 802 edges)
[2026-04-30 15:30:29] fire_attach_to_environment
[2026-04-30 15:30:31] fire_attach_to_framework
[2026-04-30 15:30:32] Phase: Traversal (started)
[2026-04-30 15:30:33] PASS test_addition_is_commutative (17 ms)
[2026-04-30 15:30:33] PASS test_subtraction_is_anti_commutative (12 ms)
[2026-04-30 15:30:34] PASS test_total_cost_is_input_plus_output... (10 examples, 84 ms)
... (3,700+ more lines) ...
[2026-04-30 16:19:38] Phase: Traversal (finished)
[2026-04-30 16:19:39] fire_diagnostic_capture_post_testrun
[2026-04-30 16:19:42] Run complete.

Summary: 3680 passed, 18 failed, 4 errored, 10 skipped (49m 27s)
Stream:  results/20260430-153012/testresults_stream.jsos

Every line traces back to a chapter. Discovery: Chapter 2. Collection / Graph: Chapter 3. fire_attach_*: Chapter 9. PASS / FAIL counts: Chapter 5. The wall-time figure: this chapter.

What this volume gives the reader

By the end of the case study a reader who started Volume 1 cold should be able to:

• Read a testbase test file and identify each construct's framework concept.
• Choose a sensible TESTBASE_MAX_EXAMPLES_CAP for their CI and full-sweep runs.
• Use the agents CLI to extract failures, summaries, and configuration from a results directory.
• Tell when a test should mock its dependency vs. depend on a real coupling.
• Tell which tests in a suite cross the single-host / distributed boundary.
• Read an arcli-decorated CLI's source and predict its behaviour from the decorators alone.

That fluency is the substrate Volume 2 builds on. Volume 2 extends TestJob to InteropLibraryTestJob to DistributedTestJob. The lifecycle does not change; the hooks do. Volume 2 picks up exactly where this case study ends — with the same suite running across multiple hosts, the same JSOS streaming format, the same agents CLI output, but with the fire_* hooks doing landscape activation instead of nothing.

End of Volume 1

The volume started by asking what's missing from unittest and pytest for automation testing (Chapter 1) and ended by watching a 3,700-test suite finish in under an hour because the framework filled the gaps (this chapter).

The chapter that comes next is in another volume. Volume 2 opens with "Why distributed automation is genuinely hard" and then walks the inheritance chain that extends what this volume built.