Chapter 7 — Testing AI Agents

Testing AI agents is unusual. The output is non- deterministic at the LLM layer; the system under test includes a service that costs money to call; the right answer often depends on subjective evaluation. The SDK's answer is mock-first testing: 305 test files, every LLM and tool mocked, no real API calls. This chapter walks the discipline and what it catches.

A reader who's tested traditional Python code is used to:

• Tests that produce deterministic output.
• Test runs that complete in seconds (no network calls).
• Test failures that point at a specific line.

AI-agent testing has different default constraints:

• LLM responses vary even with the same prompt (temperature, model versioning).
• Real LLM calls cost money and rate-limit.
• LLM "correctness" is often subjective.

The codecraftai-agent-kit's answer: don't test the LLM. Test the SDK's behaviour around the LLM. The 305 test files in the suite test the agent's plumbing — state machine transitions, tool dispatch, memory operations, workflow execution, observability emission. None of them hit a real LLM API.

This chapter walks the discipline.

The numbers

Verified from the source:

• 305 test files at codecraftai-agent-kit/source/testroots/ccai/tests/singlehost/agentkit/.
• ~78,155 LOC of test code.
• 212 top-level test files (mostly test_*.py).
• 72 property-based test files (test_*_properties.py — chapter 8 walks these).
• 5 subdirectory layers:
  • builtins/ (5 files) — built-in tools.
  • platform/ (14 files) — cloud-platform adapters.
  • resilience/ (53 files) — circuit breakers, retry, degradation, dead-letter.
  • steps/ — workflow step execution.
  • transports/ — message transport.

The most recent verified test count: 1,964 / 1,964 PASSED (commit a2e437a, Apr 27 2026).

Mock-first — ccai.testbase.mocking.Mock

The SDK's tests use the framework's standard mocking discipline (Volume 1 chapter 7). Every external dependency is mocked:

• LLM providers — Mock(spec=LLMProvider).
• Tools — Mock(spec=Tool) or test-specific subclasses.
• Memory stores — Mock(spec=MemoryStore) or in-memory fakes.
• Coordinators / external services — mocked.

A representative pattern from test_contextsummarizer.py:16–46:

def _create_mock_llm_provider(summary_text="This is a summary."):
    mock_message = LLMMessage(role="assistant", content=summary_text)
    mock_response = Mock()
    mock_response.configure(recursive=True)
    mock_response.mock_context().set_attribute("message", mock_message)

    mock_provider = Mock()
    mock_provider.configure(recursive=True)
    mock_provider.mock_context().set_attribute("provider_name", "test-provider")
    mock_provider.mock_context().mock_method("complete", return_value=mock_response)
    return mock_provider


def _create_failing_llm_provider():
    def _raise_error(**kwargs):
        raise RuntimeError("LLM failure")
    mock_provider = Mock()
    mock_provider.configure(recursive=True)
    mock_provider.mock_context().set_attribute("provider_name", "failing-provider")
    mock_provider.mock_context().mock_method("complete", side_effect=_raise_error)
    return mock_provider

The pattern:

1. Construct a Mock.
2. .configure(recursive=True) to enable nested mock attribute access.
3. .mock_context().set_attribute(name, value) for properties.
4. .mock_context().mock_method(name, return_value=...) for return values.
5. .mock_context().mock_method(name, side_effect=callable) for exception simulation.

The mock pattern is consistent across all 305 test files. A reader who learns it once reads any test in the suite.

What gets unit-tested

Looking at the test file organisation, the test surface covers:

Lifecycle and state machine

test_agentstatemachine.py exercises:

• Every valid transition.
• Every invalid transition (asserts SemanticError raised).
• Concurrent transitions (thread-safety).
• Hook invocation order.

A typical test:

def test_agent_state_machine_rejects_running_to_initialized():
    """
        Verify that RUNNING → INITIALIZED is not a valid transition.
    """
    sm = AgentStateMachine()
    sm.transition(LifecycleState.INITIALIZED)  # CREATED → INITIALIZED
    sm.transition(LifecycleState.RUNNING)       # INITIALIZED → RUNNING

    # RUNNING → INITIALIZED is not in VALID_TRANSITIONS.
    try:
        sm.transition(LifecycleState.INITIALIZED)
        rtnval = False
    except SemanticError:
        rtnval = True

    assert rtnval
    return

Tool dispatch

test_toolexecutor.py exercises:

• Valid input / valid output → success.
• Invalid input → ToolValidationError.
• Tool's execute() raises → ToolExecutionError.
• Credentials resolved correctly.
• Default values applied.
• Schema validation errors aggregated correctly.

LLM provider abstraction

test_llmprovider_properties.py exercises (a property test — chapter 8 covers these):

• Constructing with valid provider / model name → succeeds.
• Constructing with empty / None name → raises SemanticError.
• Tags and config defaults applied.

Memory operations

test_filememorystore.py, test_memoryquery.py, test_memoryindex.py exercise:

• Storing entries and retrieving by ID.
• Searching by tags / metadata / scope.
• Semantic search (with mocked embedding provider).
• Cosine similarity correctness.
• Expiration policy enforcement.

Workflow execution

test_workflowexecutor.py exercises:

• Linear workflows.
• Branching workflows (condition evaluation).
• Parallel branches.
• Checkpoint save / load / resume.
• Step retries on failure.
• Workflows with approval gates.

Observability emission

test_tracer.py, test_metricscollector.py, test_executionrecorder.py exercise:

• Spans nest correctly.
• Sampling decisions are stable (same trace_id → same decision).
• Metrics validate against the registry.
• Records have monotonic sequence numbers.

Resilience subsystems

test_circuitbreaker.py exercises:

• CLOSED → OPEN → HALF_OPEN → CLOSED transitions.
• Failure thresholds.
• Recovery timeout enforcement.
• Concurrent calls preserve state correctness.

The 53 files under resilience/ cover circuit breakers, retry policies, degradation strategies, dead-letter queues, and similar patterns. Resilience is one-fifth of the test surface — production agents fail, and the SDK treats failure handling as load-bearing.

What's not tested

A short list of things the test suite deliberately doesn't exercise:

Real LLM API calls

No tests in the SDK hit OpenAI, Anthropic, or any other real provider. All providers are mocked.

The reasoning:

• Cost. A test suite that cost $50 per run discourages running it.
• Rate limiting. CI runs that hit shared rate limits are non-deterministic.
• Vendor variability. Provider responses change over time as models update; tests against real providers would have to track the changes.
• Test scope. The SDK's job is to correctly route the LLM's responses through its plumbing, not to verify the LLM's responses themselves.

Application code that uses the SDK can write integration tests against real providers if it wants to verify end-to-end behaviour. Those tests live in the application's repo, not the SDK's.

LLM output quality

The SDK doesn't verify that the LLM's responses are correct, helpful, or well-formatted. That's the application's responsibility.

A test that wanted to verify "the agent gives sensible answers about the weather" would have to:

1. Run the agent against a real LLM.
2. Score the response somehow (LLM-as-judge, human evaluation, regex matching).
3. Decide whether the score meets a bar.

None of those are in the SDK's scope. The SDK's tests verify "the agent dispatches tool calls correctly when the LLM returns this specific tool-call response." What makes the LLM return that specific response is the application's prompt-engineering concern.

Cross-package integration

The agent-kit's tests don't import from sysman, landscape, or interop. Those packages have their own test suites that integrate with agent-kit.

The integration is one-directional:

• codecraftai-sysman-* packages may use agent-kit (downstream).
• codecraftai-landscaping may use agent-kit.
• codecraftai-interop may use agent-kit.
• agent-kit doesn't depend on or test against any of them.

Agent-kit is a library; downstream packages test their use of it.

The 49-minute wall time

Volume 1 chapter 10's case study mentioned "49 minutes wall time" for the agent-kit suite at TESTBASE_MAX_EXAMPLES_CAP=10. This figure refers to:

• Local execution (no CI workflow file in the repo).
• All 305 test files.
• Property tests at the cap (chapter 8).

There's no .github/workflows test runner. The Makefile contains only distribution targets. The 49-minute figure is from running the suite manually — likely the original SDET's measurement on representative hardware.

A reader running the suite themselves should expect similar time on similar hardware. Faster machines (Apple Silicon, modern Xeons) come in under 30 minutes. Slower machines (older laptops, constrained CI runners) take longer.

The wall time is not unusual for an SDK at this scale. 3,744 individual test invocations × ~0.8 seconds average ≈ 50 minutes. Most of that time is property tests running their max_examples=50 iterations.

Test fixtures vs factory functions

Pytest users would expect conftest.py files with shared fixtures. The agent-kit's test suite uses a different pattern: factory functions per test module.

# Not a pytest fixture.
def _create_mock_llm_provider(summary_text="default"):
    # ... construct mock ...
    return mock_provider


# Used directly in test functions.
def test_summarizer_with_custom_text():
    provider = _create_mock_llm_provider(summary_text="custom")
    # ... test logic ...

Why factories instead of fixtures?

• Visibility. A test that calls _create_mock_llm_provider(summary_text="custom") is explicit about what mock it's using. A pytest fixture is invisible until you trace its definition.
• Reuse boundary. Factories live in the test module; they're reused across tests in that module but don't leak into others. Pytest fixtures span modules unless scoped carefully.
• Independence from pytest. The agent-kit's tests run through testbase (Volume 1). Pytest fixtures would couple the tests to pytest specifically; factories work with any test runner.

The framework's broader test-fixture story is in Volume 1 chapter 3 — @testbase.attach.parameter for test-scoped resources. The agent-kit uses factories in addition to the framework's standard fixture pattern, depending on whether the resource is per-test (factory) or per-package (fixture).

A representative test walkthrough

A worked test exercising several SDK pieces at once:

import testbase

from ccai.testbase.mocking import Mock
from ccai.agentkit.contextsummarizer import ContextSummarizer
from ccai.agentkit.llmconversation import LLMConversation
from ccai.agentkit.llmmessage import LLMMessage


def _create_mock_llm_provider(summary_text="This is a summary."):
    mock_message = LLMMessage(role="assistant", content=summary_text)
    mock_response = Mock()
    mock_response.configure(recursive=True)
    mock_response.mock_context().set_attribute("message", mock_message)

    mock_provider = Mock()
    mock_provider.configure(recursive=True)
    mock_provider.mock_context().set_attribute("provider_name", "test-provider")
    mock_provider.mock_context().mock_method("complete", return_value=mock_response)
    return mock_provider


def test_summarize_messages_calls_llm_with_combined_content():
    """
        Verify that ContextSummarizer combines messages and calls the LLM.
    """
    provider = _create_mock_llm_provider(summary_text="The user asked for help.")
    summarizer = ContextSummarizer(provider=provider)

    messages = [
        LLMMessage(role="user", content="I need help with..."),
        LLMMessage(role="assistant", content="What kind of help?"),
        LLMMessage(role="user", content="Specifically..."),
    ]

    result = summarizer.summarize_messages(messages)

    # Verify the result is a system-role message.
    assert result.role == "system"
    assert result.content == "The user asked for help."

    # Verify the provider was called with a conversation that includes
    # the formatted messages.
    call = provider.mock_context().last_call("complete")
    sent_conversation = call.kwargs.get("conversation") or call.args[0]
    sent_content = sent_conversation.messages[-1].content
    assert "I need help with..." in sent_content
    assert "What kind of help?" in sent_content


def test_summarize_messages_raises_on_empty_input():
    """
        Verify that ContextSummarizer raises SemanticError on empty input.
    """
    provider = _create_mock_llm_provider()
    summarizer = ContextSummarizer(provider=provider)

    try:
        summarizer.summarize_messages([])
        rtnval = False
    except SemanticError:
        rtnval = True

    assert rtnval

Two tests covering one feature:

• The first verifies the happy path — summarize calls the LLM, returns the result.
• The second verifies the guard — empty input raises SemanticError.

The mock provider is a small fixture (factory function); the rest of the test is straightforward setup → call → assert.

What this chapter set up

A reader who's read this chapter should understand:

• The test suite's scale (305 files, 78k LOC).
• The mock-first discipline (no real LLM calls).
• What gets tested at each layer (lifecycle, tools, LLM abstraction, memory, workflows, observability, resilience).
• What deliberately doesn't get tested (real LLM calls, output quality, cross-package integration).
• The factory-function pattern instead of pytest fixtures.

The next chapter walks property tests at scale — the 72 property-based test files in the suite, what they catch that example-based tests don't, and the @given(constraints={"max_examples": 50}) discipline that keeps the suite tractable.

What's next

Chapter 8 walks the property-test layer. Examples from test_costtracker_properties.py, test_tokenusage_properties.py, test_llmconversation_properties.py. What property tests buy when LLM behaviour drifts. The recent migration from hypothesis to ccai.testbase.properties (commit 8f31798). How the suite balances coverage with wall-time budget.