Chapter 4 — Property-Based Testing with `@given`

Example-based tests verify the cases the test author thought of. Property-based tests verify the property the test author stated, against an unbounded space of inputs. The hard part is not writing the property — it is making the framework's generation budget match the time the suite has to run. This chapter is about that.

Property-based testing in codecraftai-testbase is conceptually a sibling of Hypothesis — the testing library most Python engineers know — but with two specific adaptations for automation suites:

A runtime-overridable cap on max_examples. The environment variable TESTBASE_MAX_EXAMPLES_CAP constrains how many examples each property test may generate, regardless of what the test declared. This lets a build engineer tune the whole suite without editing the individual tests.
A deterministic per-test seed derived from the job seed. The framework computes SHA-256(job_seed + ":" + test_name) and feeds it to the per-test generator. Re-running a job with the same job seed reproduces the same per-test seeds — the basis for replayable property-based runs.

The rest of the chapter walks @given, the strategies it accepts, the cap, and the per-test seed. Where Hypothesis behaviour differs, the chapter calls it out explicitly.

The `@given` decorator

# codecraftai-testbase/source/packages/ccai/testbase/properties/decorators.py:23–97

def given(*, constraints: Optional[Dict[str, Any]] = None, **strategies: Strategy) -> Callable:
    """
        Decorate a test function with strategies for property-based
        execution.

        :param constraints: optional dict with `max_examples` and/or
            `max_shrink_attempts` keys.
        :param strategies: keyword arguments mapping the test
            function's parameter names to Strategy instances.
    """
    def _decorator(func: Callable) -> Callable:
        func._property_ = {
            "strategies": strategies,
            "constraints": constraints if constraints is not None else {},
        }
        rtnval = func
        return rtnval
    return _decorator

The decorator stores metadata on the function under _property_. Per Volume 0 chapter 2's decorator discipline, nothing about the function body changes. The property runner (below) is the thing that uses the metadata.

A typical use:

import testbase
from testbase.strategies import integers, text

@testbase.given(
    a=integers(min_value=-1000, max_value=1000),
    b=integers(min_value=-1000, max_value=1000),
    constraints={"max_examples": 50},
)
def test_addition_is_commutative(a, b):
    """
        Verify that integer addition is commutative across the
        given range.
    """
    rtnval = (a + b) == (b + a)
    assert rtnval
    return

The strategy keys map onto the test function's parameter names. constraints={"max_examples": 50} says "generate at most 50 examples per run." The max_shrink_attempts key (also in the constraints dict) caps how many shrinks the runner attempts after finding a failure.

Constraint vocabulary

The constraint dict supports two keys (codecraftai-testbase/source/packages/ccai/testbase/properties/decorators.py:42–44):

max_examples — integer, default 100. How many distinct inputs to generate before declaring the property holds.
max_shrink_attempts — integer, default 100. After the runner finds an input that violates the property, how many attempts to make at finding a smaller input that still violates.

Per Volume 0 chapter 4 (the Aspects pattern), behaviour parameters live on the test, not on a wrapping decorator. There is no @retry, no @timeout, no @shrink_more decorator. The test author writes constraints={...} and the runner respects it — except where the runtime cap overrides.

`TESTBASE_MAX_EXAMPLES_CAP`: the build engineer's knob

Here is where testbase parts company with Hypothesis.

# codecraftai-testbase/source/packages/ccai/testbase/properties/propertyrunner.py:47–79

ENV_MAX_EXAMPLES_CAP = "TESTBASE_MAX_EXAMPLES_CAP"

def _resolve_max_examples_cap() -> Optional[int]:
    """
        Read the TESTBASE_MAX_EXAMPLES_CAP environment variable and
        return the integer value, or None if unset.

        :returns: integer cap or None.
        :raises: ValueError when the env var is set but unparseable.
    """
    raw = os.environ.get(ENV_MAX_EXAMPLES_CAP, None)
    if raw is None:
        rtnval = None
        return rtnval

    try:
        rtnval = int(raw)
    except ValueError as cause:
        err_msg_lines = [
            "TESTBASE_MAX_EXAMPLES_CAP must be a parseable integer.",
            "    VALUE: {!r}".format(raw),
        ]
        err_msg = os.linesep.join(err_msg_lines)
        raise ConfigurationError(err_msg) from cause

    return rtnval

The function reads the environment variable at runtime. When the property runner constructs itself, it stores the cap. When it runs a test, it picks the smaller of (the test's declared max_examples, the cap). A test that says max_examples=200 running under TESTBASE_MAX_EXAMPLES_CAP=10 runs ten examples. A test that says max_examples=5 running under the same cap runs five.

The cap is only an upper bound. It cannot make a test generate more examples than its declaration; it can only generate fewer. This is intentional: a test author who deliberately budgets 5 examples (because the strategy is expensive) does not have their budget raised by an environment variable.

The cap is the build engineer's knob. In CI, set it to 10 — the suite finishes in a CI window short enough to run on every PR. On a nightly full-sweep run, leave it unset — the suite runs at its full declared budget. In local development, set it to 1 or 5 for a fast smoke pass.

This is the Aspects pattern from Volume 0 chapter 4 lifted to the suite level. The behaviour parameter (how many examples) is set by the caller of the runner, not edited into hundreds of test files. Chapter 10 returns to this knob in detail — the case study suite uses TESTBASE_MAX_EXAMPLES_CAP=10 to finish in 49 minutes wall time.

Determinism: the per-test seed

Property-based tests are only useful in CI if they are reproducible. A test that fails non-deterministically because each run generates different inputs is a flake. Testbase addresses this by deriving the per-test seed deterministically from the job seed:

# codecraftai-testbase/source/packages/ccai/testbase/properties/propertyrunner.py:82–97

def derive_test_seed(job_seed: bytes, test_name: str) -> int:
    """
        Derive a deterministic per-test seed from the job seed and the
        test name.

        :param job_seed: the run-level seed (bytes).
        :param test_name: the fully-qualified test function name.

        :returns: integer per-test seed suitable for feeding to a PRNG.
    """
    payload = job_seed + b":" + test_name.encode("utf-8")
    digest = hashlib.sha256(payload).digest()
    rtnval = int.from_bytes(digest[:8], byteorder="big")
    return rtnval

The job seed is a fixed-length byte string set once at job start (typically by TestJob's constructor). The test name is the fully-qualified module path plus function name. SHA-256 over job_seed + b":" + test_name produces a 32-byte digest; the first eight bytes become a 64-bit integer the runner feeds to the strategy.

The properties this gives:

Re-running the same job seed reproduces every per-test seed. A failing run can be replayed end-to-end.
Adding a test does not change the seeds of existing tests. The seed is a function of that test's name and the job seed, so the inputs other tests see are unchanged.
Renaming a test changes its seed. This is unavoidable — but the build engineer who renames a test should expect to re-baseline that test's recent runs anyway.

A specific question this raises: can the test author override the per-test seed? No. Per the property runner declaration (codecraftai-testbase/source/packages/ccai/testbase/properties/decorators.py:49) there is no seed constraint key. The job_seed is the only seed input; the framework is the only seed deriver. This is deliberate: a test that gets to override its own seed cannot be re-baselined to a known-good run by an external operator.

The PropertyRunner

The runner that ties the strategies, constraints, cap, and seed together is PropertyRunner (codecraftai-testbase/source/packages/ccai/testbase/properties/propertyrunner.py:100–120):

class PropertyRunner:
    def __init__(self,
                 constraints: Dict[str, Any],
                 job_seed: bytes,
                 test_name: str,
                 recorder: ResultRecorder,
                 parent_inst: TestInstance) -> None:
        self._constraints = constraints
        self._job_seed = job_seed
        self._test_name = test_name
        self._recorder = recorder
        self._parent_inst = parent_inst
        self._cap = _resolve_max_examples_cap()
        return

Six instance variables, all set in __init__ per the Volume 0 chapter 2 discipline (no late-introduced instance variables). The runner exposes a run() method that:

Resolves the effective max_examples (min(declared, cap)).
Resolves the max_shrink_attempts (declared only — the cap does not apply).
Derives the per-test seed.
Drives the strategy generators for max_examples iterations.
For each iteration: emits a property-iteration event to the recorder, runs the test body with the generated inputs, emits a result event.
On failure, runs up to max_shrink_attempts shrinks and reports the smallest failing input.

The result events get recorded to the JSOS stream same as example-based tests; Chapter 5 covers the format.

Composition with `@testbase.attach.parameter`

A @given test can also have attached parameters from Chapter 3. The decorators compose:

@testbase.attach.parameter(db_session, identifier="db")
@testbase.given(
    invoice_number=integers(min_value=1, max_value=10**9),
    constraints={"max_examples": 25},
)
def test_invoice_lookup_round_trips(db, invoice_number):
    """
        Verify that an invoice round-trips through the lookup API.
    """
    inserted = db.insert_invoice(invoice_number)
    looked_up = db.lookup_invoice(invoice_number)
    rtnval = (inserted == looked_up)
    assert rtnval
    return

The strategy keys (invoice_number) and the attachment keys (db) are disjoint — they bind to different parameters. The attached resources are acquired once at the resource lifespan boundary (Session, Package, or Test); the strategy values are generated per iteration of the property test.

A subtle point: when db is a Test-scoped resource, the runner re-acquires db on every iteration, not just every test invocation. If db is expensive, prefer Package or Session scope — or generate the per-iteration data inside the test body rather than asking the originator to do it.

Strategy library

Testbase ships its own strategy library (under codecraftai-testbase/source/packages/ccai/testbase/strategies/). The vocabulary is similar to Hypothesis:

integers(min_value=..., max_value=...)
floats(min_value=..., max_value=..., allow_nan=...)
text(min_size=..., max_size=..., alphabet=...)
booleans()
lists(elements=..., min_size=..., max_size=...)
tuples(*element_strategies)
dictionaries(keys=..., values=..., min_size=..., max_size=...)
one_of(*strategies)
none() and just(value)

Composition is via lists(elements=integers()) and one_of(...). A user-defined strategy is a class that subclasses Strategy and implements draw(self, rng) — the per-iteration hook the runner calls.

The strategy implementations are testbase's own, not Hypothesis's. This is a deliberate choice: it keeps the framework's dependency graph free of Hypothesis (a sizable external dependency) and gives the runner control over the generation budget at a per-test level.

Shrinking

When the runner finds an input that violates the property, it attempts to shrink it — find a smaller, structurally-simpler input that also violates the property. The shrink algorithm is strategy-specific: integers shrink toward zero, lists shrink by removing elements, strings shrink toward empty.

The max_shrink_attempts constraint caps the number of shrink attempts. When the cap is reached the runner reports the smallest-found-so-far input.

The reported failure carries:

The input that failed.
The seed used.
The number of shrinks attempted.
The exception (or the failed assertion's left/right operands).

A test author re-running locally with the reported input can reproduce the failure deterministically — without re-running the entire generation budget.

What property-based testing is good for

A working list, with examples from the portfolio:

Round-trip invariants: encode/decode, serialise/parse, insert/lookup. Property: decode(encode(x)) == x.
Algebraic identities: commutativity, associativity, identity, idempotency.
Boundary discovery: for any i in [min, max], function f(i) does not raise.
Ordering invariants: a sort produces a non-decreasing sequence; a deduplicator never emits the same key twice.
Configuration loaders: for any valid config dict shaped like S, the loader produces an object whose properties match S's field semantics.

What property-based testing is not good for: end-to-end integration tests where each iteration is expensive (each one takes a database round-trip). Those should remain example-based, with a small handful of carefully chosen inputs.

What's next

Chapter 5 — Streaming JSOS results. Each iteration of a property-based test becomes a JSOS record; the format and the consumers.
Chapter 6 — The agents CLI. How agents testresult list and agents testresult failures consume the JSOS stream.

Chapter 10's case study uses TESTBASE_MAX_EXAMPLES_CAP=10 on a 3,700-test single-host suite. By the time the reader reaches that chapter the cap will be familiar — the case study just shows what the suite-level wall time is when the cap is set.

Chapter 4 — Property-Based Testing with @given

The @given decorator