Chapter 1 — Measured Interoperability — The Philosophy

Networks flake. Services take time to start. A command that succeeds nine times out of ten will fail on the tenth run and leave a test suite in an unknown state. The naive solution — time.sleep(30) before every interaction — wastes time on fast machines and still fails on slow ones. Hand-written retry loops scatter timing decisions throughout the codebase. codecraftai-interop solves this with Measured Interoperability: a clean separation of Mechanism (what to do) from Behaviour (how persistently to do it).

This chapter establishes the philosophy that shapes the entire interop layer. Every protocol package built on codecraftai-interop — SSH, REST, UPnP, Tasker, Bluetooth, CAN, serial, power — implements Measured Interoperability. The discipline is what makes the protocol stack predictable, testable, and tunable across the wildly different timing profiles real-world automation runs against.

The chapter is the volume's foundation. Chapter 2 walks the mechanism that implements the discipline — the Aspects pattern. The remaining seven chapters walk the protocol stack itself. But none of those make sense without this chapter's framing.

The premise — what the interop layer's workload actually is

Distributed automation and interop are dominated by persistence behaviours: actions that must keep trying, polling, waiting, or reconnecting until a condition is met or a deadline passes.

Concrete examples from the portfolio:

Run a command on a remote host; the command takes a variable amount of time; retry the read of stdout / stderr until the command exits.
Reboot a device; wait for it to become reachable again via SSH.
Apply a configuration change; poll until the change has propagated to all four nodes in the cluster.
Check that a service is healthy; retry the health probe if it returns transient errors.
Stream events from a device; reconnect when the wire drops.
Send a power-cycle command; wait for the device to return to a known state.

Most interop API calls are not "do X once and return." They are "do X, then verify X took effect, retrying or polling until success criteria are met." The interop layer is built around this premise; the framework's ActionPattern enum names the canonical persistence behaviours:

# codecraftai-interop/source/packages/ccai/interop/aspects.py:24–32

class ActionPattern(IntEnum):
    SINGLE_CALL = 0                    # Try once, return the result.
    SINGLE_CONNECTED_CALL = 1          # Wait for connection, then try once.
    DO_UNTIL_SUCCESS = 2               # Retry until success or deadline.
    DO_WHILE_SUCCESS = 3               # Poll while the condition holds.
    DO_UNTIL_CONNECTION_FAILURE = 4    # Retry until the wire drops.

Five named persistence behaviours. Each is a strategy the interop layer can apply to a call. Each call selects one.

What "measured" means

The userguide's framing:

codecraftai-interop solves this with Measured Interoperability: a clean separation of two concerns.

Mechanism: The act of sending bytes to a target — invoking subprocess.Popen, opening a paramiko channel, writing to a serial port. This is the agent's job.

Behaviour: How tenaciously to retry, how long to wait before giving up, when to log results, which error codes to treat as transient. This is the Aspects object's job.

— codecraftai-docs/userguide/packages/codecraftai-interop.rst:25

The word measured carries weight. The interop layer is not merely interoperable — it is measured in its interoperability. The framework's calls don't lock onto the wire and refuse to give up. They don't give up after the first hiccup. They apply measured persistence — exactly the amount the application layer asked for, no more, no less.

The discipline is what distinguishes Measured Interoperability from naive retry loops. The naive approach has timing decisions everywhere: time.sleep(30) here, for i in range(5) there, try / except blocks scattered through test bodies. Measured Interoperability has a single mechanism — the Aspects parameter — that carries timing decisions from the application layer down to the interop layer where they're applied.

The four principles

Measured Interoperability rests on four load-bearing principles. These principles describe who owns what concerns across the polling-dominated workload of the interop layer.

Principle 1 — Persistence behaviours live in the interop layer, not the application layer

Tests and helper code should not contain hand-written persistence behaviours. No for _ in range(N): try: ... except: time.sleep(...) loops in test bodies. Persistence behaviours are the interop layer's responsibility, not the test author's.

If they lived in the application layer:

Every test would reinvent persistence, with subtly different semantics.
Bugs would proliferate — off-by-one timeouts, wrong intervals, missing back-off, swallowed exceptions, retry- past-real-failure patterns that hide bugs.
Behaviour would be inconsistent across the codebase. One test polls every 0.5 seconds, another every 5 seconds, neither author thought about it.
Code review couldn't catch persistence-bug patterns because they'd be scattered.

Centralising persistence at the interop layer means: one implementation, well-tested, consistently applied. The test author calls device.power.cycle() — under the hood, the function applies the chosen ActionPattern to the call. The test author doesn't write the loop.

Principle 2 — Decisions about flow control stay in the application layer

The library / low-level protocol code does NOT make application-level decisions about what to do when things go wrong. The interop layer doesn't decide "wait 30 seconds and try again, but only twice" on its own — that's a behaviour the application layer chose.

When the interop layer does execute such decisions (which it must, to perform persistence behaviours), it does so under a delegation contract. The application layer passes the details of the behaviour to the interop layer; the interop layer follows the delegation faithfully.

The Aspects pattern is the delegation contract.

The Aspects parameter on every interop call (chapter 2 walks it in detail) is how the application layer hands the interop layer its policy:

Which ActionPattern to apply.
How long to wait (completion_timeout, connection_timeout, inactivity_timeout).
How often to poll (completion_interval, connection_interval, inactivity_interval).
What counts as success (expected_status, allowed_error_codes).
What LoggingPattern to emit while running.

Different callers use different policies on the same library function; the library code doesn't change.

This is why decorators are wrong for this purpose (Volume 0 chapter 2's discipline lands here). A @retry(times=3, delay=10) decorator hard-codes the policy at the library level — every call site gets the same policy whether or not it's the right one. An aspects=AspectsCmd(...) parameter passes the policy from the caller per call — every caller chooses.

Principle 3 — Neither too brief nor too persistent

The interop layer cannot:

Immediately give up. A single failed packet, a momentary connection blip, a transient timeout — these would manifest as test failures despite the underlying system being fine. False negatives.
Be too persistent. Retrying forever, retrying through real failures, polling past actual breakage — these hide bugs. A test that should have failed (because the system is genuinely broken) appears to pass because the interop layer kept retrying until coincidentally succeeding. False positives.

All interop code adheres to flow control contracts that specify persistence criteria. The criteria define when to stop trying:

Total time elapsed (completion_timeout).
Time without progress (inactivity_timeout).
Number of attempts.
A specific error code that means "permanent failure, don't retry."
A specific success condition that means "yes, done."

The criteria are encoded in the Aspects object. The interop layer follows them — exactly as long as they say to try, exactly when they say to stop.

The framework's defaults (from codecraftai-interop/source/packages/ccai/interop/aspects.py:43–62):

DEFAULT_COMPLETION_TIMEOUT = 600          # 10 minutes total
DEFAULT_COMPLETION_INTERVAL = 10          # check every 10 seconds
DEFAULT_CONNECTION_TIMEOUT = 16           # 16 seconds to establish
DEFAULT_CONNECTION_INTERVAL = 2           # retry connection every 2s
DEFAULT_ALLOWED_ERROR_CODES: Tuple[int] = []
DEFAULT_ALLOWED_CONNECTION_EXCEPTIONS: Tuple[BaseException] = []
DEFAULT_MUST_CONNECT = False

These defaults are sensible starting points. A specific call that needs different criteria builds its own AspectsCmd(completion_timeout=...) and passes it.

Principle 4 — When the contract's bounds are exceeded, control reverts via exception

The interop layer never makes the decision "we tried hard enough, let's give up silently and report success." It also never makes the decision "we'll keep trying past the agreed budget because we think it'll work eventually." When the persistence criteria from the application layer are exhausted — the timeout elapsed, the attempt count reached, the deadline passed — the interop layer raises an exception and returns control to the application layer.

The exception is the exit mechanism of the delegation contract. The application layer:

Delegated persistence responsibility (principle 1).
Specified the criteria via Aspects (principle 2).
The interop layer honoured the criteria (principle 3).
The interop layer reverts control when the criteria are exhausted (principle 4).

The application layer then decides what to do. Catch the exception and retry with different aspects. Fail the test. Escalate to the orchestrator. Try a fallback approach. The interop layer doesn't choose; it surfaces the situation and lets the application choose.

This is why the interop layer's exception types are typed (SshAuthError, SshSFTPError, TimeoutError, ConnectionError) and not generic RuntimeError. The exception's type tells the application layer which kind of contract bound was hit, so the application can dispatch on the type:

try:
    device.ssh.run_cmd(cmd, aspects=fast_aspects)
except TimeoutError:
    # The completion_timeout elapsed.  Try with longer aspects.
    device.ssh.run_cmd(cmd, aspects=patient_aspects)
except SshAuthError:
    # Auth failed.  Credentials might have rotated.  Re-resolve.
    device.refresh_credentials()
    device.ssh.run_cmd(cmd, aspects=fast_aspects)
except ConnectionError:
    # The wire dropped.  Different recovery — escalate.
    raise InfrastructureError("Pod is unhealthy")

The typed exceptions are the vocabulary through which the contract reverts control.

The status-vs-exception split

A subtlety worth pinning down because it surprises readers: a non-zero exit status from a remote command does NOT raise in the interop layer.

The reason: a non-zero status is within the contract. The contract was "run this command, capture status / stdout / stderr." The contract was honoured even when the command exited 1. The application layer reads the status and decides whether the non-zero is meaningful.

A test that expects a non-zero status (say, "this command should fail because the file doesn't exist") wants status == 1, not an exception. Treating the non-zero as an exception would force every caller to wrap in try / except even for expected failures.

By contrast, a transport failure (network drop, SSH auth failure, connection timeout) exceeds the contract — "capture status / stdout / stderr" cannot be honoured if the session never opened. The interop layer raises a typed exception per principle 4.

The split is:

Status as data — a non-zero exit code is data the application reads.
Exception as control reversion — a contract violation reverts control.

Together, they cover the full surface. Application code reads status when it cares about the command's exit; catches typed exceptions when the contract was exceeded.

How the principles interlock

The four principles compose into one design:

The interop layer does the persistence work (principle 1) →
under a delegation contract (principle 2) →
with persistence criteria that bound behaviour (principle 3) →
and reverts control via typed exception when the criteria are exhausted (principle 4).

The Aspects parameter is the artifact carrying all four. The application layer delegates persistence (principle 1), passes its decisions about flow control via the Aspects parameter (principle 2), the parameter encodes the persistence criteria (principle 3), and the interop layer follows the criteria — including raising on exhaustion (principle 4).

Where this philosophy applies

The whole interop layer for all protocols uses Measured Interoperability. Every protocol package in the portfolio implements it:

codecraftai-interop — the core. Local commands, SSH (paramiko- wrapped), serial-over-TCP, power, DNS / mDNS.
codecraftai-protocol-ssh — from-scratch SSH.
codecraftai-restclient-base — REST foundation; AspectsRest is the REST-shaped Aspects subclass.
codecraftai-restclient-github-public — auto-generated GitHub client; every method takes aspects=AspectsRest(...).
codecraftai-interop-ext-upnp, -tasker, -bluetooth, -can — protocol extensions.

The discipline is uniform. An engineer who understands this chapter can read any protocol agent in the portfolio and know what the Aspects parameter is for.

Why this design holds up at scale

Three properties make Measured Interoperability work in production:

Configuration consistency

Because every interop call accepts Aspects, a project that needs longer timeouts in CI (slower lab) and shorter timeouts locally (faster laptop) sets the Aspects differently per environment. No code changes; the same library calls work both places.

Aspects and its subclasses inherit from Freezable. Once constructed, an Aspects instance is immutable. This makes it safe to share across concurrent calls — no risk of one thread mutating an aspects instance another thread is using. Volume 2 chapter 11's cross-host fan-out relies on this property.

Composability with the framework

The framework's testjob lifecycle, validators, configuration profiles, and orchestration all compose with Measured Interoperability cleanly. A tandem validator (Volume 2 chapter 5) is itself an application of the four principles at a higher framework layer: validators centralise polling (principle 1), take their criteria as constructor parameters (principle 2), bound persistence (principle 3), raise AssertionError on contract exhaustion (principle 4). The discipline scales.

What this chapter set up

By the end of this chapter a reader should:

Know the formal name Measured Interoperability and the Mechanism / Behaviour separation.
Recognise persistence behaviours as the umbrella term for retry / poll / wait / reconnect patterns.
Be able to name the five ActionPattern modes: SINGLE_CALL, SINGLE_CONNECTED_CALL, DO_UNTIL_SUCCESS, DO_WHILE_SUCCESS, DO_UNTIL_CONNECTION_FAILURE.
Understand the four principles: persistence in interop, decisions in application, criteria bound persistence, control reverts via exception.
Distinguish status-as-data from exception-as-control- reversion.

The next chapter walks the Aspects pattern in code detail: the Aspects and AspectsCmd classes, their fields, their immutability, the LoggingPattern enum, the MonitoredScope deadlock-detection mechanism, and the typical aspect-passing flow through an interop call.