Chapter 10 — Remote Objects over a Wire: codecraftai-python-rpc

Every protocol in this volume moves bytes and hands the caller a result. codecraftai-python-rpc moves something stranger: a reference. A client holds an object that lives in another process, calls its methods, passes it callbacks, and lets it go — and the framework keeps the two processes agreeing about which objects still exist. This chapter is the newest package on the volume's floor and the one that depends most heavily on everything the earlier chapters established.

Chapters 3 through 8 walked protocols that expose operations: run a command over SSH, resolve a name, issue a REST call, drive a UPnP device. The caller assembles arguments, the protocol carries them, a result comes back. The object model on each side is private; only the operation crosses the wire.

codecraftai-python-rpc crosses the line those chapters stayed behind. It lets a process publish a Python object and lets another process use that object as an object — attribute access, method calls, callbacks travelling in both directions, futures for long operations — with the framework responsible for identity, lifetime and failure semantics in between. That is a considerably harder contract than "carry this request", and the package's design is mostly a catalogue of the ways the naive version goes wrong.

The chapter reads the package from the wire upward: the frame, the codec, the session, the proxy, the lease, and finally the two entry points that a reader actually types. Throughout, the question is the same one the volume has been asking since Chapter 1: what does the caller measure, and what does the framework guarantee?

What's in the package

codecraftai-python-rpc/source/packages/ccai/rpc/
├── __init__.py                 # the public surface
├── aspects.py                  # AspectsRpc, rpc_aspects_scope, remaining_time
├── endpoints.py                # endpoint descriptors and resolution
├── errors.py                   # the RPC exception vocabulary
├── rpcsocketserver.py          # RpcSocketServer
├── rpcsocketclient.py          # RpcSocketClient
├── core/
│   ├── looperqueue.py          # bounded queue with shutdown
│   ├── looper.py               # one worker thread over a queue
│   ├── looperpool.py           # elastic pool of loopers
│   ├── flattenable.py          # by-value contract for user types
│   ├── futures.py              # RemoteFuture, Progress, wait_all, as_completed
│   ├── leases.py               # ILeaseAware
│   ├── protocol/               # constants, 24-byte header, message classes
│   ├── serialization/          # tags, registry, placeholder, codec
│   ├── transport/              # duplex socket I/O, in-process pairs, fault injection
│   ├── session/                # DuplexSessionBase, dispatch, heartbeat, client, server
│   ├── proxy/                  # RemoteProxy, reference table, cache, wrappers
│   ├── dispatch/               # CallContext and ordering lanes
│   └── observability/          # metrics, live-state dump
└── security/                   # authenticators and TLS upgraders

About 4,600 lines of package code and 174 tests, including a cross-process test that starts a real server in a child process. The only internal dependency is codecraftai-foundations (Volume 0): the exception vocabulary, the base Aspects class, and Freezable.

Two things the tree announces before a line of it is read. First, there is no pickle anywhere. Second, the transport, session and proxy layers are separate directories with separate tests, which is what lets the package test session semantics over an in-process socket pair rather than a real network.

The frame

Everything on the wire is a frame: an eight-byte length prefix, a fixed twenty-four-byte header, and a payload. The header is a struct with five fields (core/protocol/header.py:23–35):

_HEADER_FORMAT = "!HBB16sI"

@dataclass(frozen=True)
class MessageHeader:
    version: int
    mtype: MessageType
    mcode: int
    correlation_id: uuid.UUID
    payload_length: int

version is the protocol version, checked on every header against a supported range (core/protocol/constants.py:15–16). mtype is one of request, reply or exception (constants.py:22), and mcode says which request or reply (constants.py:28–58). The sixteen-byte correlation_id ties a reply to the request that caused it. payload_length lets a reader allocate before it reads.

The header is decoded before the payload, and the payload never has to be parsed to route a message. That ordering is the whole point of a fixed header: the session's reader thread can match a reply to its waiter, or hand a request to the dispatcher, without touching the codec. The reader is never blocked on a slow decode, so a heartbeat reply is seen promptly even while a large reply is queued behind it.

The message classes in core/protocol/messages.py are thin dataclasses registered by (type, code) through a decorator (messages.py:90). There are twelve requests, twelve replies and one exception message. The names are the vocabulary of the rest of the chapter: Hello, Ping, Close, GetService, GetAttr, SetAttr, DelAttr, Call, Release, Renew, Cancel, Progress.

The codec: a closed set of types, no pickle

The temptation in any Python RPC design is pickle. It handles every type, it round-trips faithfully, and it is a remote-code-execution primitive by construction — a peer that can send you a pickle can run code in your process. The package refuses it outright and instead ships a tagged binary codec (core/serialization/codec.py) over a closed set of types.

The tags are an IntEnum (core/serialization/tags.py:19). The set they cover is deliberately short: None, bool, int (varint, zigzag for negatives — codec.py:39–72), float, str, bytes, list, tuple, dict with string keys, set, frozenset, timezone-aware datetime, date, timedelta, UUID, exceptions, and three extension points:

• Registered enums (registry.py:28): an enum class maps to a string tag, and the value's name travels.
• Registered Flattenable types (registry.py:37) and registered dataclasses (registry.py:46): the type maps to a tag and its flatten() dictionary travels. A dataclass is registered with one call; the codec never introspects an unregistered type.
• References (placeholder.py:19): an object that travels by reference is replaced by a ReferencePlaceholder naming the owning session and the reference id. The proxy layer turns placeholders back into proxies on the way in.

Anything outside that set fails before a byte is sent, with SerializationError naming the offending type and the path to it inside the argument tree. Chapter 2's framing applies directly: the codec is a mechanism, and the mechanism's job is to fail loudly when asked to do something outside its contract, never to guess.

Depth is bounded (tags.py:16, MAX_DEPTH = 64) so a self-referential structure cannot hang an encoder, and the maximum message size is a session setting (constants.py:19, sixty-four megabytes by default) enforced when the length prefix is read, before any allocation.

Exceptions deserve a note. A remote exception arrives as the same type when the type lives in a preloaded module — builtins and the Volume 0 vocabulary (codec.py:36) — with the remote traceback attached as remote_traceback. Any other type arrives as RemoteError carrying the original type name. The codec never imports a module on the say-so of a peer. That single rule is the difference between "an exception crossed the wire" and "the peer chose which module to import into my process".

Work dispatch: queues, loopers and pools

Before the session can be understood, the threading substrate under it needs a paragraph. Three small classes in core/ carry every unit of work in the package:

• LooperQueue (core/looperqueue.py:30) — a bounded queue with an explicit shutdown sentinel (looperqueue.py:21), so a worker blocked on get() wakes when the queue is closed instead of when its timeout happens to expire.
• Looper (core/looper.py:24) — one named thread draining one queue.
• LooperPool (core/looperpool.py:22) — an elastic set of loopers over a shared queue, growing between a minimum and a maximum worker count when the queue's depth passes a high-water mark.

The pool's bounds come from AspectsRpc (dispatch_min_workers, dispatch_max_workers, dispatch_highwater). The choice to make them Aspects rather than constructor arguments is Chapter 2 again: they are behaviour knobs, and behaviour knobs travel in the Aspects bundle so a caller who needs different concurrency for one session sets it in one place.

Every thread the package starts is named rpc-<role>-<purpose>-<session>. When a process is dumped with faulthandler or inspected in a debugger, the names say which session and which purpose each thread serves. Volume 0's first principle — debuggability over performance — is cheap to honour here and expensive to retrofit.

The session

DuplexSessionBase (core/session/base.py:103) is the centre of the package, and at 660 lines it is the largest file. A session owns one transport, one reader thread, one writer queue, a dispatch pool, a heartbeat monitor and a reference table. Client and server sessions (core/session/client.py, core/session/server.py) differ only in who speaks first.

Request correlation

send_request (base.py:306) allocates a correlation id, registers a Waiter (base.py:69), writes the frame and blocks on the waiter with a deadline. The reader thread matches replies to waiters by correlation id and never blocks on anything but the socket. Because requests are correlated rather than sequenced, a session can have many calls in flight from many threads, and — this matters for callbacks — a request arriving from the peer can be dispatched while one of our own requests is still waiting. That is what lets a server call back into a client listener during the client's own call.

Heartbeat and inactivity

HeartbeatMonitor (core/session/heartbeat.py:20) sends a Ping every heartbeat_interval seconds of silence and declares the peer gone after inactivity_timeout seconds without any inbound frame. Both are Aspects. The detection is measured, in the Chapter 1 sense: the session does not "assume" a peer is alive because a socket is open; it observes frames and acts on their absence. When the monitor gives up, every waiter is failed with ConnectionLostError and every reference the peer held is expired — which brings us to leases.

Dispatch, deadlines and lanes

Inbound requests go to a DispatchLooper (core/session/dispatch.py:27). Each request is wrapped in a CallContext (core/dispatch/context.py:25) carrying the caller's session, the remaining time the caller sent with the request, the caller's context pairs, and a cancellation flag. Service code reads these through module functions — remaining_time(), check_cancelled(), report_progress() (context.py:87–107) — so a service method needs no framework type in its signature to cooperate with deadlines and cancellation.

Ordering is the subtle part. The package has lanes (core/dispatch/lanes.py:19): a per-target queue that serialises deliveries to one object. Lanes apply to one-way notifications only (dispatch.py:53–62). The comment at that site records the reason, and it is worth quoting because it is the kind of decision that only a failing test teaches:

Ordering lanes apply to one-way callbacks only. An acknowledged call already serialises its caller, and a lane on an acknowledged call would deadlock a re-entrant chain that targets the same object.

A calls B, B calls back into A's listener, the listener calls B again: if every delivery to one target waited its turn in a lane, the chain deadlocks on itself. Acknowledged calls are already ordered by the fact that the caller is blocked waiting for them. Only fire-and-forget notifications need the lane, and only for them is it safe.

Proxies and references

RemoteProxy (core/proxy/remoteproxy.py:56) is what a caller holds. Attribute reads become GetAttr requests, writes become SetAttr, method calls become Call, and a small list of dunder methods (remoteproxy.py:22) is forwarded so a remote sequence supports len() and indexing. Access to underscore-prefixed names is refused at the owner unless the type is decorated @expose_private, and writes to the root service object are refused unless it is @mutable_service (core/proxy/wrappers.py:114–127). Both are opt-in: the default is the conservative one.

Method access returns a _RemoteMethod (remoteproxy.py:25) that carries three verbs. Calling it is the ordinary acknowledged call. rpc_notify(...) sends the call one-way and returns immediately. rpc_call_async(...) returns a RemoteFuture. A method decorated @oneway on the owner's side (wrappers.py:106) is always delivered as a notification regardless of how the caller invokes it, which is how a service declares "this method's result is never interesting" once instead of at every call site.

By value or by reference — never by guess

An argument crosses the wire in exactly one of two ways, and the caller must say which. Types in the codec's closed set travel by value. A class decorated @by_ref (wrappers.py:86) always travels as a reference: the owner registers the object in its reference table and sends a placeholder; the holder receives a proxy. A caller can also decide per call with the ByRef and ByValue wrappers (wrappers.py:48–84). An unwrapped, unregistered object raises SerializationError before the request is sent.

The design rejects the friendlier alternative — "proxy anything we can't serialise" — because the friendlier alternative hides the most expensive decision in the system. A by-value argument costs one copy. A by-reference argument costs a lease, a table entry in the owner, a proxy in the holder, and one round trip per attribute access for the rest of its life. Making the choice explicit at the type or the call site keeps that cost visible where it is paid.

The reference table and the cache

The owner's side is a ReferenceTable (core/proxy/table.py): reference id to object, plus the set of holders and each holder's lease expiry. The holder's side is a proxy cache (core/proxy/cache.py) so two placeholders for one remote object resolve to one proxy, and so garbage collection of the proxy can release the reference.

That last sentence hides a race worth recording. A proxy's method objects were originally created on each attribute access and held only a weak path back to the proxy; a caller that stored listener.on_event and dropped listener could see the proxy collected, the reference released, and the stored method fail on its next call. The fix is RemoteMethodDescriptor (wrappers.py:21) plus a rule that every derived view — a method, an Aspects view — holds a strong reference to its base proxy. A reader who inherits this package should treat that invariant as load-bearing.

Leases: agreeing about what exists

Two processes holding references to each other's objects need a rule for when an object may be freed. Reference counting across a network is a known bad idea — a lost Release message leaks the object forever, and a crashed holder never sends one. The package uses leases instead (specs/leases), and the mechanism is small enough to state completely.

• When an owner sends a reference, it records the holder with a lease of lease_duration seconds (base.py:424–432).
• The holder renews every live reference it still holds, in one Renew request carrying all of the ids, well inside the duration. The owner replies with the ids it no longer knows (base.py:520–522), so a holder learns promptly that a reference is gone rather than at its next call.
• A holder that drops a proxy sends Release as a notification (base.py:409–413).
• The owner sweeps expired leases (base.py:488), and an expired or released object with an on_lease_lapsed method (core/leases.py:22, ILeaseAware) is told so (base.py:461–476). A session-level callback, on_lease_lapse (base.py:455), sees every lapse.
• When a peer is declared gone by the heartbeat monitor, every lease it held lapses at once.

Two refinements exist because tests found them. A released reference leaves a tombstone so a late Renew for it is answered "unknown" rather than resurrecting it, and the owner remembers recently released objects briefly so that a reference released and re-sent within one round trip does not race its own release.

The effect, for the reader who only ever uses the entry points, is that a callback object passed to a server simply works for as long as the server keeps it, is freed when the server drops it or dies, and can ask to be told when that happens. The whole apparatus exists so that the user-visible rule can be that short.

Aspects, applied to a wire

AspectsRpc (aspects.py:33) extends the Volume 0 Aspects base with the knobs a wire needs:

The bundle is frozen after validation (aspects.py:38), and replace(**changes) (aspects.py:59) is the way to derive a variant. Three scopes apply it:

1. Per client or server, in the constructor.
2. Per proxy, with proxy.rpc_with_aspects(aspects), which returns a view sharing the same reference.
3. Per block, with rpc_aspects_scope(aspects) (aspects.py:124), a context manager that overrides for every call made on the current thread inside it.

The caller's completion_timeout does not stay on the caller's side. The remaining time travels in the request, the service reads it with remaining_time() (aspects.py:137 on the caller, context.py:87 on the service), and a service that forwards to a second service can pass on what is left. This is the delegation contract of Chapter 2 realised end to end: the caller states the budget once and every hop below it inherits the budget rather than inventing its own.

Long operations

RemoteFuture (core/futures.py:40) is the async form of a call. result(timeout) blocks; cancel() sends a Cancel request that the service observes through check_cancelled(); add_progress_callback receives Progress records (futures.py:24) that the service emits with report_progress(fraction, message). wait_all and as_completed (futures.py:146–155) fan a set of futures back in.

Progress and cancellation are ordinary one-way messages in the protocol (Progress, Cancel), which is why they work while the call they concern is still in flight: they share the wire with it, correlated by the same id, and the dispatcher treats them as notifications rather than queueing them behind the call they are trying to influence (dispatch.py:72).

Transport and the fault injector

Transport (core/transport/base.py) is a small duplex interface; DuplexSocketIO (core/transport/socketio.py) is the real one. Two details are worth the reader's attention. First, close() always releases the file descriptor, even when the peer closed first — the package once leaked one descriptor per peer-initiated close, and the fix is a single _released flag. Second, an in-process socket pair (core/transport/inprocess.py) lets the whole session stack be tested without a listening port, and FaultInjectingTransport (core/transport/faults.py:21) lets tests drop, delay or truncate frames on demand.

Testing a protocol by injecting faults at the transport seam, rather than by mocking the session, is the single most important testing decision in the package. It is why the lease, heartbeat and correlation logic have real coverage for the failure paths that only happen on a bad network.

The entry points

The two classes a reader types are the smallest part of the package. RpcSocketServer (rpcsocketserver.py:33) accepts connections, upgrades them if asked, authenticates them if asked, and gives each one a server session over the same service object. RpcSocketClient (rpcsocketclient.py:32) connects with the Aspects' retry budget, performs the Hello exchange, and hands back a proxy typed with the caller's protocol so an IDE sees the methods:

from ccai.rpc import AspectsRpc, RpcSocketClient, RpcSocketServer

server = RpcSocketServer(Greeter(), ("0.0.0.0", 9040))
server.start()

client = RpcSocketClient(GreeterProtocol, ("server-host", 9040),
                         aspects=AspectsRpc(max_attempts=30, completion_interval=2.0))
client.connect()
greeter = client.get_service()
greeter.greet("world", loud=True)

Security lives beside the entry points rather than inside the protocol. An authenticator (security/auth.py:20) checks a credential presented in Hello; an upgrader (security/tls.py:21–37) wraps the accepted socket in TLS or mutual TLS before the first frame. Both are optional callables handed to the constructors. The protocol itself never learns whether it is running over TLS, which is the right amount of ignorance: the wire discipline is the same either way, and the security decision is made once at the edge.

Observability follows the same edge principle. server.describe() and client.session.describe() return plain dictionaries; dump_state() (core/observability/state.py:30) renders every live server and client in the process; and install_dump_on_signal (state.py:44) wires that dump to a signal so a stuck process can be asked what it is doing without a debugger. The ccai.rpc.wire logger, at debug level, traces every frame header.

Where it sits in the portfolio

The package is the newest arrival on this floor, and its dependencies say where it belongs: only Volume 0's foundations below it, and nothing in the volume above it depends on it yet. The intended consumers are Volume 2's distributed sequencer — which today moves results over SSH and JSOS (Volume 2 ch.10, ch.13) and which a reference-carrying wire would let hold live worker objects rather than streams — and Volume 7's agent runtime, where a tool that runs in another process is exactly a remote object with a lease.

For a reader deciding whether to reach for it, the test is the volume's usual one. If the interaction is operations — run this, fetch that — one of the protocols in Chapters 3 to 8 is the smaller tool and should win. If the interaction is a conversation between objects — callbacks, long-lived handles, progress, cancellation — the leases and correlation machinery here are what you would otherwise be writing badly by hand.

What this chapter set up

• The frame: an eight-byte length, a twenty-four-byte header with a correlation id, routing decided before decoding.
• The codec: a closed type set, explicit registration for user types, references as placeholders, exceptions rebuilt only from preloaded modules — and no pickle.
• The session: correlated requests, a reader that never decodes, heartbeat-measured liveness, deadlines and cancellation carried in a CallContext, lanes for notifications only.
• Proxies: by-value or by-reference, never by guess; conservative defaults for private and mutable access; views that keep their base alive.
• Leases: the rule a holder sees is one sentence; the mechanism is renewals, tombstones and lapse callbacks.
• Aspects on a wire: one budget, stated once, inherited by every hop.

End of Volume 5

The chapters that come next are in other volumes. Volume 4 (Cross-Platform System Management) walks the sysman tier that uses these protocols heavily. Volume 7 (Building an AI Agent SDK) builds on Volume 5's REST foundations for the agent's tool-calling layer, and is the first candidate for the remote-object model this chapter closed on. Volume 8 (Time-Travel Debugging in Production) instruments wire calls for replay.

Volume 5's contribution to the series is the protocol discipline. Every other volume that involves cross-machine communication relies on what was established here: Measured Interoperability, applied to whatever protocol the situation needs, with the Aspects pattern as the delegation contract that makes the discipline scale — and, after this chapter, with a way to let objects themselves cross the wire when operations are not enough.