# Chapter 7 — Case Studies > **Abstract.** A case study is an honest answer to "what does > using this *actually look like*?" This chapter walks three > debugging scenarios that exercise the time-travel pod / server > tier against problems readers of the earlier volumes should > recognise: a flaky distributed test (cross-referencing > Volume 2's tandem testing model), an agent execution that > behaved unexpectedly (cross-referencing Volume 7's > `ExecutionRecorder`), and a sysman feature that misbehaved on > a remote host (cross-referencing Volume 4's RHEL / Ubuntu / > macOS / Windows implementations). Each case study traces a > path from "the symptom appeared" to "the recording answered > the question", with the code anchors that drove each > decision. All three are built from the source as committed — > no fictions, with the protocol mismatch from ch. 3 §3.8 > understood as the working assumption. ## 7.1 Case study 1 — The flaky distributed test ### The setup A team running Volume 2's tandem distributed test framework (`codecraftai-testplus` driving `codecraftai-pod-tcp`) has a test that fails intermittently in CI. The test exercises a distributed system: a *driver* Pod (capital P, the test- execution unit from Volume 2) sends a request through a *frontend* observer Pod, which forwards to a *backend* observer Pod, which writes to a database; the test asserts on the database state. Symptom: about one run in twenty, the assertion fails with "row not found". Twenty runs out of twenty pass when the test is invoked locally; only the CI environment shows the failure. Standard logs say only that the assertion failed. Adding more log lines to either Pod helps localise the assertion-time state but does not show the *moment* the row failed to write. ### The debugging hypothesis The team's hypothesis: a race between the database write and the driver's read. The test asserts immediately after calling the frontend; the assertion runs against the database before the backend has finished its write. The hypothesis is testable. If they had a recording of the backend Pod's write path during a failing run, they could see whether the write committed before the driver's read started. ### Instrumenting The test fixture is updated. In the backend Pod's request handler, an existing portal `db.transaction.commit` is known to the developer team — it brackets the database commit: ```python with TimeTravelPortal("db.transaction.commit"): session.commit() ``` The portal is *registered* but not *active* on the running backend Pod. The team adds a snippet to their CI test setup that activates the portal on each test run via the JSON-lines remote-control protocol (ch. 5 §5.2): ```python import socket, json def activate_db_commit_portal(pod_host: str, pod_port: int, token: str) -> None: request = { "command": "activate_portal", "params": { "name": "db.transaction.commit", "once": True, "capture_id": f"file:///tmp/ci-debug-{run_id}.bin", }, "token": token, } request_bytes = (json.dumps(request) + "\n").encode("utf-8") with socket.create_connection((pod_host, pod_port), timeout=5.0) as sock: sock.sendall(request_bytes) response_line = b"" while b"\n" not in response_line: response_line += sock.recv(4096) response = json.loads(response_line.split(b"\n", 1)[0]) assert response.get("status") == "ok", response return ``` A single round-trip activation: create a TCP connection to the backend Pod's `9876`, send the `activate_portal` JSON request, read one line of response, assert the response is `ok`. Configured with `once=True` so the portal deactivates itself after the next call exits — no ongoing overhead, just the one recording. The capture path encodes the CI run ID so concurrent test runs don't overwrite each other's recordings. The pod's portal activation (`codecraftai-timetravel-pod/source/packages/ccai/timetravel/pod/portalregistry.py:25-41`) stores the capture URI; the next entry into the `db.transaction.commit` scope routes the recording there (via `portal.py:187-250`). ### The first failed run with instrumentation CI runs the test 20 times. 19 pass. Run 16 fails — and because of `once=True`, the recording captured only that failing run's data. The recording, off-line, walked through the pod-side `reader.py`: ```python from ccai.timetravel.pod.reader import TraceFileReader from ccai.timetravel.pod.timeline import TimelineBuilder reader = TraceFileReader("/tmp/ci-debug-16.bin") events = list(reader.read_events()) builder = TimelineBuilder() builder.add_events(events) timeline = builder.build() for ev in timeline: print(ev.timestamp_ns, ev.__class__.__name__, getattr(ev, "function_name", "")) ``` The output is a sequence of `CallEvent` and `ReturnEvent` records, each timestamped in nanoseconds since the gateway's start. The team filters to function-level events under the backend's commit codepath: ``` 142_300_000 CallEvent Session.commit 142_300_500 CallEvent Session.flush 143_100_000 CallEvent Connection.execute 143_300_000 CallEvent Connection.commit 143_400_000 ReturnEvent Connection.commit 143_410_000 ReturnEvent Connection.execute 143_420_000 ReturnEvent Session.flush 143_430_000 ReturnEvent Session.commit ``` The portal's outermost call ran from 142.3 ms to 143.43 ms into the gateway-relative timeline. ### Cross-correlating with the driver The driver Pod runs its assertion check. The driver Pod's own gateway timestamps its events relative to its *own* start — different from the backend's. But the driver wrote a log line "asserting database row exists at request_id=R-XXXX" at wall-clock time `2026-04-26T14:22:05.213Z`. The backend's gateway recorded a `GatewayStartEvent` with `wall_clock_time = 2026-04-26T14:22:05.099Z` (`pod/traceevents.py:162-174`). Subtracting: the driver's assertion ran 114 ms after the backend's gateway started. But the recording shows the backend's `Session.commit` only *returned* 143.43 ms after gateway start. **The driver's assertion ran 29 ms before the backend's commit returned.** The race is confirmed. The driver was reading before the backend had finished writing. ### The fix The hypothesis pointed at a missing synchronisation. The driver's request was issued, the frontend was returning *before the backend's commit* — likely because the frontend's response was triggered by the queue handoff, not the database commit. The fix is application-side: the backend's request handler should not signal completion until the database commit returns. Implementation in the team's app code, not in the pod. The pod's role was diagnostic: it gave the team the timeline that showed the race. Without it, they had only "row not found" intermittently and a hypothesis they could not distinguish from "intermittent network glitch" or "GC pause". With it, they had the nanosecond-precise timeline and the call graph that pinned the cause. ### What this case shows about the design - **Once-shot activation** is the right granularity for a CI flake. The team did not want a continuous recording of every CI run; they wanted *the next failing run*, recorded. `once=True` makes that the default. - **Capture-id per run** is what enables CI fan-out. Twenty concurrent test runs each had their own activation with their own capture path; no recording was overwritten. - **The control plane is scriptable.** The 30-line snippet above is the entire integration with CI. No SDK to vendor, no plugin to write — a single TCP call with a JSON request. - **The pod's reader is internally consistent** (ch. 3 §3.10). The team did not need the server to make sense of the recording — they read the file directly. This is the "small-scale" debugging path: file-backend, off-line read. For larger-scale recording aggregation across many CI runs, the server tier would land later. ### Cross-reference to Volume 2 Volume 2 ch. 13 walked the *single-actor + many-observers* shape of distributed tests in the tandem model. This case study is one tier up: the Pod (capitalised) running the backend service is the *actor*; the time-travel pod (lower-case) inside it is one of the observers. The recording's role is exactly the role Volume 2 assigns to observers: *to observe the actor's state without changing it*. The portal does its work *bracketing* the application's actual behaviour — it does not modify the commit, the database, or the response; it captures what they did and produces an artefact the analyst reads later. The book's recommendation made plain: in Volume 2's distributed test scenarios, every long-running observer Pod should expose named portals at every cross-actor boundary — the points where *this Pod observes the actor's state*. When a flaky test fails, those portals are the operator's first switch. ## 7.2 Case study 2 — The agent that skipped a tool ### The setup A team running an agent built on Volume 7's `codecraftai-agent-kit` has a customer support bot. The bot should call the `lookup_order` tool first, then call `generate_response` second, with the order data threaded through. Symptom: about one conversation in fifty, the bot calls `generate_response` *without first calling* `lookup_order`, and the response is generic instead of order-specific. The team can see the symptom in their logs (the model output is visibly wrong); they cannot see *why* the model made the choice. ### The agent's recording posture The agent's lifecycle is already instrumented with portals from its declaration in Volume 7 ch. 6: - `agent.run` brackets the entire agent invocation. - `agent.tool.dispatch` brackets each tool call. - `agent.message.compose` brackets the model's final response composition. The agent's recording posture is `calls only` by default (`trace_calls=True`, others false). Cheap. All three portals are registered; none is currently active. ### The hypothesis Two competing hypotheses: 1. The model's tool-selection step somehow returned a `compose` action before completing the `lookup_order` action. Bug at the model layer — possibly a malformed tool schema, possibly a model hallucination. 2. The tool-selection logic on the SDK side has a code path that bypasses `lookup_order` under certain conditions. Bug in the agent SDK. A recording can distinguish. If `agent.tool.dispatch` was called for `lookup_order` and *returned an error*, the model was given a poison-pill response and skipped to compose (hypothesis 2). If `agent.tool.dispatch` was *never called* for `lookup_order`, the model never asked (hypothesis 1). ### Instrumenting The team activates the three portals on the production agent. This time they need *variable capture* on the `agent.tool.dispatch` portal — they need to see the *tool arguments*, which means line-and-variable granularity on that portal. Variable capture is configured at gateway construction; because of the spec gap from ch. 5 §5.3 the team cannot hot-flip a running pod's trace configuration. Their workaround is to deploy a configuration-update through the standard CI/CD path: ```python TimeTravelConfig( active_portals=["agent.run", "agent.tool.dispatch", "agent.message.compose"], trace_calls=True, trace_lines=True, trace_variables=True, record_inner_portals=True, backend_uri="tcp://timetravel.internal:9090", ) ``` This is the operational reality §5.3 describes: *granularity is set at startup*. The team accepts the deploy and rolls out the new configuration to canary, then to production. ### The recording A few hours after rollout, two more flake instances are recorded. The team queries the time-travel server for the trace sessions that bracket the failing conversation IDs (the agent logs the conversation ID at the start of each turn; the team correlates). The recording for one failure shows, on the call-graph reconstruction (`tracedata/call_graph.py:32-62`): ``` agent.run [duration: 1.842s] ├─ agent.tool.dispatch [duration: 0.009s] │ └─ ToolRegistry.dispatch [duration: 0.009s] │ ├─ ToolRegistry._lookup [duration: 0.001s] │ └─ ToolRegistry._invoke [duration: 0.005s] │ └─ └─ agent.message.compose [duration: 0.420s] └─ ModelClient.complete [duration: 0.417s] ``` The `lookup_order` tool *was* invoked. It returned an *Exception* (`Connection refused`) at the `ToolRegistry._invoke` boundary. The agent SDK's response: it fell back to `compose` *with the exception in scope*. The variable snapshot at the `compose` portal entry confirms. The team queries the variable history (`tracedata/variable_tracker.py:39-55`) for the `tool_results` variable at the timestamp of `agent.message.compose` entry: ```python { "lookup_order": { "status": "error", "error": "Connection refused", "fallback_invoked": True, } } ``` The model received a tool-result that said "lookup_order failed; fallback invoked". The model — correctly, on its training — generated a generic response. ### The fix The bug is in the order-lookup service, not the agent. The service has an intermittent connection issue that the team had not yet correlated with this symptom. The service team investigates separately; the agent team adds a retry on `ToolRegistry._invoke` for transient errors before falling back to the model. ### What this case shows - **Variable capture is the high-fidelity option.** The call-only recording would have shown the exception was raised and that compose was called next. The variable- capture recording showed *what the model received as its prompt*, which was the answer to the actual question ("did the model see the failure or did the SDK silently proceed?"). - **The deploy-time-only granularity choice is a real cost.** The team had to roll out a configuration change to enable variable capture. The spec gap from ch. 5 §5.3 is felt operationally: a runtime-flippable granularity would have closed the diagnostic loop in minutes instead of hours. - **The call graph is the right view for the agent path.** The team did not care about every line that executed; they cared about the *shape* of the call graph — what was invoked, in what order, with what return value. The call- graph builder (`tracedata/call_graph.py`) is the analysis module that answers this question. ### Cross-reference to Volume 7 Volume 7 ch. 6 introduced `ExecutionRecorder` / `ExecutionReplayer` as the agent-specific recording mechanism. This case study shows the *generalisation* in action: instead of a special-purpose recorder for agent runs, the agent's lifecycle is bracketed by ordinary portals (`agent.run`, `agent.tool.dispatch`, `agent.message.compose`), recorded by the ordinary pod, ingested by the ordinary server, queried by the ordinary analysis tier. The benefit of generalisation: an agent's debugging path is the same as any other production debugging path. The operations team has one toolset for "production debugging" and agent-driven services are not a special case. The cost of generalisation: the agent-specific affordances the SDK could provide (richer model-output capture, tool- schema-aware filtering, dedicated agent-run views) are not yet in the pod. The book recommends them as future work that sits comfortably in the agent SDK rather than in the pod. ## 7.3 Case study 3 — The sysman feature that didn't apply ### The setup A team running Volume 4's cross-platform sysman tier has a feature that *applies a firewall rule* on RHEL hosts. The feature is implemented in `codecraftai-sysman-rhel` and consumed through the standard sysman client. The implementation shells out to `firewalld`'s `firewall-cmd`. Symptom: on one specific RHEL 9 host out of a fleet of 200, the firewall rule does not apply. The sysman feature reports success. An out-of-band check (`firewall-cmd --list-all`) shows the rule is not in place. ### The hypothesis The action's stdout said success. The action's stderr was empty. The exit code was zero. The host's `firewalld` is running. But the rule is not present. A reasonable hypothesis: the action is shelling to the *runtime* `firewall-cmd`, but the rule is not being persisted through `firewalld`'s config; the next reload would lose it. But that does not explain why a `--list-all` immediately after the action shows no rule — there has been no reload in between. A second hypothesis: there is a permissions issue. The action is running as a service account that *thinks* it can call `firewall-cmd`, but the call is being silently dropped by PolicyKit. ### Instrumenting The sysman feature on the affected host has portals at every boundary: - `sysman.feature.firewall.enable` - `sysman.action.run` - `sysman.action.exit` The team activates them with variable capture. The `sysman.action.run` portal brackets the actual `subprocess.run` call; the variable snapshot at line-level captures the command, arguments, environment, and the resulting `CompletedProcess`. Activation, this time, goes through the time-travel server's REST surface: ``` POST /processes/{host_id}/portals/sysman.action.run/activate payload: {"capture_id": "file:///var/log/ccai/firewall-debug.bin"} ``` (In the implementation as committed, this endpoint returns mock data. The team's workflow uses the `CommandDispatcher` directly via a server-side script — not the REST API tier — demonstrating the §6.9 honesty point: the *services* are real; the REST wrappers are scaffold.) ### The recording A scheduled re-application of the firewall rule fires. The recording captures the action. The team reads the file: ```python from ccai.timetravel.pod.reader import TraceFileReader reader = TraceFileReader("/var/log/ccai/firewall-debug.bin") events = list(reader.read_events()) # Walk the variable snapshots inside sysman.action.run in_action = False for ev in events: if isinstance(ev, PortalEnterEvent) and ev.portal_name == "sysman.action.run": in_action = True elif isinstance(ev, PortalExitEvent) and ev.portal_name == "sysman.action.run": in_action = False elif in_action and isinstance(ev, VariableSnapshotEvent): if "completed_process" in ev.local_variables: cp = ev.local_variables["completed_process"] print(f"@ {ev.timestamp_ns} cmd={cp.args} rc={cp.returncode}") print(f" stdout={cp.stdout!r}") print(f" stderr={cp.stderr!r}") ``` The output: ``` @ 4_512_000_000 cmd=['firewall-cmd', '--add-rich-rule=...', '--permanent'] rc=0 stdout=b'success\n' stderr=b'' ``` A subsequent variable snapshot, just inside `sysman.feature.firewall.enable`, shows the team's *own* verification step: ``` @ 4_530_000_000 verified=True ``` The action ran, returned success, and a verification step *also* reported success. But the rule is not on the host. ### The clue in the next snapshot Looking past the immediate action, the next portal that fires is `sysman.feature.firewall.persist` — a different portal, in the *next* feature in the chain. Its variable snapshot shows: ``` @ 4_580_000_000 cmd=['firewall-cmd', '--reload'] rc=0 stdout=b'success\n' stderr=b'' ``` A reload, immediately after the action. The team checks the host's `/etc/firewalld/zones/public.xml`. The rich-rule the action added at 4.512 s is *not* in the file at the time of reload. ### The bug The issue is in the team's automation: The `--permanent` flag does *not* apply the rule to the running configuration. It writes the rule to the *permanent* configuration files. A subsequent reload reads those files and applies them. The team's verification step ran at 4.530 s. At that moment: - The runtime configuration had not changed (no `--add-rich- rule` without `--permanent`). - The permanent configuration files *had* been updated, but the reload that applies them had not yet run. - The verification step checked the *runtime* configuration, saw the rule was absent, and — because of a default in the team's verifier — assumed *true* if no contradicting evidence was found. A bug compounded by a bug: - The action used `--permanent` only, not `--permanent --add-rich-rule + a separate runtime call`. This is a documented `firewalld` gotcha. - The verifier's default-true was incorrect. It should have required positive confirmation. ### The fix The action is updated to call `firewall-cmd --add-rich-rule` *twice*: once with `--permanent`, once without. The verifier is updated to require positive confirmation with an explicit "rule found" predicate. ### What this case shows - **Variable capture made the bug visible.** Without the `CompletedProcess` snapshots, the team had only the action's reported success and the verifier's reported success — both of which were positive — and a host that did not behave as the reports claimed. - **Multi-portal recording provides correlation.** The action portal showed the action's behaviour; the persist portal showed the reload's behaviour; the gap between them showed the bug was the *runtime configuration not yet updated when the verifier checked it*. - **The recording made the documentation point obvious.** Once the team saw the call sequence, the `firewalld` documentation about `--permanent` was the immediate next read. The recording did not solve the bug; it pointed precisely at the place the documentation needed to be consulted. ### Cross-reference to Volume 4 Volume 4's RHEL chapter (ch. 4) walked the `codecraftai-sysman-rhel` package, including the deliberate deviation from Convention 4: RHEL-specific exception types are *not* prefixed with the platform name, because the package's contract is the OS-specific implementation of a sysman feature and the platform context is implicit. This case study exercises that implementation. The portal boundaries used here (`sysman.feature.firewall.enable`, `sysman.action.run`, `sysman.action.exit`, `sysman.feature.firewall.persist`) are the *generic* sysman boundaries — the same names appear on the Ubuntu, macOS, and Windows implementations. A bug on Ubuntu would surface through the same portal names with platform-specific variable contents. The pod abstracts away the platform- specific commands; the portal vocabulary is consistent across the cross-platform tier. ## 7.4 What the three case studies have in common Pulling the three together, three patterns repeat: **Static instrumentation of architectural boundaries was sufficient.** In all three cases, the portals the team activated were already in the source code, declared at a boundary the developer had already identified as debugging-worthy: the database commit boundary, the agent tool-dispatch boundary, the sysman action-run boundary. No ad-hoc instrumentation was added during the incident. **The right granularity was case-by-case.** The flaky test needed only `calls`; the agent needed `lines + variables`; the sysman feature needed `lines + variables`. The calls-only recording in case 1 was inexpensive and left no operational footprint. The variable-capture recordings in cases 2 and 3 cost an order of magnitude more in volume. Both choices were appropriate for the question being asked. **Off-line analysis dominated.** In every case, the operator read the recording *after* the symptom had passed. This is the time-travel debugging mode the volume's title names: the symptom occurred, the recording captured the state, the operator walked the recording to understand what happened. Replay-during-incident was not the workflow; *post-mortem state reconstruction* was. ## 7.5 What the three cases revealed about the design Each case lit a specific design seam. **Case 1 lit `once=True` activation.** The CI flake workflow demanded a single recording, not a continuous stream. The flag exists, is honoured, and is the right granularity for *next failure occurrence* debugging. **Case 2 lit the spec gap on runtime configuration.** The agent team's variable-capture decision required a deploy. The chapter has been honest: until `update_configuration` lands on the JSON-lines protocol (ch. 5 §5.3), this is the operational reality. **Case 3 lit the multi-portal correlation use case.** The firewalld bug only became visible when the team correlated *two* portals' recordings — the action portal showed the intent, the persist portal showed the side effect, the gap between them was the bug. The pod-side `timeline.py` (`pod/timeline.py:35-71`) and server-side `timeline.py` (`tracedata/timeline.py:35-63`) both support multi-source timeline merging via `heapq.merge`; this is the algorithmic foundation for cross-portal correlation. ## 7.6 Bridging to the next chapter The three case studies are *qualitative* — they describe what the recording revealed and how the team used it. The remaining question is *quantitative*: how much overhead does each granularity actually impose? How much storage does a recording consume? When does the cost cross a threshold that makes the recording impractical? The next chapter measures. It quotes the runtime-overhead budget for each granularity, the storage budget for each typical use case, and the policy framework for *when to enable, when to disable*. → Continue to [`08-performance-storage-tradeoffs.md`](08-performance-storage-tradeoffs.md). ← Back to [`06-session-management-at-scale.md`](06-session-management-at-scale.md).