Chapter 15 — Debugging a Distributed Failure

Distributed automation produces failures that don't reproduce on a laptop. The framework's debugging story is therefore not "step through the code"; it's "extract enough structured data from the failed run to reason about what happened across hosts and produce a hypothesis you can test." This chapter walks the audit-diary pattern, with the six-pass landscaping audit as the reference example.

A distributed test failed in CI. The summary says one test errored on host C, two validators caught steady-state violations, and host F's log fetch was truncated. The CI runner has been torn down. Re-running may or may not reproduce. How does an SDET diagnose what happened?

This chapter is the framework's answer. It is not a universal-debugging chapter — the framework can't replace domain knowledge of the system under test. It is a chapter on what the framework's structured data lets you do and the audit-diary pattern for working through it.

The full production-grade post-mortem story — replay-driven debugging, time-travel reconstruction, pod instrumentation — lives in Volume 8. This chapter establishes the workflow at the framework level; Volume 8 provides the tooling that makes the workflow easier in production.

The structured data the framework provides

The structured-data debugging story in this chapter is the story of an integration run — a run driven by IntegratedTestJob (in codecraftai-integration-runtime). Plain testbase runs standalone and records only to stdout (its default recorder is StdoutResultRecorder); it has no presence folder, no RuntimeVariables, and no on-disk results-directory layout to debug from. The presence-folder results directory, the JSOS-to-disk stream, the rendered HTML summary, and the CI/build metadata are all the responsibility of IntegratedTestJob, which overrides three TestJob provider hooks: _create_output_context() returns RuntimeOutputContext (artefacts land under the activated app's well-known presence folder via ccai.thisapp.paths — activation-gated), _create_run_metadata() returns RuntimeRunMetadata (build / pipeline / job from RuntimeVariables), and _create_recorder() returns JsonResultRecorder (JSOS streaming plus the rendered HTML summary). The audit-diary and evidence discipline below therefore applies to IntegratedTestJob runs — that's where the files exist to reason about.

An integration run that failed leaves a results directory shaped roughly:

results/20260501-153012-abc123/
├── testsummary.json                # high-level
├── testresults_stream.jsos         # streaming records
├── testresults.json                # consolidated
├── startup-configuration.json      # run inputs (Vol 1 ch.6)
├── host-logs/
│   ├── casey-01/
│   │   ├── syslog.log
│   │   ├── auth.log
│   │   └── application.log
│   ├── casey-02/
│   │   └── ...
│   └── casey-12/                   # may be partial — host went down
│       └── pre_only.log
├── diagnostics/
│   ├── pre_testrun/
│   │   ├── package-versions.json
│   │   ├── environment.json
│   │   └── per-host/
│   │       ├── casey-01.json       # host metadata snapshot
│   │       └── ...
│   └── post_testrun/
│       └── ...
└── tc-by-products/
    ├── test_replication_in_majority/
    │   ├── stdout.log
    │   ├── stderr.log
    │   └── per-host-output/
    │       ├── casey-01.log        # commands run on casey-01
    │       └── ...
    └── ...

That is the framework's gift to the debugger. Every structured fact about the run is recoverable from these files. The debugger's job is to reason about what combination of facts produced the failure.

The audit-diary pattern

The portfolio's debugging discipline — applied across multiple audits in real history — is the audit diary. An audit diary is a structured note an SDET keeps while diagnosing a failure. It has six sections:

1. Symptom. What the run reported. Quote the JSOS record.
2. Hypothesis-1. The first plausible explanation.
3. Evidence for / against H1. What the structured data says.
4. Hypothesis-2 (if H1 ruled out). Repeat.
5. Diagnosis. The root cause once a hypothesis fits all the evidence.
6. Remediation. The fix and how it's verified.

The discipline is not unique to this framework — it's a generic engineering pattern. The framework's contribution is making each section answerable through the structured data. Symptoms come from JSOS records; evidence comes from per-test logs and host logs; remediation is verified by re-running the suite with the same job_seed (chapter 9).

A worked audit diary

A real-shape example. Suppose a run failed with:

{
  "run_id": "20260501-153012-abc123",
  "outcome": "partial",
  "tests": {"passed": 47, "failed": 1, "errored": 1, "skipped": 3},
  "phases": {
    "operational": "partial",
    "verification": {"landscape": "partial", "topology": "success"}
  },
  "missing_devices": ["casey-12"]
}

And the failure record:

{
  "kind": "test_finished",
  "test_name": "test_quorum_after_partition",
  "outcome": "errored",
  "exception": {
    "type": "TimeoutError",
    "message": "Timeout waiting for cluster to converge",
    "traceback": [
      "  File \"tests/test_quorum.py\", line 87",
      "    wait_for_quorum(cluster, timeout=60)",
      "  File \"ccai/agentkit/cluster.py\", line 142, in wait_for_quorum",
      "    raise TimeoutError(...)"
    ]
  },
  "host_log_status": {
    "casey-01": "complete", "casey-02": "complete",
    "casey-03": "complete", "casey-04": "complete",
    "casey-05": "complete", "casey-06": "complete",
    "casey-07": "complete", "casey-08": "complete",
    "casey-09": "complete", "casey-10": "complete",
    "casey-11": "complete", "casey-12": "unreachable_at_activation"
  }
}

Symptom

test_quorum_after_partition errored with TimeoutError — "Timeout waiting for cluster to converge." The Pod was 11/12 hosts (casey-12 unreachable from activation).

Hypothesis-1

The test depends on casey-12 for quorum. With casey-12 down, the cluster cannot reach quorum and the wait_for_quorum times out. Root cause: the test should have been skipped (like test_failover_to_casey_12 was) but wasn't.

Evidence for H1

• The test has no @skip_if(host_unavailable("casey-12")) decoration. (Read the test source.)
• Tests that were skipped have explicit declarations. (Read the JSOS skip records: each carries reason: "Required host casey-12 was not in operational landscape" — the framework's deactivation chain caught them.)
• The test's host requirements are implicit — it walks the full cluster expecting all 12 hosts.

Diagnosis

The test's host requirements are implicit; the framework couldn't deactivate it because the framework didn't know what the test required. The test ran against an 11-host cluster expecting a 12-host quorum.

Remediation

Add explicit host requirements to the test. Either through a @skip_if(host_unavailable(...)) decoration matching the ones on the explicitly-host-bound tests, or by reading the integrated landscape and skipping when the configured majority cannot be reached. Re-run with the same job_seed against an 11-host Pod (recreating the failure deterministically); the test should now skip rather than error.

The audit-diary takes 15-30 minutes for a clear case like this. The framework's structured data is what makes it possible to answer each section without re-running the suite.

The cross-host log-correlation discipline

When a failure spans multiple hosts (a network partition test, a replication test, a leader-election test), the audit diary needs to correlate events across host logs. The framework's discipline:

• Use the JSOS record's timestamps as the controller's reference clock. Volume 1 chapter 5 said the controller's clock is the JSOS clock.
• Identify each host's offset from the controller at the pre-test diagnostic capture. The capture includes per-host time queries; the offset is host_time - controller_time at that moment.
• Apply the offset (with caveats about drift) when reading host logs. A controller-side JSOS record at T_controller = 15:32:00.500 corresponds to roughly T_host_C = T_controller + offset_C in casey-03's syslog.

The framework does not automatically do this correlation — chapter 13 explained the design choice to keep timelines side-by-side rather than forced-merged. But the diagnostic capture provides the offset values, and a debugging script (or AI agent) can compute the correlation when needed.

The bisection pattern for reproducing flakes

A test that fails 5% of the time is a flake. The audit diary for a flake has a different shape than for a deterministic failure.

Three phases:

Phase 1 — Reproduce the failure deterministically

Run the test repeatedly with random seeds until you have a seed that triggers the failure. Record the seed. Now you have a reproducible failure (same seed → same input → same test behaviour, modulo Pod-state non-determinism).

Phase 2 — Bisect on Pod state

Re-run the failure-inducing seed against a fresh Pod. If the failure reproduces, the failure is Pod-state-independent (it's about the test or the system under test, not the specific Pod state). If it doesn't, the failure is Pod-state-dependent — re-running won't help; you need Volume 8's time-travel tooling to reconstruct the original Pod state.

Phase 3 — Bisect on inputs

For Pod-state-independent failures, the failure-inducing seed gives you a specific input. Use the framework's shrinking machinery (Volume 1 chapter 4) to find the smallest input that still triggers. Add that input as a regression test case.

The pattern is universal but the tools differ between Pod-state-dependent and Pod-state-independent flakes. Most distributed flakes turn out to be Pod-state-dependent, which is why Volume 8's tooling is so important for production debugging.

Reference: the six-pass landscaping audit

The portfolio's history includes a six-pass audit of codecraftai-landscaping that fixed 15+ silent runtime bugs. The audits are real production-grade examples of the audit-diary pattern at scale. The commits:

• Pass 1 (42f9c86) — six refactor-leftover bugs, including Liskov-substitution violations (three protocol extension subclasses lacked **kwargs forwarding), an uninitialized ControlExtension._control_id, three string-concatenation bugs, a mutable default argument in factories.create_landscape(), and duplicate dataclass definitions of DatacenterSummary and FolderSummary.
• Pass 2 (a92e5d2) — two attribute-name mismatches: a LandscapeService.__init__ declaring _primary_ssh_credential but the search loop and property using _ssh_credential (silent AttributeError on credential lookup); an OrchestrationJob.begin() writing to _orch_results_dir but reading the nonexistent _test_results_dir.
• Pass 3 (1679fe7) — two more refactor-leftover bugs: LandscapeIntegrationLayer.__init__ initialising pool attributes to None instead of {} (breaking @property accessors that called .copy() unconditionally); a ClientSourcePackager leaking file handles in zip-file digest computation.
• Pass 4 (1545f67) — five logic / behavioural bugs: exclude-filter "duplicate-add" bug at four call sites (missing break, wrong semantics — items that passed N filters were added N times); FriendlyIdentifier.__eq__ always returning True (compared self._full_identifier to itself rather than other._full_identifier); FriendlyIdentifier.identity property dereferencing regex groups without null checks; wrong return-type annotations on get_devices() and get_services() (annotated to return single items instead of lists); inconsistent exception type in ClientSourcePackager (Exception instead of RuntimeError).
• Passes 5 and 6 — incremental, integrated into specific test fixtures.

Three lessons from the six-pass arc:

1. Audits surface progressively

Each audit pass caught bugs the previous passes missed. The first pass caught structural bugs (Liskov violations, uninitialised attributes, mutable defaults). Later passes caught semantic bugs (filter logic, equality semantics, type annotations). The progression reflects the reviewer's lens shift — each pass looks for a different class of issue.

2. Silent failures are common

Every bug fixed across the six passes was silent at import time. None of them caused an ImportError; none of them broke the package's smoke tests. They surfaced only when specific code paths exercised the bug — and the code paths were sometimes exercised only in distributed scenarios that were rare in single-host tests.

The lesson: the framework's structured-data discipline is critical for catching silent failures. An audit diary working from JSOS records can spot inconsistencies (a test that should have been skipped wasn't; a credential that should have resolved didn't) that no individual test exposed.

3. Tests added per fix

Every audit pass added regression tests for the bugs it caught. The Volume 1 chapter 10 case study mentioned that testbase's own test suite runs around 3,700 tests; the landscaping audit contributed +101 tests across 11 test files to that count. The tests are now permanent regressions ensuring the same classes of bugs can't recur.

What's not in this chapter — Volume 8

Production-grade post-mortem debugging — replaying a failed run, walking the Pod's state at a specific moment, reconstructing what happened with full instrumentation — is Volume 8 (Time-Travel Debugging in Production). Volume 8's tools assume:

• The Pod was instrumented during the run.
• The instrumentation produced a replay-able event log.
• A debugger can replay the event log against a reconstructed Pod state.

For Pod-state-dependent flakes (the hard ones), Volume 8 is the right tool. This chapter walks what an SDET can do with just the framework's structured data; that's enough for ~70% of distributed failures and a starting point for the remaining 30% that need Volume 8's tooling.

What the next chapter does

Chapter 16 is the volume's case study: a 5-host integration test for a sysman feature. The chapter walks the entire volume's machinery in one worked example — Pod provisioning, inheritance chain, landscape activation, verification, tandem validators, the wire, credentials, partial-run semantics, audit diary.

By the end of the case study, a reader who started Volume 2 cold should be able to read a real-world distributed test job's source and predict its behaviour. That fluency is the volume's contract.