Volume 2 — Distributed Automation

The TestJob Inheritance Chain and Multi-Host Test Execution

About this volume

Volume 2 is the implementation of the contracts Volume 1 set up. Volume 1 introduced TestJob, the five fire_* lifecycle hooks as no-op stubs on single-host, the tandem-testing model in abstract, and the landscape boundary at the contract level. Volume 2 walks the bodies of those hooks — what InteropLibraryTestJob and DistributedTestJob do at each activation phase, how the DistributedTestSequencer walks a distributed suite, how tandem validators attach in practice, how the SSH wire propagates results, how a partial run is handled honestly.

Volume 3 then invokes what Volume 2 builds. Volume 3's orchestration layer hands off names; Volume 2's job layer realises them, verifies them, and runs the suite.

Who it is for

Test engineers running multi-host integration suites. Platform engineers building automation that crosses machine boundaries. SDETs scaling beyond one host. This is the volume to hand a senior SDET who finished Volume 1, ran their first 12-host integration suite, and needs to understand what just happened when host 7 of 12 went down mid-run.

Source material

• codecraftai-testplus — DistributedTestJob, DistributedTestSequencer, distributed_test_entrypoint.
• codecraftai-interop — InteropLibraryTestJob, the abstract _get_landscape() template method.
• codecraftai-landscaping — Landscape, the three activation layers (Configuration / Integration / Operational), the verification methods.
• codecraftai-testbase (tandem half) — Validator, ValidatorCoupling, attach_to_test.
• codecraftai-protocol-ssh — the remote-execution wire.
• codecraftai-credentials, codecraftai-dataprofiles — the cross-machine bits Volume 3 owns at orchestration scale and Volume 2 owns at job scale.

Contents

• Chapter 1 — Why Distributed Automation Is Genuinely Hard
• Chapter 2 — Single Actor, Multiple Observers — The Determinism Model
• Chapter 3 — The Pod Model — Where Parallelism Actually Lives
• Chapter 4 — The TestJob Inheritance Chain — The Customisation Cascade
• Chapter 5 — Tandem Testing in Depth
• Chapter 6 — Landscape Activation
• Chapter 7 — The CI → Job Hand-off as a Verified Contract
• Chapter 8 — The DistributedTestSequencer
• Chapter 9 — job_seed and Reproducibility
• Chapter 10 — The Remote-Execution Wire
• Chapter 11 — Credentials at Scale
• Chapter 12 — Data Profiles in Remote Workers
• Chapter 13 — Result Aggregation
• Chapter 14 — Partial-Run Semantics
• Chapter 15 — Debugging a Distributed Failure
• Chapter 16 — Case Study: A 5-Host Integration Test for a Sysman Feature

How it connects to the other volumes

• Volume 0 — error vocabulary, waiting primitives, application identity, configuration loading, the CodeCraftX philosophy (especially ch.1's meta-thesis: skeleton + patterns).
• Volume 1 — the single-host substrate; the contract Volume 2 extends. Vol 1 ch.1's seventh gap (tandem-bicycle) introduced the model; Vol 1 ch.2's customisation cascade introduced TestJob as the customisation entry; Vol 1 ch.9 established the landscape boundary.
• Volume 3 — orchestration invokes Volume 2's distributed jobs; Vol 3 ch.21 names the verification contract Volume 2 ch.7 walks in detail; Vol 3 ch.25 walks the bridge tasks.
• Volume 5 — SSH transport in detail; Volume 2 ch.10 sketches the wire and forwards to Volume 5.
• Volume 6 — landscape and topology files in detail (Volume 2 treats them as inputs; Volume 6 walks their schemas).
• Volume 8 — time-travel debugging in production; Volume 2 ch.15 sketches the post-mortem flow and forwards.

Part of The Code Craft AI Books.