Chapter 1 — What Integration-Level Automation Is

Most CI failures are not test failures. Most CI failures are environment failures, configuration failures, or coordination failures. The discipline that separates the three is integration automation.

A pipeline run is conventionally drawn as a stack of layers:

┌─────────────────────────────────────────────────────┐
│  CI provider (GitLab CI / GitHub Actions / Jenkins) │
└─────────────────────────────────────────────────────┘
                        │
                        ▼
┌─────────────────────────────────────────────────────┐
│  Orchestration                                      │  ← integration
│   "what high-level steps run, in what order,        │     automation
│    with what fallbacks?"                            │
└─────────────────────────────────────────────────────┘
                        │
                        ▼
┌─────────────────────────────────────────────────────┐
│  Jobs                                               │
│   "given step N is 'run tests', execute the         │
│    test suite (possibly across N hosts)."           │
└─────────────────────────────────────────────────────┘
                        │
                        ▼
┌─────────────────────────────────────────────────────┐
│  Tasks                                              │
│   individual tests, sysman feature calls,           │
│   interop ops, the things the jobs are made of      │
└─────────────────────────────────────────────────────┘

This chapter is about the second layer. Most teams conflate it with the first or the third — they put orchestration logic into vendor YAML, or they put orchestration logic into job code as conditional branches keyed off environment variables. Both choices have predictable failure modes. Both choices are how the fifty-line YAML grows to six hundred lines.

What "integration-level" means

Three jobs to be done sit at the orchestration layer:

1. Sequencing. Which steps run, in what order, with what parallelism, with what fallback when one fails.
2. Parameterisation. Which configuration is in scope for this run — which landscape, which runtime profile, which credential set, which data profiles. This is not "set forty environment variables"; this is "select named typed configurations."
3. Coordination. How information flows between steps — the output of step N informing the input of step N+1; the artifacts of build informing the inputs of test; the results of test informing the deploy gate.

A well-built orchestration layer has clean expression of all three. A typical CI YAML expresses (1) clearly, (2) badly, and (3) by accident — the accidental expression being string-substitution into shell commands of values that came from environment variables that were exported by an earlier step that nobody remembers anymore.

What "integration-level" doesn't mean

Three things that are not integration-level automation, despite often being conflated with it:

• Build automation. Compiling code, linking artifacts, packaging containers — these are tasks an orchestration may invoke, but they are not orchestration. A build script is a single step in a larger flow. When the build script tries to also be the orchestration, the Makefile is fifteen hundred lines and contains a function called is_running_in_ci.

• Job execution. Running a test suite, running a deployment, running a sysman feature against a fleet — these are jobs. Jobs may run on multiple hosts; jobs may have internal sequencing of their own (Volume 2 on distributed automation is about exactly this). But a job is invoked by an orchestration; it is not an orchestration.

  A point of vocabulary worth pinning down here: the test job hierarchy in Volume 2 (TestJob → InteropLibraryTestJob → DistributedTestJob) and the orchestration job class (OrchestrationJob in codecraftai-landscaping/source/.../orchestrationjob.py:34) are separate hierarchies. OrchestrationJob inherits from ContextUser, not from TestJob. This volume treats orchestration jobs; Volume 2 treats test jobs. They meet at the bridge tasks of chapter 25 and not before.

• Provisioning. Creating cloud resources, allocating containers, setting up databases — these are resources an orchestration consumes, expressed declaratively as Specifications (chapter 4). When a CI YAML contains forty lines of aws ec2 create-instance shell, those lines are misplaced; the orchestration should consume a resource by name, not by recipe.

The discipline of integration automation is to keep these three out of the orchestration layer. The orchestration says "this Specification must be live" or "run this Job" or "publish this artifact." It does not contain the recipe for any of the three.

Why this matters

Three concrete consequences flow from getting the layer separation right:

Pipelines become readable

A workflow that invokes a Job by name (rather than inlining seventy lines of shell to do the Job's work) is short. Short pipelines fit in code review; long pipelines do not. Pipelines that fit in code review get reviewed; pipelines that do not, do not. The first place discipline pays off is in the simple act of reading what the pipeline does on a Tuesday afternoon when something has gone wrong.

Pipelines become portable

A workflow expressed in codecraftai-taskplus's YAML is identical whether it runs on GitLab CI, GitHub Actions, Jenkins, or a developer's laptop. The CI vendor's YAML shrinks to a trigger and a runner image; everything else moves into the portable workflow. The portable workflow can be checked into the same repo as the code; it can be versioned with the code; it can be diffed; it can be tested in isolation.

Pipelines become debuggable

When a pipeline fails, the question "what configuration was in scope when it failed?" has an answer. The answer is the names of the typed configurations the orchestration selected: landscape staging-east, runtime verbose, credentials ci-runner. Those are typed objects with schemas; they are not the union of forty environment variables exported in unknowable order. An engineer reproducing the failure locally selects the same names. This is the subject of Part IV.

The shape of this volume

The volume has six parts. This chapter — and the next two — present the discipline. Part II names the six configuration categories and introduces each. Part III is the credentials chapter (which is a thicker subject than the other five categories combined and earns its own part). Part IV is the local-dev → CI → staging → prod story — how the discipline pays off in real workflows. Part V is the implementation in codecraftai-taskplus. Part VI is the case study and the wiring to real CI vendors.

A reader who learns nothing else from this volume should leave with this: the orchestration layer is not the place for environment variables. Environment variables are the symptom of a missing configuration category. Every time you find yourself reaching for an export FOO=bar in a pipeline, ask which of the six categories the value belongs in — and then put it there instead.

Summary

• Integration-level automation is the orchestration layer between the CI provider and the jobs that do work.
• It has three jobs: sequencing, parameterisation, coordination.
• Most teams conflate it with build automation, job execution, or provisioning. All three conflations are how pipelines grow unmanageably.
• The discipline of this volume is to keep the orchestration layer small and to express its parameterisation as named typed configurations rather than as environment variables.
• The six configuration categories — Specifications, Landscapes, Topologies, Runtimes, Credentials, Data Profiles — are introduced in Part II.