Chapter 4 — Specifications

A pipeline that contains forty lines of aws ec2 create-instance is two pipelines pretending to be one. The first is a provisioner; the second is the actual orchestration. The Specification separates them.

The previous chapter named six configuration categories. This chapter introduces the first one: Specifications — the declarative description of the resources that must exist before an orchestration runs.

Specifications are the only one of the six categories that does not live inside codecraftai-configuration. They live in codecraftai-provisioning. The reason is structural: the other five categories are loaded by the orchestration as it starts; Specifications are consumed before the orchestration starts. They describe a state of the world that the orchestration assumes is true.

The provisioner-orchestration confusion

Most CI pipelines that automate cloud-backed environments contain two concerns smeared into one file: how to produce the environment, and what to do with it once produced. When the two are mixed, three predictable failures follow:

Recipes drift. The thirty lines of aws ec2 create-instance in pipeline A are subtly different from the thirty lines in pipeline B. The differences are accidental, accumulated over months of small fixes nobody propagated across all three pipelines that need similar VMs.
Re-runs are not idempotent. Imperative provisioning code is hard to make idempotent, because imperative code does not have a clean way to say "if the resource already exists in the right shape, leave it alone." Re-running an imperative provisioning step on a partially-provisioned environment leaves a mess.
The orchestration cannot run against a pre-existing environment. When the imperative provisioning is part of the orchestration, you cannot easily say "skip provisioning today; the environment is already up." The orchestration is welded to its provisioner.

The discipline of this chapter is to factor the two concerns apart. The orchestration declares the resources it needs. A separate provisioning step makes the declaration true. The orchestration itself is unchanged whether the provisioning step ran or whether a matching environment was already up.

The Specification format

A Specification is a YAML file with two top-level sections:

environment:
  # how to provision: where, in which account, with which credentials,
  # under which logical groupings
  venues: [...]
  groups: [...]

apod:
  # what to provision: the concrete devices that make up the
  # automation pod, each referencing a profile that describes the
  # resource shape

  devices: [...]

profiles:
  # the parameterised resource templates referenced by devices

  - {...}
  - {...}

The structure is parsed by ProvisioningEngine.create_provisioning_engine(specification=...) which loads the YAML and instantiates four kinds of model objects. The four are layered abstractions, not a flat list. Understanding the layering is the key to writing maintainable specs.

Layer 1 — CloudVenue: where we provision

A CloudVenue describes a target account in a cloud provider:

environment:
  venues:
    - name: aws-staging
      provider: aws
      management_url: https://ec2.us-east-1.amazonaws.com
      subscription: "123456789012"   # AWS account ID
      region: us-east-1
      credential: ci-runner-aws

A venue says: this is an AWS account in us-east-1, accessed with the ci-runner-aws credential. The credential field is a reference into the credentials category (chapters 9–12) — the venue does not embed secret material; it names a credential the resolution layer will look up at runtime.

Multiple venues can coexist in one Specification — you can provision across AWS and Azure in a single spec — but the typical case is one venue per spec, with multiple specs for multi-cloud setups. Per the per-provider provisioners shipped under venues/{aws,azure,google}/, the supported providers are AWS, Azure, and Google Cloud.

Layer 2 — CloudGroup: how resources are organised

A CloudGroup describes a logical grouping with associated tags:

environment:
  groups:
    - name: ci-runner-pool
      description: "Worker pool for the integration test suite"
      tags:
        - key: cost-center
          value: engineering-test
        - key: lifecycle
          value: ephemeral

Groups are the unit of cleanup. When the orchestration is finished and the provisioning step is asked to tear down, it tears down by group. Devices that share a group share a fate. Tags travel with the devices into the cloud provider's billing and management systems, so the cost-center attribution is automatic.

Layer 3 — CloudResourceProfile: what kind of resource

A CloudResourceProfile is a parameterised template for resources of a particular shape:

profiles:
  - name: runner-medium
    description: "Standard medium-sized integration runner"
    credential: ci-runner-aws
    parameters:
      instance_type: t3.medium
      image: ami-0abcd1234
      subnet: subnet-0123456789abcdef0
      security_groups:
        - sg-runner-default
      iam_profile: runner-instance-profile
      ebs_volumes:
        - device: /dev/sda1
          size_gb: 30
          type: gp3
    tags:
      - key: role
        value: runner

A profile is the recipe, but it is the recipe expressed as data, not as imperative code. The provisioner for the venue's provider knows how to turn the parameters into the right calls — the AWS provisioner reads instance_type and calls RunInstances; the Azure provisioner reads the same parameter and calls compute.virtual_machines.begin_create_or_update. The same profile, with provider-specific parameter blocks, can target either cloud.

Layer 4 — CloudDevice: which specific resources

A CloudDevice is a single instance to be provisioned, referencing a group, a profile, and a role:

apod:
  devices:
    - identifier: runner-1
      category: ec2-instance
      group: ci-runner-pool
      profile: runner-medium
      role: runner
      tags:
        - key: index
          value: "1"
    - identifier: runner-2
      category: ec2-instance
      group: ci-runner-pool
      profile: runner-medium
      role: runner
      tags:
        - key: index
          value: "2"
    - identifier: runner-3
      category: ec2-instance
      group: ci-runner-pool
      profile: runner-medium
      role: runner
      tags:
        - key: index
          value: "3"

Three devices, each pointing at the runner-medium profile, all in the ci-runner-pool group. The provisioner instantiates three EC2 instances of the same shape, tags them per-device, and collectively tears them down when the group is dismissed.

A complete worked example

Here is the assembled Specification for a three-runner integration pool, end to end:

# spec-ci-runner-pool.yaml
environment:
  venues:
    - name: aws-staging
      provider: aws
      management_url: https://ec2.us-east-1.amazonaws.com
      subscription: "123456789012"
      region: us-east-1
      credential: ci-runner-aws
  groups:
    - name: ci-runner-pool
      description: "Worker pool for the integration test suite"
      tags:
        - key: cost-center
          value: engineering-test
        - key: lifecycle
          value: ephemeral

profiles:
  - name: runner-medium
    description: "Standard medium-sized integration runner"
    credential: ci-runner-aws
    parameters:
      instance_type: t3.medium
      image: ami-0abcd1234
      subnet: subnet-0123456789abcdef0
      security_groups:
        - sg-runner-default
      iam_profile: runner-instance-profile
      ebs_volumes:
        - device: /dev/sda1
          size_gb: 30
          type: gp3
    tags:
      - key: role
        value: runner

apod:
  devices:
    - identifier: runner-1
      category: ec2-instance
      group: ci-runner-pool
      profile: runner-medium
      role: runner
      tags: [{key: index, value: "1"}]
    - identifier: runner-2
      category: ec2-instance
      group: ci-runner-pool
      profile: runner-medium
      role: runner
      tags: [{key: index, value: "2"}]
    - identifier: runner-3
      category: ec2-instance
      group: ci-runner-pool
      profile: runner-medium
      role: runner
      tags: [{key: index, value: "3"}]

The file is consumed by:

from ccai.provisioning.provisioningengine import ProvisioningEngine

engine = ProvisioningEngine.create_provisioning_engine(
    specification="config/specs/spec-ci-runner-pool.yaml"
)

# The engine has parsed the spec and instantiated:
#   1 CloudVenue   (aws-staging)
#   1 CloudGroup   (ci-runner-pool)
#   1 CloudResourceProfile (runner-medium)
#   3 CloudDevice  (runner-1, runner-2, runner-3)

The provisioning step, run before the orchestration, ensures three EC2 instances exist in the aws-staging venue, in the ci-runner-pool group, matching the runner-medium profile. If they already exist with matching identifiers and tags, the step is a no-op. If they exist with the wrong shape, the step replaces them. If they don't exist, the step creates them.

The orchestration that runs after provisioning sees a Landscape (chapter 5) populated with three runners. The orchestration does not know — and does not need to know — whether they were created fresh or pre-existed.

Idempotence is the design contract

The single most important property of a Specification is that applying it is idempotent. Apply the same Specification twice, and the second apply is a no-op.

The reason this matters is operational. In a healthy CI system, many things go wrong: a network blip mid-provision, a partial rollout, a cancelled job that left half a runner pool created. The recovery story for a non-idempotent provisioner is bespoke debugging every time. The recovery story for an idempotent provisioner is re-apply.

Idempotence is achievable because the Specification is declarative. The spec says "three EC2 instances should exist with these properties." The provisioner inspects the current state, computes the diff, and converges. Imperative provisioning code does not admit this analysis.

A consequence of declarative-and-idempotent: a Specification can be the source of truth for an environment's existence. Re-running the provisioner on a schedule keeps the environment in the desired state, repairing drift. This is the same operating model Terraform, Pulumi, and other infrastructure-as-code systems use; the codecraftai-provisioning package brings it inside the portfolio so the same toolchain that runs the orchestration can also run the provisioner.

Composition with the rest of the categories

Specifications sit at the top of the configuration stack and feed into the others:

   Specification
        │
        │ ProvisioningEngine.create_provisioning_engine
        │ (applied)
        ▼
   ┌────────────────────────────┐
   │  Live cloud resources      │
   │  (3 EC2 instances tagged   │
   │   ci-runner-pool / runner) │
   └──────────────┬─────────────┘
                  │
                  │ landscape generation
                  │ (or hand-authored landscape that
                  │  references the provisioned resources)
                  ▼
   ┌────────────────────────────┐
   │  Landscape                 │
   │  (apod members =           │
   │   runner-1, runner-2,      │
   │   runner-3)                │
   └──────────────┬─────────────┘
                  │
                  │ Topology references
                  │ landscape members
                  ▼
   ┌────────────────────────────┐
   │  Topology                  │
   │  (runner-cluster:          │
   │    members = runner-1,     │
   │      runner-2, runner-3)   │
   └────────────────────────────┘

The apod section in the Specification mirrors the apod section in the Landscape (chapter 5). This is intentional: the provisioner's output is the landscape's input. The shared apod shape is the seam between the two stages of the pipeline.

The credentials referenced by the venue (ci-runner-aws) and by the profile travel through the resolution layer (chapters 9–12) at provisioning time and again at orchestration time. The same credential set is used by both — provisioning to create the resources, orchestration to access them.

The runtime profile (chapter 7) parameterises how the provisioner runs (timeouts, retry, log levels for the provisioning step itself) without touching the Specification. Same provisioner, same spec, different runtime: useful for distinguishing production-grade provisioning from local-dev rapid iteration.

What Specifications is not

Three things to keep out of a Specification, every one of which a team will be tempted to put in:

Not the orchestration

A Specification is consumed before the orchestration runs. It describes the static shape of the environment. It does not describe what to do with the environment. Things like "run smoke tests after provisioning" or "deploy the application image" are orchestration concerns; they belong in taskplus workflows (chapter 20), not in the spec.

The boundary is sharp: the Specification's output is resources exist. The orchestration's input is resources exist. The two do not overlap.

Not a build script

Building a container image, compiling code, packaging artifacts — these are not provisioning. Provisioning creates the substrate; build steps run on top. When a Specification grows a build_image section, the section is misplaced; either the image is built once out-of-band and referenced by the spec's image: ami-0abcd1234 field, or the build is its own orchestration step that produces an artifact the spec consumes.

Not a runtime config

The Specification carries structural data — what hardware shape, what region, what security group. It does not carry behavioural data — log levels, retry counts, parallelism. When a Specification sprouts a log_level field, the field belongs in the runtime configuration (chapter 7) for the orchestration that runs against the provisioned environment.

The test of "does this belong in the spec?" is: would the answer to this question change if the orchestration ran twice against the same environment? If the answer changes, it's runtime configuration. If the answer doesn't change, it's structural and belongs in the spec.

Migration: from inlined recipes to Specifications

A pipeline that has accumulated forty lines of aws ec2 ... migrates to a Specification in three steps:

Lift the recipe. Take the imperative provisioning shell and rewrite it as a Specification YAML. Each aws ec2 run-instances call becomes a CloudDevice referencing a CloudResourceProfile. Hostnames and IPs that were hardcoded become outputs the provisioner returns.
Replace the inline shell with a provisioner invocation. The pipeline's shell call becomes:
```
codecraftai-provisioning apply --spec config/specs/spec-ci-runner-pool.yaml
```
The migration is mechanical; the spec carries the same intent the shell carried.
Generate the landscape from the provisioner output. The provisioner emits the apod section of a landscape file as part of its run. The orchestration consumes the landscape, not the provisioner output directly.

After migration, the pipeline's provisioning shell is gone. The forty lines have moved to the Specification, which is now versioned, diffable, idempotent, and shareable across pipelines.

Where Specifications doesn't fit

Honest scoping notes:

Bare-metal labs. The portfolio's provisioning model assumes cloud APIs. A lab full of physical hardware needs a different provisioning approach (PXE, Ansible against a static inventory, or the kind of orchestrated lab-power package codecraftai-interop ships). Treat the static inventory as the Landscape (chapter 5) and skip the Specification.
SaaS-only environments. If the environment is "an account in a SaaS product," there is no provisioner — the provisioning is manual, one-time, and out of band. The spec doesn't apply; the orchestration consumes a Landscape that names the SaaS endpoints.
Local-dev environments. A developer running the pipeline against docker-compose up is not provisioning cloud resources. The Specification is empty (or omitted); the Landscape names the local containers. The orchestration runs unchanged.

The discipline is robust to these cases: when there is no Specification, the orchestration sees the Landscape directly. The spec layer is additive — it earns its keep when there's cloud infrastructure to provision.

Summary

A Specification is a declarative manifest of the resources that must exist before an orchestration runs.
It lives in codecraftai-provisioning, separate from the five other configuration categories that live in codecraftai-configuration, because it is consumed before the orchestration rather than by the orchestration.
The format has four layered abstractions: CloudVenue (where), CloudGroup (how organised), CloudResourceProfile (what kind), CloudDevice (which specific instances).
Applying a Specification is idempotent: apply twice, no-op the second time. This is the design contract that makes Specs recoverable from partial failure.
The Specification's output (live cloud resources) is the Landscape's input. The shared apod shape is the seam.
Specifications does not contain orchestration steps, build recipes, or runtime configuration. Each has its own category.
Migration from inlined CI shell to Specification is mechanical: lift the recipe, swap the inline call for codecraftai-provisioning apply --spec ..., generate the landscape from the provisioner.
Where there's no cloud to provision (bare-metal, SaaS-only, local-dev), the discipline degrades gracefully — the Specification is empty or omitted, and the orchestration runs against a static Landscape directly.

The next chapter introduces the Landscape — the inventory of what's in scope for a pipeline run, whether produced by a Specification or hand-authored.