Volume 0 — Foundations of Automation Engineering with Python — contents
Volume 0 — Foundations of Automation Engineering with Python
  1. Chapter 1 — The CodeCraftX Philosophy
  2. Chapter 2 — Coding Standards as a Tool, Not a Style Guide
  3. Chapter 3 — The Error Vocabulary
  4. Chapter 4 — Interfaces and Contracts
  5. Chapter 5 — Waiting with Rich Diagnostics
  6. Chapter 6 — Application Identity and Presence
  7. Chapter 7 — Extension and Factory Machinery
  8. Chapter 8 — Runtime Variables and Well-Known Paths
  9. Chapter 9 — Configuration Loading
  10. Chapter 10 — Credentials, Data Profiles, Storage Scaffolding
Other volumes
References / The Code Craft AI books / Volume 0 — Foundations of Automation Engineering with Python

Chapter 9 — Configuration Loading

Where Volume 3 talked about what configuration to put in files, this chapter talks about how the loader actually works — the multi-source, multi-format, optionally-encrypted file machinery every higher tier relies on.

The previous chapter introduced the runtime variable layer. This chapter introduces codecraftai-configuration, the foundation-tier package whose job is to load configuration files — from disk, from HTTP, from CouchDB, from MongoDB — parse them, optionally decrypt them, validate them, and produce typed MergeMap objects the rest of the portfolio consumes.

Volume 3 used this package's machinery extensively without explaining how the machinery works. This chapter is the missing piece. It complements Volume 3 chapters 3 (the six configuration categories) and 4–8 (the per-category file formats) by walking the loader internals — the ConfigurationLoader, the ConfigurationSourceBase hierarchy, the cryptography primitives, and the resolution pipeline that drives all of it.

The loader's job

A consumer wants the data inside a configuration file by name. They say "give me the landscape named staging-east," and the loader handles:

  1. Source resolution — which place(s) to look (a local directory, an HTTP endpoint, a CouchDB database)?
  2. Filename derivation — what file in the source corresponds to staging-east? (Tries staging-east.yaml, staging-east.yml, staging-east.json in order.)
  3. Format detection — based on the extension that resolved.
  4. Reading — fetching the bytes from wherever they are.
  5. Decryption — when the bytes form an encrypted document.
  6. Parsing — turning the bytes into a Python dict.
  7. Composition — overlaying onto a MergeMap if the category supports composition.

Seven steps. The consumer sees one call:

config_uri, config_data = loader.load_configuration_by_name("staging-east")

The chapter walks the seven steps from bottom to top.

ConfigurationLoader — the public surface

The ConfigurationLoader class is the consumer-facing entry point. Constructed with a list of source URIs:

loader = ConfigurationLoader(
    sources=[
        "dir://~/.ccai/configurations/landscapes",
        "http://config.staging-east.internal/landscapes",
        "couchdb://config-couch.internal/landscapes",
    ],
    credentials={...},
    verify_certificate=True,
)

Three parameters:

  • sources — list of source URIs. Each URI's scheme selects the source class. Sources are tried in order.
  • credentials — credential mapping for sources that need authentication.
  • verify_certificate — TLS certificate verification flag for HTTPS sources.

The class exposes two main methods:

def load_configuration_by_name(self, config_name: str,
                                key: Optional[str] = None,
                                keyphrase: Optional[str] = None) -> Tuple[str, dict]:
    """Returns (source_uri, parsed_config_dict)."""

def load_configuration_from_file(self, config_file: str,
                                  key: Optional[str] = None,
                                  keyphrase: Optional[str] = None) -> dict:
    """Returns parsed_config_dict for a single explicit file path."""

The first method walks every source until one returns a configuration; the second loads a specific file directly. Both take optional key or keyphrase for encrypted documents.

When neither method finds a configuration, the loader raises ConfigurationError (chapter 3's must-bubble type) listing every source URI it tried. The error is findable; the engineer reading the failure knows where the loader looked without having to instrument anything.

The source hierarchy

Every source extends ConfigurationSourceBase, the abstract base that declares two methods:

class ConfigurationSourceBase:

    scheme: str = None    # subclasses override

    @classmethod
    def parse(cls, uri: str) -> Optional["ConfigurationSourceBase"]:
        ...

    def try_load_configuration(self, config_name: str,
                                credentials: Optional[Dict] = None
                                ) -> Tuple[Optional[ConfigurationFormat], Optional[dict]]:
        ...

Two methods, one class attribute:

  • scheme — the URI scheme this source handles (dir, http, couchdb, mongodb).
  • parse(uri) — given a URI, construct an instance of this source class if the URI matches the scheme; otherwise return None.
  • try_load_configuration(name, credentials) — try to load the named configuration; return (format, dict) on success or (None, None) if not found.

The shipped source classes:

DirectorySource — filesystem

URIs of the form dir://path/to/directory. Loads files from a local filesystem. Implementation detail worth naming: when the URI doesn't start with dir://, the source treats the entire URI as a directory path — so ~/.ccai/configurations works without the prefix.

The directory is expanded through os.path.expandvars and os.path.expanduser, then queried with the standard extension fallback (.yaml, .yml, .json in order).

HttpSource — HTTP / HTTPS endpoints

URIs of the form http://host/path or https://host/path. Loads files via HTTP GET requests. The same extension fallback: the source tries <base_url>/<name>.yaml, then .yml, then .json, and returns the first that responds with HTTP 200.

The verify_certificate flag from the loader controls TLS verification; the credentials mapping is consulted for HTTP basic-auth.

CouchDbSource — CouchDB documents

URIs of the form couchdb://host:port/database. Loads configurations as CouchDB documents — each document's _id is the configuration name. The pycouchdb (or equivalent) library is required.

The chapter-9 (Volume 3) cross-reference: a CouchDbProfile data profile points at the same kind of CouchDB. The two packages share the same protocol — codecraftai-dataprofiles's profile is the connection; codecraftai-configuration's source is the configuration storage using the same connection mechanics.

MongoDbSource — MongoDB documents

URIs of the form mongodb://host:port/database. Loads configurations as MongoDB documents. Same pattern as CouchDB; the pymongo library is required.

The format detection

Per the configuration loader's logic, file format is detected by extension:

_FORMAT_BY_EXTENSION = {
    ".yaml": ConfigurationFormat.YAML,
    ".yml": ConfigurationFormat.YAML,
    ".json": ConfigurationFormat.JSON,
}

ConfigurationFormat is the small enum from configurationformat.py:

class ConfigurationFormat(str, Enum):
    JSON = "json"
    YAML = "yaml"

A string-enum dual-citizenship class. The same pattern as chapter 6's ContextPaths — both a name and a value, greppable and rename-safe.

The format dictates the parser:

  • YAML → yaml.safe_load(content)
  • JSON → json.loads(content)

The choice of yaml.safe_load (not yaml.load) is the chapter-1 errors-never-silent discipline applied to YAML parsing — yaml.load accepts arbitrary Python object constructors, which is a security risk; safe_load rejects them.

Encrypted configurations

When the loader receives a parsed dict and detects an encrypted_content key, it activates the cryptography path.

The encrypted document format

Encrypted configurations have a known shape:

version: "1.0"
format: yaml          # the inner content's format
encrypted_content: "BASE64-CIPHERTEXT-HERE"

The codecraftai-configuration/cryptography.py module produces and consumes documents in this shape. Volume 3 chapter 11 ("Credentials in flight") detailed the cryptography primitives for credentials; the same primitives apply to any configuration file.

generate_fernet_key(keyphrase) — deterministic key derivation

def generate_fernet_key(keyphrase: str) -> str:
    rgen = random.Random(keyphrase)
    key_bytes = rgen.randbytes(32)
    key = base64.urlsafe_b64encode(key_bytes)
    return key

Deterministic — the same keyphrase always produces the same key. The trade-off (Volume 3 chapter 11 discussed in detail): shared-keyphrase distribution is easier; weak keyphrases produce weak keys; PBKDF2 / scrypt / Argon2 derivation is the project's responsibility when stronger derivation is needed.

The Fernet class from cryptography.fernet accepts any 32-byte URL-safe-base64 key, so a project that wants stronger derivation passes that key directly to encrypt_content / decrypt_content rather than going through generate_fernet_key.

encrypt_content / decrypt_content — symmetric round-trip

def encrypt_content(key: str, plain_content: str) -> str:
    cryptor = Fernet(key)
    plain_bytes = plain_content.encode("utf-8")
    cypherbytes = cryptor.encrypt(plain_bytes)
    encrypted_content = base64.b64encode(cypherbytes).decode("utf-8")
    return encrypted_content


def decrypt_content(key: str, encrypted_content: str) -> str:
    cryptor = Fernet(key)
    cipherbytes = base64.b64decode(encrypted_content.encode("utf-8"))
    plainbytes = cryptor.decrypt(cipherbytes)
    plaincontent = plainbytes.decode("utf-8")
    return plaincontent

Symmetric, base64-wrapped, UTF-8 strings on both sides. The encryption is cryptography.fernet.Fernet's standard AES-128-CBC

  • HMAC-SHA256 + base64 — a well-understood primitive, not a roll-your-own.

create_encrypted_configuration(key, plain_configuration, format)

The wrapping helper:

def create_encrypted_configuration(key: str, plain_configuration: str,
                                    format: str = ConfigurationFormat.JSON):
    encrypted_configuration = encrypt_content(key, plain_configuration)

    docinfo = {
        "version": "1.0",
        "format": format,
        "encrypted_content": encrypted_configuration,
    }
    return docinfo

Takes a plaintext string, encrypts it, wraps it in the known-shape document. The result can be written to disk or to any of the configuration sources; the loader recognises the shape and decrypts on read.

The composition layer

Loaded configurations don't exist in isolation — they compose. A single category has many files (per-environment landscapes, per-environment runtimes, etc.); each file contributes to a unified resolved configuration.

The composition primitive is MergeMap from codecraftai-xmodules (chapter 0's foundation tier). A MergeMap is a MutableMapping that holds an ordered list of component maps and composes them with overlay semantics — later maps shadow earlier maps key-by-key, but lower-priority entries remain inspectable.

In codecraftai-configuration, configurationmaps.py defines the five well-known composition targets:

class CONFIGURATION_MAPS:

    CREDENTIAL_CONFIGURATION_MAP = MergeMap()
    DATAPROFILE_CONFIGURATION_MAP = MergeMap()
    LANDSCAPE_CONFIGURATION_MAP = MergeMap()
    RUNTIME_CONFIGURATION_MAP = MergeMap(ConfigurationDefaults.DEFAULT_CONFIGURATION)
    TOPOLOGY_CONFIGURATION_MAP = MergeMap()

Five MergeMap instances. Each is the resolved view of all loaded configurations of that category, composed in load order.

resolve_*_configuration — per-category resolution

For each of the five categories, configurationmaps.py ships a resolution function:

def resolve_landscape_configuration(this_app, keyphrase=None, credentials=None):

    landscape_files = this_app.get_variable_value(ConfigVariablesDesc.AM_CONFIG_LANDSCAPE_FILES)
    landscape_names = this_app.get_variable_value(ConfigVariablesDesc.AM_CONFIG_LANDSCAPE_NAMES)
    landscape_sources = this_app.get_variable_value(ConfigVariablesDesc.AM_CONFIG_LANDSCAPE_SOURCES)

    # ... default fallbacks if names/sources unspecified ...

    loader = ConfigurationLoader(sources=landscape_sources, credentials=credentials)

    for config_name in landscape_names:
        config_uri, config_info = loader.load_configuration_by_name(
            config_name, keyphrase=keyphrase
        )
        CONFIGURATION_MAPS.LANDSCAPE_CONFIGURATION_MAP.maps.insert(0, config_info)

Three things visible:

  • Variables drive resolution. The function reads AM_CONFIG_LANDSCAPE_NAMES, _FILES, _SOURCES through the chapter-6 descriptor pipeline. These are the same env-var selectors Volume 3 chapter 7 named — AM_LANDSCAPE_NAMES is shorthand for AM_CONFIG_LANDSCAPE_NAMES.
  • Multiple files compose. Each loaded configuration is prepended (maps.insert(0, ...)) to the MergeMap. The first configuration loaded ends up last in the chain; later loads have higher priority. This is intentional — it means a chain like [base, staging, staging-east-overlay] resolves with the overlay on top of staging on top of base.
  • The resolved map is shared. Every consumer that needs the active landscape reads CONFIGURATION_MAPS.LANDSCAPE_CONFIGURATION_MAP; the same instance is updated by every loaded configuration.

resolve_configuration_maps — the umbrella entry

The umbrella function calls each of the five resolution functions in turn, conditional on the per-category USE_* flags:

def resolve_configuration_maps(this_app=None, use_credentials=None,
                                use_dataprofile=None, use_landscape=None,
                                use_runtime=None, use_topology=None,
                                keyphrase=None, credentials=None):
    if this_app is None:
        this_app = ThisApplicationSingleton()

    # ... resolve use_* flags from variables if not explicitly passed ...

    if use_credentials:
        resolve_credentials_configuration(this_app, keyphrase=keyphrase, credentials=credentials)
    if use_dataprofile:
        resolve_dataprofiles_configuration(this_app, keyphrase=keyphrase, credentials=credentials)
    if use_landscape:
        resolve_landscape_configuration(this_app, keyphrase=keyphrase, credentials=credentials)
    if use_runtime:
        resolve_runtime_configuration(this_app, keyphrase=keyphrase, credentials=credentials)
    if use_topology:
        resolve_topology_configuration(this_app, keyphrase=keyphrase, credentials=credentials)

A consumer (typically the application's startup code) calls this once after establish_app_persona and activate_profile_*. By the time the call returns, all five CONFIGURATION_MAPS are populated, and the rest of the application reads from them.

The variable model the loader uses

configurationvariables.py defines ConfigVariablesDesc, extending the chapter-8 RuntimeVariablesDesc:

class ConfigVariablesDesc(RuntimeVariablesDesc):

    AM_CONFIG_CREDENTIAL_NAMES = AppVarDesc(default=None,
        context_path=ContextPaths.CONFIG_CREDENTIAL_NAMES,
        converter=split_and_normalize_name_list)
    AM_CONFIG_CREDENTIAL_FILES = AppVarDesc(default=None,
        context_path=ContextPaths.CONFIG_CREDENTIAL_FILES,
        converter=split_and_normalize_path_list)
    AM_CONFIG_CREDENTIAL_SOURCES = AppVarDesc(default=None,
        context_path=ContextPaths.CONFIG_CREDENTIAL_SOURCES)

    # ... per-category descriptors for landscape, runtime, topology, dataprofile

    AM_CONFIG_USE_CREDENTIALS = AppVarDesc(default=False, source=VariableSource.NONE)
    AM_CONFIG_REQUIRE_CREDENTIALS = AppVarDesc(default=False, converter=parse_bool)
    # ... USE_* / REQUIRE_* per category

Per category, six descriptors:

  • AM_CONFIG_<CAT>_NAMES — the list of config names to load.
  • AM_CONFIG_<CAT>_FILES — explicit file paths to load (when not loaded by name).
  • AM_CONFIG_<CAT>_SOURCES — the source URI list.
  • AM_CONFIG_<CAT>_URIS — the resolved-URI output (read-only; populated post-load).
  • AM_CONFIG_USE_<CAT> — boolean controlling whether to load this category at all.
  • AM_CONFIG_REQUIRE_<CAT> — boolean controlling whether failure to load is fatal.

Six descriptors per category × five categories = thirty descriptors, plus their typed accessors in ConfigVariables. This is the chapter-6 descriptor model at scale.

The input normalizers

The descriptors above use converter functions to turn raw string values into typed lists: normalize.py ships three:

  • split_and_normalize_name_list(value, sep=";") — splits on ;, strips each entry.
  • split_and_normalize_path_list(value, sep=";") — splits, strips file:// prefix, expands ~ and $VARS, makes absolute.
  • split_and_normalize_source_list(value, sep=";") — splits, expands non-URI paths. When an entry starts with http://, https://, dir://, mongodb://, or couchdb://, it's left alone; otherwise it's path-expanded.

The default separator is ; rather than : because filesystem paths on Windows use : as a drive separator.

The settings layer

configurationsettings.py provides the one-time initialisation function that establishes the application persona and the default configuration:

def establish_configuration_settings(*, name: str = None,
                                       logger_name: str = None,
                                       log_basename: str = None,
                                       presence_dir: str = None,
                                       log_directory: str = None,
                                       settings_file: str = None,
                                       extensions: str = None,
                                       service_name: str = None,
                                       default_configuration: dict = None,
                                       **other):

The function is idempotent — guarded by a CONFIG_SETTINGS_ESTABLISHED flag. It calls establish_app_persona (chapter 6) with the provided parameters; if default_configuration is provided, replaces ConfigurationDefaults.DEFAULT_CONFIGURATION.

This is the typical application's first call:

from ccai.configuration.configurationsettings import establish_configuration_settings

establish_configuration_settings(
    name="my-tool",
    logger_name="my-tool",
    log_basename="my-tool",
)

One call. After this, the persona is established (chapter 6), the runtime layer is available (chapter 8), and the configuration loader is ready to be invoked through resolve_configuration_maps.

What the loader is not

Three deliberate non-features.

Not a configuration editor

The loader reads configurations. It does not provide a write-back path for modifying them at runtime. When a configuration needs to change (a new entry added, an entry modified, an entry removed), the change happens at the source (file edit, database update); the loader's next read picks it up.

For runtime overrides, the chapter-6 override_variable_value mechanism applies. An override doesn't modify the loaded configuration; it adds a higher-priority resolution that shadows it.

Not a generic schema validator

The loader reads YAML/JSON into Python dicts. Schema validation — "this dict has these specific fields with these specific types" — is the consumer's responsibility. Per Volume 3 chapter 5 (Landscape files), the loader doesn't validate the landscape's apod or infrastructure shape; the landscape consumer (typically codecraftai-landscaping) does that.

The validate classmethod pattern from Volume 3 (chapter 8 — data profiles, chapter 9 — credentials) is the per-category validation hook. The loader doesn't enforce it; the manager that builds typed instances from the loaded dict does.

Not asynchronous

The loader is synchronous. Loading a configuration blocks the calling thread until the source returns. When a configuration load is slow (large files, slow databases, far-away HTTP servers), the calling code waits.

For most consumers, configuration loading happens once at startup — synchronous loading is the right choice for a one-time cost. Consumers that need asynchronous loading (streaming configuration updates, hot reload) build the async layer themselves on top of the synchronous loader.

Cross-references

The configuration loader is consumed by:

  • Volume 3 (Orchestration) uses the loader extensively. Chapter 9 / 10 / 11 / 12 of Volume 3 explain credentials in depth; chapters 4–8 explain the per-category content; chapter 3 lists the categories. The current chapter is the loader side of all that material.
  • Volume 1 (Testing) — codecraftai-testbase calls resolve_configuration_maps at startup to populate the active configurations before tests run.
  • Volume 5 (Interop) — interop's per-protocol adapters read connection details from data profiles and credentials loaded by this layer.

The chapter-9 (Volume 3) ${VAULT:...} resolver discipline is the layer above this one — the loader produces dicts whose secret values may be ${SCHEME:...} references; the chapter-10 (Volume 3) resolver turns those references into materialised secrets. The loader doesn't know about references; the resolver doesn't know about file formats. The two layers compose cleanly.

Summary

  • codecraftai-configuration is the configuration-loading package every higher tier uses. The loader fetches configuration files from filesystem, HTTP, CouchDB, or MongoDB sources; detects format by extension; optionally decrypts; parses into Python dicts; composes into per-category MergeMaps.
  • The ConfigurationLoader is the public surface. Its two methods (load_configuration_by_name, load_configuration_from_file) take care of source iteration, filename derivation, format detection, decryption, and parsing.
  • Four shipped source classes under sources/: DirectorySource, HttpSource, CouchDbSource, MongoDbSource. Each extends ConfigurationSourceBase with a scheme attribute and parse(uri) / try_load_configuration methods.
  • Format detection is by extension: .yaml / .yml / .json in that fallback order. The format determines the parser (yaml.safe_load for YAML, json.loads for JSON).
  • Encrypted configurations have a known three-key shape (version, format, encrypted_content). The cryptography.py module ships generate_fernet_key, encrypt_content, decrypt_content, and create_encrypted_configuration — Fernet symmetric encryption with deterministic key derivation from a keyphrase.
  • The composition layer uses MergeMap to compose multiple loaded configurations per category. Five CONFIGURATION_MAPS instances (Credentials, DataProfiles, Landscape, Runtime, Topology); five resolve_*_configuration functions; one umbrella resolve_configuration_maps.
  • ConfigVariablesDesc extends the chapter-8 RuntimeVariablesDesc with thirty configuration-related descriptors — six per category — driving the resolution pipeline. The chapter-7 AM_*_NAMES family in Volume 3 is shorthand for AM_CONFIG_*_NAMES.
  • Input normalizers (split_and_normalize_*) convert string env vars into typed lists; the default separator is ; for cross-platform path safety.
  • The loader is not a configuration editor (read-only), not a generic schema validator (the consumer validates), not asynchronous (synchronous loading is the right choice for one-time startup cost).

The next chapter — the last in Volume 0 — is the survey chapter for the remaining foundation-tier packages (codecraftai-credentials, codecraftai-dataprofiles, and codecraftai-storage-database). These three are smaller in substance than the packages this chapter covered; the survey treatment fits.

View this page as Markdown