Chapter 8 — Data Profiles

A connection string in environment-variable form is a hostname, a port, a username, a password, a database name, and three optional flags concatenated into a string nobody can parse reliably. A data profile is the same six items as named, typed fields.

This chapter introduces the fifth and final category in Part II: Data Profiles — the configuration that names where data lives, how to reach it, and which credential to use against it. After this chapter Part II is complete; Part III then digs into Credentials in the dedicated detail they deserve.

The connection-string disease

Every long-lived automation pipeline has, at some point, contained a line like this:

export DATABASE_URL="postgresql://app_user:Tr0ub4dor3@db-staging-east-1.internal:5432/app?sslmode=require&pool_size=20"

This is the connection string as configuration. It is also the connection string as bug-magnet. Six failure modes follow predictably:

Credential leak. The password is embedded in the variable. It will appear in env dumps, in CI logs, in error tracebacks, in the developer's shell history. Masking helps but never completely.
Untyped concatenation. Hostname, port, database, query parameters all concatenated. No schema; one missing slash and the whole string parses wrong.
Single source of truth, scattered. Three tests, two scripts, and one CI step each parse the URL with their own regex. When the URL changes shape — Snowflake adds a warehouse segment, Mongo Atlas changes the srv form — every regex has to be updated.
Mixed transport semantics. TCP databases, Unix-socket databases, document stores, and SaaS endpoints all share the "URL string" format with subtly different conventions. Code that worked against TCP Postgres breaks against socket-only Postgres.
No environment portability. The local-dev DATABASE_URL is postgresql://localhost/dev; the staging DATABASE_URL is the string above; the prod DATABASE_URL is something else. Three different strings, three different failure modes when the wrong one leaks into the wrong run.
Credentials and connection details fused. When the password rotates, you change the URL. When the host moves, you also change the URL. Two unrelated lifecycles share one variable; each rotation requires touching the other.

The discipline of this chapter replaces the connection string with a Data Profile — a typed object naming each component as a separate field, with the credential reference stored separately from the credential material.

What a Data Profile is

A Data Profile is a named, typed description of a data source. It has three universal fields and a category-specific extension:

- identifier: staging-db-readwrite
  category: basic-tcp
  host: db-staging-east-1.internal
  port: 5432
  dbtype: postgres
  dbname: app
  credential: staging-db-rw

The three universals come from BaseDataProfile:

Field	Purpose
`identifier`	The name the orchestration uses to refer to this profile
`category`	The category discriminator that selects the concrete profile class
`credential`	A reference to a credential identifier (chapters 9–12) — never the credential value

Everything else depends on the category. Five categories ship in codecraftai-dataprofiles; each is implemented as a concrete class inheriting from BaseDataProfile with its own additional fields and its own validate classmethod.

The five shipped categories

`basic-tcp` — TCP-connected databases

For databases reached over TCP with host and port, regardless of engine. The shipped class is DatabaseBasicTcpProfile.

- identifier: staging-db-readwrite
  category: basic-tcp
  host: db-staging-east-1.internal
  port: 5432
  dbtype: postgres
  dbname: app
  credential: staging-db-rw

Required fields: credential, dbtype, host, port. Optional fields: dbname. The dbtype field carries the engine type (postgres, mysql, mariadb, redis, etc.) so consuming code can select the right driver.

`basic-socket` — Unix-socket databases

For databases reached over a Unix domain socket — same engine catalogue as basic-tcp but with a filesystem path instead of host and port. The shipped class is DatabaseBasicSocketProfile.

- identifier: local-postgres
  category: basic-socket
  socket: /var/run/postgresql/.s.PGSQL.5432
  dbtype: postgres
  dbname: app_dev
  credential: local-postgres-dev

Required fields: credential, dbtype, dbname, socket.

The split between basic-tcp and basic-socket is deliberate. A single "database profile" category that accepted either a host/port or a socket would force consuming code to branch on which fields were present. Two categories make the dispatch a single category: ... lookup.

`couchdb` — CouchDB

For CouchDB instances. The shipped class is CouchDbProfile.

- identifier: staging-couchdb
  category: couchdb
  host: couchdb.staging.internal
  port: 5984
  scheme: https
  dbname: configurations
  credential: staging-couchdb-rw

Required fields: credential, dbname, host, port. Optional: scheme (defaults to http). CouchDB has its own profile category because its connection-string format is non-standard (the database name lives in the path, the auth lives in the userinfo) and because the codecraftai-configuration loader supports CouchDB as a configuration source — the same profile shape doubles for "database we read from" and "config store we read configuration from."

`mongodb-atlas` — MongoDB Atlas

For MongoDB Atlas deployments. The shipped class is MongoDBAtlasProfile.

- identifier: staging-atlas
  category: mongodb-atlas
  connection: "mongodb+srv://<username>:<password>@cluster0.abcde.mongodb.net/?retryWrites=true"
  credential: staging-atlas-rw

Required fields: credential, connection. The connection field is the Atlas connection template — the literal string MongoDB Atlas gives you in its console — with <username> and <password> placeholders. At runtime the credential is materialised and the template is expanded by MongoDBAtlasProfile.connection_string(credobj).

This category exists separately from basic-tcp because Atlas's mongodb+srv:// scheme expands to a list of hosts via DNS at runtime — there is no fixed host/port to put in a profile. The template-with-placeholders form is the lowest-fidelity capture of what Atlas actually requires.

`snowflake` — Snowflake

For Snowflake databases. The shipped class is SnowflakeProfile.

- identifier: staging-snowflake
  category: snowflake
  account: my-org-staging
  warehouse: COMPUTE_WH_STAGING
  database: APP_STAGING
  schema: PUBLIC
  parameters:
    role: ANALYST
    timezone: UTC
  credential: staging-snowflake-rw

Required fields: account, warehouse, database, schema, credential. Optional: parameters (a dictionary of additional connection options that pass through to the Snowflake connector).

Snowflake gets its own category because the connection model is genuinely different from TCP databases — account, warehouse, database, schema are independent dimensions you select at connection time, and the same Snowflake account hosts many of each.

A complete worked example

Building on the staging-east Landscape from chapter 5 (which had a staging-db service and a staging-api service), here is the matching Data Profile file:

# dataprofiles-staging-east.yaml
dataprofiles:

    # Primary application database — the one staging-api reads/writes
    - identifier: staging-db-app-rw
      category: basic-tcp
      host: db-staging-east-1.internal
      port: 5432
      dbtype: postgres
      dbname: app
      credential: staging-db-rw

    # Same database, different credential — read-only for analytics
    - identifier: staging-db-app-readonly
      category: basic-tcp
      host: db-staging-east-1.internal
      port: 5432
      dbtype: postgres
      dbname: app
      credential: staging-db-readonly

    # Redis cache the API uses for sessions
    - identifier: staging-redis
      category: basic-tcp
      host: redis-staging-east-1.internal
      port: 6379
      dbtype: redis
      credential: staging-redis-default

    # Snowflake warehouse for analytical queries
    - identifier: staging-warehouse
      category: snowflake
      account: my-org-staging
      warehouse: COMPUTE_WH_STAGING
      database: APP_STAGING
      schema: PUBLIC
      credential: staging-snowflake-rw

    # CouchDB used for storing test artifacts
    - identifier: staging-test-artifacts
      category: couchdb
      host: couchdb.staging.internal
      port: 5984
      scheme: https
      dbname: test-artifacts
      credential: staging-couchdb-rw

Five profiles, five categories represented. Notice three things:

The same physical database appears twice (staging-db-app-rw and staging-db-app-readonly) with different credentials. The Data Profile is the access intention, not just the location.
Every credential is a reference, never an inline value. The five credential identifiers (staging-db-rw, staging-db-readonly, staging-redis-default, staging-snowflake-rw, staging-couchdb-rw) are looked up by the resolution layer at runtime.
No connection strings appear anywhere. The components are named fields; the connection string — when one is needed — is built at runtime by the profile class's connection_string() method.

The orchestration consumes profiles by name:

from ccai.dataprofiles.wellknown import DataProfileManagerSingleton

profile = DataProfileManagerSingleton().lookup_profile(
    "staging-db-app-rw"
)

# profile is a DatabaseBasicTcpProfile instance with
# host/port/dbname/dbtype/credential typed fields

cred = CredentialManagerSingleton().lookup_credential(profile.credential)
conn_str = f"postgresql://{cred.username}:{cred.password}@{profile.host}:{profile.port}/{profile.dbname}"

Or, for the categories that ship a connection_string() helper:

profile = DataProfileManagerSingleton().lookup_profile("staging-couchdb")
cred = CredentialManagerSingleton().lookup_credential(profile.credential)
conn_str = profile.connection_string(cred)

The connection string is built at the point of use, with the materialised credential, and discarded immediately. It never becomes a process-environment value.

The `category` field as discriminator

Every Data Profile has a category field that selects which concrete class to instantiate when the file is loaded. The DataProfileManager performs this dispatch:

if category == DatabaseBasicSocketProfile.category:    # "basic-socket"
    profobj = DatabaseBasicSocketProfile(**profile)
elif category == DatabaseBasicTcpProfile.category:     # "basic-tcp"
    profobj = DatabaseBasicTcpProfile(**profile)
elif category == CouchDbProfile.category:              # "couchdb"
    profobj = CouchDbProfile(**profile)
elif category == MongoDBAtlasProfile.category:         # "mongodb-atlas"
    profobj = MongoDBAtlasProfile(**profile)
elif category == SnowflakeProfile.category:            # "snowflake"
    profobj = SnowflakeProfile(**profile)
else:
    logger.warning(f"Unknown category '{category}' ...")

This is the standard ccai discriminator pattern: a string field on the data, a class attribute on the implementations, a dispatch that matches them at load time. The pattern shows up in codecraftai-credentials (chapter 9), in codecraftai-provisioning (chapter 4 — category on CloudDevice), and elsewhere. Adding a new Data Profile category — say, redis-sentinel for Redis Sentinel deployments where the single-host model breaks down — means adding a new class and adding a new branch in the dispatch.

The validate classmethod on each concrete class enforces the category-specific required fields at load time. A profile of category basic-tcp missing the port field fails to load with a clear error message naming the missing field; the orchestration never starts with a malformed profile.

Composition with credentials and landscape

The Data Profile sits at a specific composition point: it bridges where data lives (the landscape) and who's allowed to access it (credentials).

       Landscape
       ────────
       host: db-staging-east-1.internal       ◄─── inventory truth
       (the host is in the infrastructure)


       Data Profile                            ◄─── access intention
       ────────────
       identifier: staging-db-app-rw
       host: db-staging-east-1.internal
       port: 5432
       credential: staging-db-rw   ◄─────────  reference


       Credential                              ◄─── access material
       ──────────
       identifier: staging-db-rw
       type: BasicCredential
       username: app_user
       password: <looked up at runtime>

Three deliberate boundaries:

The Data Profile points at hosts named in the Landscape. The profile's host field is the same hostname the Landscape's infrastructure section names (db-staging-east-1.internal). When the host moves, the Landscape's infrastructure entry changes — and the Data Profile's host field changes alongside. The two changes are kept in sync because the same person editing the same environment's configuration touches both.

The Data Profile does not resolve the host through the Landscape automatically. This is a deliberate design choice from codecraftai-dataprofiles: the profile is self-contained data; the Landscape is queried separately. The result is that a Data Profile is a complete description of how to reach a data source, even if the corresponding Landscape isn't loaded.
The Data Profile names a credential identifier. It does not contain the credential. The resolution layer (chapter 10) produces the credential at the moment of use. The credential's lifecycle is independent of the profile's.
The Landscape's role and the Data Profile's role are different. The Landscape says "there is a service called staging-db reachable at db-staging-east-1.internal:5432." The Data Profile says "to access the app database on that host, use the staging-db-rw credential." Both can exist; both are necessary; they describe orthogonal aspects.

A common confusion: why does the Data Profile have a host field if the Landscape already has it? The answer is that the Data Profile is for data sources that may not be in the Landscape at all. A SaaS data store like Snowflake or MongoDB Atlas isn't in the infrastructure section of any Landscape — it's a third-party service the orchestration calls into. The Data Profile is the only place where its access details live.

For internal data stores that are in the Landscape, the duplication is the cost of keeping the Data Profile self-contained. A future evolution could allow Data Profile fields to reference Landscape entries by name (host: ${landscape:staging-db.host}), at the cost of making profiles dependent on a particular Landscape. The current shipped form keeps the abstraction simple at the price of one duplicated hostname.

The `DataProfileManager` and lookup-by-name

At runtime the orchestration interacts with profiles through the DataProfileManagerSingleton:

from ccai.dataprofiles.wellknown import DataProfileManagerSingleton

manager = DataProfileManagerSingleton()
profile = manager.lookup_profile("staging-db-app-rw")

Three behaviours of the manager worth noting:

Lookup by identifier returns the typed object. The orchestration receives a DatabaseBasicTcpProfile (or whichever class matches the category), not a generic dictionary. Type-checked attribute access (profile.host, profile.port) replaces dictionary fetches with the chance of typos.
Missing identifier raises ConfigurationError. The error message includes the list of source URIs the manager loaded from, so the engineer can immediately see where to add the profile. No silent None returns; loading and lookup both fail loudly.
Validation runs at load time. Errors and warnings are collected from each profile's validate classmethod; if any errors exist, the load fails before any profile is instantiated. The manager never holds a partially-valid profile set.

What a Data Profile is not

Three things to keep out, every one of which a team will be tempted to put in:

Not a credential

The most common mistake. A Data Profile names a credential by identifier. It does not embed username, password, token, key, or any secret material. When you find yourself adding a password field to a profile, stop — the password belongs in the credential record, not the profile.

A symptom of getting this wrong is that profile rotation requires editing the profile. Done right, password rotation is invisible to the profile — the credential record changes, the profile is unchanged, the next connection picks up the new password automatically.

Not a Landscape entry

A Data Profile points at data sources; a Landscape lists hosts and services. The two have similar shapes but different purposes. Putting a Data Profile in a Landscape ("infrastructure: services: ... profile fields ...") confuses the inventory with access details. Putting Landscape entries in a Data Profile file ("hosts: ... ports: ...") loses the apod-vs-infrastructure distinction.

The test of "does this belong in a Data Profile?" is: is this about how to reach a data source? If yes, profile. If no, it belongs elsewhere.

Not a runtime configuration

A Data Profile carries connection parameters that are intrinsic to the data source — host, port, account, database name. It does not carry behaviour parameters like connection-pool size, statement timeout, or retry count, even though those are commonly passed to the same connection-builder code. Behaviour parameters go in the runtime configuration (chapter 7); the Data Profile is the what to connect to, not the how to behave when connected.

The test: would two different orchestrations against the same data source need different values of this field? If yes, runtime. If no, profile.

Migration: from connection strings to Data Profiles

A pipeline that has accumulated connection strings as environment variables migrates to Data Profiles in three steps:

Enumerate every connection string. Walk the codebase, the Dockerfiles, the docker-compose files, the CI YAML, and the developer's .env files. List every variable that looks like a URL pointing at a data source. Group them by destination — DATABASE_URL, STAGING_DB_URL, PROD_DB_RO_URL, etc.
Decompose each string into typed fields. For each URL, identify the host, the port, the database, the database type, the credential. Each becomes a field in a Data Profile entry.

Replace the URL parsing in code with profile lookups. Code that did:

url = os.environ["DATABASE_URL"]
conn = sqlalchemy.create_engine(url)

becomes:

profile = DataProfileManagerSingleton().lookup_profile("staging-db-app-rw")
cred = CredentialManagerSingleton().lookup_credential(profile.credential)
conn = sqlalchemy.create_engine(
    f"postgresql://{cred.username}:{cred.password}@{profile.host}:{profile.port}/{profile.dbname}"
)

The application code now reads a typed profile and a typed credential, builds the connection string at the point of use, and never holds the password in a long-lived variable.

After migration, the DATABASE_URL and its kin are gone. The plumbing tax from chapter 7 has been paid once for AM_DATAPROFILE_NAMES; every new data source after that is a one-file change to the profile YAML.

Where Data Profiles don't fit

Honest scoping notes:

Truly ephemeral data sources. A pipeline that creates a one-time database, populates it, queries it, drops it — all in a single step — does not need a Data Profile. The connection details are local to the step; promoting them to a profile adds ceremony without payoff.
Highly dynamic discovery. Pipelines that discover data sources at runtime through service discovery (Consul, etcd, K8s service mesh) typically generate the equivalent of a profile in memory. The shipped DataProfileManager doesn't model dynamic registration; the pattern is to translate the discovery output into a profile dictionary at orchestration start and pass it to load_datasource_profiles.
In-memory or filesystem-only data. A pipeline whose "data" is a JSON file in the repository doesn't need a Data Profile. The file is in the codebase; reading it is direct.

The discipline scales from "two databases" to "dozens of services across multiple categories"; below that scale the discipline is optional.

Summary — and end of Part II

A Data Profile is a typed, named description of a data source: where it lives, how to reach it, which credential to use.
The five shipped categories are basic-tcp, basic-socket, couchdb, mongodb-atlas, and snowflake. Each is a concrete class inheriting from BaseDataProfile with category-specific fields and category-specific validation.
Three universal fields — identifier, category, credential — are required on every profile. The credential field is a reference; the credential material lives in the credential category and is resolved at runtime.
The DataProfileManager loads profiles, validates them at load time, and exposes lookup-by-identifier through the DataProfileManagerSingleton.
Profiles compose with the Landscape (the profile's host parallels the Landscape's infrastructure host but is self-contained) and with credentials (the profile names a credential; the resolution layer materialises it).
A Data Profile is not a credential (no embedded secrets), not a Landscape entry (no inventory listing), and not a runtime configuration (no behaviour parameters).
Migration from connection-string env vars is mechanical: enumerate, decompose into fields, replace URL parsing with profile lookups.

This is the last chapter of Part II. The volume has now introduced five of the six categories — Specifications, Landscapes, Topologies, Runtimes, Data Profiles — with file formats, code anchors, and worked examples grounded in real portfolio code.

The sixth category — Credentials — earns four chapters of its own, which is the next part of the volume. Part III digs into credential vocabularies, the resolution discipline (the most substantial design exercise in the volume since the portfolio ships a vocabulary layer but no resolution backends), credentials in flight (rotation, encryption at rest, audit trails), and credentials across environments (the same vocabulary, different backends, per the Part IV profile-and-environment story).