Data Contracts: The Missing Agreement Between Engineers and Business

Personal content — not employer-related. This article is published in a personal capacity and shares thoughts and experiences based on public industry patterns — not a description of employer or client systems, and not professional instructions. It does not reflect the views, systems, projects, or policies of any current or past employer, client, or financial institution. See the full site disclaimer.

Engineers need stable schemas. Business needs trustworthy definitions. Without something written down, both sides guess - and every dashboard, migration, and AI pilot pays for it.

Context: By late 2024 the “data mesh” wave had cooled into something more useful in the programs I saw: domain ownership and explicit interfaces. Data contracts became the artifact I kept pushing for - smaller than a mesh program, more real than a Slack message saying “don’t break my table”.

What I mean by a contract

A published agreement for a dataset or stream:

Schema - fields, types, meaning
SLAs - freshness, availability
Quality rules you can measure
Ownership - who decides changes
Change policy - how breaking changes are handled

It is not a 40-page legal doc. It is not a wiki page nobody reads. It is closer to an API spec for data.

Producer, contract, and consumer loop for data contracts

Piece	What it answers
Schema	What is in the table, and what the fields mean
Freshness SLA	How late can it be before consumers should not trust it
Quality rules	Which checks fail the pipeline
Owners	Who decides a breaking change
Change policy	How much notice, and where it is announced

Why it mattered more in 2024

Without contracts, every consumer reimplements the same joins, builds a private “fix” layer that drifts, and blames upstream only after an incident.

With GenAI and early copilots reading tables and documents, the cost went up. Automated consumers do not have the human intuition for “this looks wrong”.

A minimal example we used as a starting point

name: curated.orders
version: 2.1.0
description: One row per completed order; excludes cancelled before payment
owners:
  business: order-management
  technical: data-platform-core
schedule: hourly at :05
freshness:
  max_lag_minutes: 75
  measured_on: _loaded_at
schema:
  - name: order_id
    type: STRING
    constraints: [unique, not_null]
  - name: customer_id
    type: STRING
    constraints: [not_null, fk:curated.customers]
  - name: revenue_eur
    type: NUMERIC
    semantics: net of VAT; matches finance definition v4
quality_checks:
  - name: revenue_reconciliation
    rule: sum(revenue_eur) within 0.5% of finance.daily_orders
change_policy:
  breaking: 14 business days notice
  channel: "#data-contracts"
consumers:
  - bi_revenue_dashboard
  - fraud_scoring_v3

Store it in git. Validate in CI. Surface violations where owners already look.

A tiny Python check I used in personal labs to fail a pipeline when the contract freshness lag blows:

# scripts/check_contract_freshness.py  (illustrative PoC)
from google.cloud import bigquery

DATASET, MAX_LAG_MIN = "curated.orders", 75

client = bigquery.Client()
lag = list(client.query(f"""
  SELECT TIMESTAMP_DIFF(CURRENT_TIMESTAMP(), MAX(_loaded_at), MINUTE) AS lag
  FROM `{DATASET}`
""").result())[0]["lag"]

if lag is None or lag > MAX_LAG_MIN:
    raise SystemExit(f"freshness contract failed: lag={lag} min (max {MAX_LAG_MIN})")
print(f"ok: lag={lag} min")

Enforcement without freezing the org

Producers: CI on schema versus contract, quality tests on each run, and semantic versioning (patch = compatible fixes, minor = backward-compatible additions, major = breaking changes).

Consumers: pin to a version, subscribe to change notes, test against fixtures.

Platform: a registry (even a folder in git), violation dashboard, escalate when SLAs breach twice in a week.

We usually started with five critical datasets, not the whole lake.

Objections I heard

“We move too fast for contracts.” You already move fast - you pay in firefighting. Contracts reduce rework; they do not forbid change.

“Business will not read YAML.” They understand “revenue matches finance” and “ready by 8am”. Write the meaning in plain language; keep the machine checks in code.

“We already have documentation.” Docs describe intent. Contracts define obligations with tests.

During migrations

During legacy-to-cloud migrations, contracts forced the parity conversation early: which fields were really required, which quirks to keep or fix, and who would sign off when numbers shifted. Consumers knew what “done” meant.

Closing

Contracts align engineers and business on what the data promises. They sit between raw pipelines and reliable BI, ML, or copilots. Start small, enforce automatically, expand domain by domain.