Why Data Pipelines Need SLOs Like Software Services

Personal content — not employer-related. This article is published in a personal capacity and shares thoughts and experiences based on public industry patterns — not a description of employer or client systems, and not professional instructions. It does not reflect the views, systems, projects, or policies of any current or past employer, client, or financial institution. See the full site disclaimer.

Product APIs have SLOs. Data pipelines often only have “it ran green”. That gap is why dashboards lied, models trained on stale data, and on-call found out from Slack.

Context: By spring 2025 SRE practices were normal for services. In platform work I was close to, curated tables were still treated like APIs without any written guarantee. These are notes on what changed when we started writing SLOs for data products - the same idea as in SLO burn-rate alerts, applied to pipelines.

Tables are products too

Pipeline SLO observability loop

If a table feeds revenue reporting, fraud scoring, or a customer-facing feature, it has consumers with expectations. An SLO makes those expectations measurable: how stale can it be, what quality is acceptable, how often must the run succeed.

Logs tell you what happened. SLOs tell you whether consumers can rely on the data.

SLIs that worked for us

SLI	How we measured	Example target
Freshness	`now() - max(_loaded_at)`	p95 ≤ 30 minutes
Success rate	successful / expected runs	≥ 99.5% over 30 days
Quality pass rate	checks passed / checks run	≥ 99.9%
Reconciliation	variance vs source	≤ 0.5% on key metrics
Job latency	duration p95	≤ 2× baseline

Two or three SLIs per critical dataset. Not ten.

Example we used for `curated.orders`

dataset: curated.orders
consumer_tier: critical
slos:
  freshness:
    sli: minutes_since_last_successful_load
    target: 95% of days ≤ 60 minutes
  availability:
    sli: successful_hourly_runs / expected_runs
    target: 99.5% over 30 days
  quality:
    sli: quality_checks_passed / quality_checks_total
    target: 99.9% over 30 days
error_budget_policy:
  - burn > 50% in 7d -> freeze non-critical changes
  - burn exhausted -> incident review + tell consumers
owner: data-platform-core

Burn rates for data

Same idea as service SRE:

Fast burn - freshness missed several times in a few hours -> page the owner
Slow burn - quality pass rate drifting for two weeks -> ticket and trend review

Signals live in pipeline metadata and quality tables, not HTTP metrics.

What we instrumented

Run table (job, dataset, start, end, status, rows, bytes). Quality results. Freshness view. A short consumer list for who to notify. Even a simple BigQuery view for status was enough to start.

-- ops.run_log + freshness status (illustrative)
CREATE TABLE IF NOT EXISTS ops.pipeline_runs (
  run_id STRING, dataset STRING, scheduled_at TIMESTAMP,
  started_at TIMESTAMP, ended_at TIMESTAMP,
  status STRING, rows_written INT64, bytes_scanned INT64
);

CREATE TABLE IF NOT EXISTS ops.expected_pipeline_runs (
  dataset STRING, scheduled_at TIMESTAMP
);

CREATE OR REPLACE VIEW ops.data_product_slo_status AS
WITH observed AS (
  SELECT
    dataset,
    scheduled_at,
    MAX(IF(status = 'success', 1, 0)) AS succeeded,
    MAX(IF(status = 'success', ended_at, NULL)) AS successful_at
  FROM ops.pipeline_runs
  WHERE scheduled_at >= TIMESTAMP_SUB(CURRENT_TIMESTAMP(), INTERVAL 30 DAY)
  GROUP BY dataset, scheduled_at
)
SELECT
  expected.dataset,
  TIMESTAMP_DIFF(CURRENT_TIMESTAMP(), MAX(observed.successful_at), MINUTE) AS lag_min,
  AVG(COALESCE(observed.succeeded, 0)) AS success_rate_30d
FROM ops.expected_pipeline_runs AS expected
LEFT JOIN observed USING (dataset, scheduled_at)
WHERE expected.scheduled_at >= TIMESTAMP_SUB(CURRENT_TIMESTAMP(), INTERVAL 30 DAY)
  AND expected.scheduled_at <= CURRENT_TIMESTAMP()
GROUP BY expected.dataset;

What the conversation changed

“Hourly is nice; business needs 8am” -> change schedule or SLA.
“We cannot hit 99.9% with vendor delays” -> renegotiate dependency or tier.
“Critical but unowned” -> assign owner before adding consumers.

Data engineers become operators of data products, not only authors of SQL.

How we started (about a week)

Pick three critical datasets and their consumers. Define one freshness and one quality SLI each. Log results from existing jobs. Add one fast-burn alert. Publish the SLO where consumers can see it. Expand after the first incident proves the point.

Closing

Green Airflow tasks are not a reliability strategy. SLOs make expectations explicit. If copilots or scoring jobs read your tables, this is how you know the inputs were fit for use - not only that a job completed.