Your Pipeline Passed. Your Dashboard Is Still Wrong.

A green pipeline is not proof of correct data. It is proof that the code ran.

That distinction matters because some of the most expensive data failures do not throw an exception. A producer changes a column, the warehouse accepts it, the transformation finishes, and the dashboard renders. The number is plausible. It is also wrong.

Call this schema drift if the structure changed. Call it semantic drift if the structure stayed the same but the meaning changed. In practice, trustworthy pipelines have to defend against both.

A failure that looks healthy

Imagine an orders table with one row per order:

order_id       string
customer_id    string
total_amount   numeric(12,2)
status         string
created_at     timestamp

Your revenue model sums total_amount for completed orders. Then an upstream service begins sending total_amount in cents instead of dollars. The column name and type do not change. Every query still runs. Revenue jumps by roughly 100×.

Now imagine a different release renames created_at to created_on. That is structural drift: a contract can catch it. The cents-versus-dollars change is semantic drift: the schema still passes, so you need value and distribution checks too.

The useful lesson is not “install a data-quality tool.” It is that reliability comes from several checks placed at different boundaries. No single test can prove a dataset is correct.

1. Start with the data contract

A data contract turns downstream assumptions into something executable. At minimum, define:

• the grain: what one row represents
• the primary or compound key
• required columns and their types
• nullability and accepted values
• freshness expectations
• who owns the producer and who owns the consumer
• the change process for breaking fields

dbt’s model contracts enforce column names, data types, and supported constraints before a model is materialized. Its documentation gives a useful example: even a subtle change from boolean values to integers can break consumers in surprising ways.

A compact dbt contract might look like this:

models:
  - name: fct_orders
    config:
      materialized: table
      contract:
        enforced: true
    columns:
      - name: order_id
        data_type: string
        constraints:
          - type: not_null
      - name: total_amount
        data_type: numeric(12,2)
      - name: created_at
        data_type: timestamp

This is deliberately boring. Boring is good. The contract makes a breaking structural change fail loudly instead of leaking quietly into a dashboard.

But a contract is only a boundary, not a full quality system. dbt notes that support varies by materialization and data platform, and its type comparison does not inspect every detail in every case. You still need tests for business meaning.

2. Separate structural, operational, and semantic checks

Treat these as three different questions.

Structural: Is the shape still valid?

Check required columns, types, key uniqueness, and nullability. Great Expectations provides both strict and relaxed schema checks: you can require an exact ordered set of columns, or only require that critical columns exist. That choice should reflect the consumer. A CSV export read by column position needs a stricter rule than a query that selects named fields.

Operational: Did the expected data arrive?

Freshness and volume belong here. If an hourly source has not changed in six hours, a perfectly valid schema is irrelevant. Check the latest event timestamp, partition availability, and row-count bands.

dbt’s current source-freshness guidance makes an important operational distinction: a freshness check configured as the first explicit job step can stop later steps, while the platform checkbox can report failure without breaking subsequent steps. Choose the behavior intentionally. A warning that allows stale models to rebuild may produce fresh-looking dashboards from old inputs.

Semantic: Do the values still mean the same thing?

This is where many silent failures live. Add checks for:

• accepted categories and unexpected new values
• realistic numeric ranges
• null-rate and distinct-count shifts
• distribution changes against a recent baseline
• reconciliation to a trusted control total
• unit invariants, such as dollars rather than cents
• cross-field rules, such as shipped_at >= paid_at

A practical revenue canary can be simple:

select
  current_date as run_date,
  count(*) as orders,
  avg(total_amount) as avg_order_value,
  sum(total_amount) as gross_revenue,
  count_if(total_amount < 0) as negative_orders
from analytics.fct_orders
where created_at >= current_date - interval '1 day';

The query is not a universal threshold. It is an observable. Compare it with recent history and route surprising changes to a person who knows the business.

3. Put checks before the irreversible step

A test that runs after the dashboard refresh is an audit. A test that runs before downstream tables are replaced is a control.

A safer sequence is:

1. Land new data in a staging or versioned table.
2. Validate structure, freshness, volume, and critical business rules.
3. Compare key metrics with the previous successful version.
4. Promote the new table or view only if the checks pass.
5. Preserve the failed version and validation output for diagnosis.

This pattern reduces the blast radius. It also makes rollback straightforward because the last known-good dataset remains addressable.

For high-risk tables, do not overwrite first and ask questions later. Use atomic swaps, versioned views, snapshots, or platform equivalents. The exact mechanism changes by warehouse; the principle does not.

4. Use lineage to decide who must care

When a schema changes, the question is not only “What changed?” It is “Who consumes it?”

OpenLineage models jobs, runs, and datasets, and allows dataset metadata to include schema and lifecycle changes such as alter, rename, truncate, and overwrite. That matters because the same changed table may feed a finance dashboard, a churn model, and an executive report. The severity of the incident depends on downstream use, not merely the producer’s successful run.

A useful change-review checklist is:

• Which dashboards, models, exports, and APIs depend on this field?
• Is the change additive, compatible, or breaking?
• Can old and new representations coexist during migration?
• Who approves the new meaning?
• What is the rollback path?

Lineage without ownership becomes a diagram. Ownership without lineage becomes guesswork. You need both.

5. Make breaking changes explicit

The cleanest schema-drift incident is the one you prevent at design time.

For a breaking change:

• create a new field or version instead of silently reusing the old one
• document units and business semantics, not just storage types
• run old and new outputs in parallel
• give consumers a deprecation window
• measure remaining use of the old field
• remove it only after downstream migration is verified

This looks slower than changing a column in place. It is usually faster than investigating a plausible but incorrect board metric three weeks later.

A 30-minute starter implementation

If your stack has no formal data-quality layer, start with one decision-critical table.

First 10 minutes: Write its grain, key, owner, freshness SLA, and five required columns.

Next 10 minutes: Add checks for uniqueness, nulls, types, accepted values, and a reasonable row-count band.

Final 10 minutes: Add one business canary—a control total or ratio that would expose a unit change, filter change, or duplicated join. Make the pipeline stop before promotion when that canary fails.

Do not begin with 200 generic tests. Begin with the assumptions whose failure would change a decision.

What would make me wrong

This layered approach is not equally valuable everywhere. I would use lighter controls for an exploratory notebook, a disposable backfill, or a low-impact table with no automated consumers. Contracts and blocking checks impose maintenance cost, and poorly chosen thresholds can create alert fatigue.

I would also change the recommendation if your storage layer already provides strong schema evolution, versioning, and compatibility guarantees and your consumers are isolated from raw changes through stable interfaces. Even then, semantic drift remains possible: a field can retain the same name and type while its definition changes.

Finally, distribution tests are evidence, not verdicts. Seasonality, campaigns, outages, and genuine product growth can all create large shifts. The goal is not to freeze the data. It is to force surprising changes into an explicit review before they become accepted facts.

The practical standard

A trustworthy pipeline should be able to answer four questions before a dataset is promoted:

1. Is the structure what we promised?
2. Did the expected data arrive on time?
3. Are the values plausible and semantically consistent?
4. Do we know which downstream decisions this change can affect?

If you can answer only the first question, you have schema validation. If you can answer all four, you are building data reliability.

Sources

• dbt model contracts
• dbt source freshness
• Great Expectations: Validate data schema
• OpenLineage object model