Every data team has the Monday morning story. A dashboard shows zero. Or worse, it shows a plausible number that is wrong. Somewhere upstream, a column was renamed, a category was added, a system was migrated, and the pipeline either failed or, more dangerously, kept running on the wrong assumptions. Nobody told the data team because nobody knew the data team depended on that column.
This is the pattern we use to make pipelines boring, and the incident that made us adopt it.
The silent zero
A manufacturing client's production dashboard depended on a status field from their line-monitoring system. The vendor released an update that changed the status codes from numbers to words. The pipeline did not fail: the filter status = 3 simply matched nothing, every downstream table dutifully computed zero output for the week, and the dashboard reported a factory that had stopped. The plant manager called at seven on Monday.
The pipeline had done exactly what it was told. The problem was that the agreement between the source and the pipeline existed only in the head of the person who wrote the filter.
If a dependency is not written down, it is not a dependency. It is a surprise waiting for a date.
What a data contract is
A data contract is that agreement, written down and checked by a machine. For each source a pipeline consumes, it states the schema: column names, types, allowed values, nullability. It states the freshness: how often data arrives and how late is too late. It states volume expectations: roughly how many rows, so that a sudden empty load or a tenfold spike is caught. And it names an owner on each side.
The contract is not a document in a wiki. It is a file in the repository, versioned with the pipeline, and it runs as a test on every load.
Enforcing it: tests before transformation
The contract runs at the boundary, before transformation. If a load fails the contract, the pipeline stops there, the previous good data stays in place, and the alert names the exact expectation that failed and the owner who can fix it. On the manufacturing project this would have produced, at six on Monday, a message saying "status: expected integer in 1 to 5, received text values 'running', 'stopped'," addressed to the vendor's contact. The plant manager would never have called.
Downstream, transformation models carry their own tests: uniqueness of keys, referential integrity, accepted values, and business rules such as "shipped quantity cannot exceed ordered quantity." Tools like dbt make these declarative and cheap to add, and an orchestrator like Airflow makes the dependency graph explicit so a failure upstream stops the right things downstream.
Alerts that people act on
We had alerts before the contract. Nobody acted on them, because they fired constantly for things that did not matter and were routed to a channel nobody read. The fix was fewer, better alerts: contract failures and freshness breaches only, routed to the named owner, with the expectation that failed and a link to the run. Every alert is either acted on or the contract is adjusted. An alert that is ignored twice is a bug in the alerting.
Contracts change; that is the point
Sources evolve. The contract's job is not to prevent change but to make it visible and deliberate. When the vendor adds a status code, the contract fails, the owner is notified, the pipeline's handling is updated and the contract is versioned. The change history becomes documentation of how the data has evolved, which is invaluable when someone asks why the numbers changed in March.
Security lives in the same file
The contract is also the natural place to record data classification: which columns contain personal or regulated data, whether they should be masked in downstream environments, and which roles may read them. Because the contract runs on every load, a new column that appears unclassified can be caught in the same way as a renamed one. On the pipelines we now build, a sensitive column cannot reach an analytics environment without a classification, because the contract check refuses it.
Purposefully boring
The goal of this work is for nothing to happen. Loads arrive, contracts pass, dashboards update, and the data team spends Monday morning on something interesting. When something does change upstream, the right person knows within the hour, with the exact detail, before anyone downstream notices. That is the whole ambition, and it is worth more than any dashboard.
- Write a contract for every source: schema, values, freshness, volume, owners.
- Run it at the boundary; stop before transformation; keep the last good data.
- Add declarative tests to every transformation model.
- Alert only on contract and freshness failures, to a named owner.
- Version contracts; treat changes as documentation.
- Put data classification in the contract and enforce it.