TestingData engineering

How to test a data pipeline: what tests miss

Vishnujan Narayanan August 2026

A green suite is evidence about your code, not your data. The cheap tests that catch what it misses.

Yuan et al. went through 198 randomly sampled failures in Cassandra, HBase, HDFS, MapReduce and Redis, and found that 92% of the catastrophic ones came from incorrect handling of errors that were not themselves fatal. In a quarter of those, the handler simply ignored the error. The bug was in the code that ran after something had already gone wrong.

That result has stuck with me because it describes almost every pipeline failure I have had to debug. The extraction worked. The load worked. Something in between failed, got caught, got logged, and the run carried on and reported success. A test suite covering all three stages passes the whole time.

Most guides on testing data pipelines cover four things: unit tests, contract tests, data quality checks and end-to-end runs. Those are the right four and I will go through them. But I have found that the failures which actually reach production live in a different set, and I have never seen those written up: sources that will not repeat themselves, runs that die halfway, and third parties that change under you. Those take up the second half.

Standard test types: unit, contract, data quality, end-to-end

Unit tests check that a function does what its author intended, given inputs the author imagined. They are cheap and you should have them. They also cannot see a seam: two functions that are each correct, wired together wrongly, pass every unit test either of them has.

Contract tests are the ones most pipeline codebases skip, and I skipped them for a long time because I misunderstood what they were for. The idea, from Pact and consumer-driven contract testing generally, is that you record what you expect a dependency to send and what you expect to send it, and both sides check themselves against that recording. In a web service you own both sides. Against a third-party API you own neither, so a contract test degrades into an assertion that the response you saved months ago still parses. That still catches a surprising amount, but it is worth being honest that it is a weaker thing than the pattern's usual sales pitch.

Data quality tests assert on the rows rather than the code: row counts within a range, no nulls in key columns, values from a known set, freshness. This is the best-tooled part of the whole subject — dbt tests if you are in a warehouse, Great Expectations or Soda if you are not, and pandera if you just want a dataframe schema that refuses the bad row. The lightest useful version is four assertions: uniqueness on the key, not-null on what matters, accepted values on the enums, and a freshness check.

End-to-end tests run the whole thing and compare the output against what you expected. They assume the pipeline is deterministic, which is where I stopped being able to follow along.

Each of the four is worth having. What they share is that they all check the code against a world you have described in advance, and pipelines break because the world stops matching the description.

Failures a green suite misses

Four that cost me weeks, and the thing they have in common.

The first is the Yuan paper's finding at one-person scale. A configuration key was misspelled, so a lookup raised, exactly as the config reader was designed to. One level up, the orchestrator caught the exception, logged it, and continued — also by design, because a failure to send a notification should not roll back database writes that already committed. Both decisions are defensible on their own. Between them they converted a hard error into a log line, and the run finished green. What kept it invisible for weeks was that a second, unrelated function sent the run summary, so the summary arrived on schedule while the actual output never did.

I now think an exit status computed from "did we reach the end" is close to worthless. Count what you intended to produce and what you produced, and fail the run when those disagree.

The second is a test that had been failing on CI for five weeks while passing locally. It asserted that some files existed; those files had been removed from version control and still sat on my disk. The suite was red on every push, which meant nobody read it, which is how the first failure survived. The fix was to assert the property actually meant — that those paths are not tracked by git — but the habit worth stealing is smaller: run your suite inside a fresh clone of your own repository before you trust it.

rm -rf /tmp/cicheck && git clone -q . /tmp/cicheck
cd /tmp/cicheck && pytest -q

Ten seconds, and it reproduces exactly what CI sees. The same class of problem — a test that quietly depends on the machine — is what Testcontainers exists to remove for anything needing a real database.

The third was a hosted model being retired underneath a running job. No code changed; the vendor withdrew it for projects not already using it, and every call started returning 404. No test suite catches this, because every test of that path mocks the vendor. One detail from that day is worth carrying: the provider's own "list available models" endpoint still returned the model while every generation request was failing. Listing is not proof of callability.

The fourth is my favourite, because the code was correct. A chat platform rejects an entire message if any one of its buttons carries a URL it cannot reach. One unset base-URL setting meant a run that had done all of its work correctly discarded the only output it exists to produce. When you assemble a result from several optional parts, a bad part should cost you that part rather than the whole — filter to the buttons that work and send what survives. The same shape shows up in a row with one unparseable column and a digest with one dead feed.

None of those four is a bug in a transformation. They are a swallowed exception, an environment difference, a third party moving, and a config value. I would guess that is the general distribution, though I have not counted carefully enough to defend a number.

Non-deterministic sources: measuring the noise floor

Here is where the standard advice runs out. Comparing a run against a stored expected output assumes that running the same query twice gives you the same answer. Against a live API it often does not, and if you do not know your source's variance you cannot interpret any comparison you make.

I found this out while trying to answer what looked like a simple question: does running a collector with more workers in parallel corrupt the results? The obvious test is to compare a parallel run against a single-worker baseline.

workers   4   →  77.9s   IDENTICAL     ← baseline
workers  12   →  40.5s   IDENTICAL
workers  20   →  26.0s   DIFF -34/+36
workers  32   →  25.9s   DIFF -38/+40

Read casually, that says concurrency is safe up to twelve workers. It does not, and the shape of the differences is the tell: 34 records missing and 36 extra. A concurrency bug loses writes. It does not swap one record for another. Symmetric substitution looks like the source returning a different result set.

So I took concurrency out of the question and ran the collector sequentially, twice, over the same range.

SEQUENTIAL vs SEQUENTIAL   (12 assets, 26 weekly slices each)
asset     runA   runB  missing  extra
BTC       1982   1978       14     10
ETH        494    487       17     10
...
TOTAL     4390   4387       55     52     →  2.44% churn, no concurrency involved

Every asset differed. The source returns a different set of results for the same query minutes apart, at about 2.44%. The parallel runs had diverged from baseline by 1.7% to 2.6% — less than two sequential runs differ from each other. Concurrency was exonerated and the two "identical" verdicts were luck.

Measure how much your system varies when you change nothing, before you compare anything to a baseline. It costs one extra run. Without that number, "identical to baseline" and "slightly different" both carry no information.

Once you have the number, the question becomes what to assert instead of equality, and the answer has a name I did not know at the time. Metamorphic testing is asserting a relationship between two runs when you cannot state the correct output for either one: the same query with a wider date window must return a superset; sorting by a different field must return the same set; deduplicating twice must equal deduplicating once. Those hold regardless of what the source did today. Reading about it after the fact reorganised how I think about this more than anything else on the subject.

The tooling to make the comparison itself cheap is worth knowing about too. data-diff compares two tables across databases and reports the rows that differ rather than a boolean, which is the difference between "the run changed something" and "the run changed these 55 rows".

Resumability: kill-and-restart tests

A scheduled pipeline rests on something narrower than a run succeeding: that the next run fixes whatever the last one left behind. That property is fully testable offline against a fake source, and it is cheap to write.

The scenarios are the ways a run dies: budget exhausted mid-run, interrupted with a KeyboardInterrupt, killed with no cleanup, a wall of 503s, resumed with a different worker count, an item entering the set late, an item dropping out. Each test kills a run one of those ways and asserts on the dataset afterwards — no duplicate identifiers, no null timestamps, no malformed progress rows, files byte-identical where nothing should have changed.

One assertion in that group does more work than the others, and it took me an embarrassing while to see why it was needed:

expected_max = (len(fresh) * SLICES
                + sum(SLICES - n for n in partial_done.values())
                + len(complete) * 3)
assert f.requests_made <= expected_max, f"refetched too much: {f.requests_made}"

The test kills a run partway and checks the next one finishes. But a collector with no resume logic at all would also finish, by fetching everything again. Bounding the request count is what makes the test about resumption rather than about completion. The same trick catches an incremental load that quietly rebuilds the whole table, and a cache that never hits.

This is chaos engineering at the scale of one script, minus the infrastructure. For generating the inputs rather than the failures, Hypothesis is where properties like "idempotent" and "always sorted" get properly exercised, including the empty and duplicate-heavy cases nobody writes by hand.

Third-party drift: retired models and schema changes

The retired model above is one instance of a category: your code is fine and the world moved. Schema drift is the same category — a column renamed upstream, a type widened, a field that starts arriving null.

Three things help, in increasing order of effort. VCR.py records real responses once and replays them, so your fixtures are transcripts rather than something you invented — which at least keeps the mocks faithful to what the vendor really sent on the day you recorded. A single real call at startup, against the actual dependency with throwaway data, catches the retired model, the placeholder URL and the mistyped config key in one go; this is the highest-value thing on the entire page and it belongs in the schedule rather than in the test suite. And where you control the producer, a schema registry enforces compatibility before the bad message is ever published, which is the version of this problem solved properly rather than defensively.

Where you do not control the producer — which is most scraping and most third-party APIs — I have not found anything better than validating the response against a declared schema and failing the load rather than writing the row. Google's data validation paper describes generating that schema from the data itself and treating a mismatch as an alert rather than a crash, which is more sophisticated than anything I run, and is the direction I would go if this were a bigger system.

A testing checklist for a new pipeline

Roughly in order of value per hour spent.

Make "nothing happened" distinguishable from "nothing was allowed to happen". Zero new rows is both the normal outcome and the failure outcome, and that ambiguity is how breakage hides longest. An attempted/succeeded/failed count at the end of every run costs nothing.

One end-to-end run against real dependencies with throwaway data, in the schedule rather than in CI. Then the four data quality assertions — uniqueness, not-null, accepted values, freshness — because they catch the silent corruption that unit tests structurally cannot.

Measure the noise floor before comparing anything to a baseline, and when the source is non-deterministic, assert metamorphic relations instead of equality. Kill the run in every way it can die and check the next one repairs it, bounding the work so the test is about resumption.

Then the cheap ones: assert on output rather than on calls, since a test verifying a function was invoked with certain arguments passes happily while the message it produced is rejected downstream. Run the suite in a fresh clone. And log enough that when something does slip through, the run tells you which of these it was.

The thing I have not done, and probably should, is measure how long each failure was live before I noticed. The loud ones are easy — they were fixed within hours, because they announced themselves. For the quiet ones I genuinely do not know, and nothing in my setup records when a bug started, only when I fixed it.

References and further reading

Hillel Wayne's metamorphic testing is the piece to read in full on the technique named above: asserting a relationship between two runs when you cannot state the correct answer for either one, which is exactly the situation a non-deterministic source puts you in.

The OSDI paper Simple Testing Can Prevent Most Critical Failures is the one to read if you only read one. Its finding — that most catastrophic failures in distributed systems come from error-handling code that was never exercised — describes the first failure above exactly: the handler ran, caught, logged, and continued.

Testing data, not just code

Great Expectations — the standard vocabulary for assertions that belong on data: row counts within a range, columns not null, values in a set. Worth borrowing even if you never adopt the framework.
dbt data tests — uniqueness, referential integrity, accepted values and freshness. Four checks that catch most silent breakage in a warehouse.
Dagster asset checks — the same idea attached to the pipeline itself, so a failing data quality check blocks the downstream step instead of being noticed later.
pandera — declare a dataframe's schema and constraints, and fail the load rather than writing the bad row. The lightest thing on this list to adopt.
Data Validation for Machine Learning (Google, SysML 2019) — how they generate and evolve a schema from the data itself, and how they treat a schema mismatch as an alert rather than a crash.

Testing the code around it

Hypothesis — property-based testing. State the property, let it find the input that breaks it.
VCR.py — record real HTTP responses once, replay them forever. Fixtures that came from the real source rather than from your imagination.
Testcontainers — a real Postgres or Redis per test run, which removes a whole class of "passes on my machine" failures.
Pact — consumer-driven contract testing. Strongest when you own both sides; against a third-party API it degrades into "the response I saved still parses", which is still worth having.
data-diff — compares two tables across databases and tells you which rows differ, rather than whether they differ.
Soda — data quality checks as a small YAML file, for pipelines that are not in a dbt warehouse.
Confluent Schema Registry — compatibility enforced before a bad message is published. The upstream fix for schema drift, where you control the producer.
pytest — specifically fixtures and tmp_path, which is what makes kill-and-resume scenarios cheap to write in the first place.
Principles of chaos engineering — the discipline the scenario tests borrow from, minus the infrastructure.

Common data ingestion bugs — a year of real pipeline failures sorted by cause, useful for deciding what your tests should be looking for.
How to make an ETL pipeline safe to rerun — the design these stop-and-resume tests are written against: rerunning repairs data instead of duplicating it.
What a 429 really means — how to tell apart a rate limit that clears on its own, one that never will, and a quota that is simply spent: the distinction a retry loop needs.