Data engineeringReliability

Data ingestion bugs, and how to catch them

Vishnujan Narayanan August 2026

Five ways ingestion breaks, with the guard for each. Eight of about 30 bugs threw no error at all.

Ingestion is the stage that gets data out of somebody else's system and into yours, and it fails in five ways. Four of them tell you. The fifth writes wrong data, reports success, and leaves nothing in the run to find.

Here is the distribution from about 30 bugs I fixed over a year across 32 scheduled collectors — share prices, company announcements, regulatory filings, news. Eight of the 30 produced no error message of any kind, which is the only genuinely interesting number here.

Roughly 30 bugs by primary cause. The eight silent ones are the category that mattered.

Two collectors account for 16 of the 30, and both are the ones that keep long-lived state between runs: a resume point, a file of accumulated history. The ones that fetch a thing and write a file barely appear. If you are deciding where to spend review effort, spend it wherever a run's output becomes the next run's input.

Silently wrong data — 8 bugs

Every one of these ran to completion and reported success.

The worst was a deduplication key naming columns that did not exist — the self-inflicted version of schema drift, where the shape of the data and the shape your code expects stop matching. I wrote the key from memory rather than from a response, so all three of personName, acqQty and securityName were absent from the data, every row produced the same empty key, and the deduplicator concluded that every record for a company on a given day was the same record repeated. A company filing four separate transactions came through as one. Nothing raised, because building a key from a missing column is not itself a crash, and the row count still looked plausible because it was never compared to anything.

Fixing that surfaced a second bug underneath it. The dedup was implemented as a merge, and a merge against stored data that already contains duplicate keys inflates rather than deduplicates, because each new row matches several stored rows. So the pipeline went from silently dropping records to silently inventing them. An anti-join — keep the rows whose key is not already stored — cannot do that, and validate="one_to_one" on pandas merge raises on it instead of performing it.

Two more came from writing rows that should never have been written. yfinance returns a bar all-or-nothing, so a throttled request or an unsettled period gives you a row with a date and no prices; my loader stored those, and since the resume point was the newest date in the file, an empty row became the high-water mark and every real bar behind it was skipped for good. Future-dated rows do the same thing more dramatically. Both are now rejected at ingest, and a startup pass purges the ones already there.

One was self-inflicted in a way I found funny afterwards. Per-item files were named using a label looked up at runtime, which returns the full name when the source is healthy and the bare identifier when it is rate-limited. Two answers meant two filenames, a new filename looks like an item with no history, and the pipeline cheerfully re-downloaded twenty years into it. I had items sitting in three files with three overlapping partial copies.

The last two are the ones I would warn a stranger about. A site started returning Cloudflare 405s, the fetch step caught it, logged it where nobody was reading, and the run reported success with zero new rows — indistinguishable from a quiet day. And a free tier silently truncates any response at 200 rows, so a wide window returns 200 records with no error, no flag and nothing in the response indicating anything was withheld. You have to already suspect that 200 is the cap. I now treat any response landing on a suspiciously round number as truncated until proven otherwise, and I check by requesting the same range twice at different window widths.

The source fighting back — 9 bugs

The largest category and the least interesting, because every one announced itself. 403s from Scrapy until the user agent rotated and the rate dropped, a browser that crashed until it ran headless, a social platform rate-limiting hard enough to need three-minute waits on a fresh page and twelve-minute waits mid-scroll, Cloudflare twice, and eventually a rotating proxy pool because one address was never going to finish a full crawl.

What made most of them stop happening was not any individual fix but an adaptive worker pool: start at five concurrent workers, drop one on any rate-limit or API error, add one back after fifteen consecutive clean calls. It finds the source's tolerance by itself and re-finds it when the source changes its mind. Scrapy's AutoThrottle is the same idea as a setting, adapting to observed latency rather than to errors, which is gentler than my version. Mine is about twenty lines and I wrote it before I knew AutoThrottle existed.

Crashes and timeouts — 5 bugs

Unhandled exceptions in thread workers that killed a whole run's progress, and a downloader with no timeout that hung indefinitely on a stalled connection. Two rules cover the category. Every network call gets an explicit timeout, because there is no useful default and a stalled TCP connection otherwise hangs until the OS gives up, which can be hours. And worker functions return a result object rather than raising into the pool, so one bad item cannot take the run with it. Three retries, ten-second timeout, exponential backoff. These are the bugs you fix once.

Environment and config — 5 bugs

Hardcoded output directories, three times, each found by running the pipeline somewhere other than the machine it was written on. A script that broke when scripts/ became Scripts/, which only exists as a bug on case-sensitive filesystems and therefore only appeared on the server.

Individually dull, collectively a sixth of the total, which is a proportion you only see by counting. All of them are the same root cause: a value belonging to the environment written into the code instead of passed into it. Running the thing in a container from a fresh clone catches the lot in one go.

Concurrency — 3 bugs

All three are one bug approached three times. Every worker thread appended its result to a shared file by reading it, concatenating, deduplicating and writing it back.

# Before: every worker reads, merges and rewrites the whole file.
with combined_file_lock:
    combined = pd.read_csv(combined_path)
    combined = pd.concat([combined, new_rows])
    combined = combined.drop_duplicates(["Date", "Symbol"], keep="last")
    combined.to_csv(combined_path, index=False)

# After: workers return frames, one writer commits once.
frames = [f.result().df for f in as_completed(futures)]
(pd.concat(frames)
   .drop_duplicates(["Date", "Symbol"], keep="last")
   .to_csv(combined_path, index=False))

The lock stopped two threads writing at the same instant, which does not make the rest of it correct. Writing a file in place is not atomic, so an interruption mid-write leaves a truncated file that the next thread reads as the complete dataset and writes back; the fix for that part is to write a temporary file and rename it over the original. Three passes in one day went into making the write safer without changing the fact that every worker was rewriting the whole file, which is a fair indicator of where the problem actually was.

Prevention: the two habits that matter

Write more tests is the conclusion everyone agrees with and nobody acts on, and it is not quite right anyway, since a passing suite is evidence about code rather than about data. Two narrower habits would have done more.

The first is that an empty result and a blocked result are different and the run has to say which one it got. Three of the eight silent bugs are the same bug wearing different clothes — blocked by Cloudflare, truncated at 200 rows, emptied by a broken dedup key — and in all three the output was zero new rows, which is also what a quiet Tuesday looks like. Anything that can legitimately return nothing needs to distinguish the source having nothing from being prevented from finding out.

The second is not to let the data be its own checkpoint without validating it first. Resuming from the newest row is a genuinely good pattern and I still use it, but any bad row you write becomes the instruction for where to start next time, and two of the eight silent bugs are exactly that.

What neither habit gives me is a way to find silent bugs that are live right now. The only mechanism I can think of is re-fetching data I already have on a schedule and diffing it against itself, which is roughly what data-diff does between two tables, and which I have not built yet.

Caveats on the count

"About 30" is doing some work in that sentence. Small bugs fixed while doing something else do not appear, so it is an undercount by an unknown amount. The three concurrency entries are one bug I got wrong twice, and counting them as one moves the totals around. Classifying after the fact biases toward whatever story was already in my head, and while the silent-versus-loud split is stark enough to survive that, a category boundary could easily move by two or three.

Most importantly this counts bugs I found. The entire argument is that a class of bug produces no signal, so the honest position is that I have no idea how many are in there now. I also cannot tell you how long any of the quiet ones were live, because nothing in my setup records when a bug started — only when I fixed it.

References and further reading

Ding Yuan et al., Simple Testing Can Prevent Most Critical Failures (OSDI '14), looked at catastrophic failures in Cassandra, HBase, HDFS, MapReduce and Redis and found the majority came from error-handling code that was wrong or empty rather than from the original error. The silently-wrong-data category above is that paper at one-person scale.

Jepsen's analyses are the best demonstration I know that a system will happily tell you everything is fine while losing your writes, and that the only way to know otherwise is to go looking on purpose. Different scale, same problem as the dedup key.

The specific fixes, in someone else's words

pandas merge — the validate="one_to_one" argument raises on the row-multiplying join that turned my deduplication into duplication.
pandas drop_duplicates — worth reading closely for what happens when a named subset column is missing, which is the mechanism behind the first silent bug.
Postgres INSERT ... ON CONFLICT — the same problem solved properly by a database. If your pipeline can write to Postgres instead of a CSV, most of the silent category stops being possible.
Ext4 and data loss (LWN) — why "write the file in place" is not safe, and why write-temp-then-rename is the pattern.
How SQLite implements atomic commit — the long version of the same idea, and a good argument for using a database rather than reimplementing one badly around a CSV.
pandera and Great Expectations — declare what a valid row looks like and fail the load, rather than discovering it downstream.
Airflow best practices — the idempotency and "tasks must be re-runnable" rules, which is the same discipline these scripts arrived at without a scheduler.

Background

Files are hard — the survey of how many ways writing a file goes wrong. Read it before trusting any pipeline whose checkpoint is a file.
yfinance — the source behind the all-NaN rows. Its issue tracker is a useful reminder that an unofficial client's behaviour is not a contract.
NSE insider-trading disclosures — the page the dedup bug was quietly discarding filings from.

How to make an ETL pipeline safe to rerun — how to design a pipeline so that rerunning it repairs bad data instead of duplicating it.
How to test a data pipeline — the kinds of tests that catch failures like these before they ship, and why a passing unit-test suite misses them.
Why your scraper gets blocked — what actually refuses a scraper (headers, TLS fingerprint, IP address, behaviour) and which fix addresses which.