How I Scraped 20 Years of NSE Insider Filings

Insider-trade filings are one of the richest signals you can feed a quant research team, and one of the most annoying to collect. India's National Stock Exchange publishes them through its PIT (Prohibition of Insider Trading) disclosure API, but "publishes" is doing a lot of work in that sentence. The endpoint is happy to hand you a few weeks of data and distinctly unhappy if you ask for twenty years in one request. This post walks through how I backfilled roughly two decades of NSE insider-trade filings into a clean, deduplicated dataset that reruns safely — part of the wider market-data platform I built and own.

The problem with "just download everything"

The naive approach — request a date range spanning 2005 to today — fails in several ways at once. The API times out on wide ranges, silently truncates results, and applies rate limits that you only discover when your rows quietly stop arriving. A twenty-year backfill is not a single download; it is thousands of small downloads that have to be orchestrated, paced, and stitched back together without gaps or duplicates.

So the first design decision was to stop thinking of it as one job. I split the full history into chunked 180-day windows and walked them sequentially. A 180-day window is small enough that the API reliably returns a complete result, and large enough that you are not making tens of thousands of requests. Each window writes its own slice; the orchestration layer tracks which windows are done so an interrupted run picks up where it left off rather than starting over.

Backfill once, then stay current cheaply

A historical backfill and a daily refresh have completely different cost profiles, and conflating them is a classic mistake. The backfill is a one-time, expensive march through history. The daily refresh should be almost free. I built the pipeline to recognize which mode it is in: on a cold start it chunks the wide range into 180-day windows, but on subsequent runs it switches to a one-year incremental re-pull. The one-year overlap is deliberate — filings can be amended or arrive late, so re-pulling a trailing window catches corrections without re-downloading the entire history every day.

Deduplication that survives numeric drift

Once you are appending overlapping windows, duplicates are guaranteed. The interesting part is that naive deduplication does not work, because the same filing does not always look identical across fetches. A quantity field might come back as the integer 629221 in one response and the float 629221.0 in another. String versus numeric drift like this defeats a plain row-equality check, and you end up double-counting the same disclosure.

The fix was a normalized composite-key set anti-join. Instead of comparing whole rows, I build a composite key from the fields that actually identify a filing, normalize each component so 629221 and 629221.0 collapse to the same token, and then anti-join new rows against the keys already on disk. Only genuinely new filings get appended. This is what lets the pipeline rerun across overlapping windows without ever inflating the dataset — a property that matters enormously when the numbers downstream are feeding research.

Closing the reporting lag with a second source

Even a perfect scrape of NSE has a structural problem: the exchange's insider data carries a 30-to-60-day reporting lag. For research that cares about recent activity, that lag is a real blind spot. To close it, I reconciled the NSE feed against a second source, Trendlyne, which surfaces filings sooner. That meant parsing Trendlyne's HTML tables with BeautifulSoup and normalizing them into exactly the same schema as the NSE records, so the two sources merge into one consistent view rather than two datasets you have to reason about separately.

Source reconciliation is its own small discipline. The two feeds disagree on column names, date formats, and occasionally on the filings themselves. The composite-key approach from earlier pays off again here: once both sources are normalized into the same key space, deduplication across sources is the same operation as deduplication within a source.

What made it reliable

None of these pieces are exotic on their own. The reliability came from combining them: chunked windows so requests always succeed, a mode switch so daily runs stay cheap, normalized composite-key dedup so reruns repair instead of corrupt, and source reconciliation so the data is timely. The result is a pipeline I can leave alone — it fetches only what is new, heals gaps on rerun, and keeps two decades of insider activity current across ~1,665 listed companies.

If you want the broader picture of how this fits into a 28-pipeline ingestion layer, the Market Data Platform project covers the prices, options, news, and corporate-action pipelines that sit alongside this one. And the companion post on resumable ETL pipelines goes deeper on the incremental-load and backoff patterns that keep all of them running unattended.