Market Data Platform

The data backbone for a quant research team — 20 years of NSE prices, F&O bhavcopies, insider filings, news, and analyst signals across ~1,665 listed companies, all refreshed incrementally and kept consistent across sources.

What I built

• Built and own a 28-pipeline NSE market-data ingestion layer feeding 12+ datasets — prices, insider trades, options, news, corporate actions — into a partitioned CSV store.
• Engineered the data backbone for quant research: 20 years of NSE prices, F&O bhavcopies, insider filings and analyst signals across ~1,665 tickers, all refreshed incrementally.
• Designed resilient ingestion clients for a dozen financial APIs and sites, with retries, adaptive rate-limit backoff and resumable state that survives restarts.
• Assembled multi-source feature datasets — prices, insider activity, news, fear/greed and analyst signals — as clean research inputs for quant and ML modeling.

Incremental & resumable ingestion

• Developed an incremental price pipeline in yfinance and pandas that resumes each of ~1,665 tickers from its last stored bar, fetching only new data each run.
• Architected a concurrent.futures thread-pool downloader with adaptive concurrency — dropping workers and pausing 30s on HTTP 429/auth errors, then ramping back after 15 clean calls.
• Backfilled ~20 years of NSE insider-trade filings from the PIT API, chunking wide ranges into 180-day windows and switching to a 1-year incremental re-pull on reruns.
• Automated F&O bhavcopy ingestion from 2017 to date, handling NSE's mid-2024 UDiFF URL-format change and streaming each ZIP from memory into per-day CSVs.

Data quality that repairs, not corrupts

• Hardened the dataset with data-quality safeguards — NaN-price row purges, (Date, Symbol) dedup merges and duplicate-file consolidation — so reruns repair gaps instead of corrupting history.
• Deduplicated appended filings with a normalized composite-key set anti-join that collapses numeric/string drift (629221 vs 629221.0), preventing double-counts across overlapping windows.
• Reconciled insider data across NSE and Trendlyne to close a 30–60-day reporting lag, parsing Trendlyne HTML tables with BeautifulSoup and normalizing them into the NSE schema.

Scraping the stubborn sources

• Scraped 22 SEBI X/Twitter handles with async Playwright and stealth, injecting session cookies from env and harvesting the virtualized timeline via an in-page MutationObserver.
• Wrote a rate-limit recovery state machine for the X scraper — fresh-page retry cycles, 12-minute mid-scroll waits and silent-throttle detection — checkpointing progress to JSON for resumable runs.
• Extended a forked 16-publisher Scrapy news crawler with a configurable-output base spider, incremental update-mode resume from the last published date, and filesystem HTTP caching.
• Classified NSE corporate announcements into fundraising, contract-win, regulatory and FDA-approval buckets via a keyword taxonomy, scraped through stealth Playwright.

Built at CiteSert — source is private.