Hi, I’m Vishnujan Narayanan.Your friendly neighbourhooddeveloper.
Quick
Prototype
- Second-Hand Bookshop Website
- DekhLaw Legal-Emergency Platform
- Quote Search by Meaning
- Student Task Planner Web App
Blogs
Notes from the build — data pipelines, ETL, scraping and the backend and DevOps work that keeps products up in production.
How to Test a Data Pipeline: What Tests Miss
A green suite is evidence about your code, not your data. The cheap tests that catch what it misses.
Data Ingestion Bugs and How to Catch Them
Five ways ingestion breaks, with the guard for each. Eight of about 30 bugs threw no error at all.
How to Scrape a Site That Paginates by Date
Find the undocumented window cap, resume from behind, and dedup on a key that cannot fail silently.
Local LLM vs API: When to Run Your Own Model
What fits in 6GB, what it costs in latency, and the schema trick that makes a small model usable.
How to Make an ETL Pipeline Safe to Rerun
A rerun should repair data, not corrupt it. Four ways that rule breaks, and a six-point checklist.
HTTP 429: What It Means and How to Retry
One status code, three meanings — wait, stop, or out of quota. How to tell them apart in code.
Can You Predict Stock Prices Minute by Minute?
Four checks that decide whether a short-horizon model has an edge. One real attempt failed all four.
Why Your Scraper Gets Blocked and How to Fix It
Four layers block you: headers, TLS fingerprint, IP, behaviour. How to tell which one, and the fix for each.
Playwright vs Selenium: Which to Use and When
Locators, saved sessions and pinned browsers — plus the 52 fixed sleeps that show what auto-waiting does not fix.
28-pipeline NSE market-data ingestion layer feeding 12+ datasets into a partitioned store.Market Data Platform
Fraud-detection model on 6.4M transactions — 95% caught at 0.995 ROC-AUC despite a 0.13% fraud rate.Payment Fraud Scoring AppView demo 
Intraday price-direction system over 9.4M NSE ticks, raising next-minute precision from 0.51 to 0.61.Stock Price Movement PredictorView code 
Quantified how Bitcoin Fear & Greed sentiment drives trader PnL across 211K crypto trades, with a contrarian sentiment-gated signal.Crypto Mood vs Trader Profit StudyView demo 
Feed-forward classifier built in pure NumPy — 97.4% accuracy / 0.995 ROC-AUC on Breast-Cancer-Wisconsin, with hand-derived backprop.Breast Cancer Diagnosis ModelView demo 
Linear regression built end to end in pure NumPy — hand-derived gradient descent and a closed-form solver, validated vs scikit-learn.Trend Prediction ModelView code 
Sources change
without telling you.
Ingest. Sites paginate, rate-limit and rename fields without warning. A pipeline that trusts its source fails silently.
Python and Playwright scrapers with retry logic that reads what a 429 actually means, and a schema check on every load. 28 pipelines across ~1,665 listed companies.
See it: Market Data Platform · HTTP 429 retries
A rerun should
repair data, not corrupt it.
Store. Jobs crash halfway. Backfills overlap. If a second run doubles rows, nobody trusts the numbers again.
PostgreSQL with idempotent upserts and incremental, resumable loads, so any job can be killed and rerun safely.
See it: Resumable ETL · Scraping NSE filings
Data nobody can query
is dead weight.
Serve. The pipeline is only worth something once a product, a model or a person can reach what it collected.
FastAPI services in front of the data, Redis for the hot paths, pgvector when search has to understand meaning.
See it: Fraud Scoring App · Bookshop Website
Green tests
don’t mean good data.
Ship & run. Code passes CI and still loads garbage. Eight of about 30 real ingestion bugs threw no error at all.
Docker images, CI/CD on every push, jobs running on Linux and AWS, and data checks that catch the failures that stay quiet.
See it: Ingestion bugs · Testing a pipeline
QuickForge
Real-time job ingestion, pgvector matching and Telegram alerts: the same four stages, applied to a product. In development.
Swing By And Say Hi On Socials
Write-ups on data engineering — ingestion, validation, and the services on top.
FollowVishnu · Ex-chinmayite · UPES’25
Follow ↗Data Engineer | From crawlers to APIs
Ernakulam, Kerala, India 500+ connections ConnectData pipelines, from crawlers to APIs. Ingestion, data validity, and the services on top.
- Fraud_Transaction_Detection
- minute-level-stock-prediction
- Trader_sentiment_analysis
View profile ↗Notes from the build — data pipelines, scraping, and the reliability work in between.
Subscribe ↗Build logs and walkthroughs — pipelines, scraping, and the tooling around them.
SubscribeHi Vishnujan — I saw your work and wanted to ask about
Just one web away