Product Explorer

A full-stack app that scrapes a book e-commerce catalog into PostgreSQL and serves it through a Next.js UI with real-time, click-driven scraping streamed over WebSockets — backed by Redis caching, BullMQ queues, and an 8-entity relational schema.

Architecture

• Built a full-stack TypeScript app that scrapes a book e-commerce catalog into PostgreSQL and serves it through a Next.js UI with real-time, click-driven scraping over WebSockets.
• Designed a NestJS backend exposing 15 REST endpoints and a Socket.IO gateway over PostgreSQL + Redis, backed by BullMQ queues and TypeORM across 8 entities.
• Engineered a four-stage ETL pipeline (navigation → category → product → detail) that ingests an e-commerce catalog into a normalized relational schema with deduplicated upserts.
• Modeled an 8-entity PostgreSQL schema with TypeORM, enforcing unique source IDs and upserting on conflict to deduplicate scraped rows.

Real-time interactive scraping

• Streamed live scraping over Socket.IO: each client click hovers, navigates, and scrapes the matching category in a dedicated browser session, emitting product chunks and status events back to the UI.
• Implemented a NestJS WebSocket gateway that spins up a Playwright session per connection and routes hover/click/paginate/get-details actions, returning structured DATA_CHUNK and SCRAPE_STATUS events.
• Built a custom Socket.IO client wrapper with an event-emitter API, automatic reconnection (5 attempts, exponential backoff), and typed request/response contracts shared with React hooks.
• Wrote a useInteractiveScraper React hook that buffers streamed product chunks per category, dedupes by source ID, tracks scrape status, and surfaces progress via toasts across an 8-page Next.js/Tailwind UI.

Queues, caching & resilience

• Offloaded scraping to two BullMQ/Redis queues (foreground and background) with high/medium/low priorities, dedicated processors, and scrape_job rows tracking job state.
• Cut redundant scrapes with read-through caching across Redis and PostgreSQL, using per-resource TTLs (navigation 24h, category 1h, product 24h) before falling back to a live crawl.
• Hardened Playwright crawls against bot defenses with stealth launch flags, a desktop user-agent, automatic cookie-consent dismissal, and retry-on-blocked with configurable retry counts.
• Managed long-lived browser sessions per client with a 30-minute idle timeout, a 5-minute cleanup sweep, and graceful teardown on disconnect and module shutdown.
• Reduced scrape bandwidth and page-load time by aborting image, CSS, font, and media requests at Playwright's network layer during interactive sessions.

Infrastructure

• Containerized the stack with Docker Compose — Postgres 15 and Redis 7 with health-check gating and persistent volumes, plus an auto-seeded SQL schema on first boot.