Predicting Next-Minute Price Direction on 9.4M NSE Ticks

Intraday prediction is where machine learning meets market microstructure, and where it is easiest to fool yourself. This is a walkthrough of how I built an intraday price-direction system over 9.4 million NSE tick records across 31 NIFTY stocks — from raw trades and quotes to minute bars to a model and a backtest — and, just as importantly, the leakage traps that quietly inflate results if you let them. The full project lives on the Minute-Level Stock Prediction page.

From ticks to minute bars

Raw tick data is firehose-shaped: 2.9 million trades and 6.4 million quotes across 31 symbols. You cannot model on that directly, and you would not want to — the signal at the individual-tick level is mostly noise and exchange mechanics. The first job is aggregation. I ingested 31 per-symbol tick files (~9.4M rows total) with a pandas loader, tagged every record by company, and concatenated them into unified trade and quote datasets.

From there I condensed the firehose into minute bars: grouping on company and minute, then inner-joining the trade and quote aggregates on a composite key. The output is roughly 11,600 labeled OHLC feature rows — a dataset small enough to model on and structured enough to reason about. Going from 9.4M ticks to ~11,600 rows is not just compression; it is choosing the timescale at which you believe a signal exists.

Features that capture momentum and liquidity

A minute bar on its own is thin. The predictive content comes from context — what happened over the last several minutes. I constructed rolling mean and standard-deviation features over 3-, 5-, 10-, and 20-minute windows on both price and volume, using pandas groupby-transform so each window is computed per company without bleeding across symbols.

On top of the rolling features I derived microstructure signals: per-minute VWAP, OHLC statistics, and bid-ask spread computed from the raw order flow. Spread is a liquidity proxy; VWAP anchors price against traded volume; the rolling stats capture momentum and volatility regime. Together that is 20-plus engineered features describing not just where the price is, but how it is moving and how liquid the book is.

The leakage trap

Here is where most intraday backtests go wrong. If you shuffle your data and do a random train/test split, you train on the future and test on the past — and your metrics look fantastic right up until you trade real money. Time-series prediction demands that the model only ever sees the past.

I labeled the target as next-minute return direction using horizon-shifted targets, and split train and test with a per-company, time-ordered split. Every test bar comes strictly after every training bar for that symbol. No shuffling, no leakage. This is unglamorous and it lowers your headline numbers, which is exactly why it matters: the numbers you get are numbers you can believe.

Modeling for precision, not accuracy

Accuracy is the wrong objective for a directional trading signal. If you act on every prediction, you pay costs on every trade and drown a weak edge in noise. What you actually want is to act only when the model is confident. I trained a scikit-learn RandomForestClassifier on the engineered features, then tuned a probability threshold of 0.63: the system only takes a long signal when predicted probability clears that bar.

That threshold lifted next-minute precision from a coin-flip 0.51 to 0.61. The trade-off is fewer trades — you are deliberately staying out of the market when the model is unsure. For a signal, that is the right trade-off. A precise signal you act on selectively beats a noisy one you act on constantly.

Backtesting honestly

Finally, the backtest. I evaluated threshold-filtered long signals across 3,100 held-out minute bars with per-trade PnL accounting, comparing precision and trade count at 0.5, 0.6, and 0.7 cutoffs. Sweeping the threshold shows you the whole frontier: lower cutoffs trade more often at lower precision, higher cutoffs trade rarely at higher precision. Seeing that curve — rather than a single cherry-picked number — is what lets you choose an operating point deliberately.

The discipline that makes this trustworthy is the same discipline from the data-engineering side of my work: be honest about what the data can and cannot tell you. The leak-free split here is the modeling cousin of the repair-not-corrupt reruns I described in building resumable ETL pipelines. In both cases the goal is the same — results you can actually rely on, not results that merely look good in a notebook.