Blog
Blog
Visual temporarily unavailable
This section's visual is under review and a replacement is being prepared.
A 60% win rate in the backtest. Twelve weeks live, the account is down 18%. Something is structurally wrong, and the backtest was hiding it the whole time.
This happens at every experience level. Quant funds with proper infrastructure still launch strategies that underperform their backtests by 20 to 40 percentage points. The gap is not a sign of poor execution. It is what happens when you close the distance between a simulation and a real market and discover how many assumptions the simulation was quietly carrying.
Six of those assumptions account for most of the damage. Not all of them are fixable. All of them are diagnosable before you risk capital.
Every candle-based backtesting library ships with a lie embedded in its default configuration: if price reached X, you filled at X.
Real fills happen inside a market microstructure. A limit order at a specific price does not fill unless another participant actively chooses to trade against it at that exact level, at the moment you placed it, when you had the queue position to get a fill. A market order fills at whatever the orderbook offers at execution time, which may be meaningfully worse than the last traded price. On a liquid pair during a calm period, the difference is small. During the first 200 milliseconds of a price move, the orderbook thins sharply on the initiating side, and the fills a market order receives can be substantially worse than the signal entry price.
Candle data makes this invisible. A 15-minute candle records open, high, low, and close. The path between those four points is gone. A candle with a high at 100 and a low at 90 could have printed 100 then 90, or 90 then 100, opposite sequences, completely different implications for a strategy that exits on a bounce. The backtester picks one and moves forward. The live market picks the other.
This problem compounds at shorter timeframes. Below the 5-minute bar, candle-based backtests are not approximations. They are a different measurement of a different thing.
Backtests assume that the size you want to trade is available at the price you want to trade it.
Real orderbooks do not offer this. The displayed depth at any price level is a combination of genuine resting orders and orders that will be cancelled before execution. Academic work on microstructure has documented that a substantial fraction of displayed depth on crypto exchanges is never executed. Spoofing, large orders placed with the intent to cancel rather than fill, is not a minor phenomenon. It is common enough to distort any backtest that treats orderbook depth as a reliable signal.
Consider an illustrative scenario: a strategy generates a signal to buy, the orderbook shows what appears to be substantial depth at the entry level, the backtest records a fill. Live, those orders cancel as soon as a market order approaches. The fill occurs several levels deeper. The strategy's expected value was calculated against a price that was never actually available.
Strategies with larger position sizes face this asymmetrically. A backtest that worked on a small account fails on a larger one not because the signal degraded, but because the liquidity assumption that was implicit in the smaller position size no longer holds.
Most backtests assume the strategy does not move the market. Almost every real strategy does.
Kyle's Lambda, introduced by Albert Kyle in 1985 and extended in subsequent microstructure research, measures price impact per unit of signed order flow: how much the market moves for each dollar of net buying or selling. This is not a constant. Lambda varies by exchange, by pair, by time of day, by volatility regime, and by the concentration of informed traders currently active. A strategy that estimates its market impact from a calm Tuesday afternoon and then trades through a 3am thin period is using the wrong number by an order of magnitude.
The backtest has no market impact because the backtest never actually placed an order. The model's trades did not affect price. Live, they do. A strategy that enters and exits through 10 to 15 trades per day on a moderately liquid pair is leaving measurable footprints in the orderbook. Other participants read those footprints. Some of them trade against the pattern.
Backtest data is cleaner than live data.
Historical exchange data has been corrected, de-duplicated, and in many cases reconstructed from logs that were incomplete when they were first recorded. Volume figures have been partially cleaned of wash trading. Timestamp alignment issues that cause apparent arbitrage opportunities have been removed in post-processing.
Live data has none of these corrections. Exchange WebSocket feeds drop messages. REST polling creates gaps. Different exchanges record the same event at slightly different timestamps. Volume figures include trades from self-matching, circular trading, and entities operating with no directional intent. A strategy that uses volume as a filter, entry condition, or normalisation denominator is partially running on fabricated data in live markets.
Order Flow Imbalance, formalised by Cont, Kukanov, and Stoikov in the *Journal of Financial Econometrics* (12(1), 47-88, 2014), accounts for this by measuring the directional pressure of order flow rather than raw volume. Wash trades have no directional intent: by definition they are both buyer and seller simultaneously. They inflate volume, but they do not create persistent OFI. A strategy built on OFI is structurally more resilient to wash trading contamination than one built on raw volume, because the signal filters out what the fabrication cannot replicate.
The parameters that produced the backtest result were selected because they produced the backtest result.
Optimised parameters always perform better in-sample than out-of-sample. This is not a controversial claim. It is a mathematical certainty: parameters fitted to historical noise will overfit that noise and fail to generalise. Most practitioners know this. Almost none of them apply a sufficiently conservative correction for how severe the effect is.
A useful exercise: take your best-performing parameter set and test it on data from a different exchange during the same period. Not a different time period, a different market. If the parameters do not transfer, they were fitted to the specific microstructure of the original venue, not to a generalised signal. Binance BTC/USDT has different queue dynamics, different fee structure, and different participant composition than Bybit BTC/USDT. A parameter set that works on one is not portable to the other without at minimum re-examining every assumption.
Walk-forward validation is the minimum standard. Split the historical data. Fit parameters on the first 70%. Test on the remaining 30% with no further fitting. Then roll the window forward and repeat. The performance on the held-out windows is a coarser but more honest estimate of what live trading will look like.
The most overlooked failure mode is not execution quality or data cleanliness. Backtests are evaluated on historical data that includes the regime the strategy was designed for, because the strategy was designed by looking at that same data.
Markets move through regimes: trending, ranging, high-volatility, low-volatility, high-informed-flow, noise-dominated. A strategy that performed well across a two-year backtest may have performed well because 70% of that period was in the regime where the strategy has edge. Live, the first three months might be in the remaining 30%. The strategy was never designed for it, and the backtest never warned you, because the backtest did not separate performance by regime.
Regime detection from microstructure features is more reliable than price-based approaches because it measures the underlying cause rather than its lagged effect. OFI autocorrelation, spread variance, and quote-to-trade ratios provide leading indicators of regime transitions. A strategy that detects a regime shift and stops trading is not giving up edge. It is acknowledging that the edge was conditional and the conditions have changed.
This is the one gap the backtest is structurally incapable of showing you. The historical data contains the regimes your strategy was built to navigate. The market delivers regimes in whatever sequence it chooses.
None of these mechanisms argue against backtesting. They argue against the specific way most backtesting is done.
Tick-level data closes the path problem that candle data cannot solve. Microstructure features built from immutable execution records, not overlays computed from candle close prices, do not repaint when the bar settles. Walk-forward validation reduces the parameter overfitting problem, though it does not eliminate it. Regime conditioning gives you a way to describe when the strategy should trade and test whether that description holds.
The harder part is accepting what cannot be fixed in the backtest at all. Market impact scales with position size in ways that are not fully predictable before you place real orders. Data quality in live feeds differs from cleaned historical data in ways that vary by exchange and by time period. Regime transitions happen at times the backtest did not contain.
The backtest is not a simulation of your strategy. It is a simulation of a version of your strategy with perfect information, no competition, and an execution model that has never existed in any real market. Close reading of each gap is what separates a strategy worth deploying from one that looks good only when examined from the safe distance of history.
Related essays on this site cover each mechanism in more detail: the path problem inside candle data and what tick-level recording reveals that candles hide; order flow imbalance as a non-repainting directional signal; why displayed orderbook depth is a less reliable measure of liquidity than most practitioners assume; and how to detect the wash trading that contaminates volume-based signals.
The engineering required to compute microstructure features from raw exchange feeds at production quality is non-trivial. DepthSignal provides these features as a live API, drawing from the same immutable execution records used in the analysis above.
Because the backtest was usually simulating cleaner data, easier fills, lower impact, and more stable conditions than the live market actually offers. Several hidden assumptions break at once.
Slippage is one reason, not the whole reason. Execution quality, liquidity, market impact, data quality, parameter overfitting, and regime change all matter.
No. Tick data closes part of the path problem, but it does not remove market impact, venue fragmentation, bad live feeds, or changing participant behavior.
No. It can reduce the gap, not eliminate it. A stronger model is still being tested against a market that will arrive in a different sequence than history did.
Because short live stretches can still sit inside a favorable regime or unusually easy execution window. Robustness needs evidence across changing conditions, not one flattering sample.
When the assumptions behind the backtest stop matching observed execution, data quality, or regime behavior. A strategy that only works inside the historical version of the market was never production-ready.
Cont, R., Kukanov, A., & Stoikov, S. (2014). The price impact of order book events. *Journal of Financial Econometrics*, 12(1), 47-88.
Kyle, A. S. (1985). Continuous auctions and insider trading. *Econometrica*, 53(6), 1315-1335.