Blog
Blog
Most backtests fail in the same polite way. They return a result that looks professional enough to trust, smooth enough to present, and precise enough to mistake for evidence. Then the strategy touches a live orderbook and the edge starts leaking out through fills, spread changes, queue loss, and price impact that never existed in the historical simulation.
That is the real reason orderbook replay matters. It is not a prestige upgrade for people who want more complicated charts. It is the difference between testing an idea against a market summary and testing it against a closer representation of the auction where capital would actually have to trade.
This matters most in crypto because many strategies are built on data that compresses away the very mechanics that determine whether the trade was executable at all. A candle can preserve the open, high, low, close, and volume of the interval. It does not preserve the pressure path inside the interval, the displayed depth that vanished when pressure arrived, or the spread instability that turned an apparent winner into a negative-expectancy fill. That is exactly why Historical Crypto Tick Data Guide matters as a foundation rather than an optional extra.
The practical question is not whether orderbook data is more detailed. It obviously is. The practical question is when that detail changes the conclusion enough that a candle-based backtest stops being a rough approximation and starts becoming a different strategy entirely.
A candle-based backtest usually looks cleaner than reality because it embeds several assumptions that are rarely stated aloud.
The first assumption is price access. If the signal fires during the interval, the engine usually behaves as if the strategy could transact somewhere near an interval boundary or some representative price inside the bar with limited friction. The second assumption is liquidity continuity. If the bar printed that price, the engine often behaves as if enough size was available there without a meaningful queue problem. The third assumption is path irrelevance. If the interval closed profitably, the simulation frequently treats the route through the interval as secondary.
Those assumptions are not always absurd. For slower strategies on deep markets, they may be close enough. The problem is that users often keep the same assumptions while asking much sharper questions. They start with directional context and then gradually drift toward execution-sensitive claims about entry timing, slippage tolerance, spread behavior, or intrabar pressure. The backtest remains candle-based while the claim quietly becomes microstructure-sensitive.
That is where the model becomes misleading. It is not that candles are useless. It is that the strategy now depends on what the candle was never designed to preserve.
The easiest way to understand this is to compare the same strategy under two different historical records.
Imagine a short-horizon mean-reversion strategy that buys after a fast downside move once volume spikes and price begins to stabilize. On candle data, the signal looks attractive. The strategy sees a local flush, enters at a clean level, and the next interval closes higher. The simulation books a modest gain after a flat fee model and moves on. Across a long sample, this produces a plausible Sharpe ratio and a drawdown profile that looks survivable.
Now replay the same signal with orderbook history. The downside move was not a smooth excursion. It was a thin-book event. Liquidity on the bid side had already been pulled before the signal fired. The spread widened just before the apparent stabilization. The strategy's buy order would have crossed a thinner ask stack than the candle implied, filled progressively worse across several price levels, and entered just as displayed depth was becoming less trustworthy rather than more stable.
The trade still looked like a rebound on the candle. It looked like a costly chase in the book.
This is the same class of problem described in why your fill is never the price you saw. The chart does not lie about where price printed. It simply does not tell you enough about what it cost to participate in the move at the moment the strategy needed to act.
Once that pattern repeats across many trades, the strategy did not merely lose a little realism. It changed economic character.
Orderbook replay does not solve every realism problem, but it upgrades the historical test in the places that matter most for execution-sensitive work.
First, it restores event order. You no longer see only the final interval summary. You see quote changes, trades, cancellations, spread evolution, and depth changes in sequence. That allows a strategy to react to the market state that existed at the time rather than to a compressed after-image of it.
Second, it turns slippage from a flat guess into a measured historical context problem. A candle-based engine may apply a uniform cost haircut to every trade. A book-based replay can at least test what the displayed depth looked like when the order would have arrived, whether top-of-book liquidity was shallow, and whether the trade would likely have consumed multiple levels.
Third, it exposes path-dependent fragility. Some strategies appear profitable because the candle closes in the expected direction. Replay makes it possible to ask whether the strategy would have survived the path through the interval long enough to benefit from that close at all.
That is why orderbook replay is not mainly about detail for detail's sake. It restores the parts of the market that determine whether a printed result was plausibly tradable.
Many teams try to patch the realism problem without changing the dataset. They add a slippage buffer and call the test conservative. That is better than ignoring slippage entirely, but it is still a weak substitute for event-level history.
The problem is that slippage is not a constant property of a strategy. It is a state-dependent outcome of liquidity conditions. It changes with time of day, venue quality, local spread width, urgency, aggressor imbalance, and the fragility of displayed depth. A flat slippage input smooths away the very left-tail outcomes that often determine whether a strategy is deployable.
A strategy may tolerate normal fills and still fail because it consistently trades at the exact moments when the book is least reliable. A uniform haircut will not describe that honestly. It will understate the cost of the worst trades and overstate the cost of the easiest ones. The resulting average may look reasonable while the distribution remains false.
Orderbook replay is still imperfect here. Historical depth is not the same as guaranteed fill truth, and queue position remains partially modeled rather than fully known. But it is still far closer to reality than pretending that every trade paid the same friction tax regardless of the surrounding auction state.
One of the least appreciated differences between summary-data testing and orderbook-aware testing is queue logic.
If a strategy posts a passive order, the historical price print alone does not tell you whether the order would have filled. It tells you only that price traded there. In a real book, fill probability depends on where the order would have sat in queue, how much size was ahead of it, how quickly the queue changed, and whether that level remained visible long enough for the order to participate.
This matters because many strategies that look elegant on bars are quietly depending on impossible passive fills. They assume that touching the level implies participating at the level. In live execution that is often false.
Orderbook replay does not eliminate the queue problem, but it at least forces the researcher to confront it. Any engine that simulates passive participation has to make an explicit assumption about queue priority, partial fills, and cancellation behavior. That is a healthier failure mode than a candle engine that does not even admit the question exists.
Even perfect historical book data does not magically make a backtest realistic. The strategy still lives inside a system with reaction time.
If the edge depends on seeing an imbalance and acting within a very short interval, then the historical market state alone is not enough. You also need a view on how quickly the real system would have recognized the signal, generated the order, transmitted it, and received the exchange response. A strategy that looks viable with near-zero latency may degrade badly once realistic delay is introduced.
This is one reason crypto backtests often overstate quality for event-driven strategies. The market data improves, but the simulation still treats the strategy like an actor with faster reflexes than production will ever allow.
That does not make replay useless. It clarifies the next realism layer that needs to be modeled. A strategy that survives reasonable latency assumptions plus book-based fills is much more interesting than one that survives only in a zero-latency, candle-summary world.
Another limit deserves honesty. Historical replay usually treats your simulated orders as ghosts. They interact with recorded liquidity, but they do not change the historical path. That is acceptable for small sizing relative to available depth. It becomes increasingly misleading as the strategy size grows.
If your order would have consumed a meaningful share of the visible book, then your own participation would have altered subsequent liquidity and pricing. A replay that ignores this can still measure signal quality, but it cannot fully measure deployable capacity.
This matters because some strategies look robust at one size and collapse at a larger one, not because the signal stopped working, but because the strategy became part of the market condition it was trying to exploit. The backtest needs to remain explicit about which question it is answering: edge quality at small size, or capacity under realistic impact. Those are not the same question.
Once a team accepts that orderbook replay is necessary, the next surprise is that the bottleneck is rarely the model first. It is the data system.
Full historical replay requires more than raw capture. It requires continuous collection, sequence integrity checks, venue normalization, symbol continuity, storage discipline, and enough validation that gaps or out-of-order damage do not silently poison the test. This is the same hidden bill behind The Problem With Free Crypto Data. The feed is only the visible layer. The trust system around it is what makes the archive useful later.
That is why teams often discover that "we have the data" and "we can test honestly" are very different statements. Event history that cannot be trusted at scale simply moves the uncertainty to a lower layer.
This is not an argument that every strategy needs full orderbook replay from day one.
If the strategy horizon is slow enough, the order size is small enough, and the research question is mainly directional rather than execution-sensitive, then candle testing can still be an efficient first filter. It is a useful tool for rejecting bad ideas quickly before the expensive realism layers are introduced.
The mistake is treating that first filter as final evidence. Once the strategy depends on intrabar timing, fill realism, venue-specific liquidity, or short-horizon pressure mechanics, the burden changes. At that point the candle model is no longer a rough proxy for the same market problem. It is testing a simpler world than the one the strategy will actually face.
Backtesting with real orderbook data matters because markets do not pay out on summary logic. They pay out on what happened in the auction when your order had to interact with real liquidity.
A candle-based backtest can still tell you whether a broad idea is worth another pass. It cannot honestly certify execution-sensitive edge once the strategy begins depending on spread, queue behavior, slippage asymmetry, or path-dependent liquidity stress. That is where orderbook replay stops being a luxury and becomes the minimum honest test.
The right question is not whether a candle backtest looked good. The right question is whether the edge remains after the market's own friction is allowed back into the room.
Because they compress away the sequence and liquidity conditions that determine real fills. The result may describe price movement correctly while still describing execution unrealistically.
Usually not. A flat buffer cannot represent the state-dependent distribution of slippage across thin books, widening spreads, and fragile liquidity windows.
It preserves visible depth, spread changes, cancellations, and queue-sensitive context. Trade history tells you what printed. Replay helps explain what liquidity existed around the print.
No. Latency, queue modeling, and your own market impact still need explicit assumptions. Replay is stronger than summary-data testing, not magically complete.
When the strategy question is slow enough and coarse enough that losing intrabar path and liquidity detail does not materially change the conclusion.