A backtest has one job that nobody markets it for: to tell you your idea is bad before the market does, for free. Almost none of them do this. The default backtest is an agreement machine. You feed it a strategy you already believe in, and it hands back a curve that confirms you were right, because you built the test and the strategy with the same set of assumptions.
I learned this building an AI trading bot. I set out to prove it made money and ended up building a research engine rigorous enough to prove it didn’t. The engine was the valuable part, and the techniques in it have nothing to do with trading specifically. They apply to anything where you are tempted to believe a number you generated yourself. Here is what made it hard to fool.
1. Score in a unit that travels
The first lie a backtest tells is in the P&L. A strategy that “made $4,000” tells you almost nothing, because the number is tangled up with how much you bet and when. Scale the position down and the same strategy “makes $400” and looks worse, despite being identical.
I scored everything in R-multiples instead. One R is the amount you risked on a trade. A win that makes 1.5 times your risk is +1.5R. A loss that hits your stop is −1R. Now every trade, every pair, every account size speaks the same language, and you can compare a EUR/USD trade to a Bitcoin trade without the currency or the bet size distorting it. Expectancy becomes a single honest number: average R per trade.
Pick a unit that strips out the things you could tune to flatter yourself, and measure in that.
2. Ban look-ahead structurally, not by good intentions
Look-ahead bias is when your backtest uses information the live system wouldn’t have had yet. Everyone knows not to do it. Almost everyone does it anyway, because it sneaks in through the plumbing, not the logic.
The classic version: you decide to enter a trade based on a candle, and you fill the trade at that same candle’s price. But in real life you only know the candle is finished once it has closed, and by then the price has moved. So I made the rule mechanical. Entries are decided at bar close and filled at the next bar’s open, never the current one. Stops resolve against the level as it stood entering each bar, not a level computed with hindsight. The backtest is only ever allowed to know what the live bot knew at that exact moment.
Don’t rely on remembering to avoid look-ahead. Build the data flow so it is physically impossible.
3. Write down the pass mark before you look
This is the cheapest technique and the one that changes the most. Before I ran anything, I wrote down what “this works” would mean, as a hard threshold: expectancy of at least +0.10R out of sample, on at least 200 trades, positive on data the model never saw, positive across a majority of test windows. Then I saved it and didn’t touch it.
Why this matters: if you decide what counts as success after seeing the results, you will always find a story that makes your best number the important one. Pre-registration takes that away from you. The result either clears the bar you set while you were still honest, or it doesn’t.
Set the goalposts first, in writing, then don’t move them.
4. Demand it hold in both halves, not just overall
Here is the technique that killed my favorite idea. I found an entry variant that scored +0.049R over the whole test window. Positive expectancy. I was ready to celebrate.
Then I split the window in half by date and scored each half on its own. The first half: +0.131R. The second half: −0.017R. The edge flipped sign. It wasn’t an edge at all. It was one market regime where the idea happened to work, canceling out against another where it didn’t, and the average hid the whole thing.
A single good number over one period is almost meaningless. Cut your window in half and make the result survive in both.
If it flips sign across the split, you found noise, no matter how good the overall figure looks.
5. Walk it forward, don’t just split once
One split is good. A rolling series of them is better, because it mimics how you would actually use the system: train on the past, test on the near future, then step forward and repeat.
My framework did this over five years of history across several market regimes. It would fit on an expanding window of past data, test on the slice immediately after, then roll forward and do it again, so the strategy was always being judged on data that came after the data it learned from. And it held back a final slice at the very end that nothing was allowed to touch until the last moment, as a true out-of-sample exam.
If your strategy only works when you pick the one test period yourself, it doesn’t work.
6. Calibrate with a strategy you know is worthless
This is the step almost nobody does, and it is the one that lets you trust everything else. Before believing my framework when it said “no edge,” I had to know it wasn’t just saying no to everything. So I fed it a deliberately edgeless strategy, essentially random entries, and checked the score.
It came back at −0.039R, right about the cost of the spread. Exactly what a strategy with no edge should score. That told me the framework wasn’t leaking a phantom edge and wasn’t manufacturing a fake loss either. It was calibrated. Now, when it scored a real strategy near that same floor, I knew the strategy genuinely had nothing, rather than the test being broken.
Run a known-nothing input through your evaluator. If it scores positive, your evaluator is the bug, and every other result it gave you is suspect.
7. Confirm a hit before you believe it
If you test enough ideas, some will look brilliant by pure chance. This is multiple testing, and it is how most “discovered” edges are born.
I had one lead that looked real: a signal where oil price moves seemed to predict the Canadian dollar, scoring +0.234R. Big number. Before believing it, I ran a confirmation test. If oil genuinely drives the Canadian dollar, the effect should show up across the other Canadian dollar pairs too, not just the one I found it in. It didn’t. The related pairs came back flat or negative. The signal was a coincidence that happened to land, and the confirmation test exposed it.
When something looks great, ask what else would have to be true if it were real, and go test that. A real effect leaves fingerprints in places you didn’t originally look.
The mindset underneath all seven
Every one of these is a way of building the same thing: a system that is allowed to tell you no, and that you will actually believe when it does. That is harder than it sounds, because the whole gravity of a personal project pulls the other way. You want it to work. You built it because you thought it would.
The payoff is that when the honest engine finally does hand you a yes, it means something. And the engine outlives the idea you built it for. Mine started as a trading tester. The same harness would validate any strategy you point it at, including one an AI wrote, which is the version of this problem more of us are about to have. The strategy was disposable. The thing that could tell me the truth about it was not.