AI Backtesting Mistakes: Why You Should Try to Kill Your Strategy First

Let's be honest. The moment your backtest turns green, you start tweaking it. A slightly tighter stop here, a different RSI period there, maybe a time filter to cut out the losers.

That instinct isn't laziness. It's human. It's also how traders talk themselves into edges that were never real.

We get why this happens. AI makes optimizing dangerously easy. Ask Claude to test a thousand parameter combinations on your NQ strategy and it'll hand you results in minutes, not weeks.

The problem isn't the speed. The problem is what optimizing does to your judgment the second something "works."

Why Does AI Make Overfitting So Easy?

Here's the trap. With enough parameters and enough combinations, random noise starts to look like an edge. Test 700 variations of anything and a handful will look incredible, purely by chance.

That's not an edge. That's math doing what math does when you give it enough tries.

We ran into this ourselves. In one sweep across our futures strategy research on NQ, ES, and Gold, we tested 700+ parameter variations. Several "winners" came out the other side. Great win rates. Clean equity curves. The kind of results that make you want to size up immediately.

We didn't size up. We attacked them instead.

What Happened When We Stress-Tested Our "Winners"?

We pulled the best 5% of trades out of each strategy and reran the numbers. We added realistic slippage and commissions. We checked performance year by year instead of pooling a decade of data together, because a decade can hide a lot of sins.

Most of those winners fell apart.

One strategy had 45% of its total profit sitting in a single year. That's not a repeatable edge. That's one lucky stretch wearing a strategy costume. Pull that one year out and you're left with something closer to noise than to an edge you'd risk real capital on.

Another strategy died the moment we removed the top 5% of outlier trades. It wasn't actually working. One or two huge trades were carrying the entire track record on their backs, and the rest of the trades were doing nothing. Or worse, quietly losing.

This is the part traders skip. You find a strategy that looks good, and the temptation is to keep polishing it. Tighten the entry. Add a filter. Chase a smoother equity curve.

But if you haven't tried to break it yet, you're optimizing something you haven't proven is real. That's backwards.

How Do You Attack Your Own Strategy?

Our rule now: before you optimize a strategy, try to kill it first. If it survives your best attacks, then it might be real. If it doesn't, you just saved yourself from trading an illusion with real money.

Concretely, that looks like this.

Run the base case with no cherry-picked parameters.

Pull the best trades out and see if it still holds up without them.

Add real slippage and commissions, not the friction-free fantasy version.

Test each year separately instead of pooling everything together.

Check long trades and short trades independently, because a strategy that only works long isn't really a two-sided edge.

None of this is glamorous. It's the opposite of the exciting part of strategy research. But it's the part that separates a strategy you can actually trust from one that just got lucky in a spreadsheet.

This is also where TradeZella earns its keep. Logging real trades against your backtested assumptions is the only way to catch drift before it costs you money. We go deeper on that process in our TradeZella Backtesting Guide.

What Is the Placebo Test, and Why Does It Humble Everyone?

We built this one into our student tutorial roadmap because it's the fastest way to expose a fake edge. Take your clever entry signal, whatever pattern, indicator, or setup you're proud of, and replace it with a totally random entry. Same exit rule. Same risk management. Same position sizing.

Then run it.

If the random version performs about as well as your "smart" one, your edge was never really the entry signal. It was your exit and your risk management doing all the work the whole time.

This test humbles almost everyone who tries it. We've watched traders discover that their carefully designed entry added basically nothing. The strategy's real edge was a disciplined stop, a sensible target, and consistent position sizing.

That's a REPs lesson hiding inside a coding exercise. Risk management and Edge are tangled together more than most traders want to admit, and Psychology is what makes people avoid running this test in the first place. Nobody wants to find out their "genius" entry is a coin flip.

We talked through this exact process, and how we use Claude to run these attacks fast, on the Edge Up Podcast, Episode 077, "Using Claude and AI in Trading," available on Spotify. Worth a listen if you want the full walkthrough.

Optimize First, or Falsify First?

This is the real question, and it's a mindset shift more than a technical one. Optimizing first means polishing something you haven't proven is real. Falsifying first means you only spend your time and energy on ideas that already survived your best attempt to break them.

We'd rather find out our strategy is fake in a backtest than find out live, with funded capital on the line, during a losing streak that looks a lot like what we could've caught in year-by-year testing.

If that sounds like it connects to how you handle a rough patch once you're live, it does. Check out How To Handle Trading Losing Streaks for that side of it. And if you want the deeper case for why a real edge has to survive scrutiny before it earns your trust, read Positive Expectancy: Finding Your Trading Edge.

Patience is key here, and so is a little bit of self-doubt, on purpose. This is Process Over Profits in action. Before you run one more optimization pass on that strategy sitting in your TradingView or TrendSpider workspace, go try to kill it first.

Pull the best trades. Add slippage. Split it by year. Run the placebo test.

If it's still standing after all that, join the trading ohana of people who've actually earned the right to trust their edge, because they tried hardest to prove themselves wrong.

Mahalo for reading and trade well! Glenn & Reid | Hawai’i Trading Academy