Passing Tests ≠ A Working Tool: Our AI Trading Tool Audit Checklist
"If the tests pass, the tool works." That's the assumption almost everyone makes about AI-built software. We made it too. For about twenty minutes.
Earlier this year we had Claude build us a position-sizing guardrail tool. Something to catch us before we sized a trade too big. The first version came back looking sharp. Every test passed. The documentation read like a senior engineer wrote it on a good day. Our gut said ship it.
We didn't. And that decision is the whole point of this post.
What Was A "Perfect" Build Actually Hiding?
I (Reid) run point on our AI builds, so I was the one staring at that first version, ready to call it done. Then we did what we tell every student to do with a new strategy before it touches real money. We audited it instead of trusting it.
What we found wasn't a small bug. It was three of them, stacked underneath a shiny surface.
The tool was reading from a dead data file. A source that no longer existed in the pipeline it was supposedly checking. Run...