I was reading your website:
>>>One way to help protect against that is to have an in-sample data set and an out-of-sample data set. You backtest your strategies always on your in-sample data. You backtest it, you optimize it, and then when youâre all set and youâre ready to trade it, you test your system or signal on your out-of-sample data set. My final go or no-go decision is based on the out-of-sample test. If my test strategy is profitable on the out-of-sample data with realistic slippage and commission applied, then Iâll start trading it on a small level. <<<
I thought about this myself in the past. But realized, or so I think, that this is STILL backtesting. Aren't you still attempting to optimize your system based on historical data. Whether it is in-sample or out-of-sample, once the final system is in place, all the data becomes in-sample since your system becomes optimized for those historical periods of time. Granted the system was modified only on the original in-sample data, but once you have the final system, you've still created a system where it "fits" your historical data, both in and out samples.
And as a result, have you really escaped the problem to begin with? I've pondered this quite often. What are your thoughs Aaron?