I just finished Foundations of Trading by Bandy. What an eye opener. Completely changed my view of backtesting. He introduced me to the term "stationarity". Price over time is NOT stationary but most algorithms requite the data to be stationary. That is why he strongly rejects backtesting over long periods of time. This increases the non stationary aspect of the data and leads to incorrect results. He doesn't, in this book anyway, tell us how to mitigate this. He did point to a youtube video of his talking about the concept. I will watch it when I get time.
There are few things in systematic trading that become pretty obvious the more you do it, examples:
- Backtesting over any "single" period of time is pretty meaningless.
- Probability distribution in financial instruments is not uniform.
While there is nothing new here, I often saw people taking it to the wrong extremes, such as equating #1 to - backtesting is meaningless / you shouldn't test beyond last few years etc or #2 future price can not be predicted at all. Those are just few examples, there is lots of "common wisdom" circulating around which seems like it logically came out of those but is incorrect.
So for the "not stationarity" nature of price I would suggest not to take it too far. On practice you cover this by
- ALWAYS breaking down your test data into intervals. Lets say - at least 4, to give you some number to start with
- Never developing initial version of your system on more than 25% of your available data
- Never use latest data interval to develop initial version of your system as it will create, often, incurable bias towards what worked recently
- Having enough data intervals to represent reasonably diverse sample of market types (basically combinations of volatility + direction)
If all of that is done, interval duration will be almost entirely irrelevant and never too long.
If is always too long when all data is used at once, regardless of its' absolute duration.
Bandy also speaks a lot, especially in the later years, about impulse vs state systems. Which is overrated in my opinion. I am oversimplifying but will take it as far as suggesting that his initial approach to systems testing in AmiBroker was rather naive (or was it was simply a mistake reported by readers). And that's why it because a topic in a first place. He basically used to include ONLY closed trades into DD calculations. No need to start calling it a fancy name and come up with a big explanation about it, could have just said -
ALWAYS look at your DD based on OPEN & CLOSED trades while modeling your system, because in real life that's what will drive you nuts and make you quit trading it.
Val
The rest were sold online with warm wishes of luck to new owners or went straight to garbage after I've read them.
