How far back is enough for a backtest?
- Thread starter markc
- Start date
IMHO it all depends on the type of trading your "system" generates. If you generate signals based on an easily quantifiable days value (eg open/close/hi/low/some ma etc) then backtesting is a very important process so long as you build in some form of slippage. If you generate more intra-day type signals then backtesting is probably not a suitable tool.
What nitro says about optimising is indeed very important, personally my approach is to get an idea, backtest and optimse on a small section of time, then try out over all the data I can, I then p&l profile the system and investigate the areas it lost money to see if a filter can be attached to the model, then I leave it a month or two, come back to my system, fill in the missing data and backtest that, without any changes to the original paramters, if it still works I then set a maximum amount of money to lose on the system, work out suitable volume to trade to give it a chance (based max loses over the backtest period) and trade it.
nitro,
Isn't it better to at least forward test systems that have backtested well? Of course, this does not mean that one merely reverse engineer a system from past data and come out with an optimised system. What if you think of an idea that at least makes some sense and then backtest it. Save some out of sample data to test again and then finally forward test it. Backtesting definitely has its value here. No? Do you mean that you never ever backtest any idea that you come out with but only forward test them? That is too time consuming! What difference does it make if you have no idea what those past data were?
Isn't it better to at least forward test systems that have backtested well? Of course, this does not mean that one merely reverse engineer a system from past data and come out with an optimised system. What if you think of an idea that at least makes some sense and then backtest it. Save some out of sample data to test again and then finally forward test it. Backtesting definitely has its value here. No? Do you mean that you never ever backtest any idea that you come out with but only forward test them? That is too time consuming! What difference does it make if you have no idea what those past data were?
Loosenup,
Well, yes. As I said, I have nothing against backtesting, just be super careful when you do it, and more importantly, I would not try to take money from the markets by simply trying to design trading systems and backtesting them.
I do backtest - in fact, I am running the biggest one of my life now (it is a brute force search that would take four months to complete on a dual 2.4 Xeon.) But the reason for this search is not the traditional one of finding the best paremeter set of a given system.
But here is the thing. I have seen so many pitfalls by doing things this way I can't even enumerate them anymore. Here is my small objection to what you are saying. What I have found is that people look for systems _first_ and then start to backtest them, when they should be becoming good traders first, and then looking for systems based on what they learned. Or alternatively, sit together with a good trader and try to extract the way they approach trading a market.
I think that there are people that can take the route of never having been traders and come up with good systems - I have seen it with my own two eyes. They are not traders, but somehow, they are able to come up with terrific systems. But you know, even in their case, it was never really the backtesting or their programming skills that was their gift, it was the intuition of how markets react from watching them over many months/years that allowed them to design proper systems that led them to use their degrees in computer science to apply the scientific method to that intuition.
Someone just getting their hands on TS or WL and trying their luck at this game are at a great disadvantage.
nitro
Well, yes. As I said, I have nothing against backtesting, just be super careful when you do it, and more importantly, I would not try to take money from the markets by simply trying to design trading systems and backtesting them.
I do backtest - in fact, I am running the biggest one of my life now (it is a brute force search that would take four months to complete on a dual 2.4 Xeon.) But the reason for this search is not the traditional one of finding the best paremeter set of a given system.
But here is the thing. I have seen so many pitfalls by doing things this way I can't even enumerate them anymore. Here is my small objection to what you are saying. What I have found is that people look for systems _first_ and then start to backtest them, when they should be becoming good traders first, and then looking for systems based on what they learned. Or alternatively, sit together with a good trader and try to extract the way they approach trading a market.
I think that there are people that can take the route of never having been traders and come up with good systems - I have seen it with my own two eyes. They are not traders, but somehow, they are able to come up with terrific systems. But you know, even in their case, it was never really the backtesting or their programming skills that was their gift, it was the intuition of how markets react from watching them over many months/years that allowed them to design proper systems that led them to use their degrees in computer science to apply the scientific method to that intuition.
Someone just getting their hands on TS or WL and trying their luck at this game are at a great disadvantage.
nitro
Quote from LoosenUp:
nitro,
Isn't it better to at least forward test systems that have backtested well? Of course, this does not mean that one merely reverse engineer a system from past data and come out with an optimised system. What if you think of an idea that at least makes some sense and then backtest it. Save some out of sample data to test again and then finally forward test it. Backtesting definitely has its value here. No? Do you mean that you never ever backtest any idea that you come out with but only forward test them? That is too time consuming! What difference does it make if you have no idea what those past data were?
backtest on a portfolio. I think it is silly to quibble over 18 or 21 years. What is important to test a decent basket of futures. (I am assuming that you as a system's trader trade futures as 90% of them do.)
If you need to backtest on the SP or a dow it may make sense to test certain periods included - like 1987, and even the 70's - I can see why a much longer timeframe could be used. I however do not believe in systematic trading of stock indices and for me to explain and delve into why is too long.
If you need to backtest on the SP or a dow it may make sense to test certain periods included - like 1987, and even the 70's - I can see why a much longer timeframe could be used. I however do not believe in systematic trading of stock indices and for me to explain and delve into why is too long.
I keep seeing "forward testing" but I'm missing something: are you basically just saying it's trading based on sound reason, logic, knowledge, etc.? In other words, no computer simulation whatsoever?
Yes.
What has happened to me is similar to what has happened to mathematics since the turn of the last century, 1899-1900.
Before Reimann, most mathematics was about this formula or that formula (althouth there are exceptions, like Galois before him.) In other words, mathematicians were interested in a particular prime, or some other paculiarity.
Reimann changed all that. He started mathematics on a path where you would be more interested in the entire geometry of a problem - it's topological embedding if you like - or the infinite _landscape_ of a given problem.
Although I value experimental tinkering with the computer as much as anyone else, it is the pattern(s)/geometry that emerge from _ALL_ possible profitable trading systems that I am interested in. The computer then becomes another tool, not the primary one.
nitro
What a moron.
Wow! I haven't heard the name Riemann in years...
Makes sense. We all have to realize even with backtesting that we're probably just "sampling" over all possible geometries...