So far, I still havenât heard anyone quote any rules of thumb for how much testing most successful traders feel comfortable with. All of the books I have seen only tell how to trade their method, not how to run sufficient tests to develop your own. I would be glad to know from any successful traders out there how much backtesting you felt comfortable with before you started trading your system. How much for short term swing trades (i.e. buy and hold for 2-3 hours)? How much for longer swing trades(2-4 days)?
How much backtesting do I need?
- Thread starter pharaoh
- Start date
For a statistically valid result, most traders like at least 30-40 trades as a result of their backtest. Choose your data length accordingly. Personally, I want 80 to 100 trade results for each of my in sample and out of sample tests.
Now, this makes me comfortable with a daily or 60 minute chart. However, if we are talking about a 1 minute chart, I'd want a lot more to test. Here, it would be important for me to see lots of trade results (again 80-100 trades) for each high volatility, low volatility, trending and choppy markets. So that's a case where I'd want to see around 400 trades.
I test both in sample and out of sample, so double some of my estimates above.
Don't just take those 100 trades and assume you are finished though. Look carefully at your trades, making sure they worked exactly as you intended. Also look at some of your big winners and losers: do they fit with your intended method? Is there a problem with your algorithm? Is the data corrupted? Finally, if everything works as intended, can you stand to take that trade? It's often easy to accept results when looking at them on average from the 50,000 foot level. But in reality, you will be living them at the 1 foot level, and sometimes the pain of one or two trades, or a string of losers, can make a good looking system unpalatable.
Now, this makes me comfortable with a daily or 60 minute chart. However, if we are talking about a 1 minute chart, I'd want a lot more to test. Here, it would be important for me to see lots of trade results (again 80-100 trades) for each high volatility, low volatility, trending and choppy markets. So that's a case where I'd want to see around 400 trades.
I test both in sample and out of sample, so double some of my estimates above.
Don't just take those 100 trades and assume you are finished though. Look carefully at your trades, making sure they worked exactly as you intended. Also look at some of your big winners and losers: do they fit with your intended method? Is there a problem with your algorithm? Is the data corrupted? Finally, if everything works as intended, can you stand to take that trade? It's often easy to accept results when looking at them on average from the 50,000 foot level. But in reality, you will be living them at the 1 foot level, and sometimes the pain of one or two trades, or a string of losers, can make a good looking system unpalatable.
I use daily bars, and my standard procedure is to test a system over a period that includes an up market, a down market, and a sideways/rangebound market. We have had all 3 in the past 4 years, so that is currently my benchmark. I look at data going back to about 8/98, or about 1000 days/bars. I want to see results that are at least halfway decent in all of those environments, while at the same time, being really good in at least one of those environments. My next step is to test using 100 days/bars of out-of-sample data. Then, if things look good, I will start to actually risk real $$ with fairly small trades, just to see if I can really follow the signals it generates in real life. If things continue to go well, I will gradually ramp up my size until I reach my usual risk amount. Having said that, someone trading using intra-day charts may be able to get acceptable results with much less data, but I have no experience in that area to be able to render any kind of informed opinion.
I didn't think of it before...
Charles Lebeau has been around awhile. I think they have a
forum, also. You should be able to get plenty of info on system
testing.
Hope this helps.
www.traderclub.com/
Charles Lebeau has been around awhile. I think they have a
forum, also. You should be able to get plenty of info on system
testing.
Hope this helps.
www.traderclub.com/
I agree with most things said on this thread so far.
I also like to remove all trend bias by using random sets of long
and short entries. Then again what do I know
I also like to remove all trend bias by using random sets of long
and short entries. Then again what do I know
DeepHindsight's point about testing on random entries is good. You can go even further with it by synthesizing random price data and backtesting against that.
<b>Backtest with random data</b>You can help gauge if your system is just "lucky" or flawed by backtesting on unbiased synthetic random-walk data. Using unbiased random data lets you gauge the statistical range of results. You can then compare the results from testing on actual data against this statistical result on random data to see if your trading system seems to be getting non-randomly good results. (Testing on unbiased random-walk data will also highlight the presence of some types of flaws in the backtester or trading system).
<b>Generating random OHLC data</b> for those comfortable with creating their own Monte Carlo simulations, you can synthesize random OHLC data by simulating a random-walk at one high frequency bar rate (e.g., take one random walk step per simulated 15 minutes) and then processing that data into daily bars (find the high and low across each day's set of 15-minute bars). Adding an appropriate number of random walk steps overnight simulates the aftermarket, overnight, and premarket action that generates a gap between the close of one day and the opening on the next day. Generating say 20 to 100 of these time series is likely to give you a set of synthetic stock price series that looks like they have nice uptrends, downtrends, and choppy trading ranges (but these are only illusions, since by definition the data is a random walk).
The point is that if you put random-walk data into a statistical analysis or backtesting, you will gain amazing insight into how randomness can fool these analyses. It is impossible to consistently profit from unbiased random-walk data -- although it may certainly seem like some trading systems appear to be very profitable on random walk data some of the time. At the very best, you can craft a lottery-like outcome with a low chance of extremely high profits and a high chance of drawdowns.
<b>The Perils of Future-Leakage</b> If your backtesting suggests that you seem to be consistently profiting with all manner of random-walk price series, it suggests that the backtester or the trading system is flawed. Usually, the flaw in the backtester or system is a "future-leak" in which information about future price action is leaking back into the simulated trading system's decisions.
Wishing you nonrandomly good trading returns,
Traden4Alpha
<b>Backtest with random data</b>You can help gauge if your system is just "lucky" or flawed by backtesting on unbiased synthetic random-walk data. Using unbiased random data lets you gauge the statistical range of results. You can then compare the results from testing on actual data against this statistical result on random data to see if your trading system seems to be getting non-randomly good results. (Testing on unbiased random-walk data will also highlight the presence of some types of flaws in the backtester or trading system).
<b>Generating random OHLC data</b> for those comfortable with creating their own Monte Carlo simulations, you can synthesize random OHLC data by simulating a random-walk at one high frequency bar rate (e.g., take one random walk step per simulated 15 minutes) and then processing that data into daily bars (find the high and low across each day's set of 15-minute bars). Adding an appropriate number of random walk steps overnight simulates the aftermarket, overnight, and premarket action that generates a gap between the close of one day and the opening on the next day. Generating say 20 to 100 of these time series is likely to give you a set of synthetic stock price series that looks like they have nice uptrends, downtrends, and choppy trading ranges (but these are only illusions, since by definition the data is a random walk).
The point is that if you put random-walk data into a statistical analysis or backtesting, you will gain amazing insight into how randomness can fool these analyses. It is impossible to consistently profit from unbiased random-walk data -- although it may certainly seem like some trading systems appear to be very profitable on random walk data some of the time. At the very best, you can craft a lottery-like outcome with a low chance of extremely high profits and a high chance of drawdowns.
<b>The Perils of Future-Leakage</b> If your backtesting suggests that you seem to be consistently profiting with all manner of random-walk price series, it suggests that the backtester or the trading system is flawed. Usually, the flaw in the backtester or system is a "future-leak" in which information about future price action is leaking back into the simulated trading system's decisions.
Wishing you nonrandomly good trading returns,
Traden4Alpha
Even better.