backtest for 3 years, blow up in 3 days,
- Thread starter jacksmith
- Start date
You may want to look into Monte Carlo testing. It randomizes your back testing to go give you more realistic results.
In the end it's all about emotion and psychology.
Before you can control your system, your methods, your strategy, you need to control yourself
In the end it's all about emotion and psychology.
Before you can control your system, your methods, your strategy, you need to control yourself
The chance is only 10/(9 x 365). Because the rainy days, and dates has no fundamnetal relation to John's behavior as far as we know.
-it is a very good example how can mislead, distort the estimation if we taking into account unrelevant facts.
"Unrelevant facts" like indicators based on the very same market behaviors, crossrelated market indexes, etc.
First, depending on the time frame and frequency of your model three years may not be enough time.
Second, if it blew out in three days it sounds like there was some serious flaws in the risk management logic that went into the model.
Food for thought.
Second, if it blew out in three days it sounds like there was some serious flaws in the risk management logic that went into the model.
Food for thought.
Here's some food for thought. Go the Kelly account simulator and fill in 100 for the "Lines Qty" and run it a few times...
http://www.hquotes.com/tradehard/simulator.html
With the defaults it is entirely a random strategy and it will display 100 different accounts trading the same random strategy. Notice that some are doing great from the start and some are doing terribly from the start. Think about it, a strategy could do great in-sample, maybe look half as good out-of-sample and blow up your account pretty easily and still be entirely random... it could even start out making money on a real market and still be entirely random!
http://www.hquotes.com/tradehard/simulator.html
With the defaults it is entirely a random strategy and it will display 100 different accounts trading the same random strategy. Notice that some are doing great from the start and some are doing terribly from the start. Think about it, a strategy could do great in-sample, maybe look half as good out-of-sample and blow up your account pretty easily and still be entirely random... it could even start out making money on a real market and still be entirely random!
(Don't hate me for this)... but I've written several times why back testing simply cannot work. It "might" help you figure out what cannot work, but cannot tell you what "will" work.
There are simply too many variables involved that cannot be programmed/duplicated... things like.... hmm? Did the gov't decide to blow up Lehman brothers? Where was the PREM/DISC top FV at time of entry on June 11, 2004? 9/11 events. Peers CEO problems, Drug interactions for Pharms/Insurance co's (Tylenol etc.).... but not just event driven movements.
I've seen too many smart people so disappointed over the last few decades when their months or even years of back testing blew up in their face...often they simply were not aware of something that "could" be programmed in, but in their defense, they were trying to do something that I feel cannot be done.
Even going back to basic statistics.... our entire universe of recorded trading is a very small sample. And the "rotational" nature of index tracking is so mis-leading that even trying to replicate the entire market moves is impossible. For example (the most obvious lie to the investor) - "the market returns 10% historically, LOL... oh yeah? What market? The original Dow 30 or the other hundred stocks that were bankrupt and replaced? If you bought the original 30, you would have GE left, and maybe not for long, LOL.
All that being said, we do use historical charting for pairs analysis but only after digging deep into the fundamentals of each stock. Many pairs revert to the mean historically, and can give you an idea that a repetitive reversion might take place..but, again, which stock in the pair is the most subject to being taken over? Yes, this is taken into consideration when trading pairs, and a reason for trading multiple pair for risk flattening purposes.
FWIW,
Don
There are simply too many variables involved that cannot be programmed/duplicated... things like.... hmm? Did the gov't decide to blow up Lehman brothers? Where was the PREM/DISC top FV at time of entry on June 11, 2004? 9/11 events. Peers CEO problems, Drug interactions for Pharms/Insurance co's (Tylenol etc.).... but not just event driven movements.
I've seen too many smart people so disappointed over the last few decades when their months or even years of back testing blew up in their face...often they simply were not aware of something that "could" be programmed in, but in their defense, they were trying to do something that I feel cannot be done.
Even going back to basic statistics.... our entire universe of recorded trading is a very small sample. And the "rotational" nature of index tracking is so mis-leading that even trying to replicate the entire market moves is impossible. For example (the most obvious lie to the investor) - "the market returns 10% historically, LOL... oh yeah? What market? The original Dow 30 or the other hundred stocks that were bankrupt and replaced? If you bought the original 30, you would have GE left, and maybe not for long, LOL.
All that being said, we do use historical charting for pairs analysis but only after digging deep into the fundamentals of each stock. Many pairs revert to the mean historically, and can give you an idea that a repetitive reversion might take place..but, again, which stock in the pair is the most subject to being taken over? Yes, this is taken into consideration when trading pairs, and a reason for trading multiple pair for risk flattening purposes.
FWIW,
Don
Wow Don, I'm surprised by your lack of knowledge and depth on this.
Backtesting isn't supposed to "tell you what 'will' work." Nothing can. But done right, it will tell you what's likely to work. Which is the best humans can do. There will always be 9/11 events which other (non-backtested) strategies are vulnerable to also, BTW. Yes, the future's unpredictable...
Would you have benched Ted Williams because he wasn't guaranteed to get a hit every time? If you truly understood backtesting and thus were able to actually discern "Ted Williams class" technical trading patterns (as opposed to just thinking you could), would you use them? The "smart people" you mentioned who can't backtest probably can't hit a 100 mph fastball either but that doesn't mean it can't be done.
One can't use charts and also rationally dismiss backtesting because both use past data... only backtesting is more objective. So you contradicted yourself when you said you use charts and mentioned reversion to the mean... despite your caveat about "digging deep into the fundamentals."
Backtesting isn't supposed to "tell you what 'will' work." Nothing can. But done right, it will tell you what's likely to work. Which is the best humans can do. There will always be 9/11 events which other (non-backtested) strategies are vulnerable to also, BTW. Yes, the future's unpredictable...
Would you have benched Ted Williams because he wasn't guaranteed to get a hit every time? If you truly understood backtesting and thus were able to actually discern "Ted Williams class" technical trading patterns (as opposed to just thinking you could), would you use them? The "smart people" you mentioned who can't backtest probably can't hit a 100 mph fastball either but that doesn't mean it can't be done.
One can't use charts and also rationally dismiss backtesting because both use past data... only backtesting is more objective. So you contradicted yourself when you said you use charts and mentioned reversion to the mean... despite your caveat about "digging deep into the fundamentals."
I'm sorry if I didn't get deep enough into my explanation it would just take too much detail. And, yes, what you noted as contradiction was simply showing what our traders do use for reference, not a general back test in the standard sense of the term. My main point is that the time I've seen so many traders take with their "testing" could have been put to better use by "doing" - not willy nilly, or random, but a well thought plan in real time can be tested for real, not paper trading, not back testing. I'm not trying to "dis" anyone's ideas, just pointing out that, IMO, much more can be done in less "real time" for a more valuable result.
But, as I always tell new people, do what you think will help you build the confidence you need to execute.
All the best,
Don
But, as I always tell new people, do what you think will help you build the confidence you need to execute.
All the best,
Don
Don, pls no need to apologize here, especially you, a man we have learned so much from. What you wrote is correct from you OWN point of view and style of trading. Some people may not understand why. I also thought backtesting was a panacea and have invested too much time in it until I read the book by Michael Harris Profitability and Systematic Trading, chapter 6.
Basically, Michael Harris, one of the most down to earth trading experts around, argues that backtesting is generally flawed fundamentally because when you backtest a system it is not part of the market and its actions do not affect market prices. What does this mean practically? It means that if your trading method is specifically designed to cause reactions by other market players then it cannot be backtested, as all tests will be bogus. Only trading systems that do not affect market direction are suitable for backtesting. This is logical but many people do not understand it. For example, a long term trend following system makes sense to backtest but an intraday arbitrage system does not make sense to backtest.