Backtesting and Strategy Validation - How Much is Enough?

B

bhause

Hi all,

I have been trading stocks and crude oil (through leveraged ETFs) now for about 6 months. Although I don't have a formal finance education (just graduated with a degree in econ - but as many of you surely know, econ is NOT finance), I have been studying finance and the markets ravenously since my interest was first piqued 8 months or so ago. I have done well so far in the markets. I am thinking that it is time to step up to the big leagues, and I want to trade crude oil futures intraday.

After reading a lot about the common mistakes that day traders make, I decided that I would not begin trading until I had a strategy that I knew would work - something that has been backtested and demonstrated positive expectancy.

Well, I have spent a ton of time working on and studying different indicators over the past month, and I have found a strategy that seems like it would work so well, that it is down right scary.

According to the back test of 1.5 months of intraday trade data, assuming that past results are an indicator of future performance, I can expect to win about 60% of my trades with an average win of 80 ticks ($800 on 1 contract) and an average loss of about 45 ticks. This yields an expectancy of about 30 ticks, or $300/trade. If the trade volume going forward is anything like it has been over my test period, such results would yield annual pre tax earnings of nearly $200k by trading only 1 crude futures contract at a time. Trade 3 and thats over half a million dollars a year.

I base my trading decisions on a single indicator, so I have no risk of data mining, or too many degrees of freedom - my test results are statistically significant. Everything that I study with regards to the system leads me to believe that I should start trading this immediately, except for the idea that I could make nearly 200 k a year trading from my house, on one instrument, on a single contract, based on a single technical indicator. It seems unreasonable, it seems too good to be true.

So my questions to the group are -

1) Does making 200k a year by trading 1 crude oil futures contract seem outlandish? Do people actually have this type of success, and if so, why don't more people do it?

2) Have any of you ever run into a situation where you have an extremely well performing technical indicator that one day simply breaks down? I mean, has the market every changed in a basic way such that you needed to reconsider your entire trading system (I realize this happens all the time from a fundamental standpoint, but I am coming from a technical standpoint)?

3) Even though I have enough observations based on my backtest to render statistical significance, is 1.5 months of trade data too limited to be reliable. Are there any specific stress tests that I should perform on my system?

I'll appreciate any advice, criticism, or suggested reading that you can offer.

Thanks!
 
I trade intraday based on a very simple system. When I discovered it, I was sure that I would be rich in a few short years. Turns out I found it when the market was roaring and the expectation was about five times what it is now. But it still works for pin money. Like you, I started out with what I now consider to be short test durations. I now fine-tune for current market conditions on 120 days of data and keep track of robustness with 253 days of data. There are so many gotchas in backtesting that I could write a book about it. If you haven't seen the books by Bruce Babcock and Robert Pardo, you might look for them. Don't buy, just read. Have you found the Golden Fleece? Probably not. An ordinary wool fleece, maybe. Ask yourself, "Am I so fucking smart that I have found a system that has eluded thousands of human analysts for ten decades, and machine analysis for two decades?"

P.S.: Ain't no such thing as statistical significance in this racket. I'll be happy to sell you any one of a hundred "statistically significant" systems I developed.
 
Hi, bhause

1) Does making 200k a year by trading 1 crude oil futures contract seem outlandish? Do people actually have this type of success, and if so, why don't more people do it?

Anyone who owns a Senator or an island that isn't a 51st State sells crude they had delivered but never sell. More people are doing it is the point.

2) Have any of you ever run into a situation where you have an extremely well performing technical indicator that one day simply breaks down? I mean, has the market every changed in a basic way such that you needed to reconsider your entire trading system (I realize this happens all the time from a fundamental standpoint, but I am coming from a technical standpoint)?

Test, retest, and price satisfaction. Every indicator has those three phases. If you can't recognize when your indicator behaves inversely, instead of reevaluating one strategy over another, you might try to attempt to pin-point data anomalies when prices shift to the opposite extreme. For example: Prices won't rise or fall without a range break. An action preceding the wanted move. Say crude's at 98 in a bear and the range of the last bar was 17 points. A tick of action past the mean of that range is indicative of printing a better price, of which then is already considered an overvaluation. News players create noise but realistically, it's our responsibility to know which side is bias bear or bull by matter of ticks. In the S&P 500, by my research, lows beat highs by two ticks. In other words highs are faded; lows are pounded. What you want to be able to do with your indicator is forecast when highs are pounded and lows are faded.

3) Even though I have enough observations based on my backtest to render statistical significance, is 1.5 months of trade data too limited to be reliable. Are there any specific stress tests that I should perform on my system?

Raw data is already price to the mean, by market makers, residual self-prophesy and/or trend-setting old hands. Which is to say price action is a result n number of days, months, years or decades of data being market satisfied look-back (in the past), based on x factor of periods statistically weighting more in research houses. Crude oil delivers every month. So a month and a half of data seems a starting point. But in order to validate anomalies when the market reverses, you likely will have to at least have a half of decade of data. Not a problem on its face using third-party charting solutions. The rub is when a decade isn't enough. A decade's worth of data is to the letter of it 5 years forward or back.

To start, every white paper written by every budding EOD duo with too much vocabulary and not enough sense out of an Ivy Leauge school can be reduced to '3-6 months of data per 1 month's replication'.
 
Quote from bhause:

1) Does making 200k a year by trading 1 crude oil futures contract seem outlandish?
More...

Your question sounds like: "is marriage a good thing?". Well, it depends on many factors. What you say can be done and some do it. Also, this reminds me of the question: "Is it possible to win the lottery?". Of course it is, however your chances are extremely low.

Quote from bhause:

2) Have any of you ever run into a situation where you have an extremely well performing technical indicator that one day simply breaks down?
More...

This is a very common situation. Indicators usually have parameters and although the indicator may be a good one, its performance varies based on the parameter values. You can back test almost any indicator for high expectancy but that would apply to a specific parameter set.

Quote from bhause:

3) Even though I have enough observations based on my back test to render statistical significance, is 1.5 months of trade data too limited to be reliable. Are there any specific stress tests that I should perform on my system?
More...

You are nowhere near statistic significance with 1.5 months of data. Significance depends on data history but also on time history. You should test your strategy on 2 years of intraday data.

Also, like all beginners I think you place too much emphasis on backtesting. This type of analysis cannot tell you what will work. It can only tell you what may not work within the limits of Type II errors. I think you should read this blog post. The author is an expert. Read it carefully. It may save you a lot of time and money.

http://www.priceactionlab.com/Blog/2010/10/proper-use-of-back-testing/

Good luck.
 
Backtesting and statistical things are done in far too a "sterile" environment. Things happen all the time that affect different segments of the trading community differently, and they all affect prices.
 
You can datamine/curve fit just fine with only one indicator. Especially on such a short timeframe.

Pretty much everybody who ever starts backtesting finds little miracles like these. They're easy to find, that's the nature of backtesting. And none of them work going forwards.

I suggest extreme caution.

Ask yourself this - how is it you with your piddly resources could find something all the PhDs and supercomputers on wall street couldn't find?
 
Quote from bhause:

Hi all,

I have been trading stocks and crude oil (through leveraged ETFs) now for about 6 months. Although I don't have a formal finance education (just graduated with a degree in econ - but as many of you surely know, econ is NOT finance), I have been studying finance and the markets ravenously since my interest was first piqued 8 months or so ago. I have done well so far in the markets. I am thinking that it is time to step up to the big leagues, and I want to trade crude oil futures intraday.

After reading a lot about the common mistakes that day traders make, I decided that I would not begin trading until I had a strategy that I knew would work - something that has been backtested and demonstrated positive expectancy.

Well, I have spent a ton of time working on and studying different indicators over the past month, and I have found a strategy that seems like it would work so well, that it is down right scary.

According to the back test of 1.5 months of intraday trade data, assuming that past results are an indicator of future performance, I can expect to win about 60% of my trades with an average win of 80 ticks ($800 on 1 contract) and an average loss of about 45 ticks. This yields an expectancy of about 30 ticks, or $300/trade. If the trade volume going forward is anything like it has been over my test period, such results would yield annual pre tax earnings of nearly $200k by trading only 1 crude futures contract at a time. Trade 3 and thats over half a million dollars a year.

I base my trading decisions on a single indicator, so I have no risk of data mining, or too many degrees of freedom - my test results are statistically significant. Everything that I study with regards to the system leads me to believe that I should start trading this immediately, except for the idea that I could make nearly 200 k a year trading from my house, on one instrument, on a single contract, based on a single technical indicator. It seems unreasonable, it seems too good to be true.

So my questions to the group are -

1) Does making 200k a year by trading 1 crude oil futures contract seem outlandish? Do people actually have this type of success, and if so, why don't more people do it?

2) Have any of you ever run into a situation where you have an extremely well performing technical indicator that one day simply breaks down? I mean, has the market every changed in a basic way such that you needed to reconsider your entire trading system (I realize this happens all the time from a fundamental standpoint, but I am coming from a technical standpoint)?

3) Even though I have enough observations based on my backtest to render statistical significance, is 1.5 months of trade data too limited to be reliable. Are there any specific stress tests that I should perform on my system?

I'll appreciate any advice, criticism, or suggested reading that you can offer.

Thanks!
More...

There are so many inherent flaws in backtesting. What software are you using?

Are you saying that you have only backtested on 1.5 months of data? Tick data.. or?

If it is just 1.5 months of any data sorry to say it isn't close to significant.

I recommend looking into getting historical equity prices of large cap stocks and doing some research. This research will take years and not days or months.
 
Quote from Random.Capital:



Ask yourself this - how is it you with your piddly resources could find something all the PhDs and supercomputers on wall street couldn't find?
More...


supercomputers on wallstreet? you must be kidding. most of banks/wall street trade with E M O T I O N S!!!
see jerome kerviel, ltcm, madoff and legions of other m##fckers.

phds? these are one of the most clueless people who engage in trading.
they could for example perform 50 advanced methods and spend 2 yrs testing them, where nothing of it makes basic sense. seen that too many times.
 
You must log in or register to reply here.
Back
Top