Successful strategies, holy grail and backtesting

[MGJ]

I'd suggest you pay the money to purchase "deep history" data going back as far as you can. This will allow you to experiment with In-Sample / Out-Of-Sample optimization.

You begin by assuming: Today is 31 December 1993 (or any other day you select). You optimize your trading system on all historical data up to and including today (12/31/1993). You will call this the "In-Sample" data set. Then after the optimization is complete and you have chosen an optimum system, you will try it on data it has never seen, data from 1/1/1994 through 10/25/2010: the Out-Of-Sample data.

You will quickly feel the sting of disappointment and, occasionally, the joy of absence-of-disappointment, when you look at the out-of-sample results. Here are a couple examples that I screen grabbed off another site. The In-Sample period was 1/1/1973 through 12/31/1993, a period of 21 years. Think of it! These two systems were optimized over twenty one years worth of data! How could you possibly curve fit a system to 21 years x 260 days/year x 31 markets = 169,260 market-days worth of daily data? These systems HAVE TO be robust, right? Glance at the pictures to get an answer. The vertical line divides the In-Sample period (before 1/1994) from the Out-Of-Sample period (after 1/1994)

 

[silent_tunes]

Thanks MGJ

Of all the replies yours was at least a bit helpful; I did not think about the in sample/out of sample method. But after backtesting, when I paper traded and even actual money traded my method, it was moderately successful. But using a date in the past as the cutoff sounds interesting. I will try it out.

My original question remains - any firms that will take me on board based on strategy backtesting/paper money trading results?

Any trading contests out there that are legit? Sort of like a tryout/audition for trading firms? This can't be any different from American Idol - record companies are spending millions to launch a new voice. Isn't that a big risk? Think of my situation as similar to a power arm looking for a minor league contract !!

 

[abattia]

Quote from nLepwa:

... Nobody (except friends&family) will fund you on backtesting results alone...

 

[silent_tunes]

Also consider this about backtesting: as is familiar to anyone who has read a mutual fund prospectus "past performance does not guarantee future results". Very true.

However, any new strategy or "aha" moment has to be tested for validity. Otherwise it is just an idea, nothing more. A serious trader cannot be expected to risk actual money on an untested idea, even if that testing is only with historical data.

Also, it is entirely possible that a strategy that backtests successfully may not be entirely profitable going forward. One reason could be timeframe. In other words, if I tested my strategy using data from the last 20 years and it works great; now I trade it with actual money for 6 months or a year and it gives subpar performance. The reason could be different timeframes. A proper comparison is possible only after running this strategy for the next 20 years or at least a few years !! (Assuming I am still in the game using that strategy !!)

Also, tried and tested strategies could fail over short or even longer time frames. For example, buy and hold over the last 10 years has given miserable results since you would barely break even.

Also, say I was totally correct about the Internet bubble or the real estate bubble and shorted dotcom stocks in 1998 or RE firms in 2006. I would still be stop lossed out or totally wiped out by the time the market realized its folly (remember the famous maxim about markets being irrational longer than the investor being solvent). In other words, a correct strategy still has to be combined with the right timeframe to evaluate the results.

That is why I consider it more important to have not just 1 strategy but a few strategies using different timeframes, different markets, different instruments that are either uncorrelated or negatively correlated that you expect one or more (if not all) of the strategies to be successful at any given time and slowly but surely grind out singles.....

 

[HowardCohodas]

Quote from silent_tunes:

I did not think about the in sample/out of sample method.

As I mentioned in an earlier post, out of sample testing is a core part of my approach. One of the reasons I gave you the references to some texts on system development is so that you can learn the compromises associated with different approaches to out of sample testing.

A cutoff date is my least favorite out of sample testing method but is better than no out of sample testing.

 

[vikana]

Quote from silent_tunes:

...

Also, my strategies require low commissions, better trading speed, good executions/fills (in other words, a retail account trading online is not optimal). The backtesting has been done to account for real world glitches (position sizing so as not to move markets, risk management, stop losses etc.)

My question is whether there are firms out there that will let me trade my strategies real time (maybe with paper money to begin with) and if my trading results are profitable, they will let me come on board to trade real money.

A few thoughts:

1) It's very easy to fool yourself with back-testing (I've successfully done it multiple times). Unless you have actual real-time trades, it's hard to know how real it is

2) If you don't have the faith to trade your own funds, why do you expect someone else to fund you? You'll do *much* better if you can put up a 6 months record, even if its just trading 100s at a time.

3) I seriously suggest you question your assumption about not being profitable retail. If you cannot be profitable with e.g. IB or Genesis, I suspect you won't be profitable with a prop firm. I do know a few *outstanding* traders who needs sub IB pricing to be profitable, but it's not many, and they trade on a scale that most of us never reach.

Hope this helps

 

[HowardCohodas]

Quote from silent_tunes:

However, any new strategy or "aha" moment has to be tested for validity. Otherwise it is just an idea, nothing more. A serious trader cannot be expected to risk actual money on an untested idea, even if that testing is only with historical data.

Excellent!

To elaborate on requirements before putting a strategy into production.

• Back testing, like any computer simulation of the real world (think engineering) has its limitations.
  
• Tick data vs. EOD data and Slippage
  Even with tick data, you can't assume the trades you want to make will be executed in a timely fashion or at the price you want.
  
• Multiple instruments applications
  If your system applies to more that one trading instrument, then it must be tested over several as well.
  
• Out of sample testing
  With backtesting you must avoid a phenomena known as "curve fitting." This is done by dividing your sample into two sets. One set is to hone your system. The second set is to prove the results were not solely dependent on the the set you chose for development.
  
  Since markets change over time, just choosing an arbitrary date to divide the sets may end up disqualifying some systems that should be further evaluated. The multiple subset approach is my preference.
  
• Paper Trading
  Once your computer simulation shows promise, you should go to paper trading. Paper trading, like an engineering prototype, is good for working out the bugs of interfacing with the real world. Even paper trading has it's compromises. Although you will be filled at the price you request if the market trades at that level, no consideration is given to availability.
  
• Preproduction
  Your next step is to trade live (with money) but for small amounts. Consider this preproduction testing.
  
• Full Production
  Live long and prosper.

 

[JoePaterno]

Quote from silent_tunes:

I can prove that you pick any random stock from the S&P 500 and my strategy is simple enough that it will work on that stock. In fact, the criteria for my strategy is any dumb investor (starting with me) can win using a simple strategy.

...........

I did not think about the in sample/out of sample method.

these are painful on many levels...

 

[dwpeters]

Quote from silent_tunes:


To phrase the question clearly, are there firms out there that will take me on board with good backtesting results instead of actual trading results or an MBA in finance?

I suggest you check with Broad Street trading: http://www.broadstreettrading.com/
I question whether they can really get you better execution than Interactive Brokers but they do have low commissions ... but also a $180 or more monthly fee for access to their trading platform.