Backtesting Sufficiency

A few sites you might find interesting in addition to this one:

www.wealth-lab.com
www.tradestation.com
www.smartquant.com
www.wilmott.com
www.neoticker.com

and of course I highly recommend googling your favorite search terms and filtering for pdf files to get access to various related research articles.

Lastly, you might try to get some old articles from IB research groups if you can hook up with someone that works there or has a friend that does...

Best of luck. It's a long process.
 
Quote from damir00:

you're asking the wrong person. if you look back at other discussions i've had on ET you'll see that - as far as i'm concerned - there is no true out of sample data except that which has yet to be generated. the entire premise of walk-forward is IMO flawed (when it comes to the market) and you're better off just test on the entire data set at once and spending the time in frame-by-frame analysis to see where the hiccups happen.

this is not a popular position on ET. :)
More...

That is exactly what I am doing. It is not because of any particular belief about the math of the situation, it is just the only way I can do it, seems good so far. I literally do load up 1000 issues but I only test over 1 year's data. I want to be able to relate to the condition the overall market was in. Testing over a whole market cycle makes no sense to me unless you can pull out a system for each different phase.

I really think the best "out of sample" data is to trade the system for real. You can then get a better idea about slippage, non-fills of your limit orders, etc. and go back and reiterate the testing.

Max
 
The point about the one bear and bull market is to see how the system performs under different market condition. This is important because what happens if the system that you just developed and only tested since Jan 2000 on the nasdaq , would have lost more money than it made from 1996 -2000. If we test it on both the bear and bull market and it performs flat during the bull market that ok, at least we have an idea of how it performs under these conditions.

Market do change and sometimes, you can't test a system all the way back to the beginning of the data, but its up to the system for example a bond system which did not work before 1988 is ok because the bond market opening times changed and after 1988 Thirty year treasury opened before the big 8:30 reports come out. If it started working well in 1988 we can explain that. If it worked going back to 1983 that great too.
 
Quote from Lefty62151:

At this point in time, they are approaching the problem from another direction. I don't want to bore you with the details. What I willl say is that there have been a few posts relating to how fit a program to a market and vice versa. The posts were largely ignored. I assume that is because to make use of the info, you would need to have a background in statistics and programming. This is probably why institutions are willing to pay people who graduate from top 10 schools with PHds in math, statistics, computational methods and computer science so much money.
More...

I have the background, and would not be bored with the details, to the contrary. Can you point in right direction?

PM me if you'd rather.
 
I AM SURE A BEAR MARKET NEVER ACTS LIKE A BULL ONE
 
Quote from Murray Ruggiero:

I AM SURE A BEAR MARKET NEVER ACTS LIKE A BULL ONE
More...

Are you the Murray that developed I-Master? If so how is it working these days.

Cheers.
 
Gorgonzola:

I noticed that you planned to use $.07 for slippage. You did not mention what type of backtesting platform you were going to be using - homegrown or commercial.

I don't have my notes handy but I considered looking into two things - 1) the impact of moving from fractions to decimals on multi-year backtesting validity and 2) if not restricting trading to stocks in a limited price range, the change in slippage due to price and that impact on any given strategy.

If you are designing your own backtesting engine you could handle these items readily. In any of the commercial packages with which I am familiar you must compromise if you want to test a strategy across a range of equities/instruments. Depending on what you try to do you may or may not need to take my musings into consideration. I haven't a clue whether they are meaningful although I obviously think they might be.
 
Yes I am the Murray Ruggiero who has been contributing editor for Futures Magazine for over a decade. I also co-developed I-Master with Keith Fitschen. The system did well for two years after release because volatility was high. The system has had flat performance over the past year or so. This is because volatility has been low. Look at the VIX over the past 12-18 months. The performance of the system is highly correlated with that.
Even with the flat performance it is still very highly rated in Futures Truth, still in top ten all time. If you look at the long-term performance of any system, they all have flat period and drawdown. I-Master is a sound system and will come back, when volatility increase.

The stock indexes are a hard series to work with because of the creation of the mini contracts 1997, cutting the SP500 in half as well as globex trading. In this case the older data might not apply, since for example the pervious low volatility period, like 1993,1994 the mini and globex did not come into play.

In addition I am also chief consultant for TradersStudio and designed the product, a trading system platform which has been discussed in a few threads on this site.
 
Quote from Murray Ruggiero:

Yes I am the Murray Ruggiero who has been contributing editor for Futures Magazine for over a decade. I also co-developed I-Master with Keith Fitschen. The system did well for two years after release because volatility was high. The system has had flat performance over the past year or so. This is because volatility has been low. Look at the VIX over the past 12-18 months. The performance of the system is highly correlated with that.
Even with the flat performance it is still very highly rated in Futures Truth, still in top ten all time. If you look at the long-term performance of any system, they all have flat period and drawdown. I-Master is a sound system and will come back, when volatility increase.

The stock indexes are a hard series to work with because of the creation of the mini contracts 1997, cutting the SP500 in half as well as globex trading. In this case the older data might not apply, since for example the pervious low volatility period, like 1993,1994 the mini and globex did not come into play.

In addition I am also chief consultant for TradersStudio and designed the product, a trading system platform which has been discussed in a few threads on this site.
More...

I am unworthy!!
 
You must log in or register to reply here.
Back
Top