better way to do Out-of-sample test?

[PocketChange]

Your perspective is on point. The game is only getting faster and the data is volumous. The actual back testing or forward tested trade results are less important but an overall series of results are useful to test the money and risk management aspects of a trading system.

This is also a dangerous area for trading system providers because as soon as your strategy adapts to a subscriber's position and meta data the CFTC considers it as providing individualized trading advise. A very slippery slope.

From an IT perspective today there are very powerful computing clouds previously inaccessible to retail traders. A retail trader could test every possible combination of a strategy over 10 years of level II / tick data and set it to run across 1000 cores producing comprehensive results in minutes for less than $100.

This type of comprehensive testing environment is complex and beyond the capabilities of the average retail trader but as these strategies evolve the existing testing methodologies and tools need to throttle up a few levels.

Trade what ever strategy you want moving forward but if you are capable it would be worth your effort to accurately run a simulation to see what types of results would have been possible in the past. Testing is just due diligence...

Quote from Rabbitone:

In this thread and other threads I am beginning to hear rumblings from traders who are building dynamic hybrid strategies that they require different testing approaches. For example Pocket Change states:

“Unless your strategy is dynamic adjusting to market changes of course you are going to have different results.”

Or Vikana states

“I eventually settled on a different way to optimize. In my opinion the point is not to "optimize" but to "stabilize" and I therefore approach the problem as follows….”

I have spent most of my professional life programming and testing IT software. Since I retired to trade full time in 2002 (after 37 years of IT) I have developed and tested hundreds of different strategies and found some of the old maxims of walk forward testing useless for my type of dynamic hybrid swing trading strategies and the time intervals I trade in.

I now believe we are entering a different era in retail software programming and testing. Many of the simple trading strategies that were first coded in the 1990s to trade simple set ups have evolved and given way to dynamic hybrid strategies that are well aware of price, volume and volatility metadata and can adapt to changing market conditions.

Many retail traders like me are injecting “qunat” or artificial intelligence into strategies to make them adaptive. When I did this I had to question whether traditional walk forward testing was still relevant given the nature of the programs being produced. Much of testing today is more to finding a stabilized strategy to trade with (rather than an optimized strategy) as Vikana stated.

But I still notice most of these discussions on testing in ET still revolve around 1990’s straight set up strategies that are static and require a multitude forward testing periods and validation before they can be put into a production environment making profits or they will be just another “curve fit.”

I believe we need new testing methods to accommodate this changing environment. But to some in the earth is flat group what I have written is hypocrisy. For them testing is fixed and rigid. They may be right. For trading the ES the old ways may be the only way. But I live off other testing methods for trading stocks and these new methods work for me and that is all I care about.

So I’m curious how many of you have gone beyond yesterday’s simple walk forward testing procedures?

 

[TraderZones]

Quote from intradaybill:

Are you a psycho or what?
psychos like you ...
insulting attitude?
You are probably an idiot...
has never traded a single $ or who blew up in a matter of 10 days ...
takes his anger here ...
I can do is say screw you to you...
idiots like you...
You should really consider visiting a competent psychiatrist to receive treatment...
[/QUOTE]

Yes, you are showing the cordial, noninsulting, well-intentioned person, gentile, mature person you are here, as opposed to what you claim I am.

I made a coherent, thoroughly proper argument twice. You did not understand it twice, and you responded with blank stares, flames, fumes, even though what I said was thoroughly accuract to any experienced trader who has backtested.

never come back to this forum again.

That would be a favor to us all. Don't let the door hit your backside on the way out...

 

[dwpeters]

Quote from mizhael:

Any thoughts on testing the robustness of an optimized trading strategy?

And any thoughts/pointers about discovering trading strategies that can go well in all cycles, e.g. doing well in-sample (data prior to 2008) and also perform well on the out-of-sample 2008/2009?

Thanks a lot!

... and back to the original question.

I know of some very good strategies that did poorly during the worst of the bear market but then went right back to making significant profits after. Rather then throwing these away to look for the elusive strategies that do well in all cycles, look for a filter that can help you avoid the times when your strategy doesn't do so well, and then stop trading it during those times, or at least reduce your exposure.

A simple example is to avoid longs when the S&P 500 is below the sma(200). This strategy is older then dirt and yet it did quite well on avoiding much of the downside. Somewhat late getting back in but you can do similar things on all kinds of timeframes. As with adjustments to the strategy itself you have to be careful of curve fitting.

 

[mizhael]

Quote from Craig66:

So in your findings, it is not always the case that a larger training sample produces better out of sample results?

Of course not always larger training sample produces better out-of-sample results...

Think of regime switching...

 

[mizhael]

Quote from ButterMilk:

I have an automated system that trade futures. The size of the training sample varies with what is being traded and changes over time. The shortest I've seen was 36 trading days, the longest being 160 days...remember that's just one of the variables that is optimized.

So you optimize each variable separately, and for each variable, you have different length of training samples?

 

[mizhael]

Quote from intradaybill:

I have a system I developed in 1993 and it still makes money.

All universally qualified propositions, like yours, are proven false in the face of a counter-example.

May I ask what are the drawdowns of your 1993-system in 2001-2002 and 2007-2008 periods?

 

[mizhael]

Quote from Rabbitone:

In this thread and other threads I am beginning to hear rumblings from traders who are building dynamic hybrid strategies that they require different testing approaches. For example Pocket Change states:

“Unless your strategy is dynamic adjusting to market changes of course you are going to have different results.”

Or Vikana states

“I eventually settled on a different way to optimize. In my opinion the point is not to "optimize" but to "stabilize" and I therefore approach the problem as follows….”

I have spent most of my professional life programming and testing IT software. Since I retired to trade full time in 2002 (after 37 years of IT) I have developed and tested hundreds of different strategies and found some of the old maxims of walk forward testing useless for my type of dynamic hybrid swing trading strategies and the time intervals I trade in.

I now believe we are entering a different era in retail software programming and testing. Many of the simple trading strategies that were first coded in the 1990s to trade simple set ups have evolved and given way to dynamic hybrid strategies that are well aware of price, volume and volatility metadata and can adapt to changing market conditions.

Many retail traders like me are injecting “qunat” or artificial intelligence into strategies to make them adaptive. When I did this I had to question whether traditional walk forward testing was still relevant given the nature of the programs being produced. Much of testing today is more to finding a stabilized strategy to trade with (rather than an optimized strategy) as Vikana stated.

But I still notice most of these discussions on testing in ET still revolve around 1990’s straight set up strategies that are static and require a multitude forward testing periods and validation before they can be put into a production environment making profits or they will be just another “curve fit.”

I believe we need new testing methods to accommodate this changing environment. But to some in the earth is flat group what I have written is hypocrisy. For them testing is fixed and rigid. They may be right. For trading the ES the old ways may be the only way. But I live off other testing methods for trading stocks and these new methods work for me and that is all I care about.

So I’m curious how many of you have gone beyond yesterday’s simple walk forward testing procedures?

Look, no matter how adaptive and smart dynamic your trading strategies are, you still need to do forward-test before you put them into real-trading, right?

A static out-sample forward-test is the minimal due-diligence.

A walk-forward test is still applicable even when you use adaptive trading strategies.

And while I see the point of emphasizing stability of parameters/variables, I still think as a trader you still optimize in the sense that you want to minimize drawdown...

 

[mizhael]

Quote from dwpeters:



... and back to the original question.

I know of some very good strategies that did poorly during the worst of the bear market but then went right back to making significant profits after. Rather then throwing these away to look for the elusive strategies that do well in all cycles, look for a filter that can help you avoid the times when your strategy doesn't do so well, and then stop trading it during those times, or at least reduce your exposure.

A simple example is to avoid longs when the S&P 500 is below the sma(200). This strategy is older then dirt and yet it did quite well on avoiding much of the downside. Somewhat late getting back in but you can do similar things on all kinds of timeframes. As with adjustments to the strategy itself you have to be careful of curve fitting.

This is really a great piece of advice, because all-weather profitable strategies are hard to come by.

Following your idea, maybe I should study more into the regime of trading strategies.

In addition to SP500, how to dig deep into the performance of a trading strategy and find out when the broad market is friendly to my trading strategy and when it is not?

I noticed that my systems got killed when the Greece problem surfaced, when the GS problem surfaced, etc. (These are real money.) Almost all my positions went against me. Correlations seemed to be approaching 1, and diversities were completely gone. But SP500 seemed to be okay, and even advanced on my black days.

Any thoughts?

 

[dwpeters]

Quote from mizhael:

This is really a great piece of advice, because all-weather profitable strategies are hard to come by.

Following your idea, maybe I should study more into the regime of trading strategies.

In addition to SP500, how to dig deep into the performance of a trading strategy and find out when the broad market is friendly to my trading strategy and when it is not?

I noticed that my systems got killed when the Greece problem surfaced, when the GS problem surfaced, etc. (These are real money.) Almost all my positions went against me. Correlations seemed to be approaching 1, and diversities were completely gone. But SP500 seemed to be okay, and even advanced on my black days.

Any thoughts?

Thanks. I haven't really taken this beyond alternative time frames as I suggested. I have tested some alternatives but with limited success. I've found very few factors that have a significant and consistent affect on my returns, and the trend of the broad market is one of them.

I went thru a period in late Jan/early Feb where it seemed I could do nothing right. My longs went down and my shorts went up. I can not explain why this happened much less predict it in the future. (well I know the longs went down with the market but why my shorts underperformed I can't explain) In this case I took advice from the book Trading Risk and reduced my position size until my strategies were working better. My losses were very limited in February and by March I was able to raise my position size and I had a good March and in April it seemed I could do no wrong - I had an awesome month.

Here is a blog that talks about regime switching. This article is the last of 7 but gives something of a summary so you can see if you want to go back to read from the beginning. Perhaps it will give you ideas:
http://www.portfolio123.com/blog.jsp?postid=71&topic=idea

And here is a blog post on coping with abnormal markets. This fellow uses adaptive, rather then static strategies and still finds some method is required to cope with abnormal markets:
http://marketsci.wordpress.com/2008...-coping-with-abnormal-markets-shades-of-grey/

The problem with this latter approach is that abnormal markets are by definition rare, so not a lot of cases for testing.

 

[mizhael]

Quote from dwpeters:



... and back to the original question.

I know of some very good strategies that did poorly during the worst of the bear market but then went right back to making significant profits after. Rather then throwing these away to look for the elusive strategies that do well in all cycles, look for a filter that can help you avoid the times when your strategy doesn't do so well, and then stop trading it during those times, or at least reduce your exposure.

A simple example is to avoid longs when the S&P 500 is below the sma(200). This strategy is older then dirt and yet it did quite well on avoiding much of the downside. Somewhat late getting back in but you can do similar things on all kinds of timeframes. As with adjustments to the strategy itself you have to be careful of curve fitting.

I am rereading your note and I just had another question:

When do you turn the longs on?

--> "avoid longs when the S&P 500 is below the sma(200)."