better way to do Out-of-sample test?

[TraderZones]

Quote from intradaybill:

I have a system I developed in 1993 and it still makes money.

All universally qualified propositions, like yours, are proven false in the face of a counter-example.

since your glasses seem to have been misplaced: "Most of my strategies that perform well in-sample will be falling like a rock out-sample." ---> EVERYONE...

Most of EVERYONE'S strategies will fall like a rock. Because most theories do not work once you use real data - out of sample.

Your statement

---? I have a system I developed in 1993 and it still makes money. --- ignoring the fact you provided no proof, provided no metrics, and badly misuse "makes money" as even just holding a stock index since 1993 made money, and do not even say whether you traded it the entire time since 1993, or with how much money (in other words, you are naively all wet)

Likely proves what I said. For one, you did not say ANYTHING about your other systems, so it is likely MOST of YOUR systems FAILED. and you only discuss "A" system, which basically reinforces what I said - MOST systems fail for EVERYONE.

Keep up. Your underwear is showing...

 

[kandlekid]

Quote from mizhael:

If I divide data into in-sample (up till Sept. 2007) and out-sample (from Sept. 2007 to 2010),

Most strategies that perform well in-sample will be falling like a rock out-sample.

But if I don't include the extreme case of year 2008 into my out-sample test data, I am cheating myself - what if another black swan event happens?

Any thoughts on testing the robustness of an optimized trading strategy?

And any thoughts/pointers about discovering trading strategies that can go well in all cycles, e.g. doing well in-sample (data prior to 2008) and also perform well on the out-of-sample 2008/2009?

Thanks a lot!

Don't really know what you are trading ... but how about including another period of extreme volatility in your in sample data ? Say 1999 to 2001 ?

 

[intradaybill]

Quote from TraderZones:

Keep up. Your underwear is showing...

Are you a psycho or what? Why are you so offending and insulting with other people? Do operators of this website understand that it is psychos like you that chase people away with their insulting attitude? You are probably an idiot who has never traded a single $ or who blew up in a matter of 10 days and then takes his anger here on other posters who try to learn and help other people.

What I can do is say screw you to you and never come back to this forum again. I will not take insults from idiots like you. You should really consider visiting a competent psychiatrist to receive treatment.

 

[Occam]

Quote from ButterMilk:

Not true...you can account for slippage. You can not get (that I know) 6 years of bid/ask/last data.

I think Nanex / NxCore will sell you exactly this for maybe around a few $K. Not in text format, but accessed through their API, the same one they use for their real time data service, which I think is quite good.

 

[mizhael]

Quote from mizhael:

How do you decide the in-sample and out-sample sizes (two extra parameters) in Walk Forward Optimization without being curve-fitting?

I meant the two extra parameters (in-sample and out-sample sizes) used in WFO lead to a lot of variation in test-performance. You will see that different sizes lead to very different performance.

Therefore, you have to have one more layer overlay optimizing these two parameters, which leads to another layer of curve-fitting...

 

[PocketChange]

If you trade large lots you should look at back testing with CME's datamine historic BBO and Level II DOM data.

Quote from ButterMilk:

Not true...you can account for slippage. You can not get (that I know) 6 years of bid/ask/last data. I trade large lots and monitor bid/ask sizes.

 

[mizhael]

Quote from kandlekid:

Don't really know what you are trading ... but how about including another period of extreme volatility in your in sample data ? Say 1999 to 2001 ?

Very good point! I am testing FX, for some reason, my data only go back to 2001. hmm, I should probably look for more data to include 1999-2001 in the training sample...

 

[Rabbitone]

In this thread and other threads I am beginning to hear rumblings from traders who are building dynamic hybrid strategies that they require different testing approaches. For example Pocket Change states:

“Unless your strategy is dynamic adjusting to market changes of course you are going to have different results.”

Or Vikana states

“I eventually settled on a different way to optimize. In my opinion the point is not to "optimize" but to "stabilize" and I therefore approach the problem as follows….”

I have spent most of my professional life programming and testing IT software. Since I retired to trade full time in 2002 (after 37 years of IT) I have developed and tested hundreds of different strategies and found some of the old maxims of walk forward testing useless for my type of dynamic hybrid swing trading strategies and the time intervals I trade in.

I now believe we are entering a different era in retail software programming and testing. Many of the simple trading strategies that were first coded in the 1990s to trade simple set ups have evolved and given way to dynamic hybrid strategies that are well aware of price, volume and volatility metadata and can adapt to changing market conditions.

Many retail traders like me are injecting “qunat” or artificial intelligence into strategies to make them adaptive. When I did this I had to question whether traditional walk forward testing was still relevant given the nature of the programs being produced. Much of testing today is more to finding a stabilized strategy to trade with (rather than an optimized strategy) as Vikana stated.

But I still notice most of these discussions on testing in ET still revolve around 1990’s straight set up strategies that are static and require a multitude forward testing periods and validation before they can be put into a production environment making profits or they will be just another “curve fit.”

I believe we need new testing methods to accommodate this changing environment. But to some in the earth is flat group what I have written is hypocrisy. For them testing is fixed and rigid. They may be right. For trading the ES the old ways may be the only way. But I live off other testing methods for trading stocks and these new methods work for me and that is all I care about.

So I’m curious how many of you have gone beyond yesterday’s simple walk forward testing procedures?

 

[Rodney King]

Quote from Rabbitone:

price, volume and volatility metadata

"Metadata"?

 

[Rabbitone]

The term metadata comes from IT meaning data about data. In trading I use it to refer to the data conditions that are taking place during buying and selling. For example programming into a strategy what the correct volatility conditions, price direction, high or low volume, etc.. that must be in place for a trade to execute.

Quote from Rodney King:

"Metadata"?