is Out Of Sample testing useless ?

[ionone]

Most users of Optimization techniques, whether it is in a Neural Network or simple Parameter optimizations, use "out-of-sample" verification to assess if an optimization will keep working in the future.
there is only one problem with that approach : if the strategy is then not profitable on "unseen" data, then you discard the optimization and try another one. Well this is exactly what the genetic optimizer does : if a test is not profitable, it will discard it. So by doing so, you don't verify anything, and it brings nothing new to the fact that the strategy will or will not be profitable in the future. As a matter of fact, the OOS test doesn't insure that the strategy will be profitable on unseen data.

I think it is better to manually fit parameters, than using the GO to find the best settings

Jeff

 

[trend2009]

very useful

 

[2rosy]

As a matter of fact, the OOS test doesn't insure that the strategy will be profitable on unseen data.

I think it is better to manually fit parameters, than using the GO to find the best settings

Jeff

it insures that the strategy is unprofitable over test data. so you ether reject or not reject but not accept

 

[RedDuke]

Crucial step. Saves you tons of real
Capital.

 

[Snuskpelle]

Here's the key: You should ideally do just one OOS test. If you're repeatedly doing "OOS" tests you're increasing the probability of the algo/system/signal/whatever passing by accident.

And yeah, that's a downfall of naive genetic optimization unless you verify all generated strategies exactly once on a final part of unseen data. They might all fail. ;-)

 

[RedDuke]

Here's the key: You should ideally do just one OOS test. If you're repeatedly doing "OOS" tests you're increasing the probability of the algo/system/signal/whatever passing by accident.

And yeah, that's a downfall of naive genetic optimization unless you verify all generated strategies exactly once on a final part of unseen data. They might all fail. ;-)

spot on. The idea is to run once, and if it fails, there is a very high probability live will fail as well.

 

[trendmomentum]

Most users of Optimization techniques, whether it is in a Neural Network or simple Parameter optimizations, use "out-of-sample" verification to assess if an optimization will keep working in the future.
there is only one problem with that approach : if the strategy is then not profitable on "unseen" data, then you discard the optimization and try another one. Well this is exactly what the genetic optimizer does : if a test is not profitable, it will discard it. So by doing so, you don't verify anything, and it brings nothing new to the fact that the strategy will or will not be profitable in the future. As a matter of fact, the OOS test doesn't insure that the strategy will be profitable on unseen data.

I think it is better to manually fit parameters, than using the GO to find the best settings

Jeff

Not at all, but this is just a loaded question. If you do not start with a strong market hypothesis before any modelling, whatever the optimizer finds at random, whether you run it once or a million times, the chances are that the live pnl curve will almost immediately flatten out, at best.

 

[ionone]

spot on. The idea is to run once, and if it fails, there is a very high probability live will fail as well.

that's exactly what i'm saying. If it fails in OOS, you discard the strategy ? but that's exactly how the GO works
It doesn't give more information about the strategy

 

[Snuskpelle]

that's exactly what i'm saying. If it fails in OOS, you discard the strategy ? but that's exactly how the GO works
It doesn't give more information about the strategy

If you only did it one time for an unseen piece of data, there's a reasonably low probability (notwithstanding all other possible problems such as bugs etc.) it is an accidental result, i.e. it does give you more information than if you had not done the OOS test.

You don't just discard the strategy, you also discard the data used in the OOS test.

 

[Trader200K]

Not at all, but this is just a loaded question. If you do not start with a strong market hypothesis before any modelling, whatever the optimizer finds at random, whether you run it once or a million times, the chances are that the live pnl curve will almost immediately flatten out, at best.

‘Strong Market Hypothesis’ begs the question ... can only the ‘higher mind/with experience’ develop the SMH or could GO ‘stumble upon‘ the same.

I try to be open minded, but the ‘stumble upon’ method suffers from monkey/typewriter syndrome ... only this monkey can type at GHz speed.

The ‘higher mind’ suffers from the limited time/resources ability to test/find that high SMH system.

I have looked at a couple more-than-cheap GO packages, but haven’t seen anything that makes me think it can beat buy/hold or one of my many marginal system tests. Maybe I am missing it. Sure would like to see some credibly good GO results, but nada so far.

Maybe a tighter definition of what qualifies as a SMH is a starting point for improving the whole process?