Ridiculous Results Given by Genetic Optimizer

[dtrader98]

Genetic Optimizers usually work by randomly seeding and selecting populations. You should not expect results to be exactly the same each time. If you truly want to compare apples to apples, you need to brute force all of the possible outcomes. That will of course, truly over-fit the best outcome, but then you can start to try to understand generalizing.

If the default optimizer is doing brute force one by one, you'd need to let it complete all possible outcomes to expect the Gen Optimizer results be a subset of those. If they are not included, either 1) you did not run all possible outcomes or 2) there are outcomes or parameters in the Gen Optimizer that are not equivalent to the search space of the brute force method.

Either way, trying to match those isn't all that important as trying to understand generalizing and why you need it.

 

[Warren]

Genetic Optimizers usually work by randomly seeding and selecting populations. You should not expect results to be exactly the same each time. If you truly want to compare apples to apples, you need to brute force all of the possible outcomes. That will of course, truly over-fit the best outcome, but then you can start to try to understand generalizing.

If the default optimizer is doing brute force one by one, you'd need to let it complete all possible outcomes to expect the Gen Optimizer results be a subset of those. If they are not included, either 1) you did not run all possible outcomes or 2) there are outcomes or parameters in the Gen Optimizer that are not equivalent to the search space of the brute force method.

Either way, trying to match those isn't all that important as trying to understand generalizing and why you need it.

Hi dtrader98, thanks for your reply.

It is clearly the second condition you mentioned. Parameters generated by Gen Optimizer is some value in between the scan steps. And the problem is, those slightest changes in parameters, produces huge difference of performance(far more better than the brute force search). So, I guess this in only a sign of over-fit.

BTW, what do you mean by "trying to understand generalizing and why you need it" ?

 

[wwatson1]

How many parameters are you trying to optimise? I don’t understand the issue here, I get fine useable profitable results from general basic optimisation of a few key parameters. Try metatrader 4, it works for most people really well, but it’s mainly for forex traders

 

[userque]

Hi Everyone,

As there are traders said that the Genetic Optimizer in NinjaTrader 8 is not reliable, I tried to test it myself.

With the same strategy I am using and same scan range of parameters:
Para1 [0.0001, 0.01] step=0.0001;
Para2 [1,10] step=0.1;
Search Space=99*90=8910;

The default optimizer (search the space one by one by brute force), and the Genetic Optimizer are giving very different results. The ridiculous thing is, the Genetic Optimizer are giving several results far more better than the default optimizer!

For example (see the attached file), the top 5 best results given by the Genetic Optimizer have a SQN(Van Tharpe's System Quality Number) of 3.4, and win rate above 70%; while, the default optimizer's top 5 results have SQN of only 1.71, and win rate of 37%-56%.

Is this theoretically possible? Or, I did not set the Genetic Optimizer right? Is this some evidence that the Genetic Optimizer is reliable?

I notice that the Genetic Optimizer generated some parameters beyond the scan steps, and I guess this is a strong evidence of over-fit.

But, anyway, I am inclined to think the Genetic Optimizer is somehow reliable after this.

As someone already stated, the default optimizer is not considering the exact same search space that the genetic optimizer is considering. You are 'stepping' over values with the default "brute force" search.

Additionally, you may want to consider using the Walk Forward Optimizer instead of the genetic optimizer (unless you really know what you doing); otherwise, you are likely over-fitting.

 

[dtrader98]

Hi dtrader98, thanks for your reply.

It is clearly the second condition you mentioned. Parameters generated by Gen Optimizer is some value in between the scan steps. And the problem is, those slightest changes in parameters, produces huge difference of performance(far more better than the brute force search). So, I guess this in only a sign of over-fit.

BTW, what do you mean by "trying to understand generalizing and why you need it" ?

So, the GA is likely generating random continuous parameters with a granularity less than your step size of the brute force generator. The fact that many of these do much better, shows it is highly sensitive to small parameter changes and thus not likely to perform well out of sample. Not only is that overkill, but shows you that the system you are developing is overly sensitive to its parameters -- generally, not a good thing.

Generalizing is finding a way to reduce the complexity of your system and parameter fitting, such that it has a better chance of continuing to perform well out of sample. As mentioned in a prior thread, one way to perform this is using walk forward validation, some other commonly known are validation, and cross-validation.

 

[Warren]

So, the GA is likely generating random continuous parameters with a granularity less than your step size of the brute force generator. The fact that many of these do much better, shows it is highly sensitive to small parameter changes and thus not likely to perform well out of sample. Not only is that overkill, but shows you that the system you are developing is overly sensitive to its parameters -- generally, not a good thing.

Generalizing is finding a way to reduce the complexity of your system and parameter fitting, such that it has a better chance of continuing to perform well out of sample. As mentioned in a prior thread, one way to perform this is using walk forward validation, some other commonly known are validation, and cross-validation.

My focus here is not the system. The system performed badly in walk forward.

While I think this test is not a firm evidence against the reliability of the default Genetic Optimizer in NinjaTrader 8. And I will need further tests.

It is my first heard of the generalizing, though, much appreciated !!

 

[ronblack]

This thread reminded of this classic article from PAL blog since we just discussed another of their articles in another thread. Please read the article before using genetic and NN algos. When you use anything that optimizes you have to think multiple comparisons. If you optimize anything on the past then chances are it has no good use for the future.

 

[Warren]

This thread reminded of this classic article from PAL blog since we just discussed another of their articles in another thread. Please read the article before using genetic and NN algos. When you use anything that optimizes you have to think multiple comparisons. If you optimize anything on the past then chances are it has no good use for the future.

Thanks for the info, have not read that yet.

 

[shatteredx]

My focus here is not the system. The system performed badly in walk forward.

While I think this test is not a firm evidence against the reliability of the default Genetic Optimizer in NinjaTrader 8. And I will need further tests.

It is my first heard of the generalizing, though, much appreciated !!

Yeah this is just a weird bug where NT8 is somehow setting the increment on your first parameter to 1e-14 when you select genetic optimizer.

I'm not sure if that's better or worse than the genetic optimizer spitting out false results. Probably an easier fix for the NT programmer?

 

[Warren]

Yeah this is just a weird bug where NT8 is somehow setting the increment on your first parameter to 1e-14 when you select genetic optimizer.

I'm not sure if that's better or worse than the genetic optimizer spitting out false results. Probably an easier fix for the NT programmer?

Cool, never notice that.