Ridiculous Results Given by Genetic Optimizer

Warren

Warren

Hi Everyone,

As there are traders said that the Genetic Optimizer in NinjaTrader 8 is not reliable, I tried to test it myself.

With the same strategy I am using and same scan range of parameters:
Para1 [0.0001, 0.01] step=0.0001;
Para2 [1,10] step=0.1;
Search Space=99*90=8910;

The default optimizer (search the space one by one by brute force), and the Genetic Optimizer are giving very different results. The ridiculous thing is, the Genetic Optimizer are giving several results far more better than the default optimizer!

For example (see the attached file), the top 5 best results given by the Genetic Optimizer have a SQN(Van Tharpe's System Quality Number) of 3.4, and win rate above 70%; while, the default optimizer's top 5 results have SQN of only 1.71, and win rate of 37%-56%.

Is this theoretically possible? Or, I did not set the Genetic Optimizer right? Is this some evidence that the Genetic Optimizer is reliable?

I notice that the Genetic Optimizer generated some parameters beyond the scan steps, and I guess this is a strong evidence of over-fit.

But, anyway, I am inclined to think the Genetic Optimizer is somehow reliable after this.
 

Attachments

  • Optimization Given by Default Optimizer-NinjaTrader8.xlsx
    Optimization Given by Default Optimizer-NinjaTrader8.xlsx
    11.6 KB · Views: 27
  • Optimization Given by Genetic Optimizer-NinjaTrader8.xlsx
    Optimization Given by Genetic Optimizer-NinjaTrader8.xlsx
    7.8 KB · Views: 19
  • Parameters.jpg
    Parameters.jpg
    792.9 KB · Views: 45
I don't know if this is still relevant but I stopped using NT a long time ago due to the fact that the way it's database is set up, it is nearly impossible got get reliable backtests.

This "Optimizer" might just be an overfitting algo, so I wouldn't rely on it anyways. But just to put your own work process into perspective, do yourself a favour and install NT on another machine, use the exact same data with the exact same markets, timeframes and parameters....and you will probably find out that you get two different results every single time.

I could never figure out why, so I just left NT for my own stuff I made in Python.

Perhaps that helps
 
MrMuppet said:
I don't know if this is still relevant but I stopped using NT a long time ago due to the fact that the way it's database is set up, it is nearly impossible got get reliable backtests.

This "Optimizer" might just be an overfitting algo, so I wouldn't rely on it anyways. But just to put your own work process into perspective, do yourself a favour and install NT on another machine, use the exact same data with the exact same markets, timeframes and parameters....and you will probably find out that you get two different results every single time.

I could never figure out why, so I just left NT for my own stuff I made in Python.

Perhaps that helps
More...

Well, another reliability to test. Will try to test that sometime. Thanks buddy.

I am not a programmer. I tried to run backtest on R, while unfortunately those financial packages have many bugs. I tried to develope my own package, but the functions are very limited.

Python sounds cool, but learning another language seems daunting.
 
Warren said:
Well, another reliability to test. Will try to test that sometime. Thanks buddy.

I am not a programmer. I tried to run backtest on R, while unfortunately those financial packages have many bugs. I tried to develope my own package, but the functions are very limited.

Python sounds cool, but learning another language seems daunting.
More...

Learn Python. It's what the professionals use
 
MetaTrader has similar flaws, I used to get obsessive about back testing but it just wears you out. I use My favourite EA with a few know settings I like. Stick with the same EA and do lots of live market testing and basic backtests, you quickly get to grips with the workable settings, just my experience.

I do still do simple backtests sometimes to check a few changes I might make but that’s about it :thumbsup:
 
Last edited:
You might want to look at Amibroker for backtesting. Tomasz, the sole developer, takes that part of his product very seriously for speed and accuracy. Dude has been banging on that software since 1995. He's optimized that code base to no end.

You can also go over to reddit.com/r/algotrading forum and look around. One recently posted that is active was a free Python library (source on github, site is backtrader.com).
 
what @nooby_mcnoob said: learn python. There are so many libraries for backtesting, charting and financial modelling that you find something for your needs.

Start out with the Anaconda distribution and with Jupyter notebooks. Python is really easy to get started with, even easier as some scripted languages like Tradestation or ToS.
If you were able do a little bit of R, Python is two weeks of effort for you and then you can ditch this PoS spathetti code that is called NT.

https://www.quantstart.com/articles...hon-considerations-and-open-source-frameworks
 
Last edited:
A Genetic Optimizer cannot be reliable for the simple fact that it tunes to strategies that would’ve made least mistakes in the past, without being able to predict mistakes that will happen in the future, or knowing what to do about them. A few bad trades can wipe out all profits of a strategy.
Besides, if it was so easy then everyone would either be a $billionaire or would realize they can’t all make money off of each other.
That is not to say that genetic optimization is useless. It can be a part of your own development process. For me C# is more useful than Python.
 
You must log in or register to reply here.
Back
Top