Quote from zer0:
small newbie question :
if you manage to build an overoptimized system doing great perfs on a significative set of data, then validate it on a larger set of unseen data, can we consider that the system is playable for real ?
Yes.
You donât always need to validate over a larger set if the backtesting / optmization is done properly.
All systems âdiscoveredâ through backtesting will suffer from some amount of hindsight bias. Walk-forward performance may eventually decay to zero, go negative, or oscillate unpredictably unless the system is continually revised.
My personal experience with back testing and forward testing has revealed the need for robust design and self adaptation.
Robust design
Stick to plausible hypotheses that can be validated as non-random over long periods of time (many data points), over many markets, and as many trades or instances as possible. This will identify the most robust inefficiencies.
Reduce the number of degrees of freedom (DoF) optimized in hindsight. This is very difficult explain, somewhat of an art. Some DoFs can be safe to optimize. Others can lead to over optimization and a short system lifetime. Number of parameters does not always translate to the impact of over optimization. It is critical to understand the nature of the parameters, and how much each parameter can falsely fit random (non systematic) market behavior.
Donât cheat on simulated trade executions. Simulate market orders with real bid/ask quotes if you want to be safe. Use proper performance estimates.
Design a system that can decide when to trade and when not to. Determining market direction can be as simple as the slope of a simple moving average if the system knows when to trade it.
Use clean market data, and be sure your historical data is consistent in character with your real-time data. Otherwise, subtle differences in the real-time data will kill performance immediately.
Run control experiments to identify biases and better understand how your systems work.
Pursue novel indicators and ideas, less correlated to the actions public domain analysis. Performance variations will be lower.
Self Adaptation
Have the system continually adjust itâs own parameters, either using heuristics or using past performance to predict future performance for various parameter combinations. Paper trade a population of systems simultaneously. Essentially you are doing all of this already with manual back testing and forward testing. Better to automate the entire process. Better to have an architecture that lets you paper trade 1000s of systems in real-time instead of back testing them and hoping your backtest is proper.
Some systems or system performance estimates can be predicted better than others (autocorrelation of returns, etc.). It is very frustrating to work with systems that have poor autocorrelation and require a long period of time to judge performance.
