Very Interesting topic! We need a process for qualifying what data is expected to be useful going forward, and fit from that, and identify what data may be unproductive and refrain from using that. -- not an easy task
on parameter (over)fitting
- Thread starter Baozi
- Start date
I'd guess it's desirable to have out-of-sample times cover a variety of market conditions, not just very recent conditions. Then, you would have a better idea of how the system holds up over time. For example, it would be better for a trend following system to detect when the market isn't trending and avoid trades then.
If the out-of-sample time period overlaps the in-sample time period, testing on a larger basket of stocks might not show what will happen in the future well because stocks in the same time period are often highly correlated.
The acceptable way for someone knowing the basics of statistics is to explore via simulation the entire sample space (a universe of plausible sample paths).
The backtesting story, imho, is some nonsense put forward by someone with little understanding of stats, and unfortunately, the herd just copies and pastes and follows what they heard. Since Stats exists, Statistical procedures are evaluated on the entire sample space, not on a single point.
Backtesting one or a few past sample paths would lead nowhere, except to delusional reliance (clearly reinforced by the hope that everyone has). But, of course, everyone must make their own journey, and all I can do is take the insults and plant the seed of doubt...
Last edited:
The acceptable way for someone knowing the basics of statistics is to explore via simulation the entire sample space (a universe of plausible sample paths).
The backtesting story, imho, is some nonsense put forward by someone with little understanding of stats, and unfortunately, the herd just copies and pastes and follows what they heard. Since Stats exists, Statistical procedures are evaluated on the entire sample space, not on a single point.
Backtesting one or a few past sample paths would lead nowhere, except to delusional reliance (clearly reinforced by the hope that everyone has). But, of course, everyone must make their own journey, and all I can do is take the insults and plant the seed of doubt...
when you say "explore via simulation" do you mean something like a montecarlo simulation?
Monte Carlo methods is a term used with a meaning broad enough to incorporate that. At least if we take it to mean any kind of experiment where pseudorandom generators are involved.
More specifically one would put up a market simulation along the lines of Euler–Maruyama methods and use that as a tool to start a patient and methodic exploration of the "statistical" properties of the trading procedure he has devised.
Clearly, one would be careful to exclude the trivial academic simplifications, which have only interest to write (practically useless) papers (like constant volatility or constant reversion rate or constant drift, or unrealistic distributional assumptions about prices and jumps, contango, backwardation, decay, daily rebalancing, etc) and build and explore plausible and realistic sample spaces.
That is, in intuitive terms, sample paths that, submitted to any competent trader of a person with actual market experience, would have no means to exclude in any way that they could have been actual realizations.
What is called "backtesting", was put out mostly as a marketing tool, to confuse (circumvention of incapable) with some superficially appearing "quantitative evidence" the clueless investors (or ignorant bosses at work in financial institutions) with some resemblance to (very purported and fundamentally wrong) "Statistics".
Eventually, through repetition, people started to actually believe that it was an actual thing, and the herd keep following and repeating everywhere the same nonsense.
Last edited:
Exactly. This is the huge flaw with the common approach of using "rolling" in-sample optimization / out-of-sample verification. The top-level research process (R&D decisions being made) ends up optimized over the out-of-sample periods. The out-of-sample periods are no longer fully out-of-sample.
I think there are several issues when developing a strategy just to trade equities. The main problem is that stocks are highly correlated with each other, so even when you trade a basket of 500 stocks, you are essentially trading a single asset class. Furthermore, given that equity bull runs can last more than a decade, any backtest based on a couple of years worth of data will fail to capture a full equity market cycle and underestimate the risks of a bear market. Traders may be lured into over-fitting to try and seek the "perfect" system, and that system may perform well for a while until it fails completely.
I propose developing multiple strategies with a small number of rules and to keep those rules simple and loose. Each of these strategies may be designed to capture different market conditions/trends/patterns. Individual strategies that generate a decent risk/reward profile can then be combined and traded as a multi-strategy portfolio. This way, you can avoid overfitting a single strategy.
I personally trade a trend following strategy across 32 futures markets. Currently, I am working on developing a mean reversion strategy that can give me similar CAGR and Max drawdown. Combining these 2 starkly different strategies should reduce drawdowns without sacrificing too much in terms of returns. I might even develop strategies with different trading timeframes, combining short-term strategies with medium-term ones.