How to know whether I am overfitting?

ValeryN said:
Ask yourself -
  1. Why would a strategy work for those 100 stocks and not another 100? If there is a particular reason - is it likely to persist overtime?
  2. How would it do in a prolonged declining general market? 2008/2000 like.
  3. How would it do during a flash crash like 1987/2011?
  4. Do you have sufficient # of trades? 1000+?
  5. Does your data include delisted companies? Survivorship bias.
  6. What is your MaxDD for that period? It is typical to have 1:1 return to DD ratio. If that's what you have - expect to see 2x DD live at some point. So by the time you hit it you might blow up your account, which means you are sizing too aggressively.
  7. How much capital is used to generate those returns? You backtest might include unrealistic leverage
Val
More...

Thanks for your comment.

In a general declining market, the strategy will fail. This is because it is a long-only strategy. We cannot short stocks in the country I am trading. In stock futures, I can short here. However, Futures are for a very limited number of stocks. And, it is impossible to get past data for stock futures here.
I do have 1000+ trades in this system, but perhaps 200 in the other one I am using. I rarely get trades in that system, so I am not sure if it works or not.
My data does not include delisted companies, I will look into this certainly.
How do I calculate Max Drawdown in Excel? I am not proficient with VBA, so I look at the results of each stock individually. Then average out what I would get per trade from all the stocks. Therefore, I am not sure what my returns every year are. But, I can certainly look into this, because I can download each year's data and see how my strategy performs, on average, in each of those years.

I have a trend-following system. The market behaved in a weird way in the past 3 months I started trading. The trend formed, then it crashed a few thousand points, thus giving me consistent losers. The index was something like 46500 when I entered. Then it crashed to 42000. Then it went up to 46000. It took a long while for the system to pull up trades between 42000 and 46000 because the system was determining where and when the trends were formed. However, when it reached 46000 it crashed again to 43000. It went up after that recently to 47000 points but trends were determined (again) late. Is it bad luck/volatility/market trading sideways or is it an overfit?
 
You can figure out if you are over fitting or not. Check the validation metrics such as accuracy and loss. If your validation metric is declining, it may be due to over-fitting.
 
You must log in or register to reply here.
Back
Top