Is this over-fitting

M

Moptop

Lets say I have a set basket of 50 ETF's that I like to backtest and trade with. I run a backtest and get good results and decide to trade a strategy, but I notice about 10/50 of the ETF's perform poorly with the chosen parameters.

1. Would it be considered over-fitting to exclude those 10 ETF's from live trading

2. Doubling down here: if I did exclude the 10 poor peformers, how bad would it be to re-optimize on the remaining 40/50 ETF's.
 
1. Yes
2. You will get different results.

If you are doing tests on indicators and changing parameters, every month most likely have different parameters that is best. If you using chart patterns, as your experience expands and can fine tune questions to ask/code of backtesting, your stats become better.

3. Risk management should be tested on it's own merit, better risk management defines better bottom line.
 
Potentially, but eliminating the 10 ETFs doesn't in its own merit mean the strategies are overfit. If you have a trend following strategy, it doesn't mean you need to use it on all commodities. Naturally, the strategy will perform best with some commodities. I would say that you should focus on using walk forward analysis with your backtests to make sure the strategies are not overfit to the specific time period your are backtesting.
 
Moptop said:
Lets say I have a set basket of 50 ETF's that I like to backtest and trade with. I run a backtest and get good results and decide to trade a strategy, but I notice about 10/50 of the ETF's perform poorly with the chosen parameters.

1. Would it be considered over-fitting to exclude those 10 ETF's from live trading

2. Doubling down here: if I did exclude the 10 poor peformers, how bad would it be to re-optimize on the remaining 40/50 ETF's.
More...


What is in the basket?

You cannot dump very different ETFs into a basket.
Just for eg don't dump
the slow-moving grain ETF with the fast-moving oil ETF.

They have very different behavior/characteristics/characters/personalities.
 
Last edited:
To provide reasonable answer you need to give more details about your ‘train set’

Imo the number 1 question for possible overfitting would be: do you have reason to believe that your future dataset (trades) will be different then the dataset you tested up on.

know also that it’s by definition impossible to know for sure. E.g NQ/qqq could have 50 exact same weekly closes on a row. Theoretically possible. So in the end you need to jump in and risk something. Would be wise to start slow and let it prove itself.
 
Take a clue from the Turtles who were taught how to trade by Richard Dennis. They traded anything, commodities, stocks, ETFs etc that moved. Now, this is a robust system because you have different asset classes being traded all at once. The only qualification is the asset class must be trending. The reason for the diversification is to reduce risk. Most of the turtles went to to establish successful careers as traders, hedge fund managers, etc. When the program ended, Richard Dennis was reported to have earned over $400 million. He shared profits with the traders he taught and gave them fixed salaries too. They also, had to sign non-disclosure agreements with Richard Dennis for 10 years.
 
Moptop said:
Lets say I have a set basket of 50 ETF's that I like to backtest and trade with. I run a backtest and get good results and decide to trade a strategy, but I notice about 10/50 of the ETF's perform poorly with the chosen parameters.

1. Would it be considered over-fitting to exclude those 10 ETF's from live trading

2. Doubling down here: if I did exclude the 10 poor peformers, how bad would it be to re-optimize on the remaining 40/50 ETF's.
More...
1) What was your backtesting period used?
2) Once you backtest a strategy with good results, you need then to forward test it for at least half the backtesting period used. It's even a good idea to forward test on a period prior to the one used for backtesting.
3) Re-optimizing a strategy after removing the losing parts is the best way to overfit.
 
rb7 said:
1) What was your backtesting period used?
2) Once you backtest a strategy with good results, you need then to forward test it for at least half the backtesting period used. It's even a good idea to forward test on a period prior to the one used for backtesting.
3) Re-optimizing a strategy after removing the losing parts is the best way to overfit.
More...
So if the method is tested on 15 years of data, you would wait for 7,5 years? Not practical imo.
Better slowly advance with real money
 
Zwaen said:
So if the method is tested on 15 years of data, you would wait for 7,5 years? Not practical imo.
Better slowly advance with real money
More...
You're right if your definition of forward test is using real time data. So in you example, yes, you would need to wait 7.5 years, which is not very practical.
I should have given more details.
A strategy is optimized using a optimization data set.
Once you find one with good results, you then test it on a validation data set, one that was not used for the optimization step.
But it could happen that both data sets (optimization and validation) give good results but fail when using it in real time.
If profitable trading was easy, everybody would be doing it...
 
You must log in or register to reply here.
Back
Top