Difference between Over Ftting and Optimization to Current Environment

K

KMeriwetherD

I am back testing an intraday strategy for trading oil futures. It is technically driven and takes about 3 trades a day during a constrained trading period. Given I only have detailed data to test back to 2010, that is as far back as my testing has gone. The system works great over the past 2 years, weaker in the 3rd and 4th. It blows up 5 years ago. I understand the effectiveness of the system is based on it being complementary to the market environment. Of course this varies based on the other algorithms dominant at the time AND I feel the price of oil is a factor as well being that exposure per contract changes the volumes traded. Also, the driving forces for oil have changed as well as the patterns in how the market digests that information. Anyway..... is it more wise to use the parameters that give, perhaps less flattering, but more consistent results over all trials OR is it better to use the the parameters that yield the best results over the past couple years? Of couse, I understand I would have to recognize when the environment is changing so I can once again keep my parameters current (if recent optimal parameters are indeed the best choice for implementation). Basically I am confused because I know I'm not supposed to "over fit" ..... but wouldn't the recent trading environment be the one that matters the most anyway? Why should I care if it worked 3 years ago when I am actively going to be on the watch for environmental changes that would cause the system to break down in the present?

I appreciate any perspective or advice. Thanks!
 
Don't overdo your optimization.
Because the past is gone. Look forward.
Some adjust the parameters every end of week.
It's called calibration if I am not mistaken. It's fine to do that.
As you said, the conditions change, especially volatility.
Which is an element to take into consideration, IMO.
A backtest from 2010 is fine. Let's forward test.
You could have done that with historical data.
However if you've blown up backward ...
Expect to blow up forward ...
Good luck.

Over fitting is definitely to avoid.
And optimization must stay rough.
Actually, there is a meta optimization,
Which is something called survival, antifragile.
I'd say over fitting is a kind of optimization gone bad.
 
Last edited:
KMeriwetherD said:
is it more wise to use the parameters that give, perhaps less flattering, but more consistent results over all trials OR is it better to use the the parameters that yield the best results over the past couple years?
More...


I think the answer to this question depends on so many variables that the only realistic attitude is that although it's always a big and important issue, only "experience" can point one in the right direction.

Not very helpful, I know; but true, I think.


KMeriwetherD said:
Of couse, I understand I would have to recognize when the environment is changing
More...


Exactly so.

One's perspective on the extent to which that's significant depends on various issues, such as how many different systems are you trading, or whether (almost?) all your eggs are in one basket.

You need to backtest and analyze within your preferred probability-parameters and according to your own degree of risk-aversion.

You need to have a way of determining, in advance, under exactly what outcome-parameters you'll assume that something "isn't working any more because changes in the market predicate that it can't" rather than that it's "just having a really bad run at the moment". Otherwise, when you start losing, you don't know what to do.

"Robustness" in system-design is of different significance to different styles and overall methods of trading.

I think you've summarised the problem well, but I just don't think there's an easy answer to this.
 
I think the best optimisation would be a quick adaptation/response to the latest/current environment as far as a trader can see/realise.


KMeriwetherD said:
I am back testing an intraday strategy for trading oil futures. It is technically driven and takes about 3 trades a day during a constrained trading period. Given I only have detailed data to test back to 2010, that is as far back as my testing has gone. The system works great over the past 2 years, weaker in the 3rd and 4th. It blows up 5 years ago. I understand the effectiveness of the system is based on it being complementary to the market environment. Of course this varies based on the other algorithms dominant at the time AND I feel the price of oil is a factor as well being that exposure per contract changes the volumes traded. Also, the driving forces for oil have changed as well as the patterns in how the market digests that information. Anyway..... is it more wise to use the parameters that give, perhaps less flattering, but more consistent results over all trials OR is it better to use the the parameters that yield the best results over the past couple years? Of couse, I understand I would have to recognize when the environment is changing so I can once again keep my parameters current (if recent optimal parameters are indeed the best choice for implementation). Basically I am confused because I know I'm not supposed to "over fit" ..... but wouldn't the recent trading environment be the one that matters the most anyway? Why should I care if it worked 3 years ago when I am actively going to be on the watch for environmental changes that would cause the system to break down in the present?

I appreciate any perspective or advice. Thanks!
More...
 
Xela, what you say puts me at ease. I'm sure there is no text book approach. I have already begun to search the signature of the market structure that is toxic to my system. Similarly, I am seeking to understand what it is that has made it so successful over the past two years. This way, I will know what warning signs to look for in the future. I am new at this, but I get the impression the process of understanding the how and why of your system is more important that the fitting of paramters.

This brings me back to my original question.... the difference between over fitting and optimization to current environment. My new perspective is that over fitting involves "fixing" the variable without regard as to WHY they are complementary to the market environment. Optimization is more intelligent than that. It is a conscious effort to more efficiently capture probable variations in current market structure.
 
Last edited:
Curve fitting to current environment is not optimization, it's adaptive, and can be effective to be more in tune with current environment / what to expect, until market breaks out. If algo blows up, better filter in advance, bur prefer to be more adaptive instead and see if worst case can be avoided, ie. by sl, without negating returns. In reality, you're better off staying mostly on the sidelines on unclear conditions, so filtering can be an edge too. Optimization is all what we do, feedback cycle-wise and never ends.

When is the best time to enter directionally?
 
Very good article here:

"strategies fail because of many reasons. Below are the three most common:
  1. The strategy was random to start with
  2. The edge was arbitraged out
  3. A change in market conditions occurred"
Curve-fitting to current environment sets the stage for the next drawdown when the conditions change because there is always a lag before confirming the change. Any curve-fitting makes the strategy random, or relevant only to a random sample. Then even if you avoid these your edge may be taken out of the market by other traders.
 
-- If the optimization time period is too short, there would be too few trades, and thus the statistical significance would be low.
-- If the optimization time period is too long, you get statistical significance because of the large number of trades, but lose on the "adaptation", because everything is averaged out.

There is a way to diminish these undesirable effects by using a time-weighted optimization. You optimize on the entire available period, but apply progressively less weight on the gains and losses that occurred in distant past. The more distant past, the less weight. The weight can be determined with a common kernel (Epanechnikov, tricube, quartic, sigmoid). The net result is that the system is "tuned" to the current conditions (based on the more recent performance), while still taking into consideration the entire available history.
 
Sergio77 said:
Very good article here:

"strategies fail because of many reasons. Below are the three most common:
  1. The strategy was random to start with
  2. The edge was arbitraged out
  3. A change in market conditions occurred"
Curve-fitting to current environment sets the stage for the next drawdown when the conditions change because there is always a lag before confirming the change. Any curve-fitting makes the strategy random, or relevant only to a random sample. Then even if you avoid these your edge may be taken out of the market by other traders.
More...

Good article.
 
Sergio77 said:
Very good article here:
More...
After reading this article I have two thoughts.

1) Where can I learn more about the application of hypothesis testing to systems back tests? I do have a basic education in stats and hypothesis testing but I am lacking in the actual application skills.

2) Does it really matter if the edge you have discovered is because of randomness (survivorship bias) or intelligent design? Both are at risk of sudden disappearance. Both are simply recognizing a repeatable action that has a favorable probability of getting a win instead of a loss. Intelligent design would be better in that if you understand why the systems works, you would be more likely to predict when it won't. Other than that, is one kind of system really better than the other?
 
You must log in or register to reply here.
Back
Top