Curve-fitting or just prudent trade selection?

[logic_man]

Let's say I've got a model which scores a trade set-up from 1 to 100 based on a number of objective factors. Any price action that triggers the set-up is a valid trade signal, but, historically, only trades which score above a 30 are profitable, in aggregate. Not that all trades under 30 fail, just that the profit factor is below 1. Taking only the trades above 30 yields a profit factor that is much higher and indicative of a robust strategy.

Is ignoring any trade signal which scores below a 30 on the model "curve-fitting" or is it just a case of ignoring similar, yet somehow different, market action?

 

[frostengine]

The idea of "curve fitting" is not necessarily evil by definition. I rely heavily on curve fitting to identify patterns. However, there is a right way to do it and a wrong way.

When you apply the 30 threshold to out of sample data (data not tested to base your decision on using 30) how does it perform? Do the performance metrics stay reasonably in-line? If so, then your fine..

 

[logic_man]

Quote from frostengine:

The idea of "curve fitting" is not necessarily evil by definition. I rely heavily on curve fitting to identify patterns. However, there is a right way to do it and a wrong way.

When you apply the 30 threshold to out of sample data (data not tested to base your decision on using 30) how does it perform? Do the performance metrics stay reasonably in-line? If so, then your fine..

Yes, trades scoring under 30 are performing the same since I noticed that it was a cutoff point between two very different sets of outcomes in real trades. If anything, they're doing a little better than they were before.

 

[clearinghouse]

Quote from logic_man:

Let's say I've got a model which scores a trade set-up from 1 to 100 based on a number of objective factors. Any price action that triggers the set-up is a valid trade signal, but, historically, only trades which score above a 30 are profitable, in aggregate. Not that all trades under 30 fail, just that the profit factor is below 1. Taking only the trades above 30 yields a profit factor that is much higher and indicative of a robust strategy.

Is ignoring any trade signal which scores below a 30 on the model "curve-fitting" or is it just a case of ignoring similar, yet somehow different, market action?

Variance-bias trade-off, my good man. You decide how much trade-off you want to make, but just be aware of the consequences.

 

[logic_man]

Quote from clearinghouse:

Variance-bias trade-off, my good man. You decide how much trade-off you want to make, but just be aware of the consequences.

That makes sense.

I should also note that the "trigger levels" of the parameters in the model were based on an initial hypothesis of what the levels "should" be. The fact that subsequent empirical evidence has shown that the levels where they "should" be weren't, in fact, the optimal levels seems to me to just be a case of learning that the initial hypothesized levels were somewhat arbitrary, although they were in the ballpark.

 

[Wide Tailz]

Quote from logic_man:

Is ignoring any trade signal which scores below a 30 on the model "curve-fitting" or is it just a case of ignoring similar, yet somehow diffrent, market action?

Every profitable system is a curve fit to some degree, IMO. Some curves last for a few weeks, some a year and others forever. Dow theory still works on the daily, last time I checked.

Old fashioned trend lines are still killing it on the weekly.

Why do these old edges still work? Not every participant in the market has the freedom of waiting to enter. Think farmers hedging every year, or oil exporters recycling petrodollarz..........

Understand who you're taking the money from.

 

[ronblack]

Quote from Wide Tailz:

Every profitable system is a curve fit to some degree, IMO. Some curves last for a few weeks, some a year and others forever. Dow theory still works on the daily, last time I checked.

Old fashioned trend lines are still killing it on the weekly.

Why do these old edges still work? Not every participant in the market has the freedom of waiting to enter. Think farmers hedging every year, or oil exporters recycling petrodollarz..........

Understand who you're taking the money from.

+

Some should read your post over and over again. Whether curve-fititng is good or bad depends on how one does it and what are the objectives, timeframes, markets, etc. The worse type of curve-fit is the one that moves signals and their exits on a price curve until the profit is acceptable. This must be avoided. This blog talks about different types of curve-fitting and their impact on trading system performance.

 

[ssrrkk]

Quote from ronblack:

+

Some should read your post over and over again. Whether curve-fititng is good or bad depends on how one does it and what are the objectives, timeframes, markets, etc. The worse type of curve-fit is the one that moves signals and their exits on a price curve until the profit is acceptable. This must be avoided. This blog talks about different types of curve-fitting and their impact on trading system performance.

I think the original usage of curve-fitting to refer to over-fitting is the following: if you have N data points, then you can come up with a (N-1)th degree polynomial with N-1 coefficients plus a constant to perfectly fit all the data points -- i.e., you can do a curve-fit against all the data points. However, that polynomial will be completely useless in predicting future unseen data. That's because the polynomial outside of the x-range of data points will usually rapidly approach +/- infinity because the high order terms inevitably dominate as x becomes large or small. In other words, when the number of parameters of your model becomes closer and closer to the number of data points in your training set, the predictions from the model become less and less generalizable for future unseen data points.

As long as your "curve-fitting" exercise involves a lot less parameters compared to your training set the performance of the model will likely be more generalizable (i.e., will likely perform better than an overfit model).

 

[Wide Tailz]

Just yesterday I re-read part of Schwager's interview with Eckhardt, and curve fitting came up. Even the master, the partner of Richard Dennis, the multimilliondollar fund manager (still in business after 30+ years) said that there is basically no way to know the difference between an acceptable fit and an "overfit". He said it takes experience to get it right. It's a discretionary decision. He said 7-8 degrees of freedom are probably too much.

In the earlier version of the book, Ed Seykota said "system trading is ultimately discretionary".

Perhaps this subject is part of what he was talking about. You never know the future, but people are probably not going to stop eating.

 

[Wide Tailz]

Quote from ssrrkk:



As long as your "curve-fitting" exercise involves a lot less parameters compared to your training set the performance of the model will likely be more generalizable (i.e., will likely perform better than an overfit model).

That's a very interesting point. Great observation.

I have a type 1 sytem that was trained on about 22000 data points and has 5 degrees of freedom.

I changed some of the parameters and got a much better fit for the most recent 1700 data points, but the system was flat from any point before that. I debated on just refitting the system every n datapoints but threw the idea away as being too delicate for the real world.

"Ultimately discretionary"

Won't know if it's the right decision until the next 1700 data points are captured.