Curve fitting

[aqtrader]

When people say some trading method or strategy will fail, what does it actually mean? If it means it will definitely fail, the difficulty level to know it for sure is the same as saying a method will succeed. Anything in between is just a random bet.

Trying a ML (machine learning) method, curve fit or over-fit are common mistakes because it will mislead to a totally failed system, that is, a wrong design. Here the word "failed" is a totally diff thing in ML from what a trader thinks.

 

[lukepoga]

saying it will fail does not depend on the outcome. you need to think in probabilities.

i dont think the word is used differently unless they are an extremely unsophisticated trader.

 

[southall]

What about applying technical analysis to the equity curve of the system?

A trading system equity curve is not a market price chart, TA wont work on it.

 

[goodgoing]

Everyone says that you shouldn't curve fit a system perfectly because it will fail. So why not perfectly curve fit a system and then just do the opposite of that system?

Everyone says what they hear others say. Finding a perfectly curve-fit system is as hard as finding a perfectly not fitted system. The author of Fooled By Technical Analysis book offers a few lines of proof of this.

 

[Buy1Sell2]

It might not fail completely. Just give you huge drawdowns.

When testing a curve fit system it might give worst case 25% drawdowns with average 100% yearly returns. Sounds excellent.

But in real world trading, because the system was curve fit, it might actually give 75% drawdowns and only 25% return.

If you attempt to trade a system with 75% drawdowns you will eventually fail in your execution due to psychological issues. Even if you automate such a strategy you will eventually turn it off and fail.

If you attempt to fade the same system you will also eventually fail as the system is net profitable. If the system went into drawdown straight away and you faded the drawdown and made 75%, that will all be lost + more when the system recovers.

This is in part the reason that Prudent Risk Management is really the only thing that matters in trading. Everything else is just noise

 

[SimpleMeLike]

Everyone says that you shouldn't curve fit a system perfectly because it will fail. So why not perfectly curve fit a system and then just do the opposite of that system?

Hello boza,

In my opinion, there is nothing wrong with curve fitting a system perfectly using in sample data. Just make sure once you have it curve fitted how you want, you test the system again using out of sample data. Split the sample date 50%. 50% data for in sample and another 50% data for out sample. This is simple to do.

 

[lukepoga]

Hello boza,

In my opinion, there is nothing wrong with curve fitting a system perfectly using in sample data. Just make sure once you have it curve fitted how you want, you test the system again using out of sample data. Split the sample date 50%. 50% data for in sample and another 50% data for out sample. This is simple to do.

That’s not curve fitting in the sense used here. He means overfitting.

So you’re actually saying ‘there is nothing wrong with overfitting’, which is absurd

 

[fordewind]

Curve fitting is great for discretionary trading.

 

[sle]

Finding a perfectly curve-fit system is as hard as finding a perfectly not fitted system.

Actually, this is definitely not true. You can keep increasing the ratio of free parameters to the number of trades in your backtest. Eventually, by doing so you can perfectly fit any dataset.

In my opinion, there is nothing wrong with curve fitting a system perfectly using in sample data. Just make sure once you have it curve fitted how you want, you test the system again using out of sample data. Split the sample date 50%. 50% data for in sample and another 50% data for out sample. This is simple to do.

Alternatively, you can fit on the full system and then use random sampling with a regression to understand the value of the parameters your introduced. It's probably a better way of doing things, since you are not really leaving untested time periods.

In any case, the practice is somewhat dangerous. Curve fitting assumes that you are introducing extra variables in the system with the aim of improving performance such as reducing the number of drawdowns. If you are dealing with a highly skewed distribution, you are going to have a fairly small number of drawdowns and your strategy will be naturally overfit even though it would appear that you have good out-of-sample results.

 

[SimpleMeLike]

Actually, this is definitely not true. You can keep increasing the ratio of free parameters to the number of trades in your backtest. Eventually, by doing so you can perfectly fit any dataset.


Alternatively, you can fit on the full system and then use random sampling with a regression to understand the value of the parameters your introduced. It's probably a better way of doing things, since you are not really leaving untested time periods.

In any case, the practice is somewhat dangerous. Curve fitting assumes that you are introducing extra variables in the system with the aim of improving performance such as reducing the number of drawdowns. If you are dealing with a highly skewed distribution, you are going to have a fairly small number of drawdowns and your strategy will be naturally overfit even though it would appear that you have good out-of-sample results.

Hello sle,

I am not understanding your comments in detail, its a very complex writings.

Let me ask. Is the below picture I drew the goal of all system developers? I just drew this cause I just start this journey. If I test a strategy that come close to that picture, I am trading it. Please correct me if I am being to simple? Optimize in sample data, Walk Forward Out Sample, Forward test, Go Live, Make money. Is that not simple enough? I could be wrong, I don't know. Still learning as I go.