Intentional Curve Fitting

H

Here2learn

Does anybody intentionally curve fit a strategy to optimize results for the immediate time period.

A thought I've been toying with is to backtest a strategy for the most recent five days and cuve fit it to those five days and use the resulting optimum settings to trade the sixth day.

Or, perhaps operate a strategy with whatever settings showed optimum results for the last month... or whatever time frame you choose.

Essentially, the thought is that market behavior fluctuates and so for optimum results, a strategy will produce better results when optimized for those market conditions. This of course is using historical performance, so an assumption would be that there is high probability of the market behaving today as it did the last five days(or this month, as it did the last three months, or whatever your timeframe).

Any thoughts? Anybody doing something similar? I could use some input.
Thanks
 
I tried it for intra-day trading, 10 days test, with 1 day out-of-sample. This was done with machine learning and about 10000 systems. About 80% of the systems lost, and many of those lost more than they earned in the 10-day curvefit.

I did it to explore curve-fitting about a year ago, so I minimized all of the parameters that prevent curve-fitting (something I would avoid for real systems), and I used indicators that are noisy (and thus more prone to curve-fits). An interesting result was that using daily data curve-fitting is less pronounced for same number of bars. This is basically because of the trending component and the fact that daily data is less noisy than 1-minute data, so it's not rocket science.

If you are just optimizing parameters (and not creating new systems) this curve-fit might be less pronounced but still very significant. I read a paper that talked about how different strategies are more vulnerable to optimization and curvefitting, namely how you send and backtest the order fills (also, is it low volatility or high volatility strategy, etc). There are many components to it, so you won't get a straight answer, other than this: In the vast majority of cases, this is a bad idea, especially if you are not using robust in-house and accurate backtesting software.
 
I think that is a viable approach to optimization as long as you control the process and testing. By example you optimize every week-end and use the settings for the next week. Repeat weekly.

That approach is easy to back test (sometimes called walk-forward optimization) and is a solid approach.

The biggest challenge is that if you have "too many" parameters you can get a lot of parameter jumping, which I don't think is valid. I have tried imposing a "local optimization" technique where the new values have to be close to the prior values to overcome this issue. Works well at times but it highly system dependent.
 
Good comments from both of you!

Yes, walk forward optimization is exactly what I am talking about.

I think as you said, braincell, the way that the backtest fills orders is key. I have a lot of trouble getting accurate backtests due to the fill accuracy, so maybe this is not a good idea using ninja trader.

Also, I agree about not having too much variability in the parameters. I would use just a few small parameter changes, but could efffectively change the profit outcome significantly.

What is the best way to determine the length of the sample period before the out of sample? Are there any standards for this such as 4 to 1 ratio(4 sample periods to 1 out of sample)?

Thanks!
 
when they first invented computers, they would have guys that would backtest for you

I quickly found out that any system that made or lost money was curve fitted

it was very difficult to come up with something that would consistently break even minus the spread and commissions

but once you developed one, then you had something to work with
 
Quote from oldtime:

when they first invented computers, they would have guys that would backtest for you

I quickly found out that any system that made or lost money was curve fitted

it was very difficult to come up with something that would consistently break even minus the spread and commissions

but once you developed one, then you had something to work with
More...

For clarification, is that to say if you have a strategy that consistently breaks even or better(less commision/slippage) across multi-assets, multi-timeframes, and multi-parameters, that it is a viable candidate for walk forward optimization???

Note the word 'multiple', not 'all' time frames/assets/parameters.
 
Quote from Here2learn:

For clarification, is that to say if you have a strategy that consistently breaks even or better(less commision/slippage) across multi-assets, multi-timeframes, and multi-parameters, that it is a viable candidate for walk forward optimization???

Note the word 'multiple', not 'all' time frames/assets/parameters.
More...
no, if it does better than break even it is curve fitted

same if it loses money

not much point in doing it intentionally since by default almost everything already is

keep in mind, we had very little accurate historical data. NYSE and cash wheat
 
Quote from Here2learn:

Good comments from both of you!

Yes, walk forward optimization is exactly what I am talking about.

I think as you said, braincell, the way that the backtest fills orders is key. I have a lot of trouble getting accurate backtests due to the fill accuracy, so maybe this is not a good idea using ninja trader.

Also, I agree about not having too much variability in the parameters. I would use just a few small parameter changes, but could efffectively change the profit outcome significantly.

What is the best way to determine the length of the sample period before the out of sample? Are there any standards for this such as 4 to 1 ratio(4 sample periods to 1 out of sample)?

Thanks!
More...

Usually the walk forward test period is 20% of the sample data. So if I back tested and optimized a strategy on 10 years data, I would walk forward the system on two years unseen data.
 
Having lots of parameters means many degrees of freedom to match trade entries and exits serendipitously with price action. So the performance appears good, but not because the strategy with those parameters has any predictive power or statistical edge; rather it’s because 10^7 parameter sets were tried, and the one with the best results was member of the small percentage that randomly generated what appears to be acceptable performance. “But correlation [i.e. with good performance] does not imply causation.” You can end up just being fooled by randomness… Taking the “best” outcome can be risky, although perhaps suggestions like those made above re “local optimizations” might reduce this risk to some degree.

IMO looking for the parameter set that is “best in tune with the market right now” is fraught with risk, and not only for the reason outlined above. It is also because it ignores that signal-to-noise ratios for trading systems are usually pitifully small, and there can be long, extended periods when no signal at all is present. In my experience, a valid approach in these no signal situations is simply to keep trading as you were, and accept the drawdown, but remain in a position to benefit immediately once the signal comes back again. With a “walk forward optimization” approach on the other hand, this fallow period with no signal may lead to strategy parameters drifting considerably from what is best over the long term, and may even result in the return of signal being missed altogether. And in my experience to date, the period immediately following return of signal is often the best run, that more than makes up for the drawdown.
 
You must log in or register to reply here.
Back
Top