How do you avoid overfitting or over-optimization in your backtest?

M

mizhael

I heard about "Walk Forward Analysis",

to me it's just like a dynamic rolling-window out-of-sample test coupled with optimization.

The result of this "Walk Forward Analysis" is a set of "optimized" parameters that are suitable for all history(it's robust, but kind of conservative).

Am I right?

Any good ways of avoiding over-fitting or over-optimization?

Thanks!
 
create a complexity number for each function... give the functions with the least complexity the most risk capital, and test the results.

I have no idea what the results you'll get, but by evaluating the complexity of parameters should avoid over optimization

You might be able to evaluate different strategies at the same time as well:

http://en.wikipedia.org/wiki/Cyclomatic_complexity
 
Quote from mizhael:

Any good ways of avoiding over-fitting or over-optimization?

Thanks!
More...

One way is by using price action formations. You replace the problem though with selection bias. One can have either curve-fitted signals or signals that are selected over others but you cannot avoid both, at least I do not know of a way of doing that. I prefer selection bias over curve-fitting.

Examples

(1) Curve-fitting: SMA(a) > SMA(b), adjust a and b for profitability

(2) Selection-bias: close > high(2) has been profitable but not close > high(1), etc. You select the profitable one.
 
When optimizing system variables, I look for "fragile" settings. This is where say 37 is profitable but 36 and 38 perform abysmally. The 37 setting is probably profitable due to one or two lucky trades it took advantage of.

I like to see ranges of system variable values that are profitable. Then I pick one in the middle rather than the one that produced the best results.
 
Quote from stevegee58:

When optimizing system variables, I look for "fragile" settings. This is where say 37 is profitable but 36 and 38 perform abysmally. The 37 setting is probably profitable due to one or two lucky trades it took advantage of.

I like to see ranges of system variable values that are profitable. Then I pick one in the middle rather than the one that produced the best results.
More...

Agreed, which is why I prefer thinking of this process as evaluating robustness rather than optimizing parameters.
 
Quote from stevegee58:

When optimizing system variables, I look for "fragile" settings. This is where say 37 is profitable but 36 and 38 perform abysmally. The 37 setting is probably profitable due to one or two lucky trades it took advantage of.

I like to see ranges of system variable values that are profitable. Then I pick one in the middle rather than the one that produced the best results.
More...

There is no such a thing as a "somewhat pregnant" woman. The same goes about optimization. The issue is whether a system is optimized or not. Any system that utilizes a function to generate signals is optimized with respect to some objective function.
 
Well, right. Even a moving average crossing system has to be run through an optimizer to find which combinations of MA periods work and which don't. You can't just pick 2 at random.
 
Quote from stevegee58:

Well, right. Even a moving average crossing system has to be run through an optimizer to find which combinations of MA periods work and which don't. You can't just pick 2 at random.
More...

Well, the point is that regardless of the choice of paramaters, such system is always optimized. You can always find an objective function that any such system maximizes. The lesson is that if you have any parameters at all in your system, it is optimized, whether you actually optimized it or not.

I tell you, very few understand what I wrote above but I know a few do.
 
You must log in or register to reply here.
Back
Top