Optimization of a trading system with avoidance of curve fitting.

A

albertly

There is very interesting dialog between market wizard in the book of Bruce Babcock "The Business One Irwin Guide to Trading Systems". The dialog was taken from Commodity Traders Consumer Report that assembled six prominent individuals: Robert Pardo, Jack Schwager, Bo Thunman, Steven Kille, Robert Pelletier, Thomas Hoffman.

The subject of the dialog was - "Optimization".

Everyone of them has strong opinion on the subject - but no agreement.

We should understand that is was not pure academic discussion - all of them has his own commercial interests. Don't be naive.

For example Robert Pardo - sells software, so obvious he has strong optimization supporter.

On other side Jack Schwager says that there is no any evidence that optimization has predictive power more than just flipping a coin.

Thomas Hoffman has scientific approach to the problem.

After dialog the author of the book makes some conclusions:
A. limit the number of optimizable rules
B. optimize over broad ranges rather than choosing the very best values
C. maximize the number of closed trades in the test
D. test the same parameters over multiple markets
E. test the optimized system over data not used for the optimization

My conclusions and my way of optimization
1. I cannot build the system without optimization.
2. My system uses rather big number of rules. Here I get into the problem of degrees of freedom. My solution (I don't know how correct it is - the common sense is not so common) I never make optimization on 1 security. I take a set of 30-40 securities and make optimization on them. In such a way I have a few thousand closed trades. As a results, I think, I have statistical significance of optimization even with a lot of rules.
3. The found parameters I check on different sets of securities (preferable from different markets)

The concise process of my optimization:

1. divide securities into sets (each set includes at least 20 securities, mixed set more ~ 60)
1.1 Large Caps (DJ)
1.2 Mid Caps (S&P)
1.3 Small Caps (Russell)
1.4 Large Caps (Local Market)
1.5 Mid Caps (Local Market)
1.6 Small Caps (Local Market)
1.7 Bonds
1.8 Futures
2. Random Set
3. Mixed Set
4. A set that I am going to trade

5. Optimization/Test on Mixed Set
6. After getting best results on Mixed Set, check other set (but not a set I am going to trade)
7. If and only if I have profit improvements on almost all set I can make conclusion that my idea / optimized parameters are not random and not curve fitting. So I apply it to my trading set to check delta improvements of profit (profit with old system / profit with new one). All this on historical data of course.
7.1 If not I try to find what's wrong. Why it works on one market but doesn't on another. It's very important and difficult step. What's went wrong - volatility, trending, unusual gaps , whatever - I try to find an explanation.

8. As a final step, very important for me to check everything on different time periods.


P.S. I trade only stocks/ETFs.
 
short, but wrong.

There is no way to use indicators without using parameters.

Hard-coded values of parameters can be the same evil as optimized ones.

Even you have parameters if you use chart patterns technique.
stop loss, targets, position sizing - everything is parameters that somehow has values.


It is very serious topic. And your answer is pointless.
 
the old, totally false story " i dont optimize"

if you dont optimize, your strategy must produce zero profits.
such system is not optimized for sure :D
 
Thanks for your interesting post.

However, I still have mixed feelings about having a system optimized over plenty of markets and timeframes like you do.

Of course such a system should be robust but what I've found is it has usually poor results in terms of Net Profit/ Max DrawDown.

On the other hand, you can adapt to the character of a market or even find a temporary edge in 1 timeframe and 1 market which will be much more profitable.

The drawback is to be able to quickly drop the strategy once it doesn't work because it will obviously be less robust.

Optimization is everywhere, no one can claim he doesn't use it: even the mere choice of using a system over an other one is optimization.
 
Quote from albertly:

Short, but wrong
More...

If you already know, why are you asking?

If your system has even one parameter which you adjust based on some criteria, then this is a fitted system and it will fail.
 
Quote from intradaybill:
If your system has even one parameter which you adjust based on some criteria, then this is a fitted system and it will fail.
More...

I'm curious to see a system wihout any parameter: can you please give us a short example ?
 
Quote from Joman:

Thanks for your interesting post.


On the other hand, you can adapt to the character of a market or even find a temporary edge in 1 timeframe and 1 market which will be much more profitable.
More...

But what do you do not to fall into curve fitting trap?

You can measure predictive power of optimization with walk forward test.

Actually I think that only technology can make drastic changes in basic market behaviour. For example internet in 1999-2000 has very big impact on all trading industry - new crowd involved, new method of trading, discount brokers etc.
Computers earlier made big impact on involving more people in trading.

Besides of that - nothing new - up trend, down trend, side walk - period of big volatility, low volatility- nothing new under the sun.

I talk about stock market only as I have no judgment about Forex, futures, etc because it is out of my interest.
 
Quote from albertly:

But what do you do not to fall into curve fitting trap?
More...

- Check how the neighborhood of your paramaters inflences your result.

- Check what percentage of the tested parameters produces positive results.

- Test in sample and out of sample.

- Walk-forward analysis.

- Real Time demo testing.

- Use tools to quickly stop the strategy once the edge evaporates like a moving average of your equity curve.
 
You must log in or register to reply here.
Back
Top