Avoiding Curve fitting

[Rabbitone]

Quote from maninjapan:

Pretty new to Automated trading and system development, but know enough to understand the pitfalls of curve fitting.
I had an idea how to avoid this when taking a strategy live, just wanted to hear what people think, or their experiences if they have tried it.

Obviously your looking for a robust range of variables when you optimize, but just to be on the safe side, would it be best to run a number of versions of your system, each with different settings, and splitting up your total position equally among them? Obviously not going to help if your system itself is crap, but could help smooth out your total returns, no?


Just as one wouldn't bet their whole account on one trade or stock, why bet your whole system on one set of variables?

Just a thought.....

The answer to your question is simple and yet very complex. The simple answer is your trading plan had better have a pre-defined contingency section it in to tell you under what conditions you will take an automated system offline for fine tuning or discontinue its use. In my plan I call this part of contingency planning the exit plan. The exit plan has risk control exit points for the triggers that are specific to my trading style.

Risk control would take a volume to write about. So, I will give you a few simple examples. Some of the risk control measures I have put in my plan now and in the past are:
- Expectancy of trades must be in-line with realistic performance. Traders are never going to match a testing optimizations performance figures. But the trader must have some guidelines as to what is realistic. I will shut the system down and fine tune if performance goes out of line.
- Statistics Management guides risk control exit points. This is a broad area. It means the optimization the trader produced in testing was loaded with standard deviations. The question is how many deviations are you going to allow in your trading plan for the various outliers, back swans and draw downs your system produces.
- Account Management. What are the maximum risks this system can place on your account per unit of time. For example will you allow the system to take more than 3 or 4 hits of 2% in a month?

The most important of these for me is account management. Think about 2 systems running at the same time. When both of the trading systems are doing great you can make great profits trading them. If one system is doing great and the other bad they cancelled each other out. But if both are doing bad and are in drawdown your account can take quite a hit or you may be forced to shut one or both of the systems down because they break my account management rules.

The number of systems for me to trade is tied to account size, position size and my shut down rule of no more that 6% account loss allowed per month. If I’m running 3 systems my position sizes are ½% to ¾% of account size so that multiple consecutive loses (5 on average) from 2 systems does not in affect invoke my 6% max loss allowed per month rule.

There is no such thing as “ideal number” of systems to run or ideal risk control exit points. It is about how bad you will allow you trading business to go down the tubes before you do something about it.

 

[maninjapan]

Wow, looks like I hit a raw nerve with some posters here. A couple of things to point out here though, Im not talking about a system that is just a bunch of indicators thrown together. I understand that price action is king. But to say a true system based on pa can have no parameters is something I just don't buy into. I now no amount of optimization can make a bad system good, I'm talking about fine tuning a generally robust system. And even if your entries leave no room for optimization there are always those other 2 pesky little details, exits and money management to consider.

 

[TraderZones]

Quote from maninjapan:

Wow, looks like I hit a raw nerve with some posters here...

That usually indicates that you need to seriously consider these points and see if the lack is yours...

 

[Code7]

I think the OP is asking a tough question. Obviously the goal is to have a strategy that will perform well in live trading. But how do you get there? The development process can be a double edged sword. You never know if any decisions you make will improve or worsen the future real money performance. All you have to work with are assumptions, expectations and experience but you can never be certain.

Here are some guidelines I like. I am looking to
- maximize data variety = types of market behavior covered
- maximize sample size = number of trades
- maximize strength of the edge = profit factor
- minimize complexity = rules and parameters

I think no parameters at all is not possible. You always have some sort of constraints, no matter if explicit or implicit. Decisions like time frame, position size etc. have to be made and will affect your trading results. Even if you make all you can adaptive, there are still limits and boundaries.

 

[Rabbitone]

Quote from maninjapan:

Wow, looks like I hit a raw nerve with some posters here. A couple of things to point out here though, Im not talking about a system that is just a bunch of indicators thrown together. I understand that price action is king. But to say a true system based on pa can have no parameters is something I just don't buy into. I now no amount of optimization can make a bad system good, I'm talking about fine tuning a generally robust system. And even if your entries leave no room for optimization there are always those other 2 pesky little details, exits and money management to consider.

Maninjapan

LOL, you didn’t hit any raw nerves. What you are seeing is many experienced traders elaborate on different types of automated trading. Too many newbie traders believe trading with automation is simple (not implying you are a newbie). That is far from the truth. I am still learning after 12 years of automated stock trading in the daily environment (my systems don’t use indicators either but do use input parameters).

It is also difficult to give you a precise answer to your questions because we don’t know many details about how your system works:
- What is it trading? (Trading Stocks is not like futures or forex)?
- What time frame are you using? (Monthly, Weekly, Daily, Intraday minutes or ticks )?
- What is your style of trading? (Position, Swing, Day Trader, Scalper)?

And then we know little of your testing method:
- Back test only? (A single optimization)?
- Back test and a single forward test? (period to period)?
- Back test and multiple forward test? (like year to year)?
- Multiple securities back test and forward test? (For 20 stocks or 10 commodities)?

The list goes on and on. And until some of your testing methods are elaborated on it will be hard to give more intricate replies.

I do agree with intradaybill that non parametric trading is an excellent way for newbies to begin automated trading (Michael Harris I’m not familiar with). Trading with input parameters can be done but often requires a huge learning curve measured in years.

Thanks,

RabbitOne

 

[Random.Capital]

Quote from intradaybill:

Price action does not require any parameters.

Yes, it does. The only way to avoid use of any parameters at all is to completely ignore any price data other than the instantaneous one. And even that is arguable.

 

[Muskoka Joe]

The "markets change" is an excuse for the poor performance of something that was not a true edge to start with. For example a "trend following" strategy that uses a "N" bar breakout or a moving average crossover is NOT an edge. It WILL therefore cease to work when the market stops trending in one direction. Two lines crossing has no relation whatever to character of markets.

There are edges that uncover true character of markets. There are edges that will only cease to work if the market ceases up completely. Of course there will be times when the equity curve is flatter then others.

The true edge strategy has dealt with the conditions in such a manner that equity curve flattens but does not dive. These strategies are based on character of markets, and market cannot outsmart them except by really just stopping.

As far as strategies with no parameters I'd love to see how that is possible. As someone said unless you are using only the last print of price referencing nothing else, you have parameters. Threshold amounts of time or volume or price are all parameters.

 

[intradaybill]

Quote from Random.Capital:

Yes, it does. The only way to avoid use of any parameters at all is to completely ignore any price data other than the instantaneous one. And even that is arguable.

What are they?

 

[Random.Capital]

Quote from intradaybill:

What are they?

Price action is determined by comparing prices at multiple points in time. Right there you have a parameter - the time delta between observations.

 

[TSGannGalt]

Guys...

Let's try to clear each other up with what you consider a "parameter"... Do you consider a "parameter" as a numeric variable? control variable? process? etc. etc.?

Let's say I have the following:

if ([i]metX(varX).ICustomCompare(metY(varY))[/i])
{
     Buy [i]symbolX[/i] [i]sizeX[/i] shares [i]orderTypeX[/i] [i]at[/i] [i]entryPriceX[/i]
}
else
{
     Sell [i]symbolY[/i] [i]sizeX[/i] shares [i]orderTypeY[/i] [i]at[/i] [i]entryPriceY[/i]
}

metX is a delegated method.
metY is a delegated method.
varX is a numerical method.
varY is a numerical method.
ICustomCompare is a control interface.
... etc. etc.

So now I run a BruteForce Optimization with 5+ parameters...

metX = { 1 = Price, 2 = MovAve, 3 = XpoMA}
metY = { 1 = Price, 2 = MovAve, 3 = XpoMA }
varX = {1...20}
varY = {1...20}
ICustomCompare can be implemented with {1 = '>', 2 = '<', 3 = '==', 4 = '!='}
... etc. etc.

Because anyone can run stuff like above... I would consider any kind of object within a trading rule as a parameter. Obviously, this is regardless of Price Action or not. So what about the rest of the guys? What's your definition of a parameter?