Metrics to use for system selection

R

ronblack

Suppose you have developed a number of trading systems, all with positive performance (profit factor > 1). They have all tested profitable in OOS. You can use only one of them. Which is the best metric to use for the final selection? Here are some choices/combinations out of the many possible:

1. Profit factor in IS
2. Profit factor in OOS
3. Profit factor in IN+OOS
4. Total profit
5. Sharpe ratio
6. Kelly ratio
7. Mumber of trades
8. Win rate
9. Profit to drawdown ratio

etc.
 
Quote from intradaybill:

Good idea but I think if oss test is good t-stat will look great anyway.

This guy thinks win rate and at the same time maximum possible payoff ratio is the best metric. He has some good points.
More...

Consistently achieving a high win rate greater than 70% as Micheal Harris suggests in the blog post throughout your trading career is woefully unrealistic.


....Not to mention the fact that he consistently tries to make inferences on the basis of tiny sample sizes which doesn't work when it comes to market system trading. I mean come on...he will generate a pattern that had a win rate of 70%+ with a sample size of 30 and scream I have a reliable trading pattern. This is laughable.


In a nutshell, your winners must be larger than your losers AND you must have an ADEQUATE win rate that is sufficient enough to produce a profit. Anything else will blowup eventually....just a matter of time
 
Quote from HurricaneUS:

Consistently achieving a high win rate greater than 70% as Micheal Harris suggests in the blog post throughout your trading career is woefully unrealistic.
More...

I agree. This is why 95% of traders lose. I guess you failed to get the message from his post. If you have win rate of 55%, then the probability of getting 10 losers in a row is 0.03% or 3 in 10,000. If the win rate is 70% the probability is 0.0006% or 6 in 1,000,000. Do you understand the difference? It is 50 times higher risk or ruin.

Quote from HurricaneUS:

....Not to mention the fact that he consistently tries to make inferences on the basis of tiny sample sizes which doesn't work when it comes to market system trading. I mean come on...he will generate a pattern that had a win rate of 70%+ with a sample size of 30 and scream I have a reliable trading pattern. This is laughable.
More...

This is interesting. I read your post here and I went over the Harris' post a few times. At no point he talks about sample size of 30 trades. What do you mean?

Quote from HurricaneUS:

In a nutshell, your winners must be larger than your losers AND you must have an ADEQUATE win rate that is sufficient enough to produce a profit. Anything else will blowup eventually....just a matter of time
More...

Just a moment...Isn't this what Harris says in the first place? The only difference is that Harris argues that win rates below 70% have a high probability of ruin.

So what do you think is an adequate win rate and an adequate ratio of winners versus losers?
 
Quote from HurricaneUS:
....Not to mention the fact that he consistently tries to make inferences on the basis of tiny sample sizes which doesn't work when it comes to market system trading. I mean come on...he will generate a pattern that had a win rate of 70%+ with a sample size of 30 and scream I have a reliable trading pattern. This is laughable.
More...
Ability to figure out if your trading pattern or trade history are real or statistically spurious is the key to this game. Personally, I think that it all boils down to two aspects - number of variables in the system and fundamental explanation for the behavior you are exploiting.

Quote from HurricaneUS:
... you must have an ADEQUATE win rate that is sufficient enough to produce a profit. Anything else will blowup eventually....just a matter of time
More...
Not true. In assets with highly skewed distributions or highly convex products it is OK to have trading processes with win rates way below 50%. The simple example - I know a guy that does insider trading detection (i think he uses some KC methods to combine volume/price patterns), his win rate is way under 25% but he's been in the business about 15 years.
 
Quote from sle:

....Not true. In assets with highly skewed distributions or highly convex products it is OK to have trading processes with win rates way below 50%. The simple example - I know a guy that does insider trading detection (i think he uses some KC methods to combine volume/price patterns), his win rate is way under 25% but he's been in the business about 15 years.
More...

We are saying the same thing......I purposefully omitted a specific win rate. I agree that you can win with a method that wins only 20%-25% of the time so long as the winners are much much larger than the losers on average.

I will say that the vast majority of traders aren't psychologically equipped to trade something that loses money 80% of the time and then the moment you go to the bathroom you've missed out on that huge winner that would've gotten you out of drawdown and then some. That's what separates the pros from the rookies.

An example I've used in the past and how I trade is to take a bunch of scratch trades for little tiny losses while recognizing when I have a winner that I hold to recoup the losses and then some. In this case, winning percentages are irrelevant as I'm not playing the probability side but instead I'm playing the trade management side.

There was another guy that traded this way. His name was Phantom of the Pits...you may have heard of him.



"Most traders plan only for the probability side and that to them is always what they consider the winning side. This is the biggest mistake you can make in trading. You must plan for the losing side." ~Phantom of the Pits



Why is the probability side the losing side for most traders?
Because markets are constantly evolving and they will chew up and send out their anuses your carefully constructed, researched, and perfectly manicured backtested statistics.
 
You must log in or register to reply here.
Back
Top