System Performance Metrics

J

jleatherman

I have created several trading systems and would now like to assess which systems did best. I thought before reading an article on CAGR, that that is how I should measure system performance, e.g., take the accumulated CAGR's of every issue traded within a system divided by the number of issues to yield an average CAGR for the system, then compare that value to similar calculations for other systems. Then I read another CAGR article on "CAGR, the good, the bad, & the ugly" and now I'm not sure whether that is the correct approach.

Now I know that one could say one of two things:

1. I want to use a system that has the highest hit rate a.k.a. accuracy.

2. I want to use a system that has the highest total return a.k.a. return.

But what I would like to know is how to pick the best system of a group of systems that eliminates this accuracy versus return dichotomy and shows me which system was the most efficient a.k.a. the system that has the highest accuracy with return also taken into consideration when issues traded within any one system were bought and sold on different time periods?

For instance, I have two systems that I started at the beginning of this year:

System A has a hit rate or accuracy of wins to losses of 54%, a total return of 2% since January 1, 2006, and a averaged CAGR of all of the issues bought/sold under this system of 781%.

System B has a hit rate of 63%, a return of 1%, but an averaged CAGR of only 266%.

That being said, which of these two systems is the "better" system, not in return, not necessarily in accuracy, but the system that was capable of possibly predicting the best performing stocks from randomly occurring (daily closing stock prices) events?

Would you have a possible solution on how rating the better system should be done?
 
Google these terms and learn about them

MAR Ratio
Sharpe Ratio
Calmar Ratio
Sortino Ratio
Ulcer Index
Return Retracement Ratio
Semideviation Sharpe Ratio aka Downside Deviation Sharpe Ratio
% of winning days (weeks, months)
Seykota Lake Ratio
 
Quote from jleatherman:

....

Would you have a possible solution on how rating the better system should be done?
More...

There needs to be a risk parameter ... that is how much risk does the system take to achieve the return ..... depending upon your definition of accuracy this might already be included.

If the metrics are to further your own goals then simply design metrics that result in an overall quality improvement .... if the metrics are to entice resale or investors then you need to use standard measures...
 
First, I don't think there is a single reward/risk ratio that is best.
Second, which ones to use depends on what you are doing.
Third, a system going forward will rarely be better than a backtested system. So even if you optimize a system, you should probably still be more conservative.

Some good risk measure I like:
1. Standard deviation: Of course, are we talking about daily stdev, weekly, or monthly? The industry standard is monthly and that seems to work best with my long term trading system, but short term systems might want to try daily instead.
2. Volatility adjusted exposure: I'm not sure if you'll find anything on the web about this, since I made it up myself but I'm sure others have thought of it too. Example: If I'm 100% long the S&P and 100% long the 10-year note and the 10-yr note has a volatility 50% of the S&P, your volatility adjusted exposure is 150% of the S&P.
3. Max drawdown / Value at Risk (VAR): This one is self-explanatory and you can find tons of info on the web.

I've created others, but they are more system specific and wouldn't apply to most other systems.

Another suggestion, if you can, is to make a graph of reward/risk on the Y axis and annualized returns on the X axis. It can be quite informative. I've attached an example (hopefully I did it correctly).
 

Attachments

You have already learned something very important: most of what you read day in, day out is written by losers for losers.
Your questions are very important, but you are the only one who can answer those. You may need more research.
Quote from jleatherman:

I have created several trading systems and would now like to assess which systems did best. I thought before reading an article on CAGR, that that is how I should measure system performance, e.g., take the accumulated CAGR's of every issue traded within a system divided by the number of issues to yield an average CAGR for the system, then compare that value to similar calculations for other systems. Then I read another CAGR article on "CAGR, the good, the bad, & the ugly" and now I'm not sure whether that is the correct approach.
More...

Right prt.
Now you also find lots of talk about risk. But risk of what?
In fact, the only thing to watch is "risk of ruin". This is relatively easy to come by for games of chance rigorously adhering to well understood probalistic definitions.
Much of what is peddled for speculation is a carbon copy of these theories. Not very often will one hear that these theories don't necessarily mean much, given that market models are radically different.

Quote from prt_systems:
There needs to be a risk parameter ... that is how much risk does the system take to achieve the return ..... depending upon your definition of accuracy this might already be included.

If the metrics are to further your own goals then simply design metrics that result in an overall quality improvement .... if the metrics are to entice resale or investors then you need to use standard measures...
More...
 
I really appreciate the responses here and a special thanks to horribilicus for an especially quick and informative reply!

Thanks again everyone, I've got some studying to do here.

Jim
 
You must log in or register to reply here.
Back
Top