System Performance Statistics & Evaluation

Quote from Mathemagician:

The higher the ratio the better.

Yes, that is the paper.

M
More...

Thanks, MM!

Very insightful paper. Will surely reread it. Of course the higher the better...but what are your preferred/ideal levels? What do you typcially see in your experience?

Finally how can one get access to the testing software refeernced in that paper. I would surely like to use it.
 
Quote from CPTrader:

I’m curious as to what system performance statistics (profit factor, net profit, net profit/DD, Mar , Sharpe ratios, Win/Loss%, etc) are most important to systematic traders.

Also what are the most important factors in your system evaluation and in determining that a system is viable for live trading?

Please let’s open this to debate and general discussion.

Thank you.
More...


I suggest reading The Way of The Turtle by Curtis Faith.
The book answers your question in detail. For example, net profit alone is not a good indicator since most of the profit could have come from a single trade.

Also, High Probability Trading my Marcel Link has an excellent section on strategy development and backtesting.

Asking this on ET will only get you a thousand different opinions and will confuse you. The two authors above give good explanations of what to look for.
 
Quote from CPTrader:

Thanks, MM!

Very insightful paper. Will surely reread it. Of course the higher the better...but what are your preferred/ideal levels? What do you typcially see in your experience?

Finally how can one get access to the testing software refeernced in that paper. I would surely like to use it.
More...
Thank you. (I am the author of that paper, by the way.) The levels seen can vary widely and depend on many factors. The testing software is software that I wrote specifically for the purpose of developing and testing my trading models. The software itself isn't commercially available, but the algorithm is part of a 3-day workshop on system development that is being given at the end of October. Building this tool should be no sweat after that.

M
 
Quote from Mathemagician:

(I am the author of that paper, by the way.) The levels seen can vary widely and depend on many factors.
More...
Unfortunately, you can't run unlimited simulations
over trading system backtests. Resampling techniques
(bootstrap, jackknife, etc...) introduce complexities of
bias and variance beyond the ken of most ET'ers.
For this reason, historital max or average drawdown
is not a good estimator of future max or average
drawdown.

I am sure you are aware of this. I am not sure that
you know, however, that for normal or near normal
distributions of model returns, historical volatility and
mean return yeild better estimates.

For the case of zero mean return (no alpha, the case
with most models discussed here) the formula ifor
expected max drawdown is simple:

1.25 * Stdev * sqrt(Time).

For the positive mean return the estimate gets
a little more complex:

(2 * QP((Time / 2) * ((MeanReturn / Stdev)^2)) / MeanReturn

The QP function, AFAIK, does not have an
analytical solution. You can download a lookup
table for it, however, at this website:

http://www.cs.rpi.edu/~magdon/data/Qp.txt

Interestingly, for a given Sharpe Ratio this
works out to a linear relationship scaling
with the square root of time. For example,
for the Hershey method, with a claimed
daily basis annualized Sharpe Ratio of 5.5,
the equation is

0.50 * Stdev * sqrt(Time)

To sum up, the inforation in the "new" ratio
under discussion is implicit in the Sharpe
Ratio itself.
 
Quote from Kevin Schmit:

Unfortunately, you can't run unlimited simulations
over trading system backtests. Resampling techniques
(bootstrap, jackknife, etc...) introduce complexities of
bias and variance beyond the ken of most ET'ers.
For this reason, historital max or average drawdown
is not a good estimator of future max or average
drawdown.

I am sure you are aware of this. I am not sure that
you know, however, that for normal or near normal
distributions of model returns, historical volatility and
mean return yeild better estimates.

For the case of zero mean return (no alpha, the case
with most models discussed here) the formula ifor
expected max drawdown is simple:

1.25 * Stdev * sqrt(Time).

For the positive mean return the estimate gets
a little more complex:

(2 * QP((Time / 2) * ((MeanReturn / Stdev)^2)) / MeanReturn

The QP function, AFAIK, does not have an
analytical solution. You can download a lookup
table for it, however, at this website:

http://www.cs.rpi.edu/~magdon/data/Qp.txt

Interestingly, for a given Sharpe Ratio this
works out to a linear relationship scaling
with the square root of time. For example,
for the Hershey method, with a claimed
daily basis annualized Sharpe Ratio of 5.5,
the equation is

0.50 * Stdev * sqrt(Time)

To sum up, the inforation in the "new" ratio
under discussion is implicit in the Sharpe
Ratio itself.
More...
Excellent discussion, Kevin. Unfortunately, the models that I work with tend to have nowhere near a normal distribution of returns, and I suspect this is true for others as well. Historical average drawdown does work very well both in theory and in practice to the extent that the historical return series is unbiased (using a biased input series is the biggest problem with nearly all analysis done by system developers) and the underlying assumptions of any statistical procedures are satisfied. The assumptions are nearly always violated to some extent, and this must be considered when interpreting output.

M
 
Quote from Kevin Schmit:


For the case of zero mean return (no alpha, the case
with most models discussed here) the formula ifor
expected max drawdown is simple:

1.25 * Stdev * sqrt(Time).

For the positive mean return the estimate gets
a little more complex:

(2 * QP((Time / 2) * ((MeanReturn / Stdev)^2)) / MeanReturn
More...

Kevin,

Do you have a reference for the formulas you provide above. I'd like to look into this a bit more.
 
Backtesting of a system is more or less useless. Real time trading results you will get from a system will be about 20% below the results you get from backtesting. When a chart is forming it looks quite different from completed chart and you can’t simulate that in backtesting. For example, system I presently trade will give me close to 100% winners/losers ratio in backtesting, in real time I get about 80%.
 
You must log in or register to reply here.
Back
Top