System Performance Score

kut2k2 · Feb 28, 2013

This thread is in a way a continuation of my previous thread in this forum which has been closed to prevent the reappearance of trolls who posted during the new format. Thanks for the great timing, guys.

That thread got me to thinking about what an ATS is really worth, but in nonmonetary terms.

Thanks to inputs from both danielc1 and dom993, I was introduced to Van Tharp's SQN (System Quality Number) as well as its fatal flaw. As a result, I've been able to devise a much superior metric called SPS (System Performance Score) which is detailed below.

I asked myself what is the single truly best way to evaluate an ATS. After a while, it hit me: how much are you willing to bet on it. The best answer to that question is unambiguous: the Kelly ratio (or Kelly fraction).

As detailed in the first post of my Kelly thread (qv), there is an easy way to calculate a very good approximation of the Kelly fraction for a general case. I will hereafter refer to this as the NOBF (near-optimal betting fraction).

As detailed in my previous ATS thread, we must establish a minimum acceptable number of trades in order to evaluate an ATS from backtesting data. We were introduced to both the Surf Criterion (1000 trades) and the Eckhardt Criterion (1800 trades). For now we will stick with the Surf Criterion.

So calculation of SPS becomes straightforward:

SPS = NOBF*min[1, N/1000] , where N is the number of trades in the backtest.

So take the N trade returns from your backtest and sum them. If the answer is nonpositive, proceed no further: your system sucks. If the answer is positive, then divide that sum by the sum of the squared trade returns. This quotient is your NOBF.

Now multiply the NOBF by the lesser of one and N/1000. This final number is the SPS of your ATS.

Enjoy.

nonlinear5 · Mar 15, 2013

Quote from kut2k2:

So take the N trade returns from your backtest and sum them. If the answer is nonpositive, proceed no further: your system sucks. If the answer is positive, then divide that sum by the sum of the squared trade returns. This quotient is your NOBF.
Now multiply the NOBF by the lesser of one and N/1000. This final number is the SPS of your ATS.

Something is missing here. Let's apply your formula to two systems, A and B:

System A trades: [+10, +10, +10, +10]
System B trades: [+20, +20, +20, +20]

SPS for A: (10 + 10 + 10 + 10) / (10^2 + 10^2 + 10^2 + 10^2) = 0.1
SPS for B: (20 + 20 + 20 + 20) / (20^2 + 20^2 + 20^2 + 20^2) = 0.05

According to these results, system A is twice better than system B, whereas intuitively (and by common sense), it's the other way around.

kut2k2 · Mar 15, 2013

Quote from nonlinear5:

Something is missing here. Let's apply your formula to two systems, A and B:

System A trades: [+10, +10, +10, +10]
System B trades: [+20, +20, +20, +20]

SPS for A: (10 + 10 + 10 + 10) / (10^2 + 10^2 + 10^2 + 10^2) = 0.1
SPS for B: (20 + 20 + 20 + 20) / (20^2 + 20^2 + 20^2 + 20^2) = 0.05

According to these results, system A is twice better than system B, whereas intuitively (and by common sense), it's the other way around.

What you've proven is that the formula doesn't work for holy grail systems. You should be so lucky.

Remember what the NOTF is an approximation of. The exact Kelly fraction is the value of k that makes the following equation true.

0 = sum[ Ri/(1+ k*Ri) ]_i=1toN

When all values of Ri are positive, the value of k that satisfies the Kelly equation is infinity.

Go see the "Bad Kelly" thread to see other limitations on the NOTF.

nonlinear5 · Mar 15, 2013

Quote from kut2k2:

What you've proven is that the formula doesn't work for holy grail systems.

I don't think it works for unholy systems, either. Here is another example:
System A: [+10, -5, +10, -5]
System B: [+20, -10, +20, -10]

SPS(A) = 0.04
SPS(B) = 0.02

In this case, your formula says that system A is twice better than system B, but it's actually exactly the same system, scaled.

kut2k2 · Mar 15, 2013

Quote from nonlinear5:

I don't think it works for unholy systems, either. Here is another example:
System A: [+10, -5, +10, -5]
System B: [+20, -10, +20, -10]

SPS(A) = 0.04
SPS(B) = 0.02

In this case, your formula says that system A is twice better than system B, but it's actually exactly the same system, scaled.

No it's not the exact same system. In system A you lose five times your bet, in system B you lose ten times your bet. The betting fractions reflect that significant difference.

nonlinear5 · Mar 16, 2013

Quote from kut2k2:

No it's not the exact same system. In system A you lose five times your bet, in system B you lose ten times your bet. The betting fractions reflect that significant difference.

There is nothing there that indicates the bet sizes. For all we know, system A bet 5% of the account, and system B bet 50% of the account on each trade. Or it could be the other way around.

kut2k2 · Mar 17, 2013

Quote from nonlinear5:

There is nothing there that indicates the bet sizes.

The point of the NOBF formula is to determine the bet size. What I posted about was the bet returns.

kut2k2 · Mar 18, 2013

I've concluded that nonlinear5 has pointed out a valid weakness in the original formula.

So here is the revision :

SPS = NOBF*R*min[1, N/1000] , where R is the average winning trade return (%).

syswizard · Mar 18, 2013

Quote from kut2k2:

I've concluded that nonlinear5 has pointed out a valid weakness in the original formula.

So here is the revision :

SPS = NOBF*R*min[1, N/1000] , where R is the average winning trade return (%).

Is this the correct Excel UDF calculation ?

Function UDFSysPerfScore(ByRef pbRng As Range, Optional pMinTrades As Long = 1000) As Double
'
' System Performance Score for any range of returns
'
Application.Volatile False
Dim i As Long
If Not pbRng Is Nothing Then
UDFSysPerfScore = WorksheetFunction.Sum(pbRng) / WorksheetFunction.SumSq(pbRng) * WorksheetFunction.Average(pbRng) * WorksheetFunction.Min(1000, WorksheetFunction.Count(pbRng) / pMinTrades)
End If

End Function

Just place it in any cell and point to the range with the returns:
=UDFSysPerfScore(B2:B1220)

nonlinear5 · Mar 18, 2013

Quote from kut2k2:

I've concluded that nonlinear5 has pointed out a valid weakness in the original formula.

So here is the revision :

SPS = NOBF*R*min[1, N/1000] , where R is the average winning trade return (%).

Now you are mixing the absolute P&L and the percentage P&L in the same formula. This will lead to confusing results.

I think if you experiment long enough, you'll come down to this:

SPS = sqrt(trades) * (P / S),
where
trades is the total number of trades
P is the average profit per trade
S is the standard deviation of all trades

This closely relates to Sharpe's ratio and the SQN (System Quality Number). The above formula is what I use to rank and compare my trading systems.