Sanity check on sharpe ratio calculation please

Do you think *any* of this is statistically significant?

GAT
Party pooper. :D

Just to clarify, "statistical significance" is an empirical measure, and aside from violating I.I.D. assumptions by constantly stringing together sequentially-created, highly NOT I.I.D. data as pretty much *everybody* in the financial world does [:banghead::banghead::banghead:], there are certainly enough observations to qualify.

If you want to offer it into other areas of fault, you'd certainly be in good company,...
https://oxfordstrat.com/ideas/sharpe-ratio/
 
I'll be honest, the fact that I only have data going back to 2010 bothers me. As everyone knows, the world has been in a bull market for a long time and even though this is pure FX - actually the algorithm relies on relationships between countries - I need to test it at war time. ForexTester doesn't seem to have any quality data going back to 2000.
 
I don't know. How should I find out?

First you need some accurate SR figures. I'm not sure exactly what you've done, but that doesn't look like a SR 4.0 curve - that's going to be very close to a straight line. And also extremely likely to be overfitted to hell. I'd say the good bits are about SR 2.0 at best, and overall about 1.7.

[Sharpe Ratios suck in many ways, but it's very easy to get a good intuition about them]

By eye I would say your annual Sharpe Ratios are roughly:
2010-2012 and end 2015 to now about 2.0 for ~N=88 periods
akward period about 0 for ~N=30 periods
2010 - now about 1.7, for ~N=118 periods

Let's assume:

  • that returns are iid (and looking at your backtest they don't look too far away at least on a monthly basis) in which case we can use this as the standard deviation of the SR estimator: sqrt[(1+.5SR^2/N)] https://cran.r-project.org/web/packages/SharpeR/vignettes/SharpeRatio.pdf
  • you have returns every single week (if you have blank weeks with no trades then the confidence intervals will be much wider)
  • you have absolutely no forward looking overfitting. I'm very skeptical that you can get a SR 1.7 strategy without overfitting, but if you can that's wonderful.
  • your returns will be perfectly stable into the future- basically impossible, but let's live with it


Say the returns are monthly (if iid won't matter if you have higher frequency) then your monthly SRs are:
[divide annual by sqrt(12)]
2010-2012 and end 2015 to now: about 0.60 for ~N=88 periods
akward period about 0 for ~N=30 periods
2010 - now about 0.5, for ~N=118 periods

Your standard deviation of estimators is:
sqrt[(1+.5SR^2/N)]

2010-2012 and end 2015 to now: 0.115
akward period: 0.183
2010 - now: 0.098

And the 95% confidence intervals for these SR are:

u - 1.96s, u+1.96s where u is average SR and s is standard deviation of estimator

2010-2012 and end 2015 to now: 0.37 to 0.83
akward period: -0.36 to 0.36
2010 - now: 0.31 to 0.69

Shit, who understoods monthly SR. Not I. Let's convert to annual (*sqrt12)

2010-2012 and end 2015 to now: 1.3 to 2.9
akward period: -1.2 to 1.2
2010 - now: 1.1 to 2.4

So what can one conclude? Well the SR in the bad period might be pretty bad (-1.2) or actually very good (+1.2). It's just not long enough to tell. And overall you might have a very good strategy (1.1) or an extremely good strategy (2.4). The good news - if you make all the assumptions above - is that at worse it's a good strategy.

And what if you do find something 'happening' in 2012-2015? What are you going to do - change your model? Well done you've just used future data (what I call 'implicit' fitting) and your Sharpe Ratio will be overstated - at best - and at worst you will have turned something that might work in the future into an overfitted POS.

GAT

[You were in luck: I can do this stuff very fast as this very day I was working on my slides for the course I run on systematic trading in January... so had the spreadsheet ready to go!]
 
I'll be honest, the fact that I only have data going back to 2010 bothers me.

Good - now you're thinking cynically.

I though this is pure FX - actually the algorithm relies on relationships between countries - I need to test it at war time. ForexTester doesn't seem to have any quality data going back to 2000.

FX... hmmm I've never found a strategy that was profitable in retail FX as the costs are way, way too high. So I'd be even more skeptical.

GAT
 
FX... hmmm I've never found a strategy that was profitable in retail FX as the costs are way, way too high. So I'd be even more skeptical.

I just have an unnatural draw to FX. I think the 24/6 nature + the size of it makes me believe it to be more pure.
 
I just have an unnatural draw to FX. I think the 24/6 nature + the size of it makes me believe it to be more pure.

How much are your annual costs in your backtest as % of capital per year? What's your annual standard deviation of returns?

There is no evidence to backup your intuition. Apart from binary options spot FX is the biggest rip off out there.



GAT
 
First you need some accurate SR figures. I'm not sure exactly what you've done, but that doesn't look like a SR 4.0 curve - that's going to be very close to a straight line. And also extremely likely to be overfitted to hell. I'd say the good bits are about SR 2.0 at best, and overall about 1.7.

[Sharpe Ratios suck in many ways, but it's very easy to get a good intuition about them]

By eye I would say your annual Sharpe Ratios are roughly:
2010-2012 and end 2015 to now about 2.0 for ~N=88 periods
akward period about 0 for ~N=30 periods
2010 - now about 1.7, for ~N=118 periods

Let's assume:

  • that returns are iid (and looking at your backtest they don't look too far away at least on a monthly basis) in which case we can use this as the standard deviation of the SR estimator: sqrt[(1+.5SR^2/N)] https://cran.r-project.org/web/packages/SharpeR/vignettes/SharpeRatio.pdf
  • you have returns every single week (if you have blank weeks with no trades then the confidence intervals will be much wider)
  • you have absolutely no forward looking overfitting. I'm very skeptical that you can get a SR 1.7 strategy without overfitting, but if you can that's wonderful.
  • your returns will be perfectly stable into the future- basically impossible, but let's live with it


Say the returns are monthly (if iid won't matter if you have higher frequency) then your monthly SRs are:
[divide annual by sqrt(12)]
2010-2012 and end 2015 to now: about 0.60 for ~N=88 periods
akward period about 0 for ~N=30 periods
2010 - now about 0.5, for ~N=118 periods

Your standard deviation of estimators is:
sqrt[(1+.5SR^2/N)]

2010-2012 and end 2015 to now: 0.115
akward period: 0.183
2010 - now: 0.098

And the 95% confidence intervals for these SR are:

u - 1.96s, u+1.96s where u is average SR and s is standard deviation of estimator

2010-2012 and end 2015 to now: 0.37 to 0.83
akward period: -0.36 to 0.36
2010 - now: 0.31 to 0.69

Shit, who understoods monthly SR. Not I. Let's convert to annual (*sqrt12)

2010-2012 and end 2015 to now: 1.3 to 2.9
akward period: -1.2 to 1.2
2010 - now: 1.1 to 2.4

So what can one conclude? Well the SR in the bad period might be pretty bad (-1.2) or actually very good (+1.2). It's just not long enough to tell. And overall you might have a very good strategy (1.1) or an extremely good strategy (2.4). The good news - if you make all the assumptions above - is that at worse it's a good strategy.

And what if you do find something 'happening' in 2012-2015? What are you going to do - change your model? Well done you've just used future data (what I call 'implicit' fitting) and your Sharpe Ratio will be overstated - at best - and at worst you will have turned something that might work in the future into an overfitted POS.

GAT

[You were in luck: I can do this stuff very fast as this very day I was working on my slides for the course I run on systematic trading in January... so had the spreadsheet ready to go!]

I think I just had an orgasm reading this. Very clear, thank you. Feel like plugging your class? Is it online? I'd attend.

To answer your questions: this isn't a "holy grail", I know it would need tweaking as markets change. The relevant inputs are "100" and "200" and "0" just from intuition looking at recent markets. I've had this idea in my head for the last year, which is why I was able to eyeball it reasonably well for the last period. I have not optimized the parameters at all.

What I would really want to know is: can I take a look at what happens in the awkward periods and make an intuitive, correct decision about what the inputs should be.

There are basically 3 dimensions along which this algorithm works, some are more important than the other. It is very likely due to a change in human factors, that one of these changes during the awkward periods. The market-related inputs, I would be wary of changing. The human-related inputs, I think I would have a very good idea.

Trying it out now...
 
How much are your annual costs in your backtest as % of capital per year? What's your annual standard deviation of returns?

There is no evidence to backup your intuition. Apart from binary options spot FX is the biggest rip off out there.

Does it matter though that the costs are 5% or 20% if I'm achieving the desired returns? I don't see FX being the only thing I do, but I do want to get close to being able to reliably get good returns before moving onto something else. I'm not too sensitive about cost _right now_ as a result.
 
Does it matter though that the costs are 5% or 20% if I'm achieving the desired returns? I don't see FX being the only thing I do, but I do want to get close to being able to reliably get good returns before moving onto something else. I'm not too sensitive about cost _right now_ as a result.

Yes it matters. A strategy earning 10% with 1% costs is a lot safer than one earning 60% with 51% costs because costs are certain and returns aren't.

You MUST know your costs or you can't analyse your returns properly. And we can't help you check to see if your costs are realistic.

GAT
 
I think I just had an orgasm reading this. Very clear, thank you. Feel like plugging your class? Is it online? I'd attend.

To answer your questions: this isn't a "holy grail", I know it would need tweaking as markets change. The relevant inputs are "100" and "200" and "0" just from intuition looking at recent markets. I've had this idea in my head for the last year, which is why I was able to eyeball it reasonably well for the last period. I have not optimized the parameters at all.

What I would really want to know is: can I take a look at what happens in the awkward periods and make an intuitive, correct decision about what the inputs should be.

There are basically 3 dimensions along which this algorithm works, some are more important than the other. It is very likely due to a change in human factors, that one of these changes during the awkward periods. The market-related inputs, I would be wary of changing. The human-related inputs, I think I would have a very good idea.

Trying it out now...

What you propose is in sample fitting and so bad. Sorry

GAT
 
Back
Top