Fully automated futures trading

[globalarbtrader]

GAT,

After a lot of thinking and a little bit of work, I was able to implement this tracking error minimization technique in Excel. Actually, it took less than 2 hours to create and troubleshoot the necessary sheets and write the Visual Basic code.

The only problem is that my so-called greedy algorithm is slow to run because Visual Basic is notoriously slow reading and writing from Excel and there's a lot of that going on (and because I'm working on a 5-year-old laptop). Fortunately, I can run this the prior evening, so it doesn't matter too much.

How long does it take the algorithm to run in Python? I'm sure it's faster, but how much faster? (If I'm reading your code correctly, I believe my greedy algorithm is a slower methodology but one more likely to get closer to the optimal solution, though I'm sure the difference would be small. Next step will be trying to figure out ways to improve the speed of the algo.)

It's a fraction of a second for my current use case: 103 instruments with ~$500K of capital on my production machine (32gb/i7 8 threads / 3ghz). I can run a full backtest with 100 instruments in a less than five minutes, so that's approximatly 10,000 optimisations (40 years *250) on my laptop (16gb/16 threads / 1.8ghz).

That solution is relatively sparse; with more capital there are more possible solutions and hence it would take longer.

GAT

 

[newbunch]

It's a fraction of a second for my current use case: 103 instruments with ~$500K of capital on my production machine (32gb/i7 8 threads / 3ghz). I can run a full backtest with 100 instruments in a less than five minutes, so that's approximatly 10,000 optimisations (40 years *250) on my laptop (16gb/16 threads / 1.8ghz).

That solution is relatively sparse; with more capital there are more possible solutions and hence it would take longer.

GAT

Now I'm jealous.

Obviously, Excel is slow and not designed for what I'm doing. But I don't know Python. It would take a long time to learn how to use it and then even longer to convert my years of work in Excel over to Python.

 

[globalarbtrader]

Now I'm jealous.

Obviously, Excel is slow and not designed for what I'm doing. But I don't know Python. It would take a long time to learn how to use it and then even longer to convert my years of work in Excel over to Python.

Also it's pure python; I could probably make it faster by using a library like CVXPY as suggested a few posts ago (that doesn't implement the greedy algorithim, so not like for like).

GAT

 

[globalarbtrader]

Now I'm jealous.

Obviously, Excel is slow and not designed for what I'm doing. But I don't know Python. It would take a long time to learn how to use it and then even longer to convert my years of work in Excel over to Python.

It's a hard one for sure.

I've said many times before that excel is fine for running an 'out of the box' trading system, since it's just coding up some rules and then combining them linearly. And I know another guy who is running basically my original system, but on excel. But for backtesting or any kind of optimisation, I can't imagine using Excel for that. I mean it's possible, but the amount of work involved must be horrendous.

I last used Excel seriously when I was a sell side options trader, and I look back and think it's amazing what sort of things we managed to achieve; but by god was it terrifying; trillions of pounds in notional risk calculated on disparate sheets with odd hacked up VBA scripts relying on middle office people to copy and paste and press F9 in the right order...

Basically if you want to go down that route you will sure face a lot of up front work, but then you'll gradually get the benefits back in time from that investment. But then given the benefits from this kind of optimisation are not substantial (you won't double your Sharpe Ratio), unless you have quite a large portfolio and are doing this full time (or possibly if you're young and can benefit from learning to code as a career move), I can see why it wouldn't make sense.

GAT

 

[Kernfusion]

Yes I calculate the covariance matrix using the underlying returns of each instrument. I calculate the correlation and the standard deviation seperately; the correlation is configured here (it has a 6 month lookback),

Just to confirm:
in your correlation configuration you have these settings:
using_exponent: True
date_method: "expanding"
rollyears: 20
ew_lookback: 25

- does is mean that every day you add one more point into the return series and calculate correlation on an ever-longer longer series (up to 20 years of points) and the ew_lookback: 25 is only used for calculating the decay parameter for exponential weighting?

This doesn't seem right because then the most recent correlation will be calculated on 20y of data and we want a much-faster reacting measure.
My initial understanding was that we always use only 25 recent weekly returns (in the beginning it'll be less because we don't have enough data, but it never grows above 25) to calculate today's classic Pearson correlation, and then to apply exponential-weighting we also need to calculate decay:
decay = 2/(25+1)
and yesterday's expCorrelation (which was computed the same way but on the previous 25 points), and the final today's correlation will be:
todayExpCorr = yesterdayExpCorr*Decay + (1-Decay)*TodayClassicPearsonCorr

Is the second approach what's actually happening and you don't expand the series to 20 years?

 

[newbunch]

GAT,

After a lot of thinking and a little bit of work, I was able to implement this tracking error minimization technique in Excel. Actually, it took less than 2 hours to create and troubleshoot the necessary sheets and write the Visual Basic code.

It was a fun experiment but I decided not to use this tracking error minimization system. I don't like how it jumps in and out of instruments based more on changes in correlation than in changes of signals. (Perhaps there is a way to reduce this effect, but my efforts towards this goal have failed.) I also think it doesn't behave well with current method of weighting instruments based again on their correlations (perhaps there is a kind of doubling or squaring effect here). Also, in my tests, the benefits of this method are minimal (but positive) versus what I already have.

In the end, I think it's more a matter of taste that I choose not to use this methodology. If I was starting with a basic rounded contract method, I would definitely switch to this tracking error minimization method, but for now I prefer my own methods. I wouldn't be surprised if I revisit this idea in the future and I have already incorporated parts of it into my own system.

Thanks GAT for the thought-provoking post.

 

[globalarbtrader]

Just to confirm:
in your correlation configuration you have these settings:
using_exponent: True
date_method: "expanding"
rollyears: 20
ew_lookback: 25

- does is mean that every day you add one more point into the return series and calculate correlation on an ever-longer longer series (up to 20 years of points) and the ew_lookback: 25 is only used for calculating the decay parameter for exponential weighting?

This doesn't seem right because then the most recent correlation will be calculated on 20y of data and we want a much-faster reacting measure.
My initial understanding was that we always use only 25 recent weekly returns (in the beginning it'll be less because we don't have enough data, but it never grows above 25) to calculate today's classic Pearson correlation, and then to apply exponential-weighting we also need to calculate decay:
decay = 2/(25+1)
and yesterday's expCorrelation (which was computed the same way but on the previous 25 points), and the final today's correlation will be:
todayExpCorr = yesterdayExpCorr*Decay + (1-Decay)*TodayClassicPearsonCorr

Is the second approach what's actually happening and you don't expand the series to 20 years?

No basically what we do is first take the last 20 years of data, and then calculate the exponential weighting on that 20 years with the appropriate halflife.

This first step is basically optional, since the influence of a day of returns twenty years ago will be miniscule given the half life of a couple of months; it's just to speed the calculations up a bit. This is actually a hangover from my original config elements where I'm estimating the correlation of trading strategy returns, for which I use a much longer halflife. So I could reduce it but it won't make any difference (just to speed things up).

Also important to note this line: frequency: "W": We''re dealing with weekly, not daily returns. So we're talking about roughly 1000 observations (~52*20) on which we calculate the exponential weighting. The ew_lookback parameters is the span in this formula. The decay parameter alpha = 2/(span+1) = 0.0769

If my maths is right that gives a half life of just over 8 weeks: a couple of months.

(On reflection, this is a bit faster than I intended, so I will probably change it to get a half life of about 6 months- after checking that doesn't trash the backtest results)

To reiterate I could certainly reduce the 20 years figure above to speed things up (in fact I've just done it), but if I reduced it further to less than say a couple of years it would start to affect the calculations; in an extreme case if I reduced it to less than twice the halflife it would clearly affect the results quite a lot.

GAT

 

[Kernfusion]

No basically what we do is first take the last 20 years of data, and then calculate the exponential weighting on that 20 years with the appropriate halflife.

This first step is basically optional, since the influence of a day of returns twenty years ago will be miniscule given the half life of a couple of months; it's just to speed the calculations up a bit. This is actually a hangover from my original config elements where I'm estimating the correlation of trading strategy returns, for which I use a much longer halflife. So I could reduce it but it won't make any difference (just to speed things up).

Also important to note this line: frequency: "W": We''re dealing with weekly, not daily returns. So we're talking about roughly 1000 observations (~52*20) on which we calculate the exponential weighting. The ew_lookback parameters is the span in this formula. The decay parameter alpha = 2/(span+1) = 0.0769

If my maths is right that gives a half life of just over 8 weeks: a couple of months.

(On reflection, this is a bit faster than I intended, so I will probably change it to get a half life of about 6 months- after checking that doesn't trash the backtest results)

To reiterate I could certainly reduce the 20 years figure above to speed things up (in fact I've just done it), but if I reduced it further to less than say a couple of years it would start to affect the calculations; in an extreme case if I reduced it to less than twice the halflife it would clearly affect the results quite a lot.

GAT

I see, but after you've applied the exponential weighting to the 5 years of weekly returns, do you calculate Pearson correlation on all the 260(5*52) points (not on just the last 75)?
In that case, wouldn't the old returns in the series affect the correlation as much as the recent ones? I.e. the exponential weighting will practically eliminate the influence of the old returns in the most recent ones, but the oldest points themselves will still remain in the series unchanged, and if correlation is then calculated on the whole series, the old returns will be there and will impact the correlation result equally to the most recent points?
(perhaps I don't understand well how exponential weighting works for correlation or how it's implemented in pandas..)

 

[globalarbtrader]

I see, but after you've applied the exponential weighting to the 5 years of weekly returns, do you calculate Pearson correlation on all the 260(5*52) points (not on just the last 75)?
In that case, wouldn't the old returns in the series affect the correlation as much as the recent ones? I.e. the exponential weighting will practically eliminate the influence of the old returns in the most recent ones, but the oldest points themselves will still remain in the series unchanged, and if correlation is then calculated on the whole series, the old returns will be there and will impact the correlation result equally to the most recent points?
(perhaps I don't understand well how exponential weighting works for correlation or how it's implemented in pandas..)

I think you're a bit confused about how exponential weighting works. We exponential weight the last 5 years, and they will get steadily decreasing weights.

Let's take a simple example. Suppose your decay is 0.25 (an extreme example)

The weights on each correlation (or whatever we're calculating) going back in time are a*(1-a)^k:

r_t: 0.25 (k=0)
r_t-1: 0.25*0.75 = 0.1875
r_t-2: 0.25*0.75^2 = 0.140625
t_t-3: 0.1055
....
r_t-10: 0.0141
....
r_t-20: 0.0008

We basically have weights going back as far as we like. If we had five years of data, we'd still have a weight on the data points five years ago (about t-1250 for business days); but it would be only about 1.7*10^-157, or to use jargon f*** all.

The sum of those weights going back in time asymptotes to 1. After three observation it's 0.578 (so the half life is somewhere between 2 and 3). it's After 10 observations it's 0.943. After 20 it's 0.997. And so on.

Now I can cut off my data series before I calculate my correlations. So for example, I could cut it off using only the last 10 observations (assuming I haven't passed a min_periods into the pandas exponential function). That means that the weights on the last 10 observations would equal 1* instead of 0.943. And the weights on anything before the last 10 observations would be zero, instead of (1-0.943 =) 0.057. This will result in small differences in our calculations, but it won't make a lot of difference.

Similarly, if I cut off the last 5 years when my half life is about 6 months then I'm not going to be missing very much relevant earlier data with substantial weights.

* this happens because the correlation is actually calculated recursively so the weights are implicit and not explicit, calculated forward in time.

GAT

 

[Kernfusion]

I think you're a bit confused about how exponential weighting works. We exponential weight the last 5 years, and they will get steadily decreasing weights.

Let's take a simple example. Suppose your decay is 0.25 (an extreme example)

The weights on each correlation (or whatever we're calculating) going back in time are a*(1-a)^k:

r_t: 0.25 (k=0)
r_t-1: 0.25*0.75 = 0.1875
r_t-2: 0.25*0.75^2 = 0.140625
t_t-3: 0.1055
....
r_t-10: 0.0141
....
r_t-20: 0.0008

We basically have weights going back as far as we like. If we had five years of data, we'd still have a weight on the data points five years ago (about t-1250 for business days); but it would be only about 1.7*10^-157, or to use jargon f*** all.

The sum of those weights going back in time asymptotes to 1. After three observation it's 0.578 (so the half life is somewhere between 2 and 3). it's After 10 observations it's 0.943. After 20 it's 0.997. And so on.

Now I can cut off my data series before I calculate my correlations. So for example, I could cut it off using only the last 10 observations (assuming I haven't passed a min_periods into the pandas exponential function). That means that the weights on the last 10 observations would equal 1* instead of 0.943. And the weights on anything before the last 10 observations would be zero, instead of (1-0.943 =) 0.057. This will result in small differences in our calculations, but it won't make a lot of difference.

Similarly, if I cut off the last 5 years when my half life is about 6 months then I'm not going to be missing very much relevant earlier data with substantial weights.

* this happens because the correlation is actually calculated recursively so the weights are implicit and not explicit, calculated forward in time.

GAT

This makes sense, but I still didn't get whether we should apply exp weighting to returns or correlations or both.
And when calculating each new observation of correlation (before applying exp-weighting to it) should we use the whole return history up to that point or just the recent X returns (where x is the lookback, e.g. 25 as in our case).
I tried several options in excel and none of them matches with what pandas is doing with exponential correlations. I'm obviously doing something wrong.. maybe I'll just get pandas source-code and see what exactly it does there..
Here's my google-excel attempts, the first page is editable (but feel free to ignore it, as I feel that I'm just being dumb here..)
https://bit.ly/3G1SoWQ
or on GitHub: https://github.com/kernfusion43/test