Fully automated futures trading

[wopr]

Hi @cholo
Would you mind sharing the python code with the buffering technique, when you find the time? I'd be interested in that.
Thanks a lot.

I’d be interested in this as well @cholo if you get a chance.

Regarding positions, I use Rob’s DO, and similar capital and risk target, and since launching it over a year ago, I’ve always had more positions on, both in terms of just number of positions as well as risk exposure. Also not sure what’s going on.

 

[jiikoo]

Hi Rob, I'm still experimenting with AFTS chapter 25 dynamic optimisation scripts and a question about negative covariance. Along with more instruments and carry & ewmac rule variations I'm getting more and more negative covariance errors, especially on capital levels over 100k. Am I doing something fundamentally wrong here? If you have written about this, can you share a link?

 

[globalarbtrader]

Hi Rob, I'm still experimenting with AFTS chapter 25 dynamic optimisation scripts and a question about negative covariance. Along with more instruments and carry & ewmac rule variations I'm getting more and more negative covariance errors, especially on capital levels over 100k. Am I doing something fundamentally wrong here? If you have written about this, can you share a link?


View attachment 328922

Not exactly what you mean by a negative covariance?

Rob

 

[jiikoo]

Not exactly what you mean by a negative covariance?

Rob

So the problem is that optimisation runs are being interrupted by the following exception:

File ~/pysystemtrade/AFTS/chapter25.py:223, in evaluate_tracking_error_for_weights(weights, other_weights, covariance)
220 track_error_var = solution_gap.dot(covariance).dot(solution_gap)
222 if track_error_var < 0:
--> 223 raise Exception("Negative covariance when optimising!")
225 track_error_std = track_error_var**0.5
227 return track_error_std

Exception: Negative covariance when optimising!

 

[globalarbtrader]

So the problem is that optimisation runs are being interrupted by the following exception:

File ~/pysystemtrade/AFTS/chapter25.py:223, in evaluate_tracking_error_for_weights(weights, other_weights, covariance)
220 track_error_var = solution_gap.dot(covariance).dot(solution_gap)
222 if track_error_var < 0:
--> 223 raise Exception("Negative covariance when optimising!")
225 track_error_std = track_error_var**0.5
227 return track_error_std

Exception: Negative covariance when optimising!

OK so you're using the provided AFTS code and not pysystemtrade. You have to shrink the correlation matrix (see https://qoppac.blogspot.com/2021/11/mr-greedy-and-tale-of-minimum-tracking.html)

"
Essentially, once I started examining the results of my optimisation more carefully, I realised that for large (100+ instrument) portfolios there were instances when my optimisation just broke as it couldn't evaluate the utility function. One cause was inputting costs of nan, and was easily fixed by making my cost deflation function more accurate. But I also got errors trying to find the standard deviation of the tracking error portfolio.

So it turns out that pandas doesn't actually guarantee to produce positive semi-definite correlation matrices, which means that sometimes the tracking error of the portfolio can have a negative variance. I experimented with trying to find the nearest PSD matrix - it's very slow, too slow for backtesting though possibly worth a last line of defence. I tried tweaking the parameters of the exponential correlation; even tried going back to vanilla non exponential correlation estimates but still ended up with non PSD matrices.

What eventually came to the rescue, just as I was about to give up, was shrinking the correlation matrix. For reasons that are too boring to go into here (but try here), shrinkage is a good way of fixing PSD issues. And shrinking the correlation matrix in this particular application isn't really a bad thing. It makes it less likely we'll put on weird positions, just because correlations are especially high or low."

My shrinkage factor is 0.5 and I shrink to an indepedent matrix (off diagionals are zero)

Rob

 

[jiikoo]

OK so you're using the provided AFTS code and not pysystemtrade. You have to shrink the correlation matrix (see
Rob

...and it works! Thanks again, much appreciated!

 

[Kernfusion]

I'm not using pandas, (just a matrix multiplication library for C#), but I have a check in my code before taking square root of portfolio variance - if the variance is negative -> make it zero. Not sure how safe\correct it is and how often it happens, probably need to add some logging around it at least..

 

[Kernfusion]

So I added some logging and yeah, I do get quite a lot of these negative variances..
I want to try to shrink my correlation matrix, but I don't quite understand what it means (sorry if it should be obvious ).
So the paper defines shrinking as "αM1 + (1 − α)M0" where, as I understand, M1 is my actual correlation matrix that I want to 'fix', α should be 0.5 and M0 is some matrix that I will shrink towards (usually previous or average\default in some sense).
That M0, as Rob wrote above is

an indepedent matrix (off diagionals are zero)

, so what values are on-diagonal, all '1'?, i.e. M0 is just an identity matrix same size as mine?
Or we should have something else on the diagonal like average of all pairwise correlations?

And the operation itself, do we just apply "αM1 + (1 − α)M0" element-by-element in matching positions (i.e. just like the normal operations of addition, subtraction and multiplication by a number are defined for matrixes)?

i.e. so far, the way I understand this operation is this:
M1(my initial correlation matrix):
1 |0.35
0.35 |1

M0: (the matrix I shrink towards - identity matrix):
1 |0
0 |1

the result, i.e. my 'fixed' correlation matrix:
0.5*1+0.5*1 |0.5*0.35+0.5*0
0.5*0.35+0.5*0 |0.5*1+0.5*1
=
1 |0.175
0.175 |1

which boils down to simply dividing by 2 each correlation value (except of on-diagonal) in my original correlation matrix
Is that correct ?

 

[globalarbtrader]

So I added some logging and yeah, I do get quite a lot of these negative variances..
I want to try to shrink my correlation matrix, but I don't quite understand what it means (sorry if it should be obvious ).
So the paper defines shrinking as "αM1 + (1 − α)M0" where, as I understand, M1 is my actual correlation matrix that I want to 'fix', α should be 0.5 and M0 is some matrix that I will shrink towards (usually previous or average\default in some sense).
That M0, as Rob wrote above is , so what values are on-diagonal, all '1'?, i.e. M0 is just an identity matrix same size as mine?

Yes when I say 'independent' I mean all off diagonals are 0, so yes that would be the identity.

Or we should have something else on the diagonal like average of all pairwise correlations?

I do sometimes use the average correlation when I'm trying to do a more robust optimisation, but for this use case we want something closer to 0.

And the operation itself, do we just apply "αM1 + (1 − α)M0" element-by-element in matching positions (i.e. just like the normal operations of addition, subtraction and multiplication by a number are defined for matrixes)?

Yes addition for matricies is element by element

which boils down to simply dividing by 2 each correlation value (except of on-diagonal) in my original correlation matrix
Is that correct ?

Yes shrinkage by 0.5 to an identity matrix is the same as dividing everything by two.

Rob

 

[Kernfusion]

Thanks Rob, I already tried to run a backtest with this way of shrinking and the negative variances went away and the performance didn't suffer (in fact the Sharpe got ~10% higher, which is probably an accident). The number of trades went down a little, which can be corrected by the buffer value, or left as is..

regarding this (not for the DO case, but in general):

I do sometimes use the average correlation when I'm trying to do a more robust optimisation

- you mean you would put average correlation to the off-diagonal positions (the same number everywhere instead of zeros), and leave 1s on-diagonal ?