Fully automated futures trading

[globalarbtrader]

So, was anybody trading front oil contract?

I hope not Even if you were short, that was a dangerous game to play.

I've banged on enough times about avoiding the front contract in futures when you can, especially for something that is physically delivered. You don't want a perfectly decent position being murdered by weird technical stuff or game playing happening around the roll.

Sometimes we have no choice, like in bond futures, but then we roll as early as possible (which for Korean bonds, sadly, is the morning of the expiry).

A relevant anecode: I once did some research and discovered that there was a mismatch between the backadjusted price curve of US bond futures in two different databases we were using at AHL. After some digging, it turned out that one curve was using expiry dates to roll, and the other was using the dates when we actually rolled, normally a week or so before the first notice date, which itself is a while before the expiry. Further testing established that the difference was higher the earlier you rolled the contract.

There was a small but signficant premium to holding the long position in a contract longer until the expiry date. Of course you couldn't actually realise that premium if you didn't want to risk being delivered into the contract, or getting caught in the panic of the last day or so before the notice date.

Even in cash settled contracts the premium is there. An extreme example is VIX where the carry premium tends to increase dramatically the close you get to the front of the curve. You pay for that in terms of much worse kurtosis and skew.

GAT

 

[AvantGarde]

https://www.businesswire.com/news/h...-Brokers-Issues-Statement-Crude-Oil-Contracts

IB lost nearly $90m on the recent WTI futures move down to -$37:

"Several Interactive Brokers LLC (“IBLLC”) customers held long positions in these CME and ICE Europe contracts, and as a result they incurred losses in excess of the equity in their accounts. IBLLC has fulfilled the firm’s required variation margin settlements with the respective clearinghouses on behalf of its customers. As a result, the Company has recognized an aggregate provisionary loss of approximately $88 million."

 

[quiper]

Hello Rob,

as non-programmer I am trying to go slowly through your pysytemtrade, learn and try to code different stages in my platform (amibroker).

I notice there are sometimes big gaps in prices in directory adjusted_prices_csv – like in ES500 - https://github.com/robcarver17/pysystemtrade/blob/master/data/futures/adjusted_prices_csv/SP500.csv. There is a jump from 10.5 (on 12.3.2018) to 2808.75 (on 13.3.2018). Is this an error in the data or something I am missing in the overall process of rolling?

Thank you.

 

[globalarbtrader]

Hello Rob,

as non-programmer I am trying to go slowly through your pysytemtrade, learn and try to code different stages in my platform (amibroker).

I notice there are sometimes big gaps in prices in directory adjusted_prices_csv – like in ES500 - https://github.com/robcarver17/pysystemtrade/blob/master/data/futures/adjusted_prices_csv/SP500.csv. There is a jump from 10.5 (on 12.3.2018) to 2808.75 (on 13.3.2018). Is this an error in the data or something I am missing in the overall process of rolling?

Thank you.

It's an error, which I am in the process of solving (once I finish another bit of code first)

GAT

 

[Kernfusion]

Hi Rob,
You finally published the risk-overlay post that you described briefly several years ago on this forum, thanks!
Meanwhile I implemented it myself in my system based on your earlier descriptions., but now comparing it with your code here (https: //gist.github.com/robcarver17/fd921a6d0146090ec1a2335c3ec28395 ) I noticed a discrepancy., in short, are you sure that this code that calculates covariance matrix is correct?:

def sigma_from_corr_and_std(stdev_list, corrmatrix):
stdev = np.array(stdev_list, ndmin=2).transpose()
sigma = stdev * corrmatrix * stdev
return sigma

I think it should dot-multiply correlation matrix with diagonal matrices made of standard deviations instead of doing simple multiplications with transposed std-matrices., i.e. like this:

def sigma_from_corr_and_std(stdev_list, corrmatrix):
sigma = np.diag(stdev_list).dot(corrmatrix).dot(np.diag(stdev_list))
return sigma


I mean, from what I read online (e.g. here: https://www.riskprep.com/all-tutorials/36-exam-22/58-modeling-portfolio-variance), the usual way to calculate portfolio risk, having correlation matrix, standard deviations and weights of the assets in the portfolio is this:

#given w, std, corr. matrix:
w = np.array([0.5, 0.3, 0.2])
std = [0.11, 0.17, 0.22]
#correlation matrix is ones on the diagonal and pairwise correlations in other cells
cor = np.array([[1, 0.5, 0.3], [0.5, 1, 0.2],[0.3, 0.2, 1]])

#then portfolio standard deviation is
w.dot( np.diag(std).dot(cor).dot(np.diag(std)) ).dot(w.transpose()) **0.5
#0.11276790323491877

#and what your code is doing is essentially this:
w.dot( np.array(std) * cor * np.array(std) ).dot(w.transpose()) **0.5
#which gives a different result: 0.11567022088

I mean essentially to get the covariance matrix we want to multiply every correlation value in each cell by the 2 corresponding standard deviations, and the 2 multiplications with diagonal std matrices is doing just that.
Or did I get something wrong ?

 

[globalarbtrader]

Hi Rob,
You finally published the risk-overlay post that you described briefly several years ago on this forum, thanks!
Meanwhile I implemented it myself in my system based on your earlier descriptions., but now comparing it with your code here (https: //gist.github.com/robcarver17/fd921a6d0146090ec1a2335c3ec28395 ) I noticed a discrepancy., in short, are you sure that this code that calculates covariance matrix is correct?:

def sigma_from_corr_and_std(stdev_list, corrmatrix):
stdev = np.array(stdev_list, ndmin=2).transpose()
sigma = stdev * corrmatrix * stdev
return sigma

I think it should dot-multiply correlation matrix with diagonal matrices made of standard deviations instead of doing simple multiplications with transposed std-matrices., i.e. like this:

def sigma_from_corr_and_std(stdev_list, corrmatrix):
sigma = np.diag(stdev_list).dot(corrmatrix).dot(np.diag(stdev_list))
return sigma


I mean, from what I read online (e.g. here: https://www.riskprep.com/all-tutorials/36-exam-22/58-modeling-portfolio-variance), the usual way to calculate portfolio risk, having correlation matrix, standard deviations and weights of the assets in the portfolio is this:

#given w, std, corr. matrix:
w = np.array([0.5, 0.3, 0.2])
std = [0.11, 0.17, 0.22]
#correlation matrix is ones on the diagonal and pairwise correlations in other cells
cor = np.array([[1, 0.5, 0.3], [0.5, 1, 0.2],[0.3, 0.2, 1]])

#then portfolio standard deviation is
w.dot( np.diag(std).dot(cor).dot(np.diag(std)) ).dot(w.transpose()) **0.5
#0.11276790323491877

#and what your code is doing is essentially this:
w.dot( np.array(std) * cor * np.array(std) ).dot(w.transpose()) **0.5
#which gives a different result: 0.11567022088

I mean essentially to get the covariance matrix we want to multiply every correlation value in each cell by the 2 corresponding standard deviations, and the 2 multiplications with diagonal std matrices is doing just that.
Or did I get something wrong ?

You are correct.

Ironically, I've just finished writing a long twitter thread where one of my comments is about low quality research code

A confession. I often find the implementation of matrices and vectors in pandas/numpy downright confusing. Another confession, I don't write enough tests.

The difference is much clearer with a more pathological example

std = [1, 4, 9]
cor = np.array([[1, 0.25, 0.9], [0.25, 1, 0.5],[0.9, 0.5, 1]])

Although it doesn't make much difference to the results of any optimisations or the original risk overlay (as long as standard deviations are in the normal range for financial assets), I have fixed it and named the commit in your honour https://github.com/robcarver17/pysystemtrade/commit/b31ac7e2b4292db0e665d46960820dc402b0c85e

Thanks again. This is the benefit of open sourcing!

GAT

 

[JMW]

Greetings! A question for you on data and how to handle. I am deep in the muck trying to get the rolls code to successfully stitch together some adjusted prices using pysystemtrade. I gave up on quandl data and bought some historical contract price series from an Australian outfit called Premium Data: looks pretty good so far, cleaner than Quandl's.

I have the following issue with a handful of the instruments, mainly energy, e.g. BRENT_W, CRUDE_W. Your roll configuration only uses the December contract for pricing. However, with my price series data, before a certain point in time, the contract price series are shorter than one year and do not join together. e.g. my BRENT_W Dec 1992 contract has prices from 1992-02-07 to 1992-11-13 not enough to join to the Dec 1991 contract.

This could of course mean my data series is not as clean as I think it is. I cannot compare with the contracts data you used to create your roll calendars and adjusted prices (not in GitHub).

Either way, let's assume the data is good and that this is all there is. I think I have two choices. 1) ignore data before 2000 (when prices for contracts start to have a much wider date range, and December contracts can be successfully joined together) or 2) manually manipulate the roll calendar and insert some rows mid way between the December contracts, say, June.

How "incorrect" would option 2) be? I am wary that the June contract may be behave in subtly different ways to the December contract.

Thanks much!

 

[globalarbtrader]

Greetings! A question for you on data and how to handle. I am deep in the muck trying to get the rolls code to successfully stitch together some adjusted prices using pysystemtrade. I gave up on quandl data and bought some historical contract price series from an Australian outfit called Premium Data: looks pretty good so far, cleaner than Quandl's.

I have the following issue with a handful of the instruments, mainly energy, e.g. BRENT_W, CRUDE_W. Your roll configuration only uses the December contract for pricing. However, with my price series data, before a certain point in time, the contract price series are shorter than one year and do not join together. e.g. my BRENT_W Dec 1992 contract has prices from 1992-02-07 to 1992-11-13 not enough to join to the Dec 1991 contract.

This could of course mean my data series is not as clean as I think it is. I cannot compare with the contracts data you used to create your roll calendars and adjusted prices (not in GitHub).

Either way, let's assume the data is good and that this is all there is. I think I have two choices. 1) ignore data before 2000 (when prices for contracts start to have a much wider date range, and December contracts can be successfully joined together) or 2) manually manipulate the roll calendar and insert some rows mid way between the December contracts, say, June.

How "incorrect" would option 2) be? I am wary that the June contract may be behave in subtly different ways to the December contract.

Thanks much!

Neithier option is ideal of course. The main issue with switching from summer to winter is that it may cause the carry rule to behave weirdly. In some cases though, eg GAS, we just have to live with it (similar things happen with Bunds, and again we just have to live with it).

Doesn't answer your question, but might help. Personally I'd choose (1) because it's less work, but (2) is also valid.

GAT

 

[HobbyTrading]

I have a question about your latest blog post https://qoppac.blogspot.com/2020/05/when-endogenous-risk-management-isnt.html I tried to ask it over there, but I fear that my question did not come through.
In the section titled "Why does expected risk vary?" you post the formula:

position as % of capital = (instrument forecast / average instrument forecast)* (target risk / instrument risk) * instrument weight * IDM

Is the term "average instrument forecast" a fixed value? Or is it something that you allow to vary over time? I had the impression from the "Systematic Trading" book that it is a fixed value (10). However, another interpretation of the word "average" could imply that you take the average over a certain period of time. And thus that it can fluctuate as time passes by.

 

[globalarbtrader]

I have a question about your latest blog post https://qoppac.blogspot.com/2020/05/when-endogenous-risk-management-isnt.html I tried to ask it over there, but I fear that my question did not come through.
In the section titled "Why does expected risk vary?" you post the formula:

position as % of capital = (instrument forecast / average instrument forecast)* (target risk / instrument risk) * instrument weight * IDM

Is the term "average instrument forecast" a fixed value? Or is it something that you allow to vary over time? I had the impression from the "Systematic Trading" book that it is a fixed value (10). However, another interpretation of the word "average" could imply that you take the average over a certain period of time. And thus that it can fluctuate as time passes by.

No it's fixed at 10. Perhaps 'target average' is a more precise term, though also more confusing.

GAT