Fully automated futures trading

[FrankieC84]

After a few weeks of reading, made it through the whole thread (what a resource); on top of my reading of AFTS (and halfway through Systematic Trading - should have started with this one). Wanted to express my gratitude to Rob, as well as all the other insight participants on this forum.

I have been running a manual system (I don't have strong coding skills so it is a bunch of crazy excel worksheets that sound like how CTAs used to do it 30 years ago) for the past 18 months that has evolved from trying to reinvent the wheel in terms of system structure (while targeting carry & trend); to evolving to something very close to Rob's static system as I learned best practices. Currently have ~30 instruments in the system; plus 5 european equity index futures that I run with a related cross-sectional system; plus a long bias to bonds (I add a constant value to my forecast that) as I don't otherwise have bond exposure in non-futures portfolio.

Three areas where I do things slightly different that I would be curious of everyone's thoughts no.
(1) Related to Kernfusion post about asset class weights:

I think with handcrafting the top level weight allocation shouldn't really be 1\n, exactly because some groups have too few instruments in them (or we should try to come up with some even higher-level groups e.g. 'financials' for all stocks, bonds and volatility).. And on the other hand, the groups with many diversifying instruments should probably receive higher allocation.. E.g. My agriculture top-level group has 19 instruments (and it contains weird\diversifying things like Coffee, Milk and Orange Juice) and my InterestRate group only 6, so it sort of makes sense to give more weight to agriculture.. This logic makes things kind of arbitrary I think, because we could end up forcing the group weight that we (almost subjectively) like.. Here's my current top-level weights, and I can't really explain how I came up with them, they started as the result of pysystemtrade optimization, but I then adjusted them manually..
View attachment 338397

I have put a lot of thought into this issue, as for example there are a lot of grains (4) and developed market FX (5) with low vol*notional that I can trade with my relatively modest portfolio size. Having more instruments within an asset classs should provide a higher asset class level sharpe ratio (given diversification benifit), and thus lead to a higher optimal weight vs. assets classes with fewer instruments. My initial solution was to assign highest level asset class (i.e. commoditiy, FX, bonds, equity, vol) a handcrafted weight, but then assign the weight to each sub asset classes (i.e. within developed country FX and emerging country FX) based on the sqrt of the number of instruments in each assets class. So for example if I have 5 developed market FX instruments and 3 emerging market FX instruments than I allocated my FX weight as sqrt(5)/((sqrt(3)+sqrt(5)) = 0.564 to the developed market FX subgroup and sqrt(3)/((sqrt(3)+sqrt(5)) = 0.436 to the emerging market subgroup (and then equal weight within the subgroup). This approach was motivated by a post Rob awhile back. A benifit of it is that as I add more instruments to my system that the weights automatically update to account for the added diversification within each sub asset group.
I recently further refined this approach to account for that fact that by weighting by the sqrt of the # of instrumetns, I am implicitly assuming that each instrument within the subasset class has a zero correlation (which is needed for the sharpe ratio to increase by the sqrt of N and thus the sqrt weighting to make sense). I therefore modified the weigthing formula to account for the correlation between instruments between a subasset group. To simplify things I use an average of the correlation across all instruments within the subasset class. Within getting into the weeds of the forumla, if I assume that correlations of the 5 developed market FX instruments is 0.65 and correlations of 3 emerging market instruments is 0.35 than I get a 0.47 weight to developed market FX and a 0.53 weight to emerging market FX.

(2) Size of System / Relation to Overall Portfolio
It seems like most people here run the system given a certain notional size and volatility target (i.e. $500K at 25% vol). I take a different approach, which is to target risk as a proportion of my overall portfolio. Right now my risk target is 6%; so if my overall portfolio $1m, I set my vol*notional at $1m * 0.06 * 2.5 IDM = $180K. This level adjusts every day then based on how my overall portfolio does. I think that the benifit of this apporach from an asset allocation propective is that my allocation to my futures strategy stayes fixed relative to all of my other allocations. I guess the disadantage is that if my futures trading is in reality a money loser, I will keep throwing money at it (i.e. as it loses money to the rest of the portfolio, I would in effect rebalance into it to keep it a the fixed % relative to my overall portfolio).

(3) Cross Sectional System
The eurpean sector futures have always facinated me, but I didn't have enough capital to allocate to enough of them in my regular system to (in my mind) take full advantage of their existance). At some point in Rob's past to do lists, I remember him mentioning potentially running a cross-sectional system on the european sector futures (maybe I am misrembering and apologies if so). So I recently took a shot at this, and was wondering what everyones thoughts are (given my limited coding skills I havn't been able to run backtests, but think that it is well supported by academic literature on cross-sectionally targeting factors in equities). The basic set up of the system is as follows (which I am running at 1% vol- 1/6th of my regular system):

(i) Take forecasted values (from regular system) for european equity indices and demean (I am actually not perfectly demeaning as I am allowing for a small long or short bias so that I have net exposure that matches my target to the european equities in my overall system).
(ii) Otherwise allocate to instruments identically as in regular system.

There are some quirks in this approach, such as that instruments are no longer bound at +/- 20 (for example in an instrument had a +20 forecast, but the average was -5, the demeaning would cause it to have a +25 forecacst). Also because of discreate nature of instrument sizes it is not exactly market netural. Also the european equity index futures are expense to trade relative to other instruments (still below speedlimit) which is annoy.

Anyway curious of everyones thoughts about these depatures and thanks again for all the help insight that has bee provided in the 413 pages of this amazing thread. I am working further automate my currently hacked together system with proper coding (am working with a computer science friend to that end).

*And one last note. In the spirit HWM discussoin on this forum, hit a sort of HWM on April 30th, said it out loud, and have since had three days, each of which would qualify as among my worst days in recent months. Still working on the trading rule for that one.

 

[handelsmeisterei]

I have a question regarding passive rolling, if you do not mind: Let's say we want to implement the roll mechanism via DO constraints. You would have to look up the contracts in your portfolio that need rolling on the roll date and set the position constraint to the number of contracts of that instrument that do not need rolling (e.g. they already passively rolled). This way your order generator will then only close the contracts that need rolling. The next day DO may decide to reopen, if not you save some costs.

So far so good. But how do you go about deciding which contracts to passively roll? I can look up in my roll calendar when I want to roll. But what do I do before that? Do I open new positions in farther away contracts based solely on liquidity (e.g. above 100 contracts traded the prior day)?

 

[globalarbtrader]

I have a question regarding passive rolling, if you do not mind: Let's say we want to implement the roll mechanism via DO constraints. You would have to look up the contracts in your portfolio that need rolling on the roll date and set the position constraint to the number of contracts of that instrument that do not need rolling (e.g. they already passively rolled). This way your order generator will then only close the contracts that need rolling. The next day DO may decide to reopen, if not you save some costs.

So far so good. But how do you go about deciding which contracts to passively roll? I can look up in my roll calendar when I want to roll. But what do I do before that? Do I open new positions in farther away contracts based solely on liquidity (e.g. above 100 contracts traded the prior day)?

Use default parameters? [no to change]y
AUTO ROLL RULES:

Irrespective of the following, we will automatically roll if a contract has expired and no position

The test for forward being liquid:
  - if relative volume between current and forward contract > 1.000000, then considered liquid (and no need to check absolute volume)
  - if relative volume between current and forward contract > 0.010000, and if absolute volume contracts>100, then considered liquid

Forward is not liquid. Are we close to the roll point? (is distance to desired roll date<10 days)
   -  No, miles away from needing to roll. Trade as normal: NO_ROLL
   -  Yes, going to roll quite soon. Roll status should be NO_OPEN 

Forward is liquid. Do we have a position on in the price contract??
   - We have no position in the priced contract: ROLL ADJUSTED
   - If we have a position on then:
      - Do we have plenty of time? (is distance to desired roll date>10 days)?
         - Yes, We have plenty of time PASSIVE ROLL
         - No, we don't. We will prompt user for required roll state  - (recommend Force, Force_Outright or Close)

 

[handelsmeisterei]

Forward is liquid. Do we have a position on in the price contract??
   - We have no position in the priced contract: ROLL ADJUSTED

Thank you very much for that explanation! Digging through pysystemtrade, I found the following explanation for ROLL ADJUSTED:

RollState.Roll_Adjusted: "Roll adjusted prices from existing priced to new forward contract (after adjusted prices have been changed, will automatically move state to no roll",

I have trouble understanding this. I would have assumed that if the forward is liquid and no position is on, then simply buy the forward? Otherwise you will have to wait till the roll date, then forward becomes priced and the adjusted prices are "correct" again. Or does your system automatically adjust the roll calendar the next day so that forward becomes priced? As long as the state is ROLL ADJUSTED you do not open additional positions in that instrument, right? I guess I must have overseen something...

 

[globalarbtrader]

Thank you very much for that explanation! Digging through pysystemtrade, I found the following explanation for ROLL ADJUSTED:

RollState.Roll_Adjusted: "Roll adjusted prices from existing priced to new forward contract (after adjusted prices have been changed, will automatically move state to no roll",

I have trouble understanding this. I would have assumed that if the forward is liquid and no position is on, then simply buy the forward? Otherwise you will have to wait till the roll date, then forward becomes priced and the adjusted prices are "correct" again. Or does your system automatically adjust the roll calendar the next day so that forward becomes priced? As long as the state is ROLL ADJUSTED you do not open additional positions in that instrument, right? I guess I must have overseen something...

The code should only briefly go into the roll adjusted state, so that it can actually process the roll (moving the current priced contract forward one roll); once that happens it automatically goes back into the no rolling state. So you'd only stay in roll adjusted if something went wrong.

Rob

 

[handelsmeisterei]

ok, thank you! It is more of a failsafe then that can detect buggy roll calendars (if staying in that state)

 

[wopr]

I'm about a week in into my adaptation of handcrafting for forecast weights, I think I got it working.
For those looking to implement, but are not using pysystemtrade, I've found it to be a sweet spot to hook into optimiseWeightsOverTime (https://github.com/robcarver17/pysy...squant/optimisation/optimise_over_time.py#L14) - that way you can compute your own net returns, but still use Rob's entire optimization logic, which can be a bit complex. That also allows you to pass in all the estimation parameters from the config. Nice API design there Rob, not sure if you were planning for that, but definitely nice to use sort of as a library.

I'm pooling gross returns, but using individual costs to get net returns. Using 50000 lookback for everything (per the default config), so basically all the data.

I'm getting 2 issues so wanted to see if those of you that have implemented it are seeing the same, or have some tips.

1. SR adjustment can be a bit extreme - it will completely eliminate some rules that get 10% allocation before adjustment - likewise, it will allocate 10+% to some rules that had no allocation before adjustment. Typical problem is carry and ags markets, I'm getting 2% or less combined allocation to all carry rules for ags. The reason is that my data history starts at 2001, and carry did exceptionally bad in ags from 2006 to 2008, and has only just now barely recovered. I've checked the allocations before and after the SR adjustment, and can confirm that before it's has a solid allocation (eg. 30% for Cocoa), and after has almost nothing (1.4% for Cocoa). Changes in weights per year can also be extreme, I see some rules getting 5% one year, 19% next year, then back down to 6% year after.

2. Overweight to more diversifying rules, in my case skew. I sometimes see 18-20% allocation to a single skew rule (I run 3 variations).

To make the above percentages somewhat meaningful, I'm running 43 rule variations, pretty much same as Rob, without abs relative skew and mean reversion.
Appreciate the help and tips!

 

[cholo]

I'm about a week in into my adaptation of handcrafting for forecast weights, I think I got it working.
For those looking to implement, but are not using pysystemtrade, I've found it to be a sweet spot to hook into optimiseWeightsOverTime (https://github.com/robcarver17/pysy...squant/optimisation/optimise_over_time.py#L14) - that way you can compute your own net returns, but still use Rob's entire optimization logic, which can be a bit complex. That also allows you to pass in all the estimation parameters from the config. Nice API design there Rob, not sure if you were planning for that, but definitely nice to use sort of as a library.

I'm pooling gross returns, but using individual costs to get net returns. Using 50000 lookback for everything (per the default config), so basically all the data.

I'm getting 2 issues so wanted to see if those of you that have implemented it are seeing the same, or have some tips.

1. SR adjustment can be a bit extreme - it will completely eliminate some rules that get 10% allocation before adjustment - likewise, it will allocate 10+% to some rules that had no allocation before adjustment. Typical problem is carry and ags markets, I'm getting 2% or less combined allocation to all carry rules for ags. The reason is that my data history starts at 2001, and carry did exceptionally bad in ags from 2006 to 2008, and has only just now barely recovered. I've checked the allocations before and after the SR adjustment, and can confirm that before it's has a solid allocation (eg. 30% for Cocoa), and after has almost nothing (1.4% for Cocoa). Changes in weights per year can also be extreme, I see some rules getting 5% one year, 19% next year, then back down to 6% year after.

2. Overweight to more diversifying rules, in my case skew. I sometimes see 18-20% allocation to a single skew rule (I run 3 variations).

To make the above percentages somewhat meaningful, I'm running 43 rule variations, pretty much same as Rob, without abs relative skew and mean reversion.
Appreciate the help and tips!

Something that I do is to divide the computed Sharpe ratios by 2. These reduces the relative differences; similar to saying that you don't fully believe your backtested Sharpe. I also found the weights a little extreme without this modification.

 

[wopr]

Something that I do is to divide the computed Sharpe ratios by 2. These reduces the relative differences; similar to saying that you don't fully believe your backtested Sharpe. I also found the weights a little extreme without this modification.

Thanks! I thought about doing that, but was not confident that I didn't mess up something with returns calculations on my side. I just divided the SR by 2 and indeed getting more reasonable results.

 

[handelsmeisterei]

So you'd only stay in roll adjusted if something went wrong.

Sorry for being obnoxious, but I have a follow up question. I implemented the roll mechanism now and get weird results. Consider the current term structure of Crude Oil (of which December will be traded).

December 2024 has a volume of 7800 and December 2025 has a volume of 1300. I currently have no position in December 2024 and December 2025 is considered liquid. Therefore the status will be ROLLADJUST. That means that I cannot open a position in Crude and have to wait till either December 2024 expires or the liquidity changes (December 2025 has more volume than 2024 or drops below 1% of 2024).

I have absolutely no experience in the volume profiles going forward, but don't I have to wait quite some time before I can open a contract in Crude? Or is my roll calendar erroneous? Perhaps ROLLADJUST only makes sense if the current contract expires soon? Thanks for your help!