Fully automated futures trading

globalarbtrader · May 8, 2024

handelsmeisterei said:
Sorry for being obnoxious, but I have a follow up question. I implemented the roll mechanism now and get weird results. Consider the current term structure of Crude Oil (of which December will be traded).

December 2024 has a volume of 7800 and December 2025 has a volume of 1300. I currently have no position in December 2024 and December 2025 is considered liquid. Therefore the status will be ROLLADJUST. That means that I cannot open a position in Crude and have to wait till either December 2024 expires or the liquidity changes (December 2025 has more volume than 2024 or drops below 1% of 2024).

I have absolutely no experience in the volume profiles going forward, but don't I have to wait quite some time before I can open a contract in Crude? Or is my roll calendar erroneous? Perhaps ROLLADJUST only makes sense if the current contract expires soon? Thanks for your help!

View attachment 339919

Not sure you're being obnoxious but you clearly haven't understood what I was trying to say.

"December 2024 has a volume of 7800 and December 2025 has a volume of 1300. I currently have no position in December 2024 and December 2025 is considered liquid. Therefore the status will be ROLLADJUST. "

Yes

"That means that I cannot open a position in Crude and have to wait till either December 2024 expires"

No.

ROLLADJUST just means we roll - so the system would shift the current priced contract from being Z24 to Z25 and does the back adjustment, the code then resets the role state to no roll, and we're free to take a position in Z25.

Rob

Code:

=======================================================================================================================================================
                                                            Status and time to roll in days                                                           
=======================================================================================================================================================

                status  roll_expiry  price_expiry  carry_expiry contract_priced contract_fwd  position_priced  relative_volume_fwd  contract_volume_fwd
ALUMINIUM_LME  No_Roll           24            69           104        20240700     20240800              0.0                1.381                  231

Roll_exp is days until preferred roll set by roll parameters. Prc_exp is days until price contract expires, Crry_exp is days until carry contract expires
Contract suffix: p=price, f=forward, c=carry
Contract volumes over recent days, normalised so largest volume is 1.0

************************************************************************************************
Automatically changing state from RollState.No_Roll to RollState.Roll_Adjusted for ALUMINIUM_LME
************************************************************************************************
********************************************************************************

Rolling adjusted prices!

Current multiple prices

                      CARRY CARRY_CONTRACT   PRICE PRICE_CONTRACT  FORWARD FORWARD_CONTRACT
index                                                                                     
2024-05-07 09:00:00  2578.5       20240800  2565.0       20240700   2578.5         20240800
2024-05-07 10:00:00  2576.0       20240800  2560.5       20240700   2576.0         20240800
2024-05-07 11:00:00  2560.5       20240800  2544.5       20240700   2560.5         20240800
2024-05-07 12:00:00  2568.5       20240800  2549.0       20240700   2568.5         20240800
2024-05-07 13:00:00  2563.0       20240800     NaN       20240700   2563.0         20240800
2024-05-07 14:00:00  2558.0       20240800     NaN       20240700   2558.0         20240800

New multiple prices

                      CARRY CARRY_CONTRACT   PRICE PRICE_CONTRACT  FORWARD FORWARD_CONTRACT
index                                                                                     
2024-05-07 10:00:00  2576.0       20240800  2560.5       20240700   2576.0         20240800
2024-05-07 11:00:00  2560.5       20240800  2544.5       20240700   2560.5         20240800
2024-05-07 12:00:00  2568.5       20240800  2549.0       20240700   2568.5         20240800
2024-05-07 13:00:00  2563.0       20240800  2549.0       20240700   2563.0         20240800
2024-05-07 14:00:00  2558.0       20240800  2549.0       20240700   2558.0         20240800
2024-05-07 14:00:01     NaN       20240900  2558.0       20240800      NaN         20240900

Current adjusted prices

index
2024-05-03 23:00:00    2543.5
2024-05-07 08:00:00    2557.0
2024-05-07 09:00:00    2565.0
2024-05-07 10:00:00    2560.5
2024-05-07 11:00:00    2544.5
2024-05-07 12:00:00    2549.0
Name: price, dtype: float64

New adjusted prices

index
2024-05-07 10:00:00    2569.5
2024-05-07 11:00:00    2553.5
2024-05-07 12:00:00    2558.0
2024-05-07 13:00:00    2558.0
2024-05-07 14:00:00    2558.0
2024-05-07 14:00:01    2558.0
dtype: float64


********************************************
AUTO ROLLING - NO USER CONFIRMATION REQUIRED
********************************************
2024-05-08 16:50:02 DEBUG Interactive_Update-Roll-Status {'component': 'parquetFuturesMultiplePricesData', 'instrument_code': 'ALUMINIUM_LME'} Wrote 2585 lines of prices for ALUMINIUM_LME to parquetFuturesMultiplePricesData
2024-05-08 16:50:02 INFO Interactive_Update-Roll-Status {'component': 'parquetFuturesMultiplePricesData', 'instrument_code': 'ALUMINIUM_LME'} Added data for instrument ALUMINIUM_LME
2024-05-08 16:50:02 DEBUG Interactive_Update-Roll-Status {'component': 'parquetFuturesAdjustedPricesData', 'instrument_code': 'ALUMINIUM_LME'} Wrote 2585 lines of prices for ALUMINIUM_LME to parquetFuturesAdjustedPrices
2024-05-08 16:50:02 INFO Interactive_Update-Roll-Status {'component': 'parquetFuturesAdjustedPricesData', 'instrument_code': 'ALUMINIUM_LME'} Added data for instrument ALUMINIUM_LME

2024-05-08 16:50:02 DEBUG Interactive_Update-Roll-Status Successful roll! Returning roll state of ALUMINIUM_LME to RollState.No_Roll

handelsmeisterei · May 8, 2024

globalarbtrader said:
ROLLADJUST just means we roll - so the system would shift the current priced contract from being Z24 to Z25 and does the back adjustment, the code then resets the role state to no roll, and we're free to take a position in Z25.

Understood. This implies that my creation of roll calendars need more work. Instead of just searching for existing triples of current/forward/carry around the roll date I have to create the roll dates more liquidity driven.

The way I implemented the system is by not making any difference between backtest and production. Production is simply the last day of the backtest and roll calendar creation does not depend on opened positions.

In contrast your roll dates seem to depend on your (prior) positions in production. Depending on strategies/instruments your roll dates are then kind of path dependant? I guess this does not make any difference in the end. Thank you for your patient explanations!

globalarbtrader · May 8, 2024

handelsmeisterei said:
The way I implemented the system is by not making any difference between backtest and production. Production is simply the last day of the backtest and roll calendar creation does not depend on opened positions.

In contrast your roll dates seem to depend on your (prior) positions in production. Depending on strategies/instruments your roll dates are then kind of path dependant? I guess this does not make any difference in the end. Thank you for your patient explanations!

I think about it differently. Prior to 2014* I had to create backadjusted prices with some assumption about how I would have rolled had I been trading at that time. Post 2014 I can use the times when I actually did trade. I use the same prices for production and backtesting.

* or later depending on when I added the relevant instrument

handelsmeisterei · May 8, 2024

globalarbtrader said:
I think about it differently. Prior to 2014* I had to create backadjusted prices with some assumption about how I would have rolled had I been trading at that time. Post 2014 I can use the times when I actually did trade. I use the same prices for production and backtesting.

Yep, interesting view. I view it like it is always 2014. This seems conceptually more satisfying. But I have no clue if it is even possible to create meaningful roll calendars in an automatic way. I will try it and can then compare the result to your "live" calendars. Will be an interesting exercise. If I understood @Kernfusion posts correctly, he also seems to be doing it without any knowledge of positions.

Kernfusion · May 8, 2024

handelsmeisterei said:
Yep, interesting view. I view it like it is always 2014. This seems conceptually more satisfying. But I have no clue if it is even possible to create meaningful roll calendars in an automatic way. I will try it and can then compare the result to your "live" calendars. Will be an interesting exercise. If I understood @Kernfusion posts correctly, he also seems to be doing it without any knowledge of positions.

I do something similar in spirit to what Rob described (and his very early posts on rolls)., I have 3 roll states NoRoll, SoftRoll, HardRoll, Error (well, technically 4, but the last one should never happen

)
- NoRoll is when we're just normally trading in the middle of a cycle,
- SoftRoll is when we're approaching the HardRoll-Date\Expiration, and the 5-day average volume of the Forward-contract has reached 5% of the Price-contract (although, I like the idea to also require some absolute minimum number e.g. 100 traded contracts in the previous day) and so we would only open positions in the Forward contract and close only in the Price contract if the system decides to change position (if during that time we don't have any positions in the "old" cycle's Price-contract, the system will simply select the "next" cycle as current and set NoRoll to basically start trading the next contract and forget about the old one)., so the behavior does depend a little on whether we have open positions in the current cycle or not.
- HardRoll means the system will close any positions in the current cycle ASAP and will switch to the next one with NoRoll State.

So there are 2 separate things (which I'm actually doing in one function):
1-determite the current cycle (i.e. the current set of Price+Carry+Forward contracts and the Start, SoftRoll, HardRoll and End, dates). This will depend on where we are in time, i.e. how close to the expiration, do I have any positions in the current and\or next Price contracts and the relative volume of the Price contract from this and next periods.
2. Determine RollState - again based on time, positions, volume

And then the trading part of my system will be opening and closing positions according to this information (current Price+Carry+Forward contracts and RollState). E.g. if my RollState is NoRoll and the system decides to add one more long, it will buy 1 current Price contract., if we're in SoftRoll it will buy 1 Forward contract.

As for choosing the Price, Carry, Forward contracts for each period, I hardcoded 100y of those periods (starting from 1980) for every product I trade according to it's recent\current volume patterns and expiration dates. So if the volume was actually different e.g. 20y ago, then I guess my backtest is a bit unrealistic, because it assume I could roll them on the same dates as currently..
In the future I hope the volume patterns will not change in such a way that my pre-generated calendar becomes invalid (for that the volume should decrease, and it usually only grows with time)., and I still have my 5% volume check for the Soft-roll periods, and the HardRoll date depends on the expiry date (which is part of the contract's specs) not on volume..

Btw, this 'ChooseRoll' function has the most unit-tests in my system

exactly because it's quite complicated and I wasn't sure that my code covers all the cases, so I wrote a test for every situation I could think of...

Kernfusion · May 8, 2024

This probably was a messy explanation, but it is indeed a messy logic

so here's an example of my prepopulated periods or cycles for one product:

As you can see, the the periods overlap a little, and the SoftRollStartTS of the previous period matches with the StartTradingTS of the next period., so if the system wakes up at the time that satisfices only 1 period, then everything is simple - close everything that's not the current Price contract and place all the required exposure in that Price contract., but if we're in 2 possible periods, then it depends on which positions we have. Fundamentally during the SoftRoll (when we're in 2 periods at the same time) we want to reduce the old price contract and open new positions in the new one (which is Forward contract of the current period and Price contract of the next one).

This is of course just my approach to that problem, not necessarily a good one..

handelsmeisterei · May 9, 2024

Thank you so much for all this valuable information! I underestimated the complexity of the problem by quite a large margin. My first experiments yesterday showed, that identifying the roll date simply by volume checks produced strange results - a random volume spike in the forward may trigger a roll then.

As the regulars of this thread do this for quite some time and store the roll dates (not calculating everything from scratch each day), this seems to be the way to go. Unfortunately.

CSIdata has lots of options in their software to give some more inspiration. I will see if I can produce meaningful roll dates by using open interest.

andrewcraig · May 9, 2024

GAT (or if someone else can answer!), for your rob_system/config.yaml on the Github repo - are the weights based on your handcraft optimiser function in pysystemtrade or rather an external top-down handcrafting as outlined in your books?

I apologise as this has probably been answered before... I am wondering as I tried to reconstruct by removing the static weights and running it through the optimiser (with all the same forecast rules, instruments and vol target). The backtest results were fairly different and a lot less correlated than I would have expected.

An alternative thought is perhaps that's just an artifact of it being a point-in-time output from the optimiser. I'm also confused here as your repo reports suggest better (and different) performance than from backtesting the static file or optimiser output, though I'd have guessed the reverse would occur!

Thank you for all your openness and countless hours spent helping others.

andrewcraig · May 10, 2024

To clarify my previous post, here is what I was seeing.

Rob original uses the 40 instrument selection mentioned early in this thread. I did a 10-year lookback as that matches the sort of out-of-sample data that would be interesting (back to the beginning of this thread) and it looks like you were probably using about the same rules as currently - so all the systems are ran through the same config.yaml with the automatic handcrafting except for rob_system which is the repo's provided static config.

1. rob_system is the repo's config.yaml.
2. systems 53 (actually 54) to 73 are the same with the repo's static instrument suggestions.
3. system 115 was a custom selection.
4. full_optimised is the same as #1 in instruments and rules, but pushed through the automatic handcraft framework used for all the other systems and then the output from portfolio_optimised().

It's an interesting exercise anyway in how much correlations drop just based on instrument selection and forecast weight selection.

Finally combined was to see if there was a theoretical benefit to combining multiple less that perfectly correlated futures system, assuming a monthly rebalance to 50/50 return exposure weighting (to the 53 system and the rob_system, as those were least correlated).

Everything was normalised to 25% annualized volatility after running the systems.

handelsmeisterei · May 13, 2024

A short update on how to calculate a history of liquidity driven roll dates. This might be interesting to other readers that run into the same problem. I ended up adding a walk forward mode to the backtest. In this mode, the backtest simply stops after having identified a new liquidity driven roll date. As explained by Rob, these roll dates depend on the position currently on and therefore need the DO positions.

The way I implemented it, the backtest adds these roll dates to a list of manual roll dates. After creating the default roll dates from the offsets some of these dates are then replaced by the manual roll dates. To create all of them for the whole backtest, I simply put the backtest in a loop and let it run in walk forward till no new roll dates were found. This takes quite some time (for me it was a whole night) but has only to be done once. In production the manual roll dates will be added to the list as you go. You only have to re-run this exercise if you either add lots of new strategies or do a significant capital bump. Both of these changes may have meaningful impact on your DO positions and may warrant a re-run. Thanks to all of you for the clarifications on how rolling actually works!