Fully automated futures trading

globalarbtrader · Mar 11, 2022

wopr said:
The timing on this closing trade turned out to be spot on, close to the top of the day (I'm assuming times are UTC).
How did it execute at that time? I know at some point your execution algo was checking that the bid-ask spread is no bigger than X multiples of the minimum tick size. Does your execution try every X minutes and then finally it succeeds when there's enough liquidity that the spread is small enough, or something else?

No, as you can see below I submitted an offer at 112.97 which got filled very quickly (approximately 30 seconds later).

It was just luck - I don't start trading US Markets until 2pm, and this happened to be the first to go, and that just happened to be the top of the market.

Rob

Code:

 emacs /media/LACIE-D2/Rob/run_stack_handler_20220310.arch

ctrl-s BRENT

sysproduction.run_stack_handler.run_stack_handler:
None



Arguments:
[]


2022-03-10:0030.06 {'type': 'stack_handler'}  check_external_position_break will run every 0 minutes until process ends with heartbeats every 10 minutes
2022-03-10:0030.08 {'type': 'stack_handler'}  spawn_children_from_new_instrument_orders will run every 0 minutes until process ends with heartbeats every 10 minutes
2022-03-10:0030.11 {'type': 'stack_handler'}  generate_force_roll_orders will run every 60 minutes at most 10 times with heartbeats every 10 minutes
2022-03-10:0030.13 {'type': 'stack_handler'}  create_broker_orders_from_contract_orders will run every 0 minutes until process ends with heartbeats every 10 minutes
2022-03-10:0030.15 {'type': 'stack_handler'}  process_fills_stack will run every 0 minutes until process ends with heartbeats every 10 minutes
2022-03-10:0030.17 {'type': 'stack_handler'}  handle_completed_orders will run every 0 minutes until process ends with heartbeats every 10 minutes
2022-03-10:0030.20 {'type': 'stack_handler'}  safe_stack_removal will run once only on process completion
2022-03-10:0030.22 {'type': 'stack_handler'}  refresh_additional_sampling_all_instruments will run every 60 minutes until process ends with heartbeats every 10 minutes
2022-03-10:0030.24 {'type': 'run_stack_handler'}  Not starting process:because Not yet time to run

....

2022-03-10:0100.07 {'type': 'stack_handler', 'component': 'mongoIbBrokerClientIdData'}  Locked IB client ID 688
2022-03-10:0100.21 {'type': 'run_stack_handler', 'broker': 'IB', 'clientid': 688}  spawn_children_from_new_instrument_orders still alive, done 0 of unlimited executions every 0 minutes
2022-03-10:0100.23 {'type': 'stack_handler', 'broker': 'IB', 'clientid': 688, 'strategy_name': 'dynamic_TF_carry', 'instrument_code': 'BRENT-LAST', 'instrument_order_id': 32086}  Passive roll handling order (Order ID:32086) Type best for dynamic_TF_carry BRENT-LAST, qty [-1], fill [0]@ price, None Parent:no parent Children:no_children, reducing trade, entire trade in next contract 20220700
2022-03-10:0100.27 {'type': 'stack_handler', 'broker': 'IB', 'clientid': 688, 'strategy_name': 'dynamic_TF_carry', 'instrument_code': 'BRENT-LAST', 'contract_order_id': '', 'instrument_order_id': 32086} [Warning] Short of time, so allocating to algo_market
2022-03-10:0100.29 {'type': 'stack_handler', 'broker': 'IB', 'clientid': 688, 'strategy_name': 'dynamic_TF_carry', 'instrument_code': 'BRENT-LAST', 'instrument_order_id': 32086}  List of contract orders spawned [(Order ID:no order ID) Type best for dynamic_TF_carry/BRENT-LAST/20220600, qty [-1], fill [0]@ price, None Parent:32086 Children:no_children]
2022-03-10:0100.32 {'type': 'stack_handler', 'broker': 'IB', 'clientid': 688, 'strategy_name': 'dynamic_TF_carry', 'instrument_code': 'BRENT-LAST', 'instrument_order_id': 32086, 'contract_order_id': ''}  Put child order (Order ID:no order ID) Type best for dynamic_TF_carry/BRENT-LAST/20220600, qty [-1], fill [0]@ price, None Parent:32086 Children:no_children LOCKED on stack with ID 31881 from parent order (Order ID:32086) Type best for dynamic_TF_carry BRENT-LAST, qty [-1], fill [0]@ price, None Parent:no parent Children:no_children

2022-03-10:1400.06 {'type': 'stack_handler', 'broker': 'IB', 'clientid': 688, 'strategy_name': 'dynamic_TF_carry', 'instrument_code': 'BRENT-LAST', 'contract_order_id': 31881, 'instrument_order_id': 32086}  Sending order (Order ID:31881) Type best for dynamic_TF_carry/BRENT-LAST/20220600, qty [-1], fill [0]@ price, None Parent:32086 Children:no_children to algo sysexecution.algos.algo_market.algoMarket
2022-03-10:1400.08 {'type': 'stack_handler', 'broker': 'IB', 'clientid': 688, 'component': 'ibFuturesContractData'}  Reqid -1: 2119 Market data farm is connecting:usfuture
....
2022-03-10:1400.32 {'type': 'stack_handler', 'broker': 'IB', 'clientid': 688, 'strategy_name': 'dynamic_TF_carry', 'instrument_code': 'BRENT-LAST', 'contract_order_id': 31881, 'instrument_order_id': 32086}  Created a broker order (Order ID:no order ID) Type market for dynamic_TF_carry/BRENT-LAST/20220600, qty [-1], fill [0]@ price, None Parent:31881 Children:no_children (not yet submitted or written to local DB)
2022-03-10:1400.34 {'type': 'stack_handler', 'broker': 'IB', 'clientid': 688, 'component': 'ibExecutionStackData', 'strategy_name': 'dynamic_TF_carry', 'instrument_code': 'BRENT-LAST', 'contract_order_id': 31881, 'broker_order_id': ''}  Going to submit order (Order ID:no order ID) Type market for dynamic_TF_carry/BRENT-LAST/20220600, qty [-1], fill [0]@ price, None Parent:31881 Children:no_children to IB
2022-03-10:1400.37 {'type': 'stack_handler', 'broker': 'IB', 'clientid': 688, 'component': 'ibExecutionStackData', 'strategy_name': 'dynamic_TF_carry', 'instrument_code': 'BRENT-LAST', 'contract_order_id': 31881, 'broker_order_id': ''}  Order submitted to IB
2022-03-10:1400.39 {'type': 'stack_handler', 'broker': 'IB', 'clientid': 688, 'component': ...

2022-03-10:1401.02 {'type': 'stack_handler', 'broker': 'IB', 'clientid': 688, 'strategy_name': 'dynamic_TF_carry', 'instrument_code': 'BRENT-LAST', 'contract_order_id': 31881, 'instrument_order_id': 32086, 'broker_order_id': ''}  Submitted order to IB (Order ID:no order ID) Type market for dynamic_TF_carry/BRENT-LAST/20220600, qty [-1], fill [-1]@ price, 112.97 Parent:31881 Children:no_children
2022-03-10:1401.04 {'type': 'stack_handler', 'broker': 'IB', 'clientid': 688, 'strategy_name': 'dynamic_TF_carry', 'instrument_code': 'BRENT-LAST', 'contract_order_id': 31881, 'broker_order_id': 32030}  Managing trade (Order ID:32030) Type market for dynamic_TF_carry/BRENT-LAST/20220600, qty [-1], fill [-1]@ price, 112.97 Parent:31881 Children:no_children with market order
2022-03-10:1401.07 {'type': 'stack_handler', 'broker': 'IB', 'clientid': 688, 'strategy_name': 'dynamic_TF_carry', 'instrument_code': 'BRENT-LAST', 'contract_order_id': 31881, 'broker_order_id': 32030}  Trade completed
2022-03-10:1401.09 {'type': 'stack_handler', 'component': 'mongoContractOrderStackData', 'strategy_name': 'dynamic_TF_carry', 'instrument_code': 'BRENT-LAST', 'contract_order_id': 31881, 'instrument_order_id': 32086}  Changed fill qty from [0] to [-1] for order (Order ID:31881) Type best for dynamic_TF_carry/BRENT-LAST/20220600, qty [-1], fill [0]@ price, None Parent:32086 Children:[32030]
2022-03-10:1401.12 {'type': 'stack_handler', 'broker': 'IB', 'clientid': 688, 'instrument_code': 'BRENT-LAST', 'contract_date': '20220600'}  Updated position of BRENT-LAST/20220600 from 1 to 0; new position in db is 0
2022-03-10:1401.14 {'type': 'stack_handler', 'broker': 'IB', 'clientid': 688, 'strategy_name': 'dynamic_TF_carry', 'instrument_code': 'BRENT-LAST', 'contract_order_id': 31881, 'instrument_order_id': 32086}  Updated position of BRENT-LAST/20220600 because of trade (Order ID:31881) Type best for dynamic_TF_carry/BRENT-LAST/20220600, qty [-1], fill [0]@ price, None Parent:32086 Children:[32030] ID:31881 with fills -1
2022-03-10:1401.16 {'type': 'stack_handler', 'broker': 'IB', 'clientid': 688, 'strategy_name': 'dynamic_TF_carry', 'instrument_code': 'BRENT-LAST', 'instrument_order_id': 32086}  Updated position of dynamic_TF_carry BRENT-LAST from 1 to 0 because of trade (Order ID:32086) Type best for dynamic_TF_carry BRENT-LAST, qty [-1], fill [0]@ price, None Parent:no parent Children:[31881] 32086 fill [-1]
2022-03-10:1401.19 {'type': 'stack_handler', 'component': 'mongoInstrumentOrderStackData', 'strategy_name': 'dynamic_TF_carry', 'instrument_code': 'BRENT-LAST', 'instrument_order_id': 32086}  Changed fill qty from [0] to [-1] for order (Order ID:32086) Type best for dynamic_TF_carry BRENT-LAST, qty [-1], fill [0]@ price, None Parent:no parent Children:[31881]
2022-03-10:1401.21 {'type': 'stack_handler', 'component': 'mongoContractOrderStackData', 'strategy_name': 'dynamic_TF_carry', 'instrument_code': 'BRENT-LAST', 'contract_order_id': 31881, 'instrument_order_id': 32086} [Warning] Can't change order (Order ID:31881) Type best for dynamic_TF_carry/BRENT-LAST/20220600, qty [-1], fill [-1]@ price, 112.97 Parent:32086 Children:[32030] INACTIVE as inactive
2022-03-10:1401.23 {'type': 'stack_handler', 'broker': 'IB', 'clientid': 688}  Released contract order 31881 from algo control

djames · Mar 11, 2022

What goes into the timing of trading per markets? I observe the volatility of most futures explodes around the US stock market open - have you ever tested impact of targeting before / after US open? Or perhaps as @wopr states maybe a heuristic rule executing when we see lowest spread within a rolling window?

globalarbtrader · Mar 11, 2022

djames said:
What goes into the timing of trading per markets? I observe the volatility of most futures explodes around the US stock market open - have you ever tested impact of targeting before / after US open? Or perhaps as @wopr states maybe a heuristic rule executing when we see lowest spread within a rolling window?

No, although hopefully I should have some decent spread data soon I can get some analysis of spreads.

I'm also not sure I want the lowest spread - I would if I was always doing market orders. Again, I should be able to do some analysis of spreads versus execution costs.

Rob

globalarbtrader · Mar 11, 2022

Looks like today is another small up, +1% as I look at it now. A few trades, just gone short VSTOXX suggesting the drama in European equities is over, replaced corn long with red wheat, trimmed long gold position back to 2 contracts, closed KOSDAQ short and MSCI Singapore long which I guess would have been hedging each other.

The only drama was a day one margin demand from IB which was very weird because my margin usage is extremely low right now, less than 20% of the account value. Took 30 minutes on the phone to resolve (as you have to get through the first line of support to someone who actually knows what they are talking about) but looks like there was a delay sweeping between securities and futures cash subaccounts, causing a trigger happy email to fire off.

To be fair, it's the first time in eight years I've had this problem, and only the second time I've had to talk to IB customer support. People do say it is their weakness, but I'm fortunate in that I don't need much hand holding and 99% of my issues have been with the API and solved through reading forums or googling...

Rob

blink18 · Mar 11, 2022

HobbyTrading said:
Indeed. It is no longer possible to find a way around the "Sunday restart". IB has made this impossible, and they did so on purpose.
Previously I could go on a multi-week holiday and have IBCalpha take care of the restarts. Now I am no longer able to do that. Either I have to find software which lets me remotely log on to my trading server at home. Or I would need to put my software at a VPS which I can access via an internet browser. Both solutions come at a cost.

Does IB Gateway restart every Sunday regardless of account type (paper/live)? At what time does it happen?

wopr · Mar 14, 2022

Did some research into risk overlays, following the methods Rob described in https://qoppac.blogspot.com/2022/02/exogenous-risk-overlay-take-two.html.

Trying to look whether any of those would have been triggered during the recent run up in oil and its derivatives and from what I see in my portfolio, none would.
Just wanted to make sure I'm not implementing something wildly off so if any of you are using those risk overlays, could you post your values?

For what Rob calls "normal" risk, I'm currently at 12%, and I see that's in line with his value too (I run 25% annualized vol target). For "jump" risk, I'm at 20%, which is surprising to me and I'm almost certain I'm computing that incorrectly somehow.
Low vol filter certainly wouldn't have kicked in in the last few weeks

Limit to one instrument's notional value being too big % of total capital wouldn't kick in as well for me, none are over 30% (I believe Rob was setting that to 100% of the capital, mainly targeting not having, say 4 EDOLLAR's).

The problem as I see it - my optimal (unrounded) position was reduced as it is vol adjusted (went to 0.25 for heating oil) but DO still decided to go long to get the needed exposure. But then annualized risk from that single position ended up being 25% of capital (one could argue "actual risk" is much higher, because the assumption about Gaussian return distribution is kinda on shaky legs for returns of the last few weeks

). Regardless of the outcome of that trade, I'm uncomfortable with that. I think I might need an additional per-instrument risk control, that's risk adjusted, and perhaps should be *after* the DO. I'm not a fan of that method tho, for the same reason Rob mentioned in his post.
So current best solution seems to be passing in `maximum_positions` into the DO, and computing that so that's it's risk adjusted.

Interestingly enough, "normal" risk went to 40% after I added that one long, and that would trigger Rob's 1.4 factor for normal risk, but that's on a "rounded" position, after the DO.

I'm open to better ideas if anyone has any!

globalarbtrader · Mar 14, 2022

wopr said:
Did some research into risk overlays, following the methods Rob described in https://qoppac.blogspot.com/2022/02/exogenous-risk-overlay-take-two.html.

Trying to look whether any of those would have been triggered during the recent run up in oil and its derivatives and from what I see in my portfolio, none would.
Just wanted to make sure I'm not implementing something wildly off so if any of you are using those risk overlays, could you post your values?

For what Rob calls "normal" risk, I'm currently at 12%, and I see that's in line with his value too (I run 25% annualized vol target). For "jump" risk, I'm at 20%, which is surprising to me and I'm almost certain I'm computing that incorrectly somehow.
Low vol filter certainly wouldn't have kicked in in the last few weeks
Limit to one instrument's notional value being too big % of total capital wouldn't kick in as well for me, none are over 30% (I believe Rob was setting that to 100% of the capital, mainly targeting not having, say 4 EDOLLAR's).

The problem as I see it - my optimal (unrounded) position was reduced as it is vol adjusted (went to 0.25 for heating oil) but DO still decided to go long to get the needed exposure. But then annualized risk from that single position ended up being 25% of capital (one could argue "actual risk" is much higher, because the assumption about Gaussian return distribution is kinda on shaky legs for returns of the last few weeks ). Regardless of the outcome of that trade, I'm uncomfortable with that. I think I might need an additional per-instrument risk control, that's risk adjusted, and perhaps should be *after* the DO. I'm not a fan of that method tho, for the same reason Rob mentioned in his post.
So current best solution seems to be passing in `maximum_positions` into the DO, and computing that so that's it's risk adjusted.

Interestingly enough, "normal" risk went to 40% after I added that one long, and that would trigger Rob's 1.4 factor for normal risk, but that's on a "rounded" position, after the DO.

I'm open to better ideas if anyone has any!

I think you basically want to limit your exposure per instrument in risk units, but calculated in contract terms as a limit for the DO, knowing about integers.

You don't really want 25% of your risk in one instrument basically. So how I'd calculate this is work out the $ risk per contract (multiplier * % ann standard deviation * price * FX), and work out some maximum % risk concentration in one instrument (would depend on capital, but maybe 10% is a sensible limit; I rarely have more than 5% looking at past risk reports) and multiply by capital to get a max $ risk.

So for example if I take Heating oil, I think the current $ risk is $78K per contract, if I use 10% of $500k = $50k as my maximum risk per instrument, then the maximum position is 50/78, which rounds down to zero. On the other hand take the Buxl (long german bond) current risk is $32k, so maximum position is 50/32 = 1.6 which rounds down to a single contract.

So I'd put in position limits of 1 contract (Buxl) and 0 contracts (heating oil). Like the other position limits they would need updating occasionally as risk changes.

Looking at this, this is actually quite a nice and sensible thing to do. Effectively the issue with DO, especially for small account sizes, is it can lump you with risk concentration since that isn't part of the utility function of the optimiser and effectively DO throws away the diversification inherent in a large portfolio.

It will mean there are a few instruments you will never take positions in, but I think that is better than waking up on day long a contract of Ethereum (annual risk $100k) just because the DO thinks that is a good idea.

Rob

Kernfusion · Mar 15, 2022

Just noticed that I'm long V2TX, one contract, this hasn't happened before in live trading. The unrounded position of V2TX is -0.5; VIX: -0.09, but DO decided to go long.. Probably because of other correlated instrument signals..

Elder · Mar 15, 2022

Kernfusion said:
Just noticed that I'm long V2TX, one contract, this hasn't happened before in live trading. The unrounded position of V2TX is -0.5; VIX: -0.09, but DO decided to go long.. Probably because of other correlated instrument signals..

Yup just went long VIX myself today thanks primarily to this somewhat unusual state of affairs.

wopr · Mar 15, 2022

Kernfusion said:
Just noticed that I'm long V2TX, one contract, this hasn't happened before in live trading. The unrounded position of V2TX is -0.5; VIX: -0.09, but DO decided to go long.. Probably because of other correlated instrument signals..

Same here, also long one V2TX as of this morning.
Heads up, Rob will say for sure, but I think DO only generates a position in the direction of the unrounded position, so if that's -0.5, if I understood correctly, you shouldn't be long.

From https://qoppac.blogspot.com/2021/10/mr-greedy-and-tale-of-minimum-tracking.html :

The direction will always be the sign of the optimal position. So we'd normally start at zero (start_weights), and then get gradually longer (if the optimal is positive), or start at zero and get gradually shorter (if the optimal position is a negative short). This means we're only ever moving in one direction which makes the greedy algorithim work. Note: This is different with certain corner cases in the presence of constraints. See the end of the post.