Fully automated futures trading

[HobbyTrading]

Want to add a long-term bond, probably BTP (which I badly missed this year ), a precious and a base metal, probably platinum and copper, maybe try adding micro Nasdaq(MNQ) or ESTX50, and maybe a currency e.g. GBP, also maybe crude.. Everything else is either too big or too correlated..
Was also thinking about ice-futures, like sugar\coffee\cotton and contracts from other exchanges like DJ600, EU600, XINA50, NIFTY, SGXNK - but these need more work for collecting historical data and implementing market orders as there's no way I'm paying 200$\month for real-time data from ICE

I think that MES has higher trade volume than MNQ, which is why I selected MES instead.
I don't use ICE precisely because of what you mention: the high monthly data subscription fee.
I am using both ESTX50 and DJ600. They are both traded on the same EUREX exchange and appear highly correlated. The DJ600 has a smaller contract size and value volatility per contract, so this might be the better choice for a smaller account size. The EUREX market data subscription costs a few dollars per month.
XINA50 and NIFTY have rather low value volatility per contract and are thus suitable for a smaller account. SGXNK and K200 (Korean KOSPI equity index) are much larger in size. The Singapore (SGX; XINA50, NIFTY, SGXNK) market data subscription is only 1~2 US dollars per month. You already have access to K200 data because you are subscribed to 3KTB data.
For long term bonds you could consider the US 10 year bond (ZN), or the Korean 10 year bond (FLKTB). From the ones that I track are those the ones with the smallest value volatility per contract.

 

[Kernfusion]

I don't use ICE precisely because of what you mention: the high monthly data subscription fee.

What about trading them with 15-min delayed free data and market orders ? I mean how bad can it be? yes I'll be giving away bid-ask spread every time, and can potentially execute on a 15-min stale price, but compared to the benefit of adding some good uncorrelated contracts it should be small?

For long term bonds you could consider the US 10 year bond (ZN), or the Korean 10 year bond (FLKTB). From the ones that I track are those the ones with the smallest value volatility per contract.

I wanted something very long-term, as this time it was pretty clear that very long-term bonds do behave differently, both US 20y and BTP made a lot of money in my paper account, when ZN didn't.
Maybe I'll even try to squeeze both ZN and BTP, that would be ideal as they cover different geographic regions..
I'll check out MES, although, I already set up data collection for MNQ, so it'll be more work. Others will probably follow in a next batch when I save up another bag of cash

 

[HobbyTrading]

What about trading them with 15-min delayed free data and market orders ?

I have not tested this as I was not aware that this is possible. 15 minutes delayed data would not affect the way I'm running the system. Does the IB API allow you to download data if you don't have a subscription?

If you want a really long term US bond you could look into UB. But the value volatility is larger than for US 20y (ZB). I'm tracking UB but I don't think that my system ever opened a position in it.

 

[Kernfusion]

I have not tested this as I was not aware that this is possible. 15 minutes delayed data would not affect the way I'm running the system. Does the IB API allow you to download data if you don't have a subscription?

I always assumed yes, I think I even experimented with it a long time ago. There's an API flag or something like that.. Also on TWS UI you can still see "real-time" quotes of unsubscribed instruments, it's marked with special symbol when the data is delayed..

Just checked, it looks like there's a special API method "IBApi::EClient::reqMarketDataType"
https://interactivebrokers.github.io/tws-api/delayed_data.html
"10-15 minute delayed streaming data is available for many types of instruments without market data subscriptions."
It does say this though: "It is important to note that historical data still requires market data subscriptions".
So maybe even though it'll give 15 min delayed RT quotes, it'll not return any EOD historical prices, which would be a bummer for me as I need daily closing prices for signal calculation. It's probably possible to just sample the RT data around the time when the day ends and use that as EOD..

 

[HobbyTrading]

Just checked, it looks like there's a special API method "IBApi::EClient::reqMarketDataType"
https://interactivebrokers.github.io/tws-api/delayed_data.html
"10-15 minute delayed streaming data is available for many types of instruments without market data subscriptions."
It does say this though: "It is important to note that historical data still requires market data subscriptions".
So maybe even though it'll give 15 min delayed RT quotes, it'll not return any EOD historical prices, which would be a bummer for me as I need daily closing prices for signal calculation. It's probably possible to just sample the RT data around the time when the day ends and use that as EOD..

Thank you for investigating this for me (it confirms what I thought to be the case). My system uses historical data for all days prior to the current trading day, and uses an intraday (near end-of-day) price as proxy for today's close price. By combining these prices I then calculate forecast, volatility, and desired position size. And places a trade if the actual position size deviates from the desired position size. So my system would not be able to operate if I can't download the historical prices.

 

[Kernfusion]

Thank you for investigating this for me (it confirms what I thought to be the case). My system uses historical data for all days prior to the current trading day, and uses an intraday (near end-of-day) price as proxy for today's close price. By combining these prices I then calculate forecast, volatility, and desired position size. And places a trade if the actual position size deviates from the desired position size. So my system would not be able to operate if I can't download the historical prices.

Yeah, I just tested downloading historical daily-close prices through API "reqHistoricalData" for "KC" (Coffee "C") and got "Historical Market Data Service error message:No market data permissions for NYBOT FUT".

There's still an option to collect near-EOD prices from the 15-min delayed RT-prices every day for every "active" contract by running a job at a certain time before close (need to test first if that truly works, but looks like it should).. And prepopulate long-term history once from some other source.

Also, interestingly enough, TWS UI actually displays the closing prices if I add a contract to the watch-list and go to right-click->analytical Tools->Price History., it even allows to download them in a csv file, but it's not giving the same closing prices in the API..
So yeah, I suppose I could write a job to sample near-close RT prices every day and store them in the db, but it needs some work.. Maybe I'll do that for the next batch of contracts I add.. Some of these contracts seem really attractive, sugar and cotton have only 1.5 and 3.3k margins..

 

[HobbyTrading]

There's still an option to collect near-EOD prices from the 15-min delayed RT-prices every day for every "active" contract by running a job at a certain time before close (need to test first if that truly works, but looks like it should).. And prepopulate long-term history once from some other source.

True. But I'm using about 1 year of historical data. I'm too lazy to write software to stitch prices from some other source together with near-EOD data from IB.

Also, interestingly enough, TWS UI actually displays the closing prices if I add a contract to the watch-list and go to right-click->analytical Tools->Price History., it even allows to download them in a csv file, but it's not giving the same closing prices in the API..

There is indeed a difference. There are cases where you can get data in TWS which you are not allowed to download via the API. These futures on ICE are an example. I recall that there was also a Chinese stock exchange which allows displaying data (thus in TWS), but does not allow downloading the data (thus no luck with the API).
If you do find a working solution for ICE instruments, can you please let me know? I am interested in those products as I do see the diversification to my portfolio they offer.

 

[globalarbtrader]

What about trading them with 15-min delayed free data and market orders ? I mean how bad can it be? yes I'll be giving away bid-ask spread every time, and can potentially execute on a 15-min stale price, but compared to the benefit of adding some good uncorrelated contracts it should be small?


I wanted something very long-term, as this time it was pretty clear that very long-term bonds do behave differently, both US 20y and BTP made a lot of money in my paper account, when ZN didn't.
Maybe I'll even try to squeeze both ZN and BTP, that would be ideal as they cover different geographic regions..
I'll check out MES, although, I already set up data collection for MNQ, so it'll be more work. Others will probably follow in a next batch when I save up another bag of cash

Yeah, I just tested downloading historical daily-close prices through API "reqHistoricalData" for "KC" (Coffee "C") and got "Historical Market Data Service error message:No market data permissions for NYBOT FUT".

There's still an option to collect near-EOD prices from the 15-min delayed RT-prices every day for every "active" contract by running a job at a certain time before close (need to test first if that truly works, but looks like it should).. And prepopulate long-term history once from some other source.

Also, interestingly enough, TWS UI actually displays the closing prices if I add a contract to the watch-list and go to right-click->analytical Tools->Price History., it even allows to download them in a csv file, but it's not giving the same closing prices in the API..
So yeah, I suppose I could write a job to sample near-close RT prices every day and store them in the db, but it needs some work.. Maybe I'll do that for the next batch of contracts I add.. Some of these contracts seem really attractive, sugar and cotton have only 1.5 and 3.3k margins..

This is something I'm going to be doing myself this year; adding a lot more markets to my portfolio. This will be in the context of the 'optimal small portfolio' project, where I have a huge number of markets, but only take positions in a fraction of them. So there are a number of different options for me here.

One is to get EOD data from another source (probably Barchart, as I already have a subscription to that; but I'd prefer an IB based solution if someone can get one to work!), but then mark the markets as 'non-tradable'. It will still contribute to the overall signal, for example a very strong trend in Brent Crude (which is ICE) would end up as a larger position in NYMEX Crude (which I already trade). I would need this technology anyway, as there are markets for which I'd be unlikely to afford the margin (eg JGB).

The second option is to do as you say, and trade 'blind'. It will mean paying a bit extra in slippage vs the slightly better deal I usually get with my simple algo (that obviously requires live tick data), but then when I backtest I ignore the algo and assuming I always get filled at market anyway. For a system whose orders are conditional only on the previous close it should be fine. Certainly when I last looked into this, seven years ago now, the IB API didn't allow it. Again I'd be curious if anyone gets it working (if I don't get around to it first).

GAT

 

[globalarbtrader]

Big day yesterday, both from a markets perspective (a large number of price movements triggered my standard deviation price filter) and I was up 2.2% which is my best for a while; but also I finally got rid of the 'stock+hedge' part of my portfolio: closing out my short postion of two Eurostoxx futures. The original logic for this (not having enough cash to fund the account and having to borrow against stock I couldn't sell without triggering capital gains) is long gone.

I removed most of the hedge in March anyway, after selling off a big position in a European high yield ETF that was finally underwater, allowing me to take a capital loss to offset against my rather splendid futures trading profits from last year. All that was left was a ragtag of small ETF positions which weren't especially well hedged by the short I had on in Eurostoxx. At the time you may also recall that I bought a position in a US treasury bill ETF using cash not needed for margin. This was very much a cash subsitute; as only 85K of the cash in the account is covered by government insurance. However this was leading to unneccesary volatility from FX changes.

I was slightly impulsive with the timing; I took a capital loss on the whole transaction, which will be wasted if I don't make a certain level of profit this tax year. I ought to have waited until the tax year was nearly done; and perhaps delayed the transaction until the first day of the new tax year. Still it's nice to have a clean sheet for 2021, and this way my next full tax year will be completely clean.

Whilst the stocks + hedge did lose money in 2020; overall it's been profitable, and in two years it pushed my account into the green when my futures trading lost money. But this was always more by luck than judgement. It's a messy distraction and I'm glad to get rid of it.

The portfolio is now mostly cash. I say mostly because there was a small position I wasn't able to sell due to trading restrictions (not sure how I was able to buy it! and it's weird I can trade as many Bund futures as I like within reason but not a few dozen shares in a vanilla German governmen bond ETF!); I need to dig into that.

I also put about 12% of the account value into a GBP money market fund, again as a cash subsitute. Was going to add more, but I need to think about it. I'm paying the ETF fee, plus swap spreads, and getting negative interest rates. That's quite a high cost to purchase insurance against IB going bust! A better bet might be this ETF: ERNS. It's an ultra short corporate bond fund. Fee is only 0.09%, and returns are still positive. There is a little bit of shock risk (returns in March 2020 were -0.4%) but bar a complete meltdown of the financial system, it should be safe.

Or maybe I should stop worrying and let IB look after my money. My plan this year when I create my new funkier system is to make better use of margin, including it in my optimisation calculations. When I get round to adding RV systems they will probably chew up some more margin anyway, SPAN notwithstanding. So there will be less 'spare' cash lying around anyway.

GAT

 

[HobbyTrading]

when I last looked into this, seven years ago now, the IB API didn't allow it.

(underscore added by me) What do you precisely mean by "it"? I guess I don't fully know how you place your orders, and what kind of algorithm you use.