Fully automated futures trading

[Kernfusion]

First blog post on dealing with small capital:

https://qoppac.blogspot.com/2021/06/optimising-my-way-out-of-small-fund.html

GAT

So as I understand, this part will be the most computationally expensive? :

"For the discrete optimisation we're probably going to want to use some kind of grid search."

Grid-search should parallelizable (that word again )., i.e. if we just need to evaluate a function of several parameters multiple times on different parameters-groups and pick the parameter group which gives the highest(lowest) value of the function., each of these evaluations is independent of each other and therefore, they can be done simultaneously on multiple threads, then the results of the function-evaluations returned back to the main thread, where we simply pick the best result. On a 4-8 core CPU with hyperthreading can give an order of magnitude speed improvement..

 

[globalarbtrader]

So as I understand, this part will be the most computationally expensive? :

"For the discrete optimisation we're probably going to want to use some kind of grid search."

Grid-search should parallelizable (that word again )., i.e. if we just need to evaluate a function of several parameters multiple times on different parameters-groups and pick the parameter group which gives the highest(lowest) value of the function., each of these evaluations is independent of each other and therefore, they can be done simultaneously on multiple threads, then the results of the function-evaluations returned back to the main thread, where we simply pick the best result. On a 4-8 core CPU with hyperthreading can give an order of magnitude speed improvement..

Yes that's exactly what I was thinking of doing. Might even try and run it across multiple machines (at least for backtesting).

GAT

 

[wopr]

each of these evaluations is independent of each other and therefore, they can be done simultaneously on multiple threads, then the results of the function-evaluations returned back to the main thread, where we simply pick the best result. On a 4-8 core CPU with hyperthreading can give an order of magnitude speed improvement..

Quick note, if you're doing this in Python, and I know Rob is, this won't do it. Python has GIL (Global Interpreter Lock) and you can't have parallelism with threads, you need to use processes. Effectively, within a process, only one thread can be executing Python code at any time. The way that's implemented is that each thread, once assigned to run by the OS, tries to grab the global lock, and only the one that gets it, gets to run.
That complicates things slightly as sharing data is simpler between threads, but I've used Python's multiprocessing (https://docs.python.org/3/library/multiprocessing.html) before with a lot of success. Specifically, there's a queue implementation which has a nice API.

Edit: I think I recall you mentioning somewhere that your system is C# or something like that, so this doesn't apply to you.

 

[Kernfusion]

Quick note, if you're doing this in Python, and I know Rob is, this won't do it. Python has GIL (Global Interpreter Lock) and you can't have parallelism with threads, you need to use processes. Effectively, within a process, only one thread can be executing Python code at any time. The way that's implemented is that each thread, once assigned to run by the OS, tries to grab the global lock, and only the one that gets it, gets to run.
That complicates things slightly as sharing data is simpler between threads, but I've used Python's multiprocessing (https://docs.python.org/3/library/multiprocessing.html) before with a lot of success. Specifically, there's a queue implementation which has a nice API.

Edit: I think I recall you mentioning somewhere that your system is C# or something like that, so this doesn't apply to you.

Yeah, the execution engine, which I also use to run back-tests by pumping historical prices through it, is in C#. I'm actually in the process of improving it's performance, as most of it was written during very late after work hours and a lot of that code is like "I don't care how optimal, I just need this thing to do that thing and I don't have the whole night to finish it"
It works satisfactory now with <= 40 futures, but it's not going to fly well if I extend it to 200.. So far I was able to reduce the full backtest time from 5 hours to 1h 20min, most of it by just rewriting completely dumb\lazy code.. Fun fact: never thought I'd need to use solved earlier Leetcode problems for anything useful, but this one https://leetcode.com/problems/sliding-window-maximum/ is precisely applicable to the BreakOut forecast calculation, and I was able to shave off maybe 40 minutes of backtest time by using the O(n) solution
My reporting part with graphs and analytics is vectorized and in Matlab, which essentially has the same parallel model as Python - you can run things in parallel and there's even some easy-to-use syntactic sugar, but behind the scenes it actually starts a separate process for every parallel stream (they call it parallel pool), but only 1 per core, hyperthreading is ignored..

 

[globalarbtrader]

Quick note, if you're doing this in Python, and I know Rob is, this won't do it. Python has GIL (Global Interpreter Lock) and you can't have parallelism with threads, you need to use processes. Effectively, within a process, only one thread can be executing Python code at any time. The way that's implemented is that each thread, once assigned to run by the OS, tries to grab the global lock, and only the one that gets it, gets to run.
That complicates things slightly as sharing data is simpler between threads, but I've used Python's multiprocessing (https://docs.python.org/3/library/multiprocessing.html) before with a lot of success. Specifically, there's a queue implementation which has a nice API.

Edit: I think I recall you mentioning somewhere that your system is C# or something like that, so this doesn't apply to you.

Correct; I didn't read the OP properly, it would be processes not threads that are spawned.

GAT

 

[Elder]

Quick note, if you're doing this in Python, and I know Rob is, this won't do it. Python has GIL (Global Interpreter Lock) and you can't have parallelism with threads, you need to use processes. Effectively, within a process, only one thread can be executing Python code at any time. The way that's implemented is that each thread, once assigned to run by the OS, tries to grab the global lock, and only the one that gets it, gets to run.
That complicates things slightly as sharing data is simpler between threads, but I've used Python's multiprocessing (https://docs.python.org/3/library/multiprocessing.html) before with a lot of success. Specifically, there's a queue implementation which has a nice API.

Edit: I think I recall you mentioning somewhere that your system is C# or something like that, so this doesn't apply to you.

Correct; I didn't read the OP properly, it would be processes not threads that are spawned.

GAT

I use ProcessPoolExecutor in concurrent.futures which is a more user friendly wrapper to multiprocessing.pool for wimps like me.

Separately I am trying to answer a very reasonable question a friend asked me about the more limited diversification opportunities from futures relative to cash markets (there are of course a lot more of the latter). My answer was that it was either hard or expensive or both to trade certain assets classes through the cash markets. Examples of these are commodities, STIR and volatility. Do people agree with this answer? If I recall correctly Jerry in the podcast is a big fan of the cash markets (which presumably is why he also prefers to trade slowly) as a diversifier.

 

[globalarbtrader]

I use ProcessPoolExecutor in concurrent.futures which is a more user friendly wrapper to multiprocessing.pool for wimps like me.

Separately I am trying to answer a very reasonable question a friend asked me about the more limited diversification opportunities from futures relative to cash markets (there are of course a lot more of the latter). My answer was that it was either hard or expensive or both to trade certain assets classes through the cash markets. Examples of these are commodities, STIR and volatility. Do people agree with this answer? If I recall correctly Jerry in the podcast is a big fan of the cash markets (which presumably is why he also prefers to trade slowly) as a diversifier.

So I have a lot of experience with this, since one of my main achievements was adding a lot of non futures markets to the AHL portfolio.

As well as costs and logistical issues, there are problems leveraging and going short cash markets; plus the difficulty of modelling them compared to futures markets (for the latter to get to a total excess return series you just have to do rolls and backadjustment, fiddly but easy once you know how; and then contract selection and rolling when you're actually trading).

Stocks:
Mostly cheap to trade and no issues with small tick sizes. Vast market with plenty of diversification. Leverage is relatively difficult to find, you eithier use CFDs (UK) which are expensive, or a margin account in the US (on which the interest margins are also quite steep for retail). It's not always easy to go short. There are technical difficulties with systematic trading of stocks: companies dying, dividends, stock splits, scrip issuance, mergers and takeovers... Plus if you're going to use leverage and go short you need history or estimates of funding and stock borrowing cost.

ETFs:
Similar to stocks, although can be simpler especially if you avoid ETFs that pay dividends, and don't worry about borrowing or going short. History tends to be limited, but can be a useful way of filling in gaps in futures markets especially for corporate bonds and EM equity indices.

Commodity markets:
Cash markets really are only for the industry professionals. Transaction costs and logistical difficulties abound. I suppose at a push you could buy some physical gold and hold it as part of your overall portfolio, if you have a decent safe in your house... but trading it would cost a fortune.

STIR:
There isn't really a STIR cash market per se. The reference rate (although this is changing) is supposed to be based on interbank lending, so really you need to be a bank.

Volatility:
You're basically talking about option trading. Leverage comes for free. Spreads can be large for OTM options. There are technical difficulties in dealing with different strikes and expiries, building surfaces, translating between notional 25 delta strangles and what you can actually trade.

Bonds:
The corporate bond market is highly fragmented, and for retail traders the spreads to be wide. Government bonds are better, and I think in the US at least it's relatively easy to buy bonds as a retail trader, but not in the UK. Leverage isn't too bad, but again for retail traders not cheap. They are technically harder to fit into a systematic strategy due to coupons, ageing... but not as bad as stocks.

You're better off with ETFs.

OTC fixed income:
Swaps et al. Basically almost inaccesible to retail traders. But relatively easy to turn into synthetic futures.

Bitcoin:
Buying and trading cash bitcoin is still (IMHO) a fraught, difficult and dangerous business.

Summary:
The most attractive cash markets are ETFs, stock and vol. It's no coincidence that most CTAs that have expanded into other asset classes are focused mainly in these areas. A lot of time and investment would be needed to build backtesters and trading strategies for stock and vol trading, but you will be rewarded with uncorrelated strategies. For stocks you are probably better off focusing on long only, or perhaps long only with a futures hedge.

ETFs are probably an easier proposition, and they are a nice way of covering gaps in your futures exposure. In many ways they are better than the cash markets they represent. They are relatively easy to get total return series for, if you exclude dividend payers. Trading them on a long only basis is going to be quite nice and easy, although obviously not as capital efficient as futures trading.

GAT

 

[Elder]

So I have a lot of experience with this, since one of my main achievements was adding a lot of non futures markets to the AHL portfolio.

As well as costs and logistical issues, there are problems leveraging and going short cash markets; plus the difficulty of modelling them compared to futures markets (for the latter to get to a total excess return series you just have to do rolls and backadjustment, fiddly but easy once you know how; and then contract selection and rolling when you're actually trading).

Stocks:
Mostly cheap to trade and no issues with small tick sizes. Vast market with plenty of diversification. Leverage is relatively difficult to find, you eithier use CFDs (UK) which are expensive, or a margin account in the US (on which the interest margins are also quite steep for retail). It's not always easy to go short. There are technical difficulties with systematic trading of stocks: companies dying, dividends, stock splits, scrip issuance, mergers and takeovers... Plus if you're going to use leverage and go short you need history or estimates of funding and stock borrowing cost.

ETFs:
Similar to stocks, although can be simpler especially if you avoid ETFs that pay dividends, and don't worry about borrowing or going short. History tends to be limited, but can be a useful way of filling in gaps in futures markets especially for corporate bonds and EM equity indices.

Commodity markets:
Cash markets really are only for the industry professionals. Transaction costs and logistical difficulties abound. I suppose at a push you could buy some physical gold and hold it as part of your overall portfolio, if you have a decent safe in your house... but trading it would cost a fortune.

STIR:
There isn't really a STIR cash market per se. The reference rate (although this is changing) is supposed to be based on interbank lending, so really you need to be a bank.

Volatility:
You're basically talking about option trading. Leverage comes for free. Spreads can be large for OTM options. There are technical difficulties in dealing with different strikes and expiries, building surfaces, translating between notional 25 delta strangles and what you can actually trade.

Bonds:
The corporate bond market is highly fragmented, and for retail traders the spreads to be wide. Government bonds are better, and I think in the US at least it's relatively easy to buy bonds as a retail trader, but not in the UK. Leverage isn't too bad, but again for retail traders not cheap. They are technically harder to fit into a systematic strategy due to coupons, ageing... but not as bad as stocks.

You're better off with ETFs.

OTC fixed income:
Swaps et al. Basically almost inaccesible to retail traders. But relatively easy to turn into synthetic futures.

Bitcoin:
Buying and trading cash bitcoin is still (IMHO) a fraught, difficult and dangerous business.

Summary:
The most attractive cash markets are ETFs, stock and vol. It's no coincidence that most CTAs that have expanded into other asset classes are focused mainly in these areas. A lot of time and investment would be needed to build backtesters and trading strategies for stock and vol trading, but you will be rewarded with uncorrelated strategies. For stocks you are probably better off focusing on long only, or perhaps long only with a futures hedge.

ETFs are probably an easier proposition, and they are a nice way of covering gaps in your futures exposure. In many ways they are better than the cash markets they represent. They are relatively easy to get total return series for, if you exclude dividend payers. Trading them on a long only basis is going to be quite nice and easy, although obviously not as capital efficient as futures trading.

GAT

GAT, this is brilliant, thanks. I am going seem so smart when I go back to explain in more detail. "Plagiarism!" I hear you complain. Fair enough, I will mention that it was you that (once again) provided the real truly valuable insights. Separately, I wonder if this would make a good topic/question for your next podcast with Niels?

 

[globalarbtrader]

GAT, this is brilliant, thanks. I am going seem so smart when I go back to explain in more detail. "Plagiarism!" I hear you complain. Fair enough, I will mention that it was you that (once again) provided the real truly valuable insights. Separately, I wonder if this would make a good topic/question for your next podcast with Niels?

Good idea, will make a note of that. I think it's the 26th June.

I note in passing that I've interpreted 'cash' to mean 'not futures'. Clearly Vol and swaps, and arguably ETFs, aren't really 'cash' since they're all still derivatives (admittedly ETFs are delta-one).

GAT

 

[Kernfusion]

Today IB sent me several bid\ask prices of "1" for BTP 2021-09 and 2021-12 contracts. Not all prices for them are coming like this, I see about 20 messages in the system's log. The actual current price for these contracts is around 152. I guess nothing too unusual, but haven't seen this particular one before .
My system generates a warning when RT price deviates by more than 9 sigma from last EOD and I throw the price away if it's different by more than 12 sigma..