Fully automated futures trading
- Thread starter globalarbtrader
- Start date
Fascinating to see someone from the 'dark side' in this discussion!
1) The markets where you could be waiting tens of minutes... or longer... for fills tend to be those with large size (much of which isn't intended for execution) and little volatility. I'm thinking of STIR markets, especially near the front of the curve (where I tend to avoid trading anyway). I could tell you about the time I was helping out on our execution desk and I whiled away 6 hours waiting for a 2 lot Euroswissy order to be passively filled before finally bailing and crossing the spread... but it was boring then, and it would be boring now
2) Is an interesting idea.
It's fair to say I've never really done much with my execution algo; it works well enough, and I don't really trade often enough to conduct experiments in changing it, or seeing if some markets do better or worse than others with a given set of parameters.
(One cool idea would be to set up a system whereby retail traders can pool execution data in some common format, producing a much larger dataset)
3) "I feel you should 're-evaluate' your 'signal' (I'll be honest I don't understand exactly how you guys do this), when the price move against you"
This is the critical point of difference. I generate signals daily overnight, and once a required trade is generated it is not changed by any further price movement that may occur after the generation has occured. The behaviour is that of a slow hedge fund, with a portfolio management team that cogitate and decide what rebalancing is required, then throw the trades over a one way wall to the execution team which has one simple mandate: execute this order at the best price you can, irrespective of what happens next, within this time frame (by the close of the market today).
Over the other side of this wall could be a sell side algo, a bank of human traders, or as we have here an in house algo (at AHL we used all three routes to market, and ran them in competition against each other). It doesn't matter.
So in practice, we could be trading the next day at a price far away from the price at which the trade was generated. The backtest deliberately simulates this behaviour, assuming the price is executed at the following days close. In practice we'll be filled somewhere between the open and the close. A good thing to check is that the difference between 'signal price' and 'mid price when execution begins' is a source only of small random noise, and not of bias. If you consistently lose money through this delay then your underyling system is capturing a source of alpha that is too fast for this lazy execution strategy.
With such a system it is true that the price moving away from you isn't neccessarily a source of panic (unless the market closes in 5 minutes); the logic behind this design decision is purely simplicity (my intention was literally to build the dumbest execution algo I could that would still add some value). Without it you'd need to implement some kind of utility function that balanced risk and value, and /or had some capacity to predict short term movements in price. You'd basically be grafting a short term strategy on to a longer term strategy, and breaking down the wall.
Of course you could just rerun the system every few seconds (which I guess is what you are alluding to), and with enough computational power and some smart coding this would be possible. It does make things complicated in the execution stack since you need to have the ability to cancel existing unfilled orders when they are no longer required to hit the optimal position. I toyed with including this logic in my own code, and it got very complicated very quickly.
Yet another way of doing this is to keep the seperation, but include more complex conditional orders. So you could have orders that said things like 'buy 50 but only if the price is 30 or better'. Or 'buy 20 NOW' for a stop loss. Or 'buy 20 but only if the price has gone above 10 but is still below 20'. You get the idea. Then your overnight run generates these more complex orders, and throws them over the wall.
This is hard for a system set up like mine to achieve, as it uses contionous signals you'd need to effectively monte carlo the system to see what position it would have on at different price levels. In fact there used to be a very large CTA that used exactly this system.
The problem with all of these solutions is that 99.9% of the time it doesn't affect what the daily system was going to do anyway: even though the price may have changed a fair bit since the original signal was generated we'd still want to do the same trade. And it's even less likely that a small change in price during the few seconds or minutes we've been executing would affect the trading system.
However there are some systems for which the conditional order solutions would make sense. One (which I'm thinking of testing) is a relatively fast (holding period a couple of days) mean reversion system; for this 'buy if the price is 30 or better' makes perfect sense. Another (which I'm not implementing) would be an option delta hedger.
4) Another reason to use limit orders rather than market is that with IB you can't change an order type, only the price. So if you start trying to executive passively with a limit order, and you want to change it to a market order to be aggresive, you can't: instead you have to change the limit price so it's crossing the spread.
GAT
Thanks GAT for the insights! Very interesting to see how things work on the buy side. A few follow-up questions if you don't mind:
1) Nice example! Now i remember I gave up and crossed the spread in a roll contract after 2 hours, because no one cares about the 200 bucks I could potentially save, and I wanted to go home early
3)
- "execute this order at the best price you can, irrespective of what happens next, within this time frame (by the close of the market today)"
I understand that funds like AHL has a lot of money
, and hence 'big' positions, but how 'gradual' do you normally change your positions? For 'small' sizes, I think you can essentially 'trade at mid' at the opening auction/closing auction? This makes sense to me if you don't have intraday opinion/don't care enough about a day's move. If your size is big enough for market impact to be an issue, of course you need alternative solutions. I imagine options include something like execution algo on screen/block request OTC/ BTIC markets?
- 'the difference between 'signal price' and 'mid price when execution begins' is a source only of small random noise, and not of bias'
This makes sense. I'm wondering if you take into account 'slippage' per contract in signal generation? I imagine it's probably insignificant for 'slow' systems. This leads me to the following question.
- Do you measure/predict/backtest % return per trade? If so, what are some benchmark numbers? From my understanding, this is not a typical statistic CTAs look at, and you are trying to maximise Sharp given a target annualised vol (i.e. max return for fixed risk)? Please correct me if I'm wrong!
I mention this because I believe % return/trade is the stat which dictates how much slippage you can 'afford', and not necessarily how 'fast/slow' you trade, although the two are obviously related. An unrealistic example: you have a system which trades 'slow' (say holding period 6 months) by buying and holding 1000 uncorrelated stocks, Sharpe would be off the roof even if % return/trade was low, and high slippage would essentially kill the system.
Yup 100% agree what I'm saying is pretty much irrelevant for small sizes, only jumped in when I see discussions on execution algos
Thank you for jumping in and adding to the conversation.Yup 100% agree what I'm saying is pretty much irrelevant for small sizes, only jumped in when I see discussions on execution algos
Can't say that easily, I record the initial and fill times and all changes in the state of the order (passive\aggressive) but not the reason for the change. (Actually I do record it, but only in the unstructured free-text log-table and it's a real pain to extract it from there). I can upper-bound it though, by counting how many orders ended up in the aggressive state and took more than the maximum passive time to execute - it's 12 out of 296, so yes not many..
Here's some more stats I can easily extract:
296 total orders (in about 9 months)
95 filled in Passive mode, all 95 made >=0$ compared to the worst side of the spread(no losers in passive mode)
201 filled in Agressive mode, 56 of which still made money compared to paying full spread (not sure how that happened, perhaps lucky price moves?)
31 orders took more than 10 min to fill:
3KTB 6
GBM 2
GE 4
N225M 1
NG 1
V2TX 6
ZT 11
151 orders took less than 1 minute to fill
Yes, most of the time it will still be worth trading because (even though I reevaluate my forecasts in real-time) the system is mostly trend-following and crossing the spread will mostly happen when that trend continues away from me and I'm chasing it.
It is possible though, that the price will be dancing around the point of buying and not buying that additional contract, but first when my system places an order and while it's active it considers it filled for the purpose of calculating my current position in this instrument., and second I have a "signal thresholding\buffering" which sort of runs position-sizing logic backwards from my current position (and that position counts active orders too) and outputs the theoretical signal which corresponds to my current position., and then I require the difference between that "signal of my current position" to be different by more than 30% of the average signal(10) from the current real-time signal to make a trade (oh, I really overthink things sometimes
). So basically yes, apart from bugs, the system will rarely want to cancel an order because of forecast change.Should I then always sit on my side of the spread unless the max timeout expires or the volume starts looking unfavorable ? - maybe.. But it needs experimentation on the PROD system and at least several months worth of data to be able to say if it makes things better.. Don't know, maybe I can try it on my paper system, first, though I would be relying on IB execution simulation..
- so you mean there's never a good reason to use market orders, and even if you just want a fast guaranteed fill it's still better to submit a limit order which crosses the spread but will not get out of it by a catastrophic amount? (e.g. if the spread is 110/112 and I want to buy at market, it's better to submit a limit order at, say 114, and I'll still most likely get a fill at 112, and if something goes wrong I get a partial fill at 112\113\114 but never worse than that)? - Sure, makes sense..
In fact, I'm thinking that at some point I'll add ICE-futures into the system, without paying for the real-time data (the only way they get that 200$\month is from my cold dead hands!
)., which means I will have 15 minute delays for every bid\ask update and will have to deal with this somehow.. One simple approach could be to actually change nothing and keep using the same algo, and just pretend that I have a reeeally big internet latency from my provider. Or I could use some IB's algo orders, or just as you're proposing here, submit limit-orders just outside of the spread (towards a worse price from my perspective), but at least limiting the worst-case..
Last edited:
In fact, I'm thinking that at some point I'll add ICE-futures into the system, without paying for the real-time data (the only way they get that 200$\month is from my cold dead hands!
I've had similar thoughts about adding futures without real time data. Problem is, if your price is too far away from the current spread (which you can't see) the order will be rejected. Do IB have an order type which is just 'put in a limit at the current side eg buy at the offer, sell at the bid'? Because that would solve the problem.
GAT
Thanks GAT for the insights! Very interesting to see how things work on the buy side. A few follow-up questions if you don't mind:
1) Nice example! Now i remember I gave up and crossed the spread in a roll contract after 2 hours, because no one cares about the 200 bucks I could potentially save, and I wanted to go home early
3)
- "execute this order at the best price you can, irrespective of what happens next, within this time frame (by the close of the market today)"
I understand that funds like AHL has a lot of money
Depends on the contract. In an extreme case, we'd be liquidating a position over a few days. But normally it would take at least a week.
No we didn't go near the auctions. With the size we were doing it was better to trade continously throughtout the day rather than drop a huge order in at the open or close.
AHL used to trade OTC of course, but for future our attitude was that our algos (and human traders) were better than any kind of outsourced solution (bank algo, block trade). Of course we didn't take this for granted, but we would check by giving some of our flow to external providers and constantly running experiments to see if they could do better.
Block trades make sense if the large marginal holders in a market are often doing different things, so you can genuinely exchange large blocks between parties with different axes. That's true of actively traded discretionary equities, but not so true of eg equity market neutral hedge funds with a similar kind of factor model. I'm not sure it's that true in futures where you have large CTA's mostly doing similar kinds of trade, mostly trading against smaller players who are doing the opposite.
Not in signal generation per se, but in terms of designing the system. So what I do is calculate a risk adjusted total cost of trading (commission plus slippage), and that drives the decision about which speed of signals make it into the model for that instrument, basically how frequently I trade.
You're right that we don't normally look at things on a 'per trade' basis. Of course I can measure the % return per trade, but it's not a key statistic. And I will measure the slippage and commission per trade, but that is then translated into annualised risk adjusted terms so it's now a Sharpe Ratio comparable with everything else.
(For the 'fixed risk' I don't actually do that, but there is a longer discusison here)
I think the key here is to look at the (risk adjusted) return per instrument, not on the entire system. So for example, suppose I can get a pre-cost SR of 0.4 trading any given instrument, and with diversification I can get that up to a SR of 1.0 (ie the diversification multiplier is 2.5). Now suppose I am willing to pay half my pre-cost SR in costs (which I'm not, that's much higher than I'd do, but for the sake of the maths....) .
Well in theory I can easily pay 0.5 SR in costs, but of course I can't; I'd be losing 0.1 SR on each of my instruments. What I actually do is look at each instrument in turn, and from that make a decision on how much I can afford to pay in costs, and in this example I'd pay 0.2 SR in costs leaving 0.2 SR net per instrument. So my after cost SR would actually be 2.5 * 0.2 = 0.5 and I'd be paying 0.5 SR in costs on the entire system.
GAT
Can't say that easily, I record the initial and fill times and all changes in the state of the order (passive\aggressive) but not the reason for the change. (Actually I do record it, but only in the unstructured free-text log-table and it's a real pain to extract it from there). I can upper-bound it though, by counting how many orders ended up in the aggressive state and took more than the maximum passive time to execute - it's 12 out of 296, so yes not many..
Here's some more stats I can easily extract:
296 total orders (in about 9 months)
95 filled in Passive mode, all 95 made >=0$ compared to the worst side of the spread(no losers in passive mode)
201 filled in Agressive mode, 56 of which still made money compared to paying full spread (not sure how that happened, perhaps lucky price moves?)
31 orders took more than 10 min to fill:
3KTB 6
GBM 2
GE 4
N225M 1
NG 1
V2TX 6
ZT 11
151 orders took less than 1 minute to fill
Yes, most of the time it will still be worth trading because (even though I reevaluate my forecasts in real-time) the system is mostly trend-following and crossing the spread will mostly happen when that trend continues away from me and I'm chasing it.
It is possible though, that the price will be dancing around the point of buying and not buying that additional contract, but first when my system places an order and while it's active it considers it filled for the purpose of calculating my current position in this instrument., and second I have a "signal thresholding\buffering" which sort of runs position-sizing logic backwards from my current position (and that position counts active orders too) and outputs the theoretical signal which corresponds to my current position., and then I require the difference between that "signal of my current position" to be different by more than 30% of the average signal(10) from the current real-time signal to make a trade (oh, I really overthink things sometimes
Should I then always sit on my side of the spread unless the max timeout expires or the volume starts looking unfavorable ? - maybe.. But it needs experimentation on the PROD system and at least several months worth of data to be able to say if it makes things better.. Don't know, maybe I can try it on my paper system, first, though I would be relying on IB execution simulation..
- so you mean there's never a good reason to use market orders, and even if you just want a fast guaranteed fill it's still better to submit a limit order which crosses the spread but will not get out of it by a catastrophic amount? (e.g. if the spread is 110/112 and I want to buy at market, it's better to submit a limit order at, say 114, and I'll still most likely get a fill at 112, and if something goes wrong I get a partial fill at 112\113\114 but never worse than that)? - Sure, makes sense..
In fact, I'm thinking that at some point I'll add ICE-futures into the system, without paying for the real-time data (the only way they get that 200$\month is from my cold dead hands!
Thanks for sharing your stats! Honestly you are probably already spending more time than you should on your algo with your trade frequency.
'201 filled in Agressive mode, 56 of which still made money compared to paying full spread (not sure how that happened, perhaps lucky price moves?)'
From my understanding of your algo, this only happens when you decide to cross the spread straight away instead of putting in a passive limit order (due to favourable conditions), and by the time your aggressive order arrives, the market has already moved in your favour and you got filled at a better price than you expected. This is happening quite often, and I think this suggests your speed is slow. Probably not worth trying to improve your algo further, as I think most 'clever' ideas will simply not work due to latency.
And yes personally I'll never use market orders, I see them as limit orders with 0/infinity price, which look very dangerous to me. I've also seen MMs deliberately going wide straight after open in normally tightly-quoted products (admittedly I've done this myself, and it has worked more often than you think....) , just to catch potential market orders. Yes they are predatory......
Again thanks for the insights GAT.
1) Yup I do believe your in-house algos will do better than outsourced solutions. Even if someone has a slightly better algo, I doubt it's worth what they charge you for.
2) Interesting to see ultimately who's holding opposite positions against CTAs. I have no idea, but my guess is that there are enough random players out there to absorb the flow over a long enough time?
3) Yes I agree with looking at SR/instruments makes the most sense. I'm not sure if this can always be calculated/estimated though, depending on how your 'signal' is generated. E.g. Say I believe there is mean-reversion in ES/NQ spread, but splitting the SR in half may not really make sense.