Fascinating to see someone from the 'dark side' in this discussion!
1) The markets where you could be waiting tens of minutes... or longer... for fills tend to be those with large size (much of which isn't intended for execution) and little volatility. I'm thinking of STIR markets, especially near the front of the curve (where I tend to avoid trading anyway). I could tell you about the time I was helping out on our execution desk and I whiled away 6 hours waiting for a 2 lot Euroswissy order to be passively filled before finally bailing and crossing the spread... but it was boring then, and it would be boring now
2) Is an interesting idea.
It's fair to say I've never really done much with my execution algo; it works well enough, and I don't really trade often enough to conduct experiments in changing it, or seeing if some markets do better or worse than others with a given set of parameters.
(One cool idea would be to set up a system whereby retail traders can pool execution data in some common format, producing a much larger dataset)
3) "I feel you should 're-evaluate' your 'signal' (I'll be honest I don't understand exactly how you guys do this), when the price move against you"
This is the critical point of difference. I generate signals daily overnight, and once a required trade is generated it is not changed by any further price movement that may occur after the generation has occured. The behaviour is that of a slow hedge fund, with a portfolio management team that cogitate and decide what rebalancing is required, then throw the trades over a one way wall to the execution team which has one simple mandate:
execute this order at the best price you can, irrespective of what happens next, within this time frame (by the close of the market today).
Over the other side of this wall could be a sell side algo, a bank of human traders, or as we have here an in house algo (at AHL we used all three routes to market, and ran them in competition against each other). It doesn't matter.
So in practice, we could be trading the next day at a price far away from the price at which the trade was generated. The backtest deliberately simulates this behaviour, assuming the price is executed at the following days close. In practice we'll be filled somewhere between the open and the close. A good thing to check is that the difference between 'signal price' and 'mid price when execution begins' is a source only of small random noise, and not of bias. If you consistently lose money through this delay then your underyling system is capturing a source of alpha that is too fast for this lazy execution strategy.
With such a system it is true that the price moving away from you isn't neccessarily a source of panic (unless the market closes in 5 minutes); the logic behind this design decision is purely simplicity (my intention was literally to build the dumbest execution algo I could that would still add some value). Without it you'd need to implement some kind of utility function that balanced risk and value, and /or had some capacity to predict short term movements in price. You'd basically be grafting a short term strategy on to a longer term strategy, and breaking down the wall.
Of course you could just rerun the system every few seconds (which I guess is what you are alluding to), and with enough computational power and some smart coding this would be possible. It does make things complicated in the execution stack since you need to have the ability to cancel existing unfilled orders when they are no longer required to hit the optimal position. I toyed with including this logic in my own code, and it got very complicated very quickly.
Yet another way of doing this is to keep the seperation, but include more complex conditional orders. So you could have orders that said things like 'buy 50 but only if the price is 30 or better'. Or 'buy 20 NOW' for a stop loss. Or 'buy 20 but only if the price has gone above 10 but is still below 20'. You get the idea. Then your overnight run generates these more complex orders, and throws them over the wall.
This is hard for a system set up like mine to achieve, as it uses contionous signals you'd need to effectively monte carlo the system to see what position it would have on at different price levels. In fact there used to be a very large CTA that used exactly this system.
The problem with all of these solutions is that 99.9% of the time it doesn't affect what the daily system was going to do anyway: even though the price may have changed a fair bit since the original signal was generated we'd still want to do the same trade. And it's even less likely that a small change in price during the few seconds or minutes we've been executing would affect the trading system.
However there are some systems for which the conditional order solutions would make sense. One (which I'm thinking of testing) is a relatively fast (holding period a couple of days) mean reversion system; for this 'buy if the price is 30 or better' makes perfect sense. Another (which I'm not implementing) would be an option delta hedger.
4) Another reason to use limit orders rather than market is that with IB you can't change an order type, only the price. So if you start trying to executive passively with a limit order, and you want to change it to a market order to be aggresive, you can't: instead you have to change the limit price so it's crossing the spread.
GAT
