HFT robots seeing orders before the exchange ?

[DeeDeeTwo]

Quote from baglunch:

...we're competing to be second in line at best vs the big banks and hft guys.

A lot of strategies still work if you are 2nd in line.

-------------------------------------------------------------------

The simple, key point that people seem to miss it this:

The bad guys can ONLY deny you a fill.

So if you put out 10 orders and DO NOT CARE which one fills...
Then you are virtually unaffected by the Bad Guys.

 

[trader422]

A lot of strategies still work if you are 2nd in line.

A pearl of wisdom for anyone who doesn't yet fully understand this truth ... I would put it slightly differently, replacing "2nd in line" with "last in line". Believe it ... if you don't and you are still trying to make money trading then you are either insane or really, really dumb.

So if you put out 10 orders and DO NOT CARE which one fills...
Then you are virtually unaffected by the Bad Guys.

This is another of several DeeDeeTwo quotes in this thread that are simply wrong not because DeeDeeTwo isn't making a good point but rather because the language is too extreme. If DeeDeeTwo had said "... Then you can have a successful trading program in spite of the Bad Guys" rather than "..Then you are virtually unaffected by the Bad Guys" I would be in more or less complete agreement ... however, it's a simple matter of basic logical reasoning using the available facts that the predators are sucking a very large portion of the profits out of the market that was not being sucked out in former years ... If you want to start with just one well-documented, big-hammer fact, focus on the data showing that for lower-volume stocks something like 50% - 70% of total volume now gets done via broker-dealer internalization or in dark pools (a huge increase since the mid-2000s), with almost none of the high-profit, low-risk uninformed order flow available to cross with the general public even within these venues. One needn't be a logician to infer that traders who don't get a shot at this volume, particularly the uninformed volume, are something other than than "virtually unaffected".

P.S. Although DeeDeeTwo's trading program and my own have major differences, they also have key similarities ... in particular, in apparent similarity with DeeDeeTwo's program, my program has about 1500 - 3000 orders working on about 1000 stocks all day long; I average about 350 executions per day with about 200 on a really dead day and 500-600 on a particularly active day. My point here is that anyone who has plowed through this whole thread can see that I must be in agreement with a lot of DeeDeeTwo's basic points. As I think I've said before in this thread, the key difference between me and DeeDeeTwo seems to be that I'm essentially a liquidity provider and DeeDeeTwo is essentially a liquidity taker ... the conclusion being that on many (but not all) points we can both be right.

 

[DeeDeeTwo]

Quote from trader422:

As I think I've said before in this thread, the key difference between me and DeeDeeTwo seems to be that I'm essentially a liquidity provider and DeeDeeTwo is essentially a liquidity taker ... the conclusion being that on many (but not all) points we can both be right.

I'm purely a liquidity provider = Market Maker.

I bracket the NBBO with orders (but emphasize one side)...
So an order that's $0.03 away and, maybe, 6th in line is valuable.

To simplify, If I have 10 orders like the above on 10 similar stocks...
Why would I care which one or two get filled?

When you focus on one order on one side on one symbol...
It's easy to explain how Bad Guys can make your trading sub-optimal.

But when you have 300-400 orders out there on > 100 stocks on both sides of the market...
And Bad Guys block 25% of those orders... who cares?

It's very hard to make money today.
It's all about experience and custom infrastructure.

You cannot buy a car that will win the Indy 500...
YOU HAVE TO BUILD IT FROM SCRATCH, baby.

 

[trader422]

It's very hard to make money today.
It's all about experience and custom infrastructure.

You cannot buy a car that will win the Indy 500...
YOU HAVE TO BUILD IT FROM SCRATCH, baby.

Something on which we can agree completely ... I'm thinking of writing a trading book titled "How I did it: 10 years and 1 million lines of code", but I think only my mother would read it ...

To simplify, If I have 10 orders like the above on 10 similar stocks...
Why would I care which one or two get filled?

But when you have 300-400 orders out there on > 100 stocks on both sides of the market...
And Bad Guys block 25% of those orders... who cares?

Very simple: A few years ago it was "which 3 or 4 get filled, with 2 or 3 against uninformed order flow at prices that get you in with much lower risk at much greater profit potential" ... now, as you say, it's "which 1 or 2 get filled with extreme adverse selection at prices that get you in with much higher risk and much lower profit potential".

When you focus on one order on one side on one symbol...
It's easy to explain how Bad Guys can make your trading sub-optimal.

I understand and agree with your point about focusing on one or a few orders (a wretched way to fail at liquidity provision), but as for me I have made it clear (I hope) that I'm a passionate devotee of your philosophy of working thousands of orders, no particular one or few of which I give a fig about.

I'm not sure I understand your point about one-sided market making ... it's great being two-sided with flexible but limited bias when it works for particular symbols, but besides the obvious benefits I can't see how being two-sided directly ameliorates the damage done by the predators.

I'm purely a liquidity provider = Market Maker.

I bracket the NBBO with orders (but emphasize one side)...
So an order that's $0.03 away and, maybe, 6th in line is valuable.

Based on these comments and previous posts you've made to this thread, I'm starting to get the feeling that perhaps our basic difference may be the data we process. I use market depth extensively and dump megabytes of logs telling me what the predators are doing in and around my markets, and how those activities impact my risk and reward ... plus I have a big discretionary component that forces me to experience all the indignities in real time for hours every day. In comparison, your trading sounds like a blessedly data-light pursuit. I am envious.

 

[baglunch]

Yes, some strategies do work if you are 2nd in line but that wasn't really what I was trying to say. What I was trying to imply was that at BEST you'll get 2nd in line and that it is an up hill battle using the consolidated feeds (the SIPs) vs proprietary feeds.

It also doesn't help that NYSE is a parity based exchange and the liquidity gets split 3 ways upon matching so the person 2nd in line is really 4th after the broker and market maker.

Can you make money being 2nd in line? absolutely. Can you expect to be towards the front of the line consistently if you only use the consolidated feed? unlikely. The consolidated feeds have come a long way since the 90s but still have a lot of work ahead of them.

What you said about the 10 orders is absolutely correct. Lets say you're a market maker/liquidity provider, for example, if you're doing index-arb and providing on the futures and buying the underlying equities you can't rely on 1 order to buy 1000 shares of xyz. I'd split it into 10 orders, because you never know when one or two of the gateways you're connecting to can lag at different times. 800 filled is better than 0.

Quote from DeeDeeTwo:

A lot of strategies still work if you are 2nd in line.

-------------------------------------------------------------------

The simple, key point that people seem to miss it this:

The bad guys can ONLY deny you a fill.

So if you put out 10 orders and DO NOT CARE which one fills...
Then you are virtually unaffected by the Bad Guys.

 

[PocketChange]

You should read up on some of the SEC findings and Nanex research documents as to the accuracy of the NMS.

SEC stated in one of their advisory notices 7% of all trades are misreported.

Nanex did a research piece where they conclude the systems deployed by the exchanges do the NBBO match in memory without any audit trail.

http://www.nanex.net/Research/IsNBBOIgnored.html

Reality is you can't rely on the regulators to keep a fair and orderly game. Assume the cheaters will always prosper and focus on evolving your game.

 

[DeeDeeTwo]

Quote from trader422:

Based on these comments and previous posts you've made to this thread, I'm starting to get the feeling that perhaps our basic difference may be the data we process.

I use market depth extensively and dump megabytes of logs telling me what the predators are doing in and around my markets, and how those activities impact my risk and reward ... plus I have a big discretionary component that forces me to experience all the indignities in real time for hours every day. In comparison, your trading sounds like a blessedly data-light pursuit. I am envious.

This is your best insight...
That I think in terms of concepts that simplify complex systems...
As opposed to what most "brilliant" people do...
They drill down and down... and complicate the hell out of everything.

I'm obsessed with "normalizing" the stocks in a Universe in such a way...
That I'm indifferent to which 10 out of 50 I go long...
And I'm indifferent to which 10 out of 50 I go short...
Which relegates algos and predators to just ambient noise.

And ambient noise is something you don't think much about...
My competitors will cost me a managable 20-30% of my profits or whatever...
And that is the end of it.

 

[trader422]

And that is the end of it.

Bravo ... although of course some degree of data diarrhea holds an essential place for many, depending on the details of their trading program ... the most obvious example being our HFT friends, who have extremely successful trading programs that are the epitome of low-risk, and which depend critically on the most massive, comprehensive market microstructure data mining/monitoring/managing imaginable. The lesson being, as it so frequently is, that there is a continuum here.

However, as I've had to learn repeatedly over my 19 years of trading, the degree to which having your nose down in data and models can waste your time and divert your attention from more worthwhile priorities (like actually making money) cannot be overemphasized.

But (hopefully in conclusion) I would like to bring this thread back to what I see as its most noble purpose, which is to benefit one or more fellow elitetraders ... to that end, I would point out that most of the DeeDeeTwo posts seem to rely on the thesis that DeeDeeTwo's opinions on the subject of predatory trading are proved right by the performance and/or design of DeeDeeTwo's trading program ... but unfortunately this makes little logical sense because DeeDeeTwo has repeatedly emphasized that the DeeDeeTwo trading program takes a thousand-mile-up view of market microstructure and the details of the activities of the predators. So the only sub-thread of this thread that I have ever been interested in - whether or not, and if so to what degree, the predators have (1) had a severe negative impact on retail liquidity provision in low-volume listed stocks and (2) are likely to continue to increase their negative impact going forward, and to a very significant degree, has really not been addressed by DeeDeeTwo's otherwise excellent posts. In conclusion: To any elitetraders who are getting something out of this thread: If your goal is to trade exactly like DeeDeeTwo then that sounds pretty good to me - go for it. But otherwise, if you are interested in liquidity provision strategies for low-volume listed stocks and wondering if it is a good idea to expend your time and energy on this pursuit at the present time - then pay careful attention to what folks like me, Haim Bodek, Sal Arnuk and Joe Saluzzi have to say ... read "Broken Markets", and then read a selection of papers from the extensive bibliography on the Themis Trading web site ... then look at the data yourself, with reference to your own trading program design ... and thus educated, decide for yourself.

 

[softdown]

I think it's important to distinguish between two things:

-"Fair" HFT competitors which have a big speed/latency advantage on the regular traders but don't have any "unfair" advantage.
-"Unfair" HFT competitors or "dishonest" brokers which have "unfair" advantages.

Example of "unfair" advantages:

1. Seeing the order flow BEFORE it reaches the order book. This was possible a few years ago with the "flash trades".
If anything like this is still possible, that's a BIG UNFAIR advantage for the big boys.

2. Brokers selling the order flow or internalizing it. The effect of this is that buyers of the order flow can see the orders before they reach the exchange.
So in that regard it's quite similar to no. 1 above, but they cannot see all the orders, just the ones from a certain broker.

A counter measure for this is to use directed orders that are supposed to be sent directly to the exchange.
I don't know if the brokers really send all the directed the orders to the exchange or not. It should be possible to check.

3. Brokers doing shenanigans with the orders like triggering stop where they shouldn't etc..


If I might have missed some ways in which they can screw you, please post them below.

 

[trader422]

If I might have missed some ways in which they can screw you, please post them below.

I could name a few things that have been publicly documented in recent months, but identifying the issues outside the internalization/dark pool area is a game of whack-a-mole: Someone could cite a recent well-documented story, then someone else could claim with some degree of credibility that that issue is resolved, but new issues will continually come to light because the HFTs are the exchanges' best customers (and in come cases part owners), so the exchanges cater to them with better, faster data feeds (aka legal, SEC-blessed front-running) and a la carte new order types designed by the predators, for the predators. Also the issues are different for high-liquidity stocks compared to low-liquidity ...

The analyses can get quite sophisticated for high-liquidity stocks and it's easier to argue both sides, but for low-liquidity stocks anyone can do a little homework and see for themselves what is going on. There are lots of moving parts, particularly with new order types, but the big glaring issue is the hijacking of (1) broker-dealer internalization; (2) payment for order flow; (3) dark pools by the predators. Notice I stress that the "hijacking" is the problem, not (1) - (3) per se ... (1) - (3) have been around forever and are arguably not so bad (they certainly never hurt me much before late 2009), but only since roughly 2007 have they been progressively commandeered by the predators and made a primary focus with the goal of hyper-efficiently sucking out all the available easy money.