Detecting stuck quotes automatically

stephencrowley · Jan 24, 2006

So I'm analyzing data.. on one of my days it appears my ARCA feed has a stuck quote for AAPL because the best price never changes after about 8:45 for that day. However, my INET, TOTAL-VIEW and BRUT feeds are updating just fine.

Anyone have any approachs to automatically detecting stuck quotes such as this?

I thought about comparing some average quote time.. if the feed hasnt had a best price change in some interval beyond the average, and the price is some percent away from the other feeds then take action.

Possible actions
1) delete all entries near the top that are significantly older than the other entires near it (a few cents away)
2) disconnect and reconnect the quote stream for that symbol.

#1 is the only option for historical simulations, #2 is the best for realtime.

Thoughts?

nononsense · Jan 25, 2006

Quote from stephencrowley:

So I'm analyzing data.. on one of my days it appears my ARCA feed has a stuck quote for AAPL because the best price never changes after about 8:45 for that day. However, my INET, TOTAL-VIEW and BRUT feeds are updating just fine.

Anyone have any approachs to automatically detecting stuck quotes such as this?

I thought about comparing some average quote time.. if the feed hasnt had a best price change in some interval beyond the average, and the price is some percent away from the other feeds then take action.

Possible actions
1) delete all entries near the top that are significantly older than the other entires near it (a few cents away)
2) disconnect and reconnect the quote stream for that symbol.

#1 is the only option for historical simulations, #2 is the best for realtime.

Thoughts?

Hi Steph,

In all automated systems, trading like industrial, screening of the incoming data is wise.
Anomalies in data feeds are not so uncommon and it is sometimes hard to get those fixed by the source.
In fact you are almost in uncharted territory - only experience can tell. In some cases connecting - disconnecting is all that is required, in others not, hard to know.

I use a time interval monitoring between successive ticks approach. I monitor individual issues and I also monitor this interval for the complete set. No hard rules for this. Anything you set yourself is bound to be off one day.

I also screen a single tick against the earlier ones received in order to catch outliers. However, I assume that a way out tick was correct if it gets repeated. The danger here is that you react to a real jump with a delay. I always think (or hope) that such jumps occur at moments when the incoming tick rate is rather high.

Having feeds come in from different sources over separate ISP's would be nice.

The nasty thing is what to do in case you have a trade on. I only have an alarm go off and I take care of this manually. I still have not found much better.

squeeze · Jan 25, 2006

Quote from stephencrowley:

So I'm analyzing data.. on one of my days it appears my ARCA feed has a stuck quote for AAPL because the best price never changes after about 8:45 for that day. However, my INET, TOTAL-VIEW and BRUT feeds are updating just fine.

Anyone have any approachs to automatically detecting stuck quotes such as this?

I thought about comparing some average quote time.. if the feed hasnt had a best price change in some interval beyond the average, and the price is some percent away from the other feeds then take action.

Possible actions
1) delete all entries near the top that are significantly older than the other entires near it (a few cents away)
2) disconnect and reconnect the quote stream for that symbol.

#1 is the only option for historical simulations, #2 is the best for realtime.

Thoughts?

I try to use quotes from different sources to create a real-time updating median price. You can weight the quotes so that more recent quotes have more weight in the median. Quotes falling a certain distance from the median are then rejected.

Runningbear · Jan 25, 2006

Try using trade volume, as price can trade at the same number for a while. Most stocks will do volume every few seconds so a 1 minute monitor on volume should make a good alert.

Runningbear

stephencrowley · Jan 25, 2006

Quote from nononsense:
I use a time interval monitoring between successive ticks approach. I monitor individual issues and I also monitor this interval for the complete set. No hard rules for this. Anything you set yourself is bound to be off one day.

Interesting. Do you monitor the interval between updates at the quote source level, or the price-point level? It seems that in a multicast scenario, it is possible that you (or your brokers feed from the exchange) could miss a quote which deletes a price level, then the market moves away, but that limit is still on the books and is thus never updated again for the rest of the day while all of the depth beyond the stale "best" quote is updated as usual.

I also screen a single tick against the earlier ones received in order to catch outliers. However, I assume that a way out tick was correct if it gets repeated. The danger here is that you react to a real jump with a delay. I always think (or hope) that such jumps occur at moments when the incoming tick rate is rather high.

I thought about this as well, delay is definately not good, but it depends on how sensitive the strategy is to this delay.

The same concept can be applied to detecting outliers in sync with other quote source instead of detecting outliers by comparing successive quotes.

Having feeds come in from different sources over separate ISP's would be nice.

The nasty thing is what to do in case you have a trade on. I only have an alarm go off and I take care of this manually. I still have not found much better.

I'm afraid I'm having to do the same, I'm just trying to limit the cases.. if the system can handle a signifacant amount of stuck-quote cases automatically then that is less time that I have to be bothered with it.

stephencrowley · Jan 25, 2006

1-minute is much too long for me.. and I monitor quotes and really don't do much with trade info. Quote data is much more informative and high frequency.. and trade data isn't even reported by all exchanges.

Quote from Runningbear:

Try using trade volume, as price can trade at the same number for a while. Most stocks will do volume every few seconds so a 1 minute monitor on volume should make a good alert.

Runningbear

stephencrowley · Jan 25, 2006

Do you use best-price data only, or dig further into the book? In my case I use a hypothetical-marker order of some fixed size, walk the book and calculate the average price paid and then use that as some sort of "realized quote", so if some stale quote is sitting between the best price, and the realized price it would be harder to detect.

Quote from squeeze:

I try to use quotes from different sources to create a real-time updating median price. You can weight the quotes so that more recent quotes have more weight in the median. Quotes falling a certain distance from the median are then rejected.

squeeze · Jan 25, 2006

Quote from stephencrowley:

Do you use best-price data only, or dig further into the book? In my case I use a hypothetical-marker order of some fixed size, walk the book and calculate the average price paid and then use that as some sort of "realized quote", so if some stale quote is sitting between the best price, and the realized price it would be harder to detect.

Best price only as this is what is currently executable. The method I described is particularly good for forex where you have access to quotes from a number of different banks. I am sure you could use depth information for othe markets.

rufus_4000 · Jan 25, 2006

Quote from stephencrowley:

So I'm analyzing data.. on one of my days it appears my ARCA feed has a stuck quote for AAPL because the best price never changes after about 8:45 for that day. However, my INET, TOTAL-VIEW and BRUT feeds are updating just fine.

Anyone have any approachs to automatically detecting stuck quotes such as this?

I thought about comparing some average quote time.. if the feed hasnt had a best price change in some interval beyond the average, and the price is some percent away from the other feeds then take action.

Possible actions
1) delete all entries near the top that are significantly older than the other entires near it (a few cents away)
2) disconnect and reconnect the quote stream for that symbol.

#1 is the only option for historical simulations, #2 is the best for realtime.

Thoughts?

I won't be much help on this problem, since I get my data stream directly from the exchanges. So if I have a quote stuck, then the exchange is experiencing problems (I know, I know, but I trade products traded on a single exchange). I do, however, measure quote latency like a hawk (not the Tibco product), if I start to experience delays in receiving time and the exchange time stamp, long latencies would cause my system to shutdown.

About a year ago, when I was trading options, I had someone in my group writing a viewer for about 10,000 option chains so that if one exchange (say PCX) is emitting bad quotes (say a $3 Ask, no bid, and CBOE is showing a $.50/.65 bid ask), then alert an monitoring application. But the firms I was with didn't do any automated trading, so a manual reconnect was acceptable.

stephencrowley · Jan 25, 2006

That's really interesting. What resolution were the timestamps from the exchange? And were clocks synchronized in any way? I receive millisecond timestamps for each quote and then I assign it my own timestamp when it reaches my system. I think i'll do some quick analysis and plot a histogram of the latencies.. fit some distribution function to it and then alerts when the latency deviates from the expected distriubtion.

Quote from rufus_4000:

I won't be much help on this problem, since I get my data stream directly from the exchanges. So if I have a quote stuck, then the exchange is experiencing problems (I know, I know, but I trade products traded on a single exchange). I do, however, measure quote latency like a hawk (not the Tibco product), if I start to experience delays in receiving time and the exchange time stamp, long latencies would cause my system to shutdown.

About a year ago, when I was trading options, I had someone in my group writing a viewer for about 10,000 option chains so that if one exchange (say PCX) is emitting bad quotes (say a $3 Ask, no bid, and CBOE is showing a $.50/.65 bid ask), then alert an monitoring application. But the firms I was with didn't do any automated trading, so a manual reconnect was acceptable.