Detecting stuck quotes automatically

[nononsense]

Quote from stephencrowley:

Interesting. Do you monitor the interval between updates at the quote source level, or the price-point level? It seems that in a multicast scenario, it is possible that you (or your brokers feed from the exchange) could miss a quote which deletes a price level, then the market moves away, but that limit is still on the books and is thus never updated again for the rest of the day while all of the depth beyond the stale "best" quote is updated as usual.
...

Steph,
To be more precise, for each incoming data item, trade, bid or ask I record the time of my local clock (millisecond resolution). It is on this basis that I monitor the feeds being alive. To detect a stuck symbol turns out to be difficult due to the greater variability between issues. Making the trigger interval long enough makes it easier but increases your reaction time.

Lumping many symbols together in order to ascertain the liveness of the data source and/or communication channel is more straightforward.

 

[rufus_4000]

Quote from stephencrowley:

That's really interesting. What resolution were the timestamps from the exchange? And were clocks synchronized in any way? I receive millisecond timestamps for each quote and then I assign it my own timestamp when it reaches my system. I think i'll do some quick analysis and plot a histogram of the latencies.. fit some distribution function to it and then alerts when the latency deviates from the expected distriubtion.

The exchange time stamps are at millisecond level. Like your system, I have my own pub/sub bus, so I aggregate the exchange timestamp, my own feedhandler timestamp, the admin module timestamp, and feed from a network latency monitor. Then the system compares the differences with a static model, to see if it is "far" out, and alert if it is. Since I have my lines to the exchanges, dedicated feedhandler (hw and sw), my latency is in a very tight range 98-99% of the time.

Of course, I do a similar level of measurement on my orders, just to be safe.

 

[nononsense]

Quote from rufus_4000:

The exchange time stamps are at millisecond level. Like your system, I have my own pub/sub bus, so I aggregate the exchange timestamp, my own feedhandler timestamp, the admin module timestamp, and feed from a network latency monitor. Then the system compares the differences with a static model, to see if it is "far" out, and alert if it is. Since I have my lines to the exchanges, dedicated feedhandler (hw and sw), my latency is in a very tight range 98-99% of the time.

Of course, I do a similar level of measurement on my orders, just to be safe.

Yeah, but that only enables you to track excessive delay in transmission. It doesn't enable you to detect a stuck quote. You simply won't be getting any timestamps, neither from the exchange nor local arrival. The problem really is to detect such stuck quote as early as possible. IMHO, you can do this only by monitoring successive arrival times.

 

[stephencrowley]

I tend to agree.. there are a few different types of failures.. 2 major ones

1) me to broker
2) exchange to broker

If the connection to broker is broken, then you wont find out until the socket times out.. which can is usually a very long time (minutes) on most systems. In this case, time between successive quotes would be perfect.. setup some watchdog timer and reset it every time a quote comes in. The watchdog would need to take into account seasonal patterns of activity so it wouldnt generate false alarms in pre/post market hours.

If only one quote source from the exchange to the broker is down then you only need to compare one quote source to the others. This still might need to take into account seasonalities though, but would probably be less of an issue.

Quote from nononsense:

Yeah, but that only enables you to track excessive delay in transmission. It doesn't enable you to detect a stuck quote. You simply won't be getting any timestamps, neither from the exchange nor local arrival. The problem really is to detect such stuck quote as early as possible. IMHO, you can do this only by monitoring successive arrival times.

 

[stephencrowley]

Replying to myself here.. but I just tested my latency and it looks like exchange -> my colo in austin is peaked at 45ms with an average of 60ms. The raw ping time is 40ms.. if i co-locate that should change to peak 5ms and average 15.. seems reasonable to sound alerts if quotes stop coming for more than a second.

 

[joe_blo]

Quote from nononsense:

Yeah, but that only enables you to track excessive delay in transmission. It doesn't enable you to detect a stuck quote. You simply won't be getting any timestamps, neither from the exchange nor local arrival. The problem really is to detect such stuck quote as early as possible. IMHO, you can do this only by monitoring successive arrival times.

Monitoring arrival latency is ok for detecting lost feeds, but doesn't help detect stuck quotes which are a result of the occasional dropped multicast packet. The quote update never comes, and that 8:45 arca order just sits there, even though the data feed continues to flow.

The solution in a live situation is to try to execute it! After all, crossed market, free money, right? And if the suspect quote remains even after your IOC is sent back, you can clear it from your local book.

 

[stephencrowley]

Right.. I was suggesting to monitor the last-update-time of each price level in the book. As you get further away from the best the update frequency should decrease unless some joker is toying around with quotes on completely unrealistic prices.. map this historically and then come up with some trigger that way.

Quote from joe_blo:

Monitoring arrival latency is ok for detecting lost feeds, but doesn't help detect stuck quotes which are a result of the occasional dropped multicast packet. The quote update never comes, and that 8:45 arca order just sits there, even though the data feed continues to flow.

The solution in a live situation is to try to execute it! After all, crossed market, free money, right? And if the suspect quote remains even after your IOC is sent back, you can clear it from your local book.

 

[stephencrowley]

I've created a pretty robust filter based on the moving standard deviation of the spread.

If the spread suddenly jumps some # of standard deviations away from its moving average then I look at both changes in the bid from the last bid, and the ask and the last ask.

If either jumps by too much then it is simply replaced with its last value.

This looks like its working very well for me so far..

 

[nitro]

The only person that knows what he is talking about, although stuck quotes can happen for more reasons than dropped MC packets.

nitro

Quote from joe_blo:

Monitoring arrival latency is ok for detecting lost feeds, but doesn't help detect stuck quotes which are a result of the occasional dropped multicast packet. The quote update never comes, and that 8:45 arca order just sits there, even though the data feed continues to flow.

The solution in a live situation is to try to execute it! After all, crossed market, free money, right? And if the suspect quote remains even after your IOC is sent back, you can clear it from your local book.

 

[stephencrowley]

lol. That is definately a good idea. Doesn't help for historical quotes though.

Quote from nitro:

The only person that knows what he is talking about.

nitro