using IB, should I consider a different market data feed?

[vicirek]

well, you ask a lot of questions all of a sudden for someone who made bold and false claims a few post above (and has badly accused and attacked me in the past)

To answer a few of your questions, IB samples quote and trade data (note that I use the term trade "data", it does not mean they delay your executions, but simply means that all you see about real-time data is in effect not entirely real-time because it is sampled. That included bids, offers, trades, volume, anything you can imagine to see changing in "near" real-time is sampled. IB does aggregate, not the exchange. That you get data in different callbacks only means that the gateway that everyone runs on their local machines is implementing an API that segregates the callbacks. But still fact remains that any and all incoming data by IB is sampled.

My question is where are the specifications or information available? How do you know that IB is doing the aggregation not their data suppliers? How much data conditioning occurs before it gets to IB?

Since IB data is for the purposes of updating quote board in TWS UI both bid and ask would be treated differently than last trades as they are in market quotation systems because they serve different purpose (one is pre-trade the other post-trade but TWS user see both of them with small delay). The data sent to client app through API is just an attempt to emulate tick data from market quotation displayed on the screen with bid ask as a main feature there plus sizes and last. API was just afterthought hence this unusual "tick" data structure and separation of price and size in separate callbacks with no time stamps. The sole purpose of API is to help automate what is available in TWS. Therefore it is not only pure sampling issue but what is prioritized when providing data for the UI updates so it will not freeze.

API documentation says that data is sent when it changes but this is just in regards to what is happening on the TWS end but not what is the relationship between what IB servers receive and then send to TWS/client application. There is very little verifiable information what is going on beyond that.

 

[volpunter]

That you need to take up with IB's tech department, I do not have an interest in knowing how IB samples its data beyond the fact that it does as it does not affect my particular approach to systematic trading [I do use different data providers and an entirely different deployment setup when a full data feed is required]. I am not here to win battles but to inform and your statement...

IB returns several data points including last bid ask etc. Bid and Ask should update when they change. Only Last or so called tick would be sampled. I am not sure if IB documentation is clear separating bid ask updates frequency from tick updates.

...is simply inaccurate. Bids and Asks are sampled equally as is last trades.

My question is where are the specifications or information available? How do you know that IB is doing the aggregation not their data suppliers? How much data conditioning occurs before it gets to IB?

Since IB data is for the purposes of updating quote board in TWS UI both bid and ask would be treated differently than last trades as they are in market quotation systems because they serve different purpose (one is pre-trade the other post-trade but TWS user see both of them with small delay). The data sent to client app through API is just an attempt to emulate tick data from market quotation displayed on the screen with bid ask as a main feature there plus sizes and last. API was just afterthought hence this unusual "tick" data structure and separation of price and size in separate callbacks with no time stamps. The sole purpose of API is to help automate what is available in TWS. Therefore it is not only pure sampling issue but what is prioritized when providing data for the UI updates so it will not freeze.

API documentation says that data is sent when it changes but this is just in regards to what is happening on the TWS end but not what is the relationship between what IB servers receive and then send to TWS/client application. There is very little verifiable information what is going on beyond that.

 

[moonmist]

Globex FIX FAST feed provides tick by tick real-time data containing relevant info for every trade. Associated with each tick or each trade is a tag called "Aggressor Side". The most accurate way to calculate bid/ask volume is to examine this field. High-quality tick by tick data feeds like Sierra Chart real-time feed will process the CME FAST FIX data feed directly. Hence, they are capable of generating highly accurate bid/ask volume. For some unknown reasons, some data feeds may filter out the "Aggressor Side" tag from CME. Then, the bid/ask volume will be determined by other less reliable methods, eg. an algorithm like the following:

If the trade occurs at the bid price or lower it is a Bid trade, and if the trade occurs at the ask price or higher it is an Ask trade.

IB's data feed are snapshots. It would be meaningless to talk about "Aggressor Side" for each trade. Hence, determining accurate bid/ask volume may be an impossible task.

 

[volpunter]

sure, if you want to segregate bid and ask volume, but total volume coming via IB is still 100% accurate for the time stamp of the snapshot.

Globex FIX FAST feed provides tick by tick real-time data containing relevant info for every trade. Associated with each tick or each trade is a tag called "Aggressor Side". The most accurate way to calculate bid/ask volume is to examine this field. High-quality tick by tick data feeds like Sierra Chart real-time feed will process the CME FAST FIX data feed directly. Hence, they are capable of generating highly accurate bid/ask volume. For some unknown reasons, some data feeds may filter out the "Aggressor Side" tag from CME. In these cases, the bid/ask volume will be determined by other less reliable methods, eg. an algorithm like the following:

If the trade occurs at the bid price or lower it is a Bid trade, and if the trade occurs at the ask price or higher it is an Ask trade.

IB's data feed are snapshots. It would be meaningless to talk about "Aggressor Side" for each trade. Hence, determining accurate bid/ask volume may be an impossible task.

 

[moonmist]

sure, if you want to segregate bid and ask volume, but total volume coming via IB is still 100% accurate for the time stamp of the snapshot.

I am not sure about that. Need to do some testing first. IB's data feed can be full of surprise.

But according to Sierra Chart documentation, customers need to enable Record True Real-Time Data in Intraday Charts

http://www.sierrachart.com/index.php?l=doc/IB.php#TrueRealTimeData

to get accurate volume value for each bar in the chart. This second data feed from Interactive Brokers updates every 5 seconds.

Have any fellow members done any testing about the volume data from IB's two data feeds ?

 

[rwk]

This was discussed on the TWSAPI group (https://groups.yahoo.com/neo/groups/TWSAPI/conversations/messages). The conclusion seemed to be that the reqRealTimeBarsEx() method preserves true high and low for the time period. Nothing is lost in the sampling. So someone building time bars should probably use that function.

Regarding the accuracy of volume data, the most commonly used data request was the reqMktDataEx() method, which returns streaming data in the tickPrice() and tickSize() events. The user would then estimate volume by summing size events. The results were pretty unsatisfactory. The tick event returned bid, ask, and last, but matching tick and size was also never entirely straightforward.

More recently IB introduced the RTVolume ticktype which returns trade price, size, time, cumulative volume, VWAP, and single-trade flag in one event.

It's my understanding that CME conflates some trade data before they release it. I am not clear whether that a more complete data feed is available or who can get it.

 

[volpunter]

Sierra Chart, a third rated platform selling connectivity to a full tick data feed? Wow, funny times we living in...there must be guys who look at chart updates every few millisecond, lol.

I am not sure about that. Need to do some testing first. IB's data feed can be full of surprise.

But according to Sierra Chart documentation, customers need to enable Record True Real-Time Data in Intraday Charts

http://www.sierrachart.com/index.php?l=doc/IB.php#TrueRealTimeData

to get accurate volume value for each bar in the chart. This second data feed from Interactive Brokers updates every 5 seconds.

Have any fellow members done any testing about the volume data from IB's two data feeds ?

 

[moonmist]

Sierra Chart, a third rated platform selling connectivity to a full tick data feed? Wow, funny times we living in...there must be guys who look at chart updates every few millisecond, lol.

Yes. They have their CME real-time data server at Cermak:

http://www.datacenterknowledge.com/...ters/worlds-largest-data-center-350-e-cermak/

https://www.sierrachart.com/SupportBoard.php?ThreadID=7489#P32745

 

[fullautotrading]

I've been using a rather lame brokerage, and am in the process of switching over to IB in the near future. It seems like they have a pretty decent array of market data feeds. The cost also seems quite reasonable, since they're making money on the commissions.

I'm curious about two points:

a) Why would someone get feeds covered by IB from a different provider?
b) Even if there's a good reason, not for "real" HFT but for something where seconds matter, it seems like it would complicate the placement of servers. If you're really only working with the IB api, then it seems like you'd want to park your servers close to wherever their's are and not have to hedge the distance from multiple providers.

Thanks so much.

imho, that should be the last of your worries. IB feed is good enough for automatic trading at any "frequency" which can make any sense for automated real trading (via API and internet connection).

First, focus on methodology and implementation issues. If, later on, should any problem arise with data, you con focus on it later (should you have some time left for that). But it's likely you will never need to worry about any data feed "improvement".

(It's obvious that third party vendors or affiliated have to invent all sort of excuses and fantasy reasons to justify their marketing efforts: it's actually their right and duty.)

 

[alphagad]

Sierra Chart, a third rated platform selling connectivity to a full tick data feed? Wow, funny times we living in...there must be guys who look at chart updates every few millisecond, lol.

Sierra Chart is a third-rated platform? Rated by you, you mean? Wow, that's a first. I've tried a lot of platforms and can confidently say that Sierra is the best, bar none. Maybe you didn't spend enough time with it?