IB - data generation process

kiwi_trader · Jul 21, 2006

I use Sierra Chart with IB TWS providing data and have got into a debate with another trader about how "good" IB data is. I contend that its pretty good although it will miss a price or two. He contends that its pretty bad. To quote:

"SC/IB's highs and lows are frequently short of one tick on ES. I know that eSignal's tick data is correct because I can see its streaming bid/ask/trade chart. This is a chart that shows 3 lines for bid, asd, and trade. This chart moves when any of bid/ask/trade changes.

I feel that IB's TWS is xxx interval based xxx. This might be the reason why SC/IB's data is off. "

"It hardly matters if you trade big time frame or slower markets. But if you trade world's fastest market(S&P e-mini) with small timeframe, it makes a big deferrence. Error occurs only when markets are moving so fast that software's interval update approach couldn't handle it in time."

"IB misses a high or low about every few minutes on ES due to big trading volumes."

and finally his belief about how you represent the data:

"If TWS is using interval update method (as I suspect) to get data from its data servers, it's just imposible to transmitt all trades. No matter how small its update interval is, you can not bundle all trades ocurred in that interval in time before passing it to TWS from IB data server. This is because data arrival time from exchange to IB data server could be after the batch has been sent out. Next batch excludes the previous interval so late arrivals are squeezed out. For what TWS does, I wouldn't want IB to delay its batching for late arrivals. So what we have from IB is the best they can do. It's a trading station, not a data service provider. "

We all agree that we'd prefer fast timely data to absolutely accurate data but there is this belief that ib data is "very" inaccurate that leads people to choose other sources and disparage IB.

So, guys, is it as bad as he claims and how do you handle batching of data out to the TWS data feed?

dcraig · Jul 21, 2006

Quote from kiwi_trader:

"If TWS is using interval update method (as I suspect) to get data from its data servers, it's just imposible to transmitt all trades. No matter how small its update interval is, you can not bundle all trades ocurred in that interval in time before passing it to TWS from IB data server. This is because data arrival time from exchange to IB data server could be after the batch has been sent out. Next batch excludes the previous interval so late arrivals are squeezed out. For what TWS does, I wouldn't want IB to delay its batching for late arrivals. So what we have from IB is the best they can do. It's a trading station, not a data service provider. "

This makes no sense at all. Ever heard of a queue or FIFO ?

kiwi_trader · Jul 21, 2006

LOL ... I didn't say it had to be right. I just wanted to quote the view so that IB could explain what they actually do.

One thing that was said a few days back that also surprised me was with regard to prices turning up in reverse order (ie a big buy order filled 887 888 and 889 yet the t&s shows 889 888 887). In that case it was caused because the asks are locked and then filled from high down to low.

So I won't assume what happens --- I keep getting new surprises in the marketplaces

Hoi · Jul 21, 2006

Quote from dcraig:

This makes no sense at all. Ever heard of a queue or FIFO ?

This make all the sense....Understand you have two types of data-providers:
a) the streaming-data (FIFO like eSignal)
b) the snapshot-data (snapshots of the current situation).

It's a trading station, not a data service provider...
This is 100% correct...for real-time trading (especially scalping) you need the snapshot-provider (like IB, and I believe X-trader does this as well).

For charting you need FIFO-providers...or you need to "live" with some missings of High/Lows per bar in your bar-chart. (it's up to you if that is important to your trading, if yes then add (not replace) a FIFO-provider to your trading-arsenal)

Hope it helps
Hoi

dcraig · Jul 21, 2006

Quote from Hoi:

This make all the sense....Understand you have two types of data-providers:
a) the streaming-data (FIFO like eSignal)
b) the snapshot-data (snapshots of the current situation).

It's a trading station, not a data service provider...
This is 100% correct...for real-time trading (especially scalping) you need the snapshot-provider (like IB, and I believe X-trader does this as well).

For charting you need FIFO-providers...or you need to "live" with some missings of High/Lows per bar in your bar-chart. (it's up to you if that is important to your trading, if yes then add (not replace) a FIFO-provider to your trading-arsenal)

Hope it helps
Hoi

Yes, but it depends on the the algorithm that is used to aggregate ticks from the exchange - not because a tick has fallen in the crack between two time windows. I have seen IB reps here deny that their feed is 'snapshot' (without really specifying how it works).

Hoi · Jul 21, 2006

Quote from dcraig:

Yes, but it depends on the the algorithm that is used to aggregate ticks from the exchange. I have seen IB reps here deny that their feed is 'snapshot' (without really specifying how it works).

It does not aggregate at all...it shows you the current best Bid and best ask and the current last trade, the moment that it is your turn to get an update....If there is aggregation at all then it is not done by IB (why would they?..that costs processing power), but is done by the exchange who distributes the quotes to IB (more and more exchanges do).

kiwi_trader · Jul 21, 2006

Quote from dcraig:

Yes, but it depends on the the algorithm that is used to aggregate ticks from the exchange - not because a tick has fallen in the crack between two time windows. I have seen IB reps here deny that their feed is 'snapshot' (without really specifying how it works).

Thats what I'd like too dcraig - to understand how it works

Quote from Hoi:

It does not aggregate at all...it shows you the current best Bid and best ask and the current last trade, the moment that it is your turn to get an update....If there is aggregation at all then it is not done by IB (why would they?..that costs processing power), but is done by the exchange who distributes the quotes to IB (more and more exchanges do).

There is (I believe, but do not know) aggregation because IB limits the number of packets it sends out (and hence gets smaller transaction (tick) counts).

I'd like to understand it better please IB

Hoi · Jul 21, 2006

To add to this discussion:

Aggregation is done by Streaming-providers (not the snapshot ones), to reduce the bandwidth in order to transport as many as they can (this in contrast with the snapshot-models: they have tuned the bandwidth by the snapshot frequency).

Hoi · Jul 21, 2006

Quote from kiwi_trader:

There is (I believe, but do not know) aggregation because IB limits the number of packets it sends out (and hence gets smaller transaction (tick) counts).

I'd like to understand it better please IB

No then you still do not understand the concept of a snapshot...it is a photograph of the current situation...I say it another time: photograph !

Your TWS-sheet-page, is stored on the IB-server (yes in core), and all quotes are streaming in realtime from the exchanges to those in-core pages..but are not transported in realtime..they are updated in realtime only in Server-core...then when it is YOUR turn to take a snapshot (once every 1/3 second) it will take a snapshot of all changed fields (compared to the previous snapshot 1/3 second ago)..and ONLY that will be transported to your PC...

Hope it clears things up ?

dcraig · Jul 21, 2006

Quote from kiwi_trader:

Thats what I'd like too dcraig - to understand how it works

There is (I believe, but do not know) aggregation because IB limits the number of packets it sends out (and hence gets smaller transaction (tick) counts).

I'd like to understand it better please IB

We may never find out if IB considers this to be confidential information ie proprietary intellectual property. I hope this is not the case.

A bit off topic, but has anybody any experience with opentick http://www.opentick.com feed ? I realise that it's still early days for them, but I recently noticed that their client API and protocol specification is being open sourced. IMHO a real plus.