IB - data generation process

[Hoi]

Quote from Trader_Herry:

Sorry for the confusion.
I realise the delay is small, but that's the delay I'm talking about. Am I right about it?

{oops my post and ddanbars crossed ...and I see we both wrote about the Flim }

Well, in my opinion a delay, is something you get, but at a (much) later point-in-time (so already history, because on the moment you get your eyes on it, the real and current situation is already changed)....it is a film that is shown to you seconds after the event..you get all the frames of the film, but because it takes bandwidth to transport this film to you, it is delayed.

In the snapshot-situation, you do not get the frames of the film at all..you get only frame 3, frame 7, frame 15, frame 30, as a photograph presented to you, and because it does not take bandwidth to transport it to you, you will get each photo in real-time without delay. This photo shows you the Last-trade, and the current best-Bid and best-ask...and THAT is what an active-trader (scalper) wants to see..the most current and actual best-bid and best-ask.
But for charting and analyses you need the Film-version as well, for that you need all frames of the film.

 

[Trader_Herry]

Quote from ddunbar:

Again, that's not a delay in any sense. It's a snapshot. Think of it in terms of a digital movie camera. It takes pictures 30 frames every second. Does your eye notice the missing frames? Movements seem fluid enough don't they?

Well, your connection (ping time) to any broker will be in the 100ms range round trip. Unless you're using a fractional T1 or higher direct to the brokerage. This is where a delay comes in. There is no trading strategy except arbitage that would be affected in the least by a 3/10's of a second snapshot. Markets don't move that fast anyway. At most, 1 tick maybe even 2. But usually, no tick.



Yes and no. The inaccuracy of high and low
is usually a few ticks. However, IB transmits the high and low from the exchange seperately. For instance, a week or two ago, the S&P emini futures high was off by 3 ticks - 3 ticks to low on their 30 min chart. Yet, in TWS, the high was reported accurately. I double checked with CME. So simple deduction was that the true high occured during the 30 min time frame whose high was 3 ticks off.

It's not going to get much worse than that - 3 to 4 ticks off. And it's not as if it happens everyday. Will this affect trading? Doubtful, unless one is looking to scalp day's range extremes. Or if they're scalping off 1 min or tick charts towards the high and/or low of the day.

Next question is, are intraday high and lows, that are not new daily high and/or lows off? Haven't seen it. They seem to only happen at day's high and lows.



They perform well because of their chosen method of dissemination. The snap shot method, as mentioned, doesn't "bog" down with information like streaming every single message event.

Thanks, ddunbar.
You are always very helpful.

I'm no longer worried about the refresh rate as you have explained this to me previously. I just point it out anyway.

As to high/low, I don't know. But I think intraday highs/lows should affect as IB just sends the last (close) price.
If the price goes higher/lower and then retrace and a snapshot is made, it just reflects the close without the high/low. So if it happens to be the final high/low of the intraday bar, the high/low will vary.

As to periods of news break, it is very good if it can still keep 0.25-0.3 second refresh rates. To be picky, I may expect it refresh a bit faster at this period.

 

[alanm]

T_H: You seem to be poking at a lot of things that you think might be a problem for someone, some day, but shouldn't you be thinking about the things that might actually be a problem for you? Choosing a broker is not like getting married without the possibility for divorce. Most active traders/investors have multiple brokers (for backup, different features, etc.), and change them a number of times throughout their lifetime.

What exactly is your trading style, or do you have one yet? That might help people better understand where you're coming from.

 

[Trader_Herry]

Quote from alanm:

T_H: You seem to be poking at a lot of things that you think might be a problem for someone, some day, but shouldn't you be thinking about the things that might actually be a problem for you? Choosing a broker is not like getting married without the possibility for divorce. Most active traders/investors have multiple brokers (for backup, different features, etc.), and change them a number of times throughout their lifetime.

What exactly is your trading style, or do you have one yet? That might help people better understand where you're coming from.

Thanks for your advice - a good piece of advice actually

But I'm not here only to choose my own broker. I help others & my friends too.

The more I know about the broker, the better I can assist when other people ask for advice. You know, different traders have different situations & different trading styles. That's why I ask different kinds of questions.

By the way, sometimes I ask questions on soneone's behalf.

 

[kostia00]

Quote from kiwi_trader:
...
"IB misses a high or low about every few minutes on ES due to big trading volumes."
...

Starting with TWS version 866, the charts are receiving the 5-second bar data from the IB servers, and starting yesterday the calculation of the 5-second bars for US SMART stocks and GLOBEX futures have been changed to include every tick, so the TWS charts should never miss high or low of the day and the bar volume displayed should be 100% accurate. We are in the process of rolling this out to all exchanges and products.

We will be upgrading the API in the coming weeks and giving third-party applications access to the same real-time and accurate 5-second bar feed.

 

[dcraig]

Quote from kostia00:

Starting with TWS version 866, the charts are receiving the 5-second bar data from the IB servers, and starting yesterday the calculation of the 5-second bars for US SMART stocks and GLOBEX futures have been changed to include every tick, so the TWS charts should never miss high or low of the day and the bar volume displayed should be 100% accurate. We are in the process of rolling this out to all exchanges and products.

We will be upgrading the API in the coming weeks and giving third-party applications access to the same real-time and accurate 5-second bar feed.

What does this mean ? Are you streaming OHLCV 5 second bars ? One every 5 seconds ?

 

[kostia00]

Quote from dcraig:

What does this mean ? Are you streaming OHLCV 5 second bars ? One every 5 seconds ?

That's correct, IB servers are streaming 5-second bars, one every 5 seconds, to TWS. TWS 866 and later uses these bars to build charts. Besides OHLCV we are sending # of ticks/trades in the 5-second bar and sum(price*size/time) so that VWAP (for trades)/TWAP(for bid/ask/midpoint) can be calculated.

 

[dcraig]

Quote from kostia00:

That's correct, IB servers are streaming 5-second bars, one every 5 seconds, to TWS. TWS 866 and later uses these bars to build charts. Besides OHLCV we are sending # of ticks/trades in the 5-second bar and sum(price*size/time) so that VWAP (for trades)/TWAP(for bid/ask/midpoint) can be calculated.

5 Seconds !!! Not exactly real time. What do you expect this to be used for ? Who wants data with 5 second delays ? I can't see the point other than to catch true high/lows. Please enlighten me if I'm missing something but this sounds like a terrible idea.

 

[jimrockford]

Quote from dcraig:

5 Seconds !!! Not exactly real time. What do you expect this to be used for ? Who wants data with 5 second delays ? I can't see the point other than to catch true high/lows. Please enlighten me if I'm missing something but this sounds like a terrible idea.

Customers requested this type of enhancement. IB will be providing a combination of real-time and delayed data. The real-time data serves a different purpose than the delayed data.

The purpose of the 5 second bars, which are delayed instead of real-time, is to ensure that charts can be constructed without missing any ticks. This is needed because IB's real-time data are sampled a few times per second, which sometimes misses highest or lowest values occuring in between snapshots.

The purpose of IB's real-time data snapshots, occuring a few times per second, which will continue to be real-time instead of delayed, will be to indicate the market's present level, without any delay (unless you consider a small fraction of a second a delay).

You are missing something. It seems obvious that you haven't read previous postings and threads on ET and on IB's own forums, which led to these changes. You are confusing the purpose of the delayed data with the purpose of the real-time data.

I'm sure that if you researched the history, all of this would make sense to you.

I think that an even better design would be to include, with each real-time snapshot occuring a few times per second, the highest and lowest trade prices, bids, and aks, which have occurred since the time of the previous snapshot, as well as the most recent trade price, bid, and ask, as of the time of the snapshot. This way, charts would reflect highest and lowest values immediately, instead of after a 5 second delay. I don't understand why it wasn't done this way. I presume there was some reason for not doing it this way. I think that the change IB are making, even if it is not the optimal solution, is a welcome step in the right direction

 

[kostia00]

Quote from dcraig:

5 Seconds !!! Not exactly real time. What do you expect this to be used for ? Who wants data with 5 second delays ? I can't see the point other than to catch true high/lows. Please enlighten me if I'm missing something but this sounds like a terrible idea.

The goal is to facilitate accurate bar charting (any bar size that's a multiple of 5 seconds). Besides catching true high/low and allowing one to compute VWAP/TWAP, it can be used to chart GLOBEX futures volume based on outright trades only (excluding the spread): the volume reported by IB in those 5-second bars is the sum of the sizes of all trades reported by GLOBEX. Currently QT and others I believe are calculating the volume for a chart by taking a difference in overall volume for the day captured at 5-second (or other) interval, and that figure includes spread trades during future roll days.