IB; Biggest Possible Improvement?

[kiwi_trader]

Quote from alanm:
I don't care if they manage to give you more accurate tick data, as long as it does not result in additional bandwidth/processor usage (on either the server or client) for those of us that don't need it.

Yes, I agree. That wasn't included because I consider it a given.

Both of the approaches to including the missing data avoid adding significant numbers of extra packets or extra data in the same packets. The timeliness of data is very important but the overheads of TCP/IP protocols dwarf the extra size involved in adding one more number to the packet.

 

[alanm]

I realized later that I should have used the word "significant", too, but I thought that was a given also

 

[Hoi]

Quote from jimrockford:

My understanding of your suggestion is that IB preserve the basic principle of updating only snapshots taken at time intervals, and of throwing away those ticks falling in between snapshots; but that each update give not only the last price occurring within the snapshot's interval, but also the lowest and highest prices falling within that interval. This would not have the costs and disadvantages and delays of a tick-by-tick feed, and would preserve the advantages of IB's existing snapshot approach, but would also have the new advantage of accurately including all highs and lows on every time scale.

That's well explained, and gets my vote !

By the way: it is not only the third-party software who needs this, it is needed for TWS's own charting as well....
If IB takes their own charting-tools seriously (and I'm sure they do), then it is definitely needed. The real-time-charts will only show the correct highs/lows per bar when you do a backfill-request (at start up), but there after the charts are filled with the real-time snapshot feed, and will miss quite a lot of highs/lows..

When programmed well (at the IB-server-side where the snapshots are taken), it does not need to give much overhead.

 

[fraserhub]

Having changed to IB in the last 6 months I really appreciate the stability as compared to some other vendors with every tick data feeds, where I often experienced delays and timeouts etc.

So your suggestion to get the additional info and mantain the stability is great and gets my vote.

Thanks for starting this thread and I sure hope IB will act on this.

Regards
Fraser

 

[AussiesRule]

Quote from jimrockford:

I agree that it would be a good idea to improve the data the way you suggest.

My understanding of your suggestion is that IB preserve the basic principle of updating only snapshots taken at time intervals, and of throwing away those ticks falling in between snapshots; but that each update give not only the last price occurring within the snapshot's interval, but also the lowest and highest prices falling within that interval. This would not have the costs and disadvantages and delays of a tick-by-tick feed, and would preserve the advantages of IB's existing snapshot approach, but would also have the new advantage of accurately including all highs and lows on every time scale.

My vote for this version kwi. IB, what do you think?

 

[kiwi_trader]

Is there any chance that we can get an opinion from one of the IB ( Interactive Brokers ) people who post on this board?

 

[kiwi_trader]

Someone challenged me to prove that the data was doubtful. I was curious about how bad it might be so instead of using "other data" I used IB's own backfill.

According to ane earlier post here when you get IB backfill you get every price they know of but when you get real time they skip all but the last price in each 200ms period. So what happened?

The upper price series is the real time ib series (30second bars)
The middle is collected by backfill (should include all prices)
The bottom shows red wherever the ohl or c varies


See the pictures:

 

[dcraig]

Quote from kiwi_trader:

Someone challenged me to prove that the data was doubtful. I was curious about how bad it might be so instead of using "other data" I used IB's own backfill.

According to ane earlier post here when you get IB backfill you get every price they know of but when you get real time they skip all but the last price in each 200ms period. So what happened?

The upper price series is the real time ib series (30second bars)
The middle is collected by backfill (should include all prices)
The bottom shows red wherever the ohl or c varies


See the pictures:

Remember the TWS data stream is not time stamped, so bar boundaries are determined from the PC clock. If the latter is off, then your bars boundaries will be off too. If the PC clock is not synced frequently via NTP, it will certainly drift - by quite a lot. Add in internet propagation delay and local PC processing time. You will never get short period bars to match exactly.

I've been through this exercise some time ago, and I found the match up was quite good over the few samples I tested.

By frequent clock synchronization I mean something like every 30 mins or so.

So what I am saying is that some of the differences are not due to missing ticks.

 

[def]

Quote from kiwi_trader:

Is there any chance that we can get an opinion from one of the IB ( Interactive Brokers ) people who post on this board?

Since you asked... This isn't my area of expertise so I'm not going to touch this but will note that a few people who might be willing to share their thoughts are on vacation at the moment.

 

[kiwi_trader]

Quote from dcraig:

Remember the TWS data stream is not time stamped, so bar boundaries are determined from the PC clock. If the latter is off, then your bars boundaries will be off too. If the PC clock is not synced frequently via NTP, it will certainly drift - by quite a lot. Add in internet propagation delay and local PC processing time. You will never get short period bars to match exactly.

I've been through this exercise some time ago, and I found the match up was quite good over the few samples I tested.

By frequent clock synchronization I mean something like every 30 mins or so.

So what I am saying is that some of the differences are not due to missing ticks.

Good point. Onto that - I had a drift problem a few months back so I run Dimension 4 and it synchs every 15 minutes. The correction during that period never exceeded 15ms.

I'll run one when the afternoon session starts with 1 minute time corrections off the HK govt timeserver (I normally use a close Telstra timer thats slaved off their internal time server).


Edit: And thanks to def. Would appreciate their input when they get back. IM(H)O it would be more important for IB to improve the data quality by one of the measures suggested here than to, say, make the next couple of improvements to IB charts with "weak" data.