IB Chronic Data Feed Issue

Quote from rcj:

Thnx for the update, Dst.

i havent seen any problems with YM data in past several days.

If IB hasnt finished with the fix, then im wondering why the
YM data has stayed "clean" . I guess im still a bit(Pun) suspicious
that the YM problem originated some where at or between the matching engine hardware or comm link to IB.
Probably not, but ...
More...

Be aware that this problem is not in YM alone. It’s in all MD-tickers when market gets volatile (see my previous post about DAX, ES, NQ). Further the MD-protocol itself is vulnerable for transport-glitches (Internet disturbance).

Actually there are two issues here:
a) the construction of the MarketDepth-window on the back-end server in the Account-memory-block
b) The as fast as possible, and as less overhead as possible, transport to the MD-window in the TWS-client.

Problems like (a) can be solved quite quickly, to which “dst” pointed in his posts.
Problems like (b) are design decisions and when needed to re-design can be costly and take time to implement.

As I wrote before: the MD-transport protocol is designed to be fast without overhead (it doesn’t have sequence-numbering nor checksums, etc). It uses the same transport-mechanism as for the Quote-feeds, but there lies its flaw: for the Quote-feed it doesn’t matter if one price-tick-update isn’t received correctly (the next update will correct it), but it is devastating for the MD-window to miss one row-update (the whole MD-window will become corrupt and will stay corrupt until it is re-initialized completely). In other words: if (b) isn’t taken care of, the MD will keep giving problems to clients having a bad internet-connection from time to time.

Hope it helps.
 
i am glad people are here 'whining' - if it weren't for this how

would we all establish the good from bad brokerages - i thought

that was the whole point. it may have been mentioned time

and time again - but there will always be people who are new

to the industry/this website and who will find this type of

feedback very useful indeed without having to trawl through

historical pages to find what is useful and what is not - so please

'whine' away if i get bored of reading it i will skip that post and

move on.
 
Hoi, I may be misinterpreting what you are describing in your point (B) but I just want to clarify. As far as the transport from our backend server to the TWS client, we use the TCP protocol. TCP uses checksums, and guarantees reliable sequenced packet delivery. The TWS cannot miss an update or get a corrupt price packet. So it is not at all vulnerable at to internet glitches.



Quote from Hoi:

Be aware that this problem is not in YM alone. It’s in all MD-tickers when market gets volatile (see my previous post about DAX, ES, NQ). Further the MD-protocol itself is vulnerable for transport-glitches (Internet disturbance).

Actually there are two issues here:
a) the construction of the MarketDepth-window on the back-end server in the Account-memory-block
b) The as fast as possible, and as less overhead as possible, transport to the MD-window in the TWS-client.

Problems like (a) can be solved quite quickly, to which “dst” pointed in his posts.
Problems like (b) are design decisions and when needed to re-design can be costly and take time to implement.

As I wrote before: the MD-transport protocol is designed to be fast without overhead (it doesn’t have sequence-numbering nor checksums, etc). It uses the same transport-mechanism as for the Quote-feeds, but there lies its flaw: for the Quote-feed it doesn’t matter if one price-tick-update isn’t received correctly (the next update will correct it), but it is devastating for the MD-window to miss one row-update (the whole MD-window will become corrupt and will stay corrupt until it is re-initialized completely). In other words: if (b) isn’t taken care of, the MD will keep giving problems to clients having a bad internet-connection from time to time.

Hope it helps.
More...
 
Samaritan,

You are correct about TCP in as far as you go. But if you have repeated losses because of corrupted packets two things will happen:
- you will definitely get delays in delivery
- you may time out because at a certain point most apps give up

So if Hoi comes back in the affirmative you need to look at what you do when the lower layers of your Comms stack are not responding in a timely manner or if their counters run over. You might check it anyway.
 
My only point was to clarify that the the market depth window will not go out of sync because of missed or out of sequence packets. Yes, if your connection is bad you will have delayed market data. If the delays are bad enough or the packets are being delivered corrupt for long periods then TCP itself will disconnect. But TCP itself will not deliver a corrupt or out of sequence packet to the application.


Quote from kiwi_trader:

Samaritan,

You are correct about TCP in as far as you go. But if you have repeated losses because of corrupted packets two things will happen:
- you will definitely get delays in delivery
- you may time out because at a certain point most apps give up

So if Hoi comes back in the affirmative you need to look at what you do when the lower layers of your Comms stack are not responding in a timely manner or if their counters run over. You might check it anyway.
More...
 
Maybe Hoi is talking about UDP protocal ??? It doesnt have the
overhead compared to TCP - faster with less than half the overhd
in bytes of TCP. Just server to client(TWS) - nothing else.
 
Quote from samaritan:

My only point was to clarify that the the market depth window will not go out of sync because of missed or out of sequence packets. Yes, if your connection is bad you will have delayed market data. If the delays are bad enough or the packets are being delivered corrupt for long periods then TCP itself will disconnect. But TCP itself will not deliver a corrupt or out of sequence packet to the application.
More...

Well, don’t bet your life on TCP/IP’s reliability. Sure it’s much better than UDP (which was used by TWS before 2002, if I remember well), but this TCP communication-layer doesn’t ensure by itself that corrupted data is not received by the application-layer.

This because it uses only a 16 bit CRC checksum, which is quite unreliable as it cannot detect all possible errors. A simple swap of two 16 bit values gives the very same checksum, on corrupted data.
See for instance these two sentences, both give the same checksum (43087):
“hoi is my name”
“hoisi my name”

Further I have read an empirical study about the question on how many times such an 16 bit error can occur on the Internet, without detection by the checksum. They found 350 cases in 30 million messages, which sounds a small chance. But this is actually one per 85,000. Still small? Not really, as with 3 Market-depth windows with 3 updates per second, this will give you 10,800 messages per hour. So hypothetically once every 8 hours a TWS-client can encounter a Market-depth corruption, not detected by the TCP-layer.

Luckily the Trading connection (for Orders, and Executions), uses the FIX-protocol-layer on top of the TCP-layer. The FIX protocol has its own sequence-checks and uses a much stronger 32-bit checksum. Actually TWS uses a ZIP-compression for the FIX-messages as well (this gives 3 checksums in total: 16 bit TCP, 32-bit ZIP and 32-bit FIX).

I don’t ask IB to implement the same reliability (and overhead) for MarketDepth as for the Trading-information. But something should be done in the application-layer to ensure reliability and recovery of corrupted MarketDepth windows. I suggested in an earlier post to implement a MD-monitor in TWS and on the Server, to check if the same results occur as empirical study above. And if so it would deserve a redesign.

Hope it helps
Hoi
 
Quote from samaritan:

My only point was to clarify that the the market depth window will not go out of sync because of missed or out of sequence packets. Yes, if your connection is bad you will have delayed market data. If the delays are bad enough or the packets are being delivered corrupt for long periods then TCP itself will disconnect. But TCP itself will not deliver a corrupt or out of sequence packet to the application.
More...

Kiwi's point, is that in the event of a TCP connection reset, is the mechanism for resynchronizing the TWS notion of the state of the order book with the server's notion of the state of the order book reliable ?
 
Quote from dcraig:

Kiwi's point, is that in the event of a TCP connection reset, is the mechanism for resynchronizing the TWS notion of the state of the order book with the server's notion of the state of the order book reliable ?
More...

In the event of a disconnect, the market data is requested again and the backend server sends the full book again. So there is no problem with synchronization.
 
The problem is showing itself again ....
<img src=http://www.elitetrader.com/vb/attachment.php?s=&postid=2577663>

The ES is populated 10 levels deep but not showing on ES BookTrader.
 

Attachments

You must log in or register to reply here.
Back
Top