IB data feed
- Thread starter simki
- Start date
There are several problems associated with IB data:
1. IB sends ticks improperly (as Oneway already mentioned). So anyone will definitely have incorrect number of ticks. Take a look at the logs of the trades:
[ 12:40:12 ] Total Volume= 13049
[ 12:40:18 ] Last= 1125.75
[ 12:40:18 ] LastSize= 1
[ 12:40:18 ] LastSize= 1
[ 12:40:18 ] Total Volume= 13050
According to the ticks we should generate two ticks for 12:40:18 with the same volume=1. It looks like the price didnât change and IB sends just update for the last size.
However it is not true, because total volume is not changed twice (compare 13049 and 13050) so one tick is redundant.
It is clear that Total Volume is updated by exchange so we can trust it.
So MultiCharts uses Total volume to recognize new tick event. To test if our assumption is correct we compared two ticks charts - one from IB and another from TRAD. 99.9% of ticks are the same for both datafeeds. This confirms our assumption. They were completely different before we added the Volume change event filtering.
This trick will not work for bids/asks because there is no objective source of events and we still have redundant data.
2. IB real-time and history ticks are not the same, because IB doesnât provide historical ticks. It provides second bars instead. It is clear that this isn't the same thing. For instance:
a) You may have 5 ticks per second, but it will be only 1 second bar.
b) You may have only 100 ticks in the night for 1 hour, but IB will send 3600 second bars. 99% of them will be with zero volume. MultiCharts has a zero-volume filter to make second bars closer to ticks.
3. 1 minutes from real-time ticks and backfill will work relatively accurately in MultiCharts. See attached screenshot and dumps. There is a difference on some bars. Maybe it is due incorrect tick time stamps as discussed above.
1. IB sends ticks improperly (as Oneway already mentioned). So anyone will definitely have incorrect number of ticks. Take a look at the logs of the trades:
[ 12:40:12 ] Total Volume= 13049
[ 12:40:18 ] Last= 1125.75
[ 12:40:18 ] LastSize= 1
[ 12:40:18 ] LastSize= 1
[ 12:40:18 ] Total Volume= 13050
According to the ticks we should generate two ticks for 12:40:18 with the same volume=1. It looks like the price didnât change and IB sends just update for the last size.
However it is not true, because total volume is not changed twice (compare 13049 and 13050) so one tick is redundant.
It is clear that Total Volume is updated by exchange so we can trust it.
So MultiCharts uses Total volume to recognize new tick event. To test if our assumption is correct we compared two ticks charts - one from IB and another from TRAD. 99.9% of ticks are the same for both datafeeds. This confirms our assumption. They were completely different before we added the Volume change event filtering.
This trick will not work for bids/asks because there is no objective source of events and we still have redundant data.
2. IB real-time and history ticks are not the same, because IB doesnât provide historical ticks. It provides second bars instead. It is clear that this isn't the same thing. For instance:
a) You may have 5 ticks per second, but it will be only 1 second bar.
b) You may have only 100 ticks in the night for 1 hour, but IB will send 3600 second bars. 99% of them will be with zero volume. MultiCharts has a zero-volume filter to make second bars closer to ticks.
3. 1 minutes from real-time ticks and backfill will work relatively accurately in MultiCharts. See attached screenshot and dumps. There is a difference on some bars. Maybe it is due incorrect tick time stamps as discussed above.
Here is dump screenshot.
You are making invalid assumptions. Just because LAST SIZE was updated twice does NOT mean that there were 2 trades of one contract each. All that means is that at that time, the last trade size was 1. That is all. Why it was sent twice is irrelevant. The ONLY thing that could be interpreted as a trade is a change in total volume
j_medved,
Our assumption is correct and you just confirmed that trade volume change is correct way to get new tick event. Moreover all tests confirm that tick count is correct.
What was wrong is the example. Actually two ticks are not a big deal. I just misunderstood the developer who is responsible for the IB data source plug-in. The real big issue is illustrated in the example below:
[ 12:49:20 ] Total Volume= 13341
[ 12:49:29 ] LastSize= 1
[ 12:49:29 ] Total Volume= 13342
[ 12:49:33 ] Total Volume= 13343
You can see that we have 1 last size, but actually total volume changed twice. So we must generate two trades instead of 1 as we would conclude according to ONE last size message.
This is the basic issue with IB real-time.
Andrew Kirillov
Our assumption is correct and you just confirmed that trade volume change is correct way to get new tick event. Moreover all tests confirm that tick count is correct.
What was wrong is the example. Actually two ticks are not a big deal. I just misunderstood the developer who is responsible for the IB data source plug-in. The real big issue is illustrated in the example below:
[ 12:49:20 ] Total Volume= 13341
[ 12:49:29 ] LastSize= 1
[ 12:49:29 ] Total Volume= 13342
[ 12:49:33 ] Total Volume= 13343
You can see that we have 1 last size, but actually total volume changed twice. So we must generate two trades instead of 1 as we would conclude according to ONE last size message.
This is the basic issue with IB real-time.
Andrew Kirillov
A representative of IB discussed this issue somewhere on these boards. As I understand it, the number of tick (ie LAST_SIZE) events you get (and I extrapolate this to mean the possible aggregation of ticks) depends on the number of instruments you have requested market data for. ie more instruments then fewer ticks per instrument. He did not elaborate on the algorithm. Possibly it is also dependent on the traded volume of the instruments subscribed to as well. Presumably, this is to constrain the load on IB servers and network infrastructure.
OK, I see. If you are trying to extract ticks and started with volume 13341, you would then have to add one tick for each volume change. You would NOT add a tick just because LastSize=1 was sent, unless a) The LastSize is different from previous LastLize sent on the security and b) there is a corresponding TotalVolume change.
IB by design is not a tick by tick datafeed and you are not guaranteed to get every event. The example you gave is not a problem with the data. At least from the point of view that its working as intended.
Had you received something like this:
[ 12:49:20 ] LastSize= 1
[ 12:49:20 ] Total Volume= 13341
[ 12:49:29 ] LastSize= 2
[ 12:49:29 ] LastSize= 1
[ 12:49:29 ] Total Volume= 13349
[ 12:49:33 ] Total Volume= 13350
then you could interpret that at least 4 trades occured after the 13341 volume, since LastSize actually changed.
Trade 1 - LastSize change to 2
Trade 2 - LastSize Change to 1
Trade 3 - Change in volume from 13341 to 13349 is 8 contracts. The 2 assumed trades total 3 contracts. Therefore there was at least one other trade for 5 contracts (or multiple trades, you cannot tell)
Trade 4 - Change in Volume to 13350
Jerry Medved
http://www.quotetracker.com
Jerry,
I donât think that our discussion constructive, because you just don't read what I wrote.
1. I explained in my first post that we use Total volume as a source of events to generate ticks. I don't understand why you suggest me to do it if it is already done 6 months ago. It sounds strange.
2. IB has a definite problem for time series. Nobody cares what initial IB intention was. It is clear that they thought about quote window only. It is fact that Ask/Bid charts will be wrong, because there is not event to generate ticks properly.
3. I believe your example and general understanding is not valid, because Total volume will be always changed for each new tick and you always know tick size of a trade = Current Total volume â Previous Total Volume.
So if your software uses the algorithm you described it will have incorrect tick sizes, because "or multiple trades, you cannot tell".
We ignore tick size field and take last price only. As soon as we see difference between total volumes we use it as a tick size. Our tests demonstrated that 99% of TRAD and IB ticks/tick sizes are the same.
Andrew Kirillov
I donât think that our discussion constructive, because you just don't read what I wrote.
1. I explained in my first post that we use Total volume as a source of events to generate ticks. I don't understand why you suggest me to do it if it is already done 6 months ago. It sounds strange.
2. IB has a definite problem for time series. Nobody cares what initial IB intention was. It is clear that they thought about quote window only. It is fact that Ask/Bid charts will be wrong, because there is not event to generate ticks properly.
3. I believe your example and general understanding is not valid, because Total volume will be always changed for each new tick and you always know tick size of a trade = Current Total volume â Previous Total Volume.
So if your software uses the algorithm you described it will have incorrect tick sizes, because "or multiple trades, you cannot tell".
We ignore tick size field and take last price only. As soon as we see difference between total volumes we use it as a tick size. Our tests demonstrated that 99% of TRAD and IB ticks/tick sizes are the same.
Andrew Kirillov