Interactive Brokers and pacing violations

[d08]

What I meant is

a) one needs to put in a delay in between requests of certain types

AND

b) one can only submit x requests in a 10 minute window.

You talked about a 10 second throttle (not sure what it is)

If we talk about one and the same thing then apologies...

Never heard of a need to put in a delay between requests, I've sent many in quick succession. 10 seconds (actually more like 8 to cut it close) is because that's how many requests you can send without running into pacing violations over time. Tried this with both 15min/monthly and 1min/weekly.
But if you're just going to send say 10 requests, you can do it without any breaks. Pacing violation numbers are averaged, not based on individual requests.

 

[Zzzz1]

I think its max 60 requests in 10 minutes, and

• Making six or more historical data requests for the same Contract, Exchange and Tick Type within two seconds.

This is off the IB website and I did run into pacing violations when I violated that (as recent as October 2016, so I assume it still applies).

So when making requests and you look to retrieve a longer time series for one and the same symbol and identical tick type (by breaking the desired amount of data down into smaller requests) you need to pace or ensure 6 or more requests of identical symbol do not fall within 2 seconds when requesting data.

Never heard of a need to put in a delay between requests, I've sent many in quick succession. 10 seconds (actually more like 8 to cut it close) is because that's how many requests you can send without running into pacing violations over time. Tried this with both 15min/monthly and 1min/weekly.
But if you're just going to send say 10 requests, you can do it without any breaks. Pacing violation numbers are averaged, not based on individual requests.

 

[Stymie]

I think you misunderstand, I doubted that NxCore would provide 90% global exchange coverage and I actually mentioned IB as a broker that does provide around 90% coverage of major markets. But I disagree with you when it comes to core services a broker should provide. A broker is in the business of processing orders and/or making markets. They are historically not in the business of providing historical data feeds. That is the core business of data providers, such as Bloomberg, Reuters,...I mentioned that it would be a huge revenue bump for IB if they provided a better real time data feed (not snapshots) and a comprehensive historical data feed product for which they could charge quite a premium precisely because they cover so many markets. One reason I stay away from most non Bloomberg/Reuters data providers is because they have very poor coverage of Asia and smaller European markets.

Ok - for the sake of discussion, your suggesting that a broker providing faster real time feeds and a clean historical database could make significant revenue from it given they cover so many markets already. I would disagree with that presumption.

First - there are already providers of low latency feeds to each exchange and global historical databases for decades. Bloomberg/Thomson Reuters have their own FX Spot trading platforms and data will need to be bought from them by any vendor wanting to compete along side them. This does happen today.

Second - Given this is not core to their transaction based fee revenue model, they would have to build out the infrastructure and massive cost of upscaling/maintaining these databases. They would need to sign agreements with all exchanges and dark pools to buy the data and store as a reseller of the data which increases the costs substantially. This whole effort would be thwarted by the fact that the competitors have been doing this for decades and have already paid the price of going up the learning curve.

Third - as brokers use a transaction fee model, their clients expect data to be free. This especially holds true if they are active traders. That's why there are so many "Soft Dollar" deals done when clients are huge and brokers pay for all their other services. They want commission fees.

IN summary, Brokers might want to offer data as a differentiator to keep or attract customers but why invest in the data infrastructure when they can simply white label it from vendors. This is what IB does by offering premium data from vendors for a fee of which they get a small cut. This simply required some software programming on the front end to automate the process with end users and all the exchange agreements and one agreement back with a single large vendor. The revenue is small relatively for both providers but helps to differentiate.

 

[Stymie]

OK - I was able to get intraday prices on charts and add the virtual security into a quote list using the "~" in front of the name for my virtual FX. I'm happy to find a solution to a more complex example if you want to share.. We can do everything but trade with the virtual instrument created. Platform works if your permissioned for the data at least for me.

 

[Zzzz1]

We are talking about the retail side here not institutional feeds or clients. And IB already has relationship with all exchanges already. They signed all agreements and all they would need to do is to run a dedicated market data server which they already do albeit on a smaller scale hence their throttling end user requests. There is no reinventing the wheel here. All IB has to do is to allocate some resources to this but it would pay off because many retail end users, me included would easily pay them 100 to 200 dollars a month if we could request historical data without limits and get all the real time updates. Any retail oriented data feed vendor today does not deliver historical nor real time data for most exchanges at price points under 500 dollar. Why do you think there are so many threads raised on this site that exactly target this problem? ESignal, NxCore, cqg, iqfeed.... All charge much more for data feeds that include futures, stocks, and options for all exchanges ib offers, if they even offer data for those exchanges IB connects to. Another problem is that most users don't want to be charged twice for real time data subscriptions, once on the broker level and again on the data connectivity level. Hardly any exchange offers arrangements that allow the same client to pay once not twice.

Ok - for the sake of discussion, your suggesting that a broker providing faster real time feeds and a clean historical database could make significant revenue from it given they cover so many markets already. I would disagree with that presumption.

First - there are already providers of low latency feeds to each exchange and global historical databases for decades. Bloomberg/Thomson Reuters have their own FX Spot trading platforms and data will need to be bought from them by any vendor wanting to compete along side them. This does happen today.

Second - Given this is not core to their transaction based fee revenue model, they would have to build out the infrastructure and massive cost of upscaling/maintaining these databases. They would need to sign agreements with all exchanges and dark pools to buy the data and store as a reseller of the data which increases the costs substantially. This whole effort would be thwarted by the fact that the competitors have been doing this for decades and have already paid the price of going up the learning curve.

Third - as brokers use a transaction fee model, their clients expect data to be free. This especially holds true if they are active traders. That's why there are so many "Soft Dollar" deals done when clients are huge and brokers pay for all their other services. They want commission fees.

IN summary, Brokers might want to offer data as a differentiator to keep or attract customers but why invest in the data infrastructure when they can simply white label it from vendors. This is what IB does by offering premium data from vendors for a fee of which they get a small cut. This simply required some software programming on the front end to automate the process with end users and all the exchange agreements and one agreement back with a single large vendor. The revenue is small relatively for both providers but helps to differentiate.

 

[Zzzz1]

You are again not carefully reading my post. I did not say that prices don't update on quote sheets for virtual securities. I said that the intraday and daily charts don't update with changes in the value of virtual securities. Has nothing to do with how complex the virtual security basket is.

OK - I was able to get intraday prices on charts and add the virtual security into a quote list using the "~" in front of the name for my virtual FX. I'm happy to find a solution to a more complex example if you want to share.. We can do everything but trade with the virtual instrument created. Platform works if your permissioned for the data at least for me.

 

[d08]

I think its max 60 requests in 10 minutes, and

• Making six or more historical data requests for the same Contract, Exchange and Tick Type within two seconds.

This is off the IB website and I did run into pacing violations when I violated that (as recent as October 2016, so I assume it still applies).

So when making requests and you look to retrieve a longer time series for one and the same symbol and identical tick type (by breaking the desired amount of data down into smaller requests) you need to pace or ensure 6 or more requests of identical symbol do not fall within 2 seconds when requesting data.

Yeah, 60 requests in 10 minutes or 6 requests in a minute or a request every 10 seconds, that's how I arrived at the number.
Didn't remember the 2 second rule, that's not that relevant as you need to follow the 60/10 rule anyway.

IB data gathering is painful but it's not that bad if you're patient and only retrieve data that you need.

 

[Zzzz1]

The approach you describe works and in that case you can ignore the 2 second rule, true. However, most data vendors and myself implemented it another way, which is to observe both rules and in exchange get data back faster than with your approach. What sucks, however, is not exactly knowing how many data points one can request at once in a single request, IB constantly changes that and does not update its api docs. Because of that one does not know which requests get broken down by ib into even smaller requests and pacing violations can still occur. Let's admit that IB's api is incredibly dirty and badly implemented. But I laud your spirit and positivity, after all one has to work with it when wanting to get data from ib.

Yeah, 60 requests in 10 minutes or 6 requests in a minute or a request every 10 seconds, that's how I arrived at the number.
Didn't remember the 2 second rule, that's not that relevant as you need to follow the 60/10 rule anyway.

IB data gathering is painful but it's not that bad if you're patient and only retrieve data that you need.