database

[Ivano]

I am building a database with historical oclh and volume data, including delisted US securities, based on the total NASDAQ universe, and simple day dataframe.

I would like to switch to IBKR API because they are doing a nice REST API and mainly because I would like to execute the orders from there. But i see there is a
> limit of 60 requests per ten minutes.


If i need to update all the nasdaq tickets daily it will take hours to update, even if i use advanced async techniques.
How do quant build their db?

I need the OCLH because need to do my studies.
SO if IBKR is not good to go for this use case how is possible that eveybody uses? DO they just use the scanner and execute on the basis of the scan results?
ps. I am assuming that if i get hourly i can easy calculate for free the OCLH of the daily right in another sql table right?

 

[rb7]

How do quant build their db?

They don't use IBKR for their market data.

If you're serious about this, look for a real market data provider. But it ain't cheap.
You pay for what you get.

 

[Peter8519]

Since you need historical intraday or tick data, take a look at iqfeed, firstrate data or tiingo. Yahoo finance provides free daily data. You may want to consider this for your initial system testing.

 

[trader221]

A simple and almost free alternative would be to connect your account to Ninjatrader afterwards.
Download NASDAQ Listing Data Daily

 

[HobbyTrading]

But i see there is a limit of 60 requests per ten minutes.

That used to be the case, but has since been removed for bar sizes of 1 minute and above. There are "soft limits" though, and IBKR is not really specific how they determine those limits.
In my own experience: I update daily a list of about 300 tickers, sequentially. It takes about 5 minutes to get the data. I am not familiar with IB's REST API, I use the regular API.

From this page: https://interactivebrokers.github.io/tws-api/historical_limitations.html

1. At this time Historical Data Limitations for barSize = "1 mins" and greater have been lifted. However, please use caution when requesting large amounts of historical data or sending historical data requests too frequently. Though IB has lifted the "hard" limit, we still implement a "soft" slow to load-balance client requests vs. server response. Requesting too much historical data can lead to throttling and eventual disconnect of the API client. If a request requires more than several minutes to return data, it would be best to cancel the request using the IBApi.EClient.cancelHistoricalData function.

 

[Ivano]

That used to be the case, but has since been removed for bar sizes of 1 minute and above. There are "soft limits" though, and IBKR is not really specific how they determine those limits.
In my own experience: I update daily a list of about 300 tickers, sequentially. It takes about 5 minutes to get the data. I am not familiar with IB's REST API, I use the regular API.

From this page: https://interactivebrokers.github.io/tws-api/historical_limitations.html

300 are not enough too me, I need to update all the nasdaq universe, i think I should call them and will ask what they expect as fair server bandwith consumption.

I guess most of the people that pay for an API want to have this functionality, why to rely on scanner that are limited as a strategy when having them in a DB i can build every kind of technical i want?

I wanted to go with IBKR also because I base my trading decisions also on the basis of options statistics.

 

[HobbyTrading]

300 are not enough too me

I was only giving an example, to give you an indication of the time it takes. If 300 symbols are done in 5 minutes, then 3000 symbols can be done in 50 minutes. You want to download historical data, so you can do that e.g. during the night. It doesn't really matter whether it takes one hour or a couple of hours.

 

[Ivano]

I was only giving an example, to give you an indication of the time it takes. If 300 symbols are done in 5 minutes, then 3000 symbols can be done in 50 minutes. You want to download historical data, so you can do that e.g. during the night. It doesn't really matter whether it takes one hour or a couple of hours.

this is cool, I think I should stick with IBKR at this stage,their data should be reliable considering their role as brokers and as i need daily candles(ideally hourly), if i would be proficient in ES future could cut the cost focusing on one index, but i am still not ready to mess around with futures, although they would give me an edge from day 1 in some way, and often isolated stocks and sectors could do wonders if I really select my trades.

 

[HobbyTrading]

Both daily and hourly candles are available at IBKR. How far back in history would you like to go? How many candles?
Futures are not the "holy grail" for everyone. They have benefits, but also disadvantages. Depending on what your ultimate goal is, and your time horizon is, you might be better of with ETFs and single stocks.

 

[Ivano]

I called them and at the moment they confirmed me 50 request per second, and historical data go back a lot of years, They tested by phone MSFT and is going back to 86/87. Looks pretty good for my purposes, just have to do my self (of course) the thread concurrency hassle, but gonna be fine.

Coming down to the DB i thought to use a strategy used by ERP wharehouses etc, basically I am gonna build an SQL with the OCHLV and then I will inner join all the indicators, etc in separate tables, it looks as a solid way to proceed, just doubting maybe would be better to start with MongoDB from the beginning, but do not want to add layers of complexity at this stage, hope will not regret and gonna be easy to migrate.