Where and how do you acquire historical prices

P

pmuvb

I thought to ask for an advice before I start coding into acquiring the historical data to avoid getting into a dead-end solutions. I am planning to use one year worth of daily bars and 1-min bars of all US stocks for back testing my programs. I've found lists of the historical data providers from Reuters to Quandl and yahoo. I am willing to pay for the data, say, up to couple of hundred bucks a month, hence Reuters is out of equation.

These are the steps and processes that involve acquiring historical prices (at least in my mind).

  1. Downloading the initial snap. I'd prefer for a provider to have an option for bulk download, that is to download a single zip file that contains all the daily bars and a zip file that contains all the 1-min bars. I realize the 1-min bar file is going to be a large one - around few GB by my estimate. Versus, for instance, yahoo, where you can only download one symbol at a time so I would have to go into trouble of writing some sort of script to download the data for all symbols, if it's possible at all with yahoo - I hear the have been going against allowing to use scripts on their website. Also, yahoo has daily bars but not intra-day bars.

  2. Updating historical data every evening. I see the tasks that can be broken down into:
a) downloading today's prices for all symbols and then merging them into the main database.

b) symbols that have been deleted, then delete then from my database (I do not want those).

c) new symbols that have been added, then add them to the database.

d) symbols which prices have bee back adjusted due to splits - those have to be downloaded the full one year history.

Having said that, it looks to me that it would be easier to keep the historical data in a single file per each symbol (or a table per each symbol if I use a database in the future). The main point is I want an easy (if possible) download of the initial snap. Then I want to download the update every evening. I'd like to have a list of the symbols that have been added, deleted, and back-adjusted so I can update those accordingly, that's to avoid to download the whole snap every night.

How do you handle all of that and what provider would you recommend for me to have an easy life.



TX
 
I used IQFeed and AMibroker for awhile. Didn’t need the real-time/intraday so ditched it as it was overkill.

Amibroker handled all the getting the data but theirs a popular product for IQFeed called QCollector. Believe it’ll get you around the 500 limit that IQFeed has for symbol subscription.

IQFeed doesn’t advertise as a historical data provider and outside of trading hours they’re or we’re lenient with rotating symbol subscriptions.

QCollector I believe pulls data through IQFeeds API and puts into csv files. Might be worth looking into. Don’t think it’s expensive, maybe 100$ for QCollector, IQFeed is maybe 100$ a month unless you start adding level 2 and exchange packages.

EOD theirs lots of options. I’d check out Norgate Data. Their tool for downloading will handle a lot of the adjustments and such. But their data gets released a few hours after market close so may be later then you expect.

CSI believe has a download tool but they only do EOD.

Yahoo will throttle you if your not timing your requests and if you pull too much. See it pop up here and there on Amibroker forums where users use Amiquote which will pull from yahoo. Generally if your symbol count is less then 2k and your requests are every second or so it’ll work but some modify request time or try and pull 10k stocks multiple times a night and they get blocked.

Intraday leaves you a lot less options or at least affordable optionsbb
 
Last edited:
%%
I get mine[price+ volume] out of my notebooks;
+ off paper candle charts....................................................................... Wisdom is profitable to direct
 
I am using polygon.io and they are pretty good and I am very much content with them.

Tick Data, Bloomberg, Refinitiv, Reuters, etc are institutional level vendors hence very expensive.
 
pmuvb said:
[...] Bloomberg, Refinitiv, Reuters, etc are institutional level vendors hence very expensive.
More...

Institutional doesn't have to be more expensive. I'm obviously biased :) and will recommend our own product, but we're still in the stealth stage. But you could try Cboe DataShop if you're only after 1 min bars at a few hundred per month. Likewise, our pricing will be quite competitive to that.

We have a flat file batch download service which has both a UI (see below) and API. By default, you get all tickers in the entire market unless you specify filters. Most of our testers are using this feature as a drop-in replacement for Refinitiv DataScope or Tick History.

2022_06_30_Databento_Download_Manager.png
 
You must log in or register to reply here.
Back
Top