Look to spec out a project that can load and store historical data

C

conduit

I like to ask you guys about your experience and thoughts regarding a project that would import historical data sets from data vendor(s).

I essentially want to be able to have a processing engine that can load data from an API and store and make those data available.

The bottle neck here and technical challenges I face is on the data acquisition side, not data storage or internal retrieval

IB API is an option but they limit data requests (throttle), Reuters Eikon has a lot of data but the API is linked to an only works when the front end is also running (and the API is not really stable), I could consider CQG, Esignal, or others. (any others come to mind? Please bear in mind I only look for stable solutions even if I need to pay up, I am not interested in web scraping, Yahoo API or other stuff that may go away any day)

Goal is to have an independently operating Windows application (could also consider Linux) that listens to internal historical data requests and then can acquire the data via data vendor APIs through a one-time request or a subscription to also keep the data sets updated going forward.

I have several solutions in place but none of them are particularly stable and are error prone. I need such solution for my research efforts but also to run analytics in Excel (essentially even Excel would via add-in have access to such solution to retrieve data).

I am experienced in programming and have no issue coding something up from scratch because I do not see mature and stable products out there that exist already.

For example: From within Excel I like to get an array of 100 1-minute bars of a stock or future and the Excel add-in would then relay the request to my custom application which in turn would acquire the data from data vendor APIs and store the data in a data structure and return the requested data.

Any thoughts, ideas, solutions? Am also happy to team up for a concerted effort given there are others who bring value to the table (not just free loaders). Please only serious replies, as I am in this business as main career and take this job seriously.
 
Thanks, though I do not look for an Excel only solution so RTD is out of question. I want to store the data internally and later make the app accessible from other applications, including Excel. And I forgot to mention I look for a data vendor(s) with international coverage, not just US stocks and futures, but also currencies, international stocks and futures (UK, European markets, Japan, Australia, Hong Kong, ...)


Shay said:
you can get data from TOS using RTD.
and check -
http://www.savetodb.com/
they have some great tools to get it for you.

Shay
More...
 
Thanks, I agree, only caveat I found is that they do not go much beyond US market data, but I may have to take another look...

Edit: I stand corrected, they do provide access to quite a number international markets. Will check them out.

promagma said:
IQfeed has a good API, the historical data isn't comprehensive, but really a great value for the price.
More...
 
I was in a similar situation to you and searched for a reliable data vendor for ages. I still haven't found a reasonable intra-day data vendor that supplies historical data for worldwide assets without costing the earth. For end-of-day data I use Metastock Datalink which is backed by Thompson Reuters data. I too focus on global assets - US, UK, Europe, Singapore, etc. equities, as well as currencies and futures from around the world. The solution is a little clunky but is good enough for my data needs at this time.

I'm a seasoned professional programmer and have essentially developed my own solutions since I couldn't find a suitable platform that met my requirements. I use my software as my full-time career for investing and trading.

For me, I separated the problems into two, data acquisition ad storage, and secondly research and backtesting.

Firstly, I conduct research using end-of-day data from Metastock Datalink, which I download everyday, screen and then convert into my own custom propriety format for storage in compressed binary files. I don't an SQL database for storage since I have stringent storage and memory constraints and need to process data for 1000's of instruments.

Secondly, I have also developed by own research and backtesting engine which uses this data and runs against various strategies I have developed. Again I couldn't find a suitable platform that allows multi-currency, portfolio style management of long-short strategies using multiple types of global assets (equities, currencies and futures) so I wrote my own. Using platforms like TradeStation (and similar) are a joke for this type of thing.

I'm now moving onto intra-day research and will apply the same techniques as above, however I'm still searching for a suitable data vendor.

Everyone's requirements vary, and what I have done may seem crazy to some, but for me it works. The bottom line is be prepared to code your own since a lot of the canned products out there are limited in one way or another. Of course you can leverage a lot of open source stuff to assist which is a good way forward. I suggest you code your own solution then scale it appropriately.
 
runtrader said:
Firstly, I conduct research using end-of-day data from Metastock Datalink, which I download everyday, screen and then convert into my own custom propriety format for storage in compressed binary files. I don't an SQL database for storage since I have stringent storage and memory constraints and need to process data for 1000's of instruments.
More...

i have EOD for all US stocks and options data from 2002,
and i insert them to SQL Server.
1.3 Billion (it's not a mistake) records !
storage is an issue...
:-)

Shay
 
to clarify, when you say "end-of-day" does that mean intraday data that are made available at the end of each trading day? If that is the case what is the definition of "end of each trading day"? Is that a global fixed cutoff time or the end of trading in each local market regardless of time zone? Or are we merely talking daily compressed data?


runtrader said:
I was in a similar situation to you and searched for a reliable data vendor for ages. I still haven't found a reasonable intra-day data vendor that supplies historical data for worldwide assets without costing the earth. For end-of-day data I use Metastock Datalink which is backed by Thompson Reuters data. I too focus on global assets - US, UK, Europe, Singapore, etc. equities, as well as currencies and futures from around the world. The solution is a little clunky but is good enough for my data needs at this time.

I'm a seasoned professional programmer and have essentially developed my own solutions since I couldn't find a suitable platform that met my requirements. I use my software as my full-time career for investing and trading.

For me, I separated the problems into two, data acquisition ad storage, and secondly research and backtesting.

Firstly, I conduct research using end-of-day data from Metastock Datalink, which I download everyday, screen and then convert into my own custom propriety format for storage in compressed binary files. I don't an SQL database for storage since I have stringent storage and memory constraints and need to process data for 1000's of instruments.

Secondly, I have also developed by own research and backtesting engine which uses this data and runs against various strategies I have developed. Again I couldn't find a suitable platform that allows multi-currency, portfolio style management of long-short strategies using multiple types of global assets (equities, currencies and futures) so I wrote my own. Using platforms like TradeStation (and similar) are a joke for this type of thing.

I'm now moving onto intra-day research and will apply the same techniques as above, however I'm still searching for a suitable data vendor.

Everyone's requirements vary, and what I have done may seem crazy to some, but for me it works. The bottom line is be prepared to code your own since a lot of the canned products out there are limited in one way or another. Of course you can leverage a lot of open source stuff to assist which is a good way forward. I suggest you code your own solution then scale it appropriately.
More...
 
Daily compressed data - i.e. daily OHLC. I download using a follow the sun technique, when LSE (UK) closes at 1630 GMT/BST I download and convert data for UK based assets for that day, likewise for EU, US, Singapore, etc.

I'm still searching for reliable intra-day data solution, although IB data seem OK for now. I do find a few discrepancies when comparing intra-day data from IB with corresponding EOD from Metastock. I use EOD data as the reference, although to complicate things further EOD changes on subsequent downloads, i.e. the data vendor cleans and republishes data, and this needs to be catered for.
 
You must log in or register to reply here.
Back
Top