Does anyone download data and what do they do with it?? The existence of this forum and the various data providers out there suggests that there might be some demand for data but from what I see and hear most investors/traders have no desire to build their own database, source their own data and do their own analytics/quality checks. I assumed that algo traders and quants would build their own test harnesses for research as well as professional firms such as funds, money managers etc but again see little evidence. Presumably this is all done behind closed doors with their own reinventing the wheel/bespoke solutions? Who exactly uses market data and what do they do with it - or do most (including professional firms) just sign up to services and use/trust the software/data combo that is provided?
Who uses datasets and feeds?
- Thread starter moneyweaver
- Start date
Ninja Mobile Trader VPS
Sponsor
Data is king. If your data provider allows for it, it is a good idea to data warehouse everything you can.
I do. In fact right now I'm expanding the list of futures I trade in my systematic trading system, so I'm downloading large amounts of data.
Depends what you mean by 'build your own database'. Nobody is going to produce an alternative to kdb, sqllite, mysql or mongodb. But plenty of people are happy to build their own schema and API on such a database.
What do you mean by 'source your own data'? You can easily get price data for almost anything from a huge variety of sources. Even fundamental or alternative data is just a google search away. Why would I pay someone to do this for me?
I disagree. Frankly, I don't trust anyone else to do it. Commercial data from the likes of BPIPE is too pricey, and for a retail level service, the quality of the checking is very poor.
Yes they do. I have, and any serious algo trader will build their own system. Almost no professional firms use commercially available software for this (though they will for backoffice and other more generic services).
GAT
I considered Bar chart a while ago to purchase data from, but strangely many of the time series they offer don't go back far enough, even on daily data the AXJO Australian index did not go back far enough and as far as I remember they don't offer New Zealand's stock index either. Also have trouble finding UK data on short term yields and guilts.
Yeah I don't trade any of those. But you're right, some of the series are a bit short. But that doesn't bother me, I only need a year or so of data. It's a pretty cheap subscription, and once I've downloaded everything I need I won't need it any more so will have cost me very little really.
GAT
Good to hear that someone is downloading large amounts of data! Like the previous reply said data is king. Whatever processes or systems you are using good decisions have to be based on high quality data inputs. I agree that data vendor or exchange-purchased historical datasets (let alone freely downloadable data) rarely provide the level of data quality needed to properly evaluate an algorithm or an trading strategy. While institutional level data comes at a price.
Regarding 'build your own database' I was referring to schema/API infrastructure solutions. I get the impression some people are willing to slap together some bespoke script/API/database schema but a lot more don't even attempt this (presumably because of the amount of support and time it takes unless you only need a one off solution). I am sure that there are plenty of pricey infrastructure off-the-shelf solutions out there for professional firms (ie infrastructure not the "magic" sauce a firm/fund is running on) but I don't see anything for serious retail or emerging money managers/funds (ie using an rdbms to centralise and schedule/trigger imports,exports and analyse all of the data from various websites and APIs).
Last edited:
I find 1 year of data is too short, especially if trading in daily time frame. Maybe good enough for trading by the minute time frame.
Not sure if you encounter this. Sometimes, I find funny spikes in IB data sometimes. It is not high quality enough.