Database organization

[volpunter]

It is not a stretch given that virtually not a single quant desk at any hedge fund or ibank uses sql to handle time series data. There is a reason they use KDB and other columnar specialized data bases. Take my advice or leave it, but tucking gigabytes or terabytes of time series based data into a rdbms/sql is just the wrong way to approach this. I explained why so you have the free choice of course to do anything that pleases you. Good luck with your development.

Who's using that level of data though in a strategy (talking code running constantly in an alpha/risk/trx cost/execution/pc model).

I can't imagine any possible theoretical or evidentially based strategy that needs tick data beyond a few days to run, maybe a few weeks at most.

I'm talking about lugging the data around in an RDMS for backtesting, in which case it doesn't really matter (within reason) what can run circles around what.

I believe it's worth pointing out that the Nasdaq exchange itself runs on SQL Server. So to say that financial applications consuming the data provided by a RDMS can't handle it if they themselves are a RDMS seems like a stretch.

 

[Butterfly]

RDBMS are perfectly capable of handling time series, not sure why volpunter, in his infinite "wisdom", think only "KDB and other columnar specialized data bases" can handle those. The storage engine of such databases doesn't exclude them from handling SQL requests and create relations between tables. Actually, some columnar based BigData DB engine will do exactly that in SQL, see MonetDB for example.

Again, volpunter confusing concepts of storage engine with the full functionality of a RDBMS.

 

[Butterfly]

I fail to see how quants at hedge funds and Investment Banks could manage all those data by themselves with low level tech when in reality they would focus more on strategies, since that's why they are being paid for.

Also, I don't see the IT departments at those huge Investment Banks letting "individuals" plugin on their workstation and company servers, software that has no been approved or not gone through some kind of formal process for integration by the company IT department. Recipe for disasters if they let individuals "play" with their software and create IT incidents because of their incompetence.

There is no way that KDB and other proprietary binary files would make the cut for approval by the Enterprise Architects for inclusion in the company IT resources.

Those binary files full of useless market data remains what they are, toys for geeks in their basement.