SQL for trading...?

[dcraig]

Quote from thstart:

SSD disks right now are reported to deteriorate with time and the access time to increase after usage. It is still not clear why but this info is not popularized. They still expensive and indeed storage logic is half of the story.

Decent article here that discusses the issues:

http://www.anandtech.com/storage/showdoc.aspx?i=3531

The ageing problem is mostly with writes and is highly dependent on the controller chip and firmware used in the drive. The forthcoming ATA TRIM command already supported by some new drives will also mitigate the problem. Linux kernel filesystem/block device support for TRIM will be out very soon. Windows 7 will support it sometime in the future.

 

[thstart]

Quote from stevegee58:

...I feel a little embarrassed mentioning this (what with all the high powered DB guys posting here) but I've been using SQLite...

I believe for your needs SQLIte is probably the right solution. If you r needs grow then you will need something more powerful.

The simplest solution depending from the task is the best one - SQLite is very capable and if your tasks are small and don't require a lot of data probably it can fit your needs.

I mentioned there is a tendency among the vendors to rewrite all they products to .NET which is not always right. Too much overhead, resources and it is slow, I don't see any other benefit other than development and maintenance is faster under .NET - that is indisputable. But we need performance.

You have to cut all what is unneeded to get the speed. SQLite is a good example of this - Dr.Hipp is not willing to add anything more which is good sign it would not get bloated. It is not right for what we do but I like the approach. By the way - it is used from Bloomberg.

The problem we had finding a solution using conventional approach is we have a lot of data and extensible algorithms to test the best strategy - it needs lot of experimentation and the time to get your answer right or wrong is the critical factor. That is why we decided to develop our own solution. Not easy decision because it takes time and investment, but it will compensate in the long run - it is patent pending and we can offer it to the trading public at some point - a very fast screener and back-testing platform. That is what I think is lacking on the security analysis market today and most needed.

 

[thstart]

Quote from Rodney King:

Their homepage now discourages its use and nudges you toward the non-inexpensive [we demo'd and priced them] Vertica tools:

<i>Consequently, recently people have been having trouble compiling and running C-Store. No more releases of the code are expected -- this code is mostly useful for repeating experiments in the papers listed below. If you want to play with a generic column-store, we recommend you check out Vertica, the commercialization of the C-Store project,</i>

That is because C-Store developer moved to Vertica

 

[thstart]

Quote from dcraig:

Decent article here that discusses the issues:

http://www.anandtech.com/storage/showdoc.aspx?i=3531

The ageing problem is mostly with writes and is highly dependent on the controller chip and firmware used in the drive. The forthcoming ATA TRIM command already supported by some new drives will also mitigate the problem. Linux kernel filesystem/block device support for TRIM will be out very soon. Windows 7 will support it sometime in the future.

There is a different access pattern with SSD and the optimal way to write/read is still not researched well. When they stop the deterioration problem we will wait for 1 year to look for any other hidden surprises. they will get cheaper too.

 

[nitro]

Quote from dcraig:

How much size do you want ?...

Storing just the level one OPRA feed and associated greeks goes through ~3Gb a day, uncompressed.

Falcon SSD $900 here

http://www.guru3d.com/article/gskill-falcon-128gb-ssd-review/'

That is 8x more expensive than the same size high end SCSI drive, with, AFAIK, no support for RAID (for redundancy).

 

[nitro]

Quote from Rodney King:

Their homepage now discourages its use and nudges you toward the non-inexpensive [we demo'd and priced them] Vertica tools:

...

What is your review of Vertica?

 

[thstart]

Quote from nitro:

What is your review of Vertica?

A quick note - Vertica is more oriented toward large warehouses like inventory analysis, etc. We need more time series oriented DB.

 

[CloroxCowboy]

Anyone with experience using hybrid drives? I haven't but it seems like a possible solution. Not sure if they can be used in a RAID array though...

 

[Rodney King]

Quote from thstart:

A quick note - Vertica is more oriented toward large warehouses like inventory analysis, etc. We need more time series oriented DB.

Quote from nitro:

What is your review of Vertica?

Mr Start's comment is roughly correct. Vertica is often used by financial-markets firms in conjunction with tools from Streambase, which is a quasi-affiliated company. Both firms [and C-Store, as mentioned elsewhere on the thread] are associated with Mike Stonebraker, a rockstar computer scientist. Both firms offer excellent products, but [ to answer Mr Nitro's question] we came to believe the heavy cost load of buying both Vertica and Streambase wasn't appropriate relative to our scope and requirements, and we moved toward open source solutions instead, at least for now. But YMMV.

 

[nitro]

Quote from Rodney King:

...wasn't appropriate relative to our scope and requirements, and we moved toward open source solutions instead, at least for now. But YMMV.

Thanks.

What open source solution did you decide on, for column store dbs anyway?