Tick Database Implementations

nitro · Jan 30, 2007

Quote from Sparohok:

Martin

Martin I don't agree with 85% of what you are saying about the use of HDF5 in relation to it's use in financial time-series databasing, but I don't have time to go into it now. I am responding so that I remember the challenge that I made and so that you can hold me to respond to that challenge later.

nitro

Sparohok · Jan 30, 2007

Quote from ktmexc20:

I am not aware of another free, open source database that will top hdf5 or it's hierarchical model for storing, analyzing, and utilizing (both off-line and in real-time) financial data

My suggestion is reiserfs. Although any modern journalling filesystem will have broadly similar properties.

Some would say that a filesystem isn't really a database; but hdf5 isn't really a database either. The difference lies in their respective shortcomings. A filesystem falls short of being a "real" database because it doesn't manage structure at the record level. hdf5 falls short of being a "real" database because it can't handle concurrency. Otherwise they are really quite similar. Different applications will be able to cope differently with these compromises.

One thing is certain though, filesystems are faster than hdf5. After all, hdf5 is layered on top of a filesystem.

Anyway, it's always interesting to discuss this. As long as I've shared my own concerns with HDF5 I feel like I've done my part.

Martin

nitro · Feb 18, 2007

I just realized that the K programming language is a derivative of APL and a cousin to the J programming language. Now I see why this is so desirable...

Are you saying they distribute free versions of k+ and kdb+ for personal use?

nitro

Quote from Batman28:

hahahaha...

JUST IMAGINE your software programmed in k+ linked to kdb+ with ksql enquries!!

p.s. I hear they give out free k+ interpreters and if u contact them with nice enquiries for personal use they distribute free packages.. but be aware, you require a super multi CPU system to even set up something like kdb+tick..

Joel Reymont · Feb 18, 2007

The answer was NO when I asked them recently.

Sparohok · Feb 19, 2007

Has anyone looked into Vertica? A column major database from Michael Stonebraker's latest academic spinoff. www.vertica.com

I signed up for their early access program but haven't heard boo. It looks a little too heavily read-optimized for my needs but still, pretty darn cool.

nitro still looking forward to your comments on the 85% you disagreed with me.

Martin

Joel Reymont · Feb 19, 2007

I don't remember anyone mentioning MonetDB so here it is:

http://monetdb.cwi.nl/

syswizard · Feb 19, 2007

http://www.proxy80.com/cgi-bin/nph-...urceforge.net/forum/forum.php?forum_id=584739
MonetDB: Fast, but not much more, if any, than MySQL 5.0
Still, benchmarks for retrieving sequential, time-series data may show it to be a winner.
Still awaiting the ODBC documentation....none to be found.

WallstYouth · Feb 19, 2007

This might be useful to some.

Simple Persistence for Java is an open source object-relational persistence library that uses a custom query language and built-in database support to simplify object persistence in Java applications. In this article, software architect Sami Salkosuo introduces the library and walks you through its zero-admin, zero-config approach to object persistence.

http://www-128.ibm.com/developerworks/java/library/j-sp4j/?ca=dgr-lnxw07Java

Joel Reymont · Feb 20, 2007

I'm using OCaml to build my (commercial) algorithmic trading tookit. Apart from performance on par with C, OCaml has built-in marshalling which can be used for persistence.

The approach to tick storage for OCaml is to just dump the tick arrays to disk and organize them in directories.

Also check out http://cristal.inria.fr/~starynke/persil/ which uses marshalling with various data stores.

nitro · Feb 20, 2007

Quote from Joel Reymont:

I'm using OCaml to build my (commercial) algorithmic trading tookit. Apart from performance on par with C, OCaml has built-in marshalling which can be used for persistence.

The approach to tick storage for OCaml is to just dump the tick arrays to disk and organize them in directories.

Also check out http://cristal.inria.fr/~starynke/persil/ which uses marshalling with various data stores.

If I were going to go this route, there is no question I would use the J programming language.

nitro