Tick Database Implementations

[nitro]

Quote from ktmexc20:

...
The question I still have though, is with your original assertion that hdf5's packet table interface is "no where near" a streaming db. How is that?

Boeing's Flight Test Instrumentations Group and the HDF5 development group at the University of Illinois have developed a library that is particularly suited for "packet" data, data that arrives in streams of packets from instruments at potentially very high speeds.


-kt

Kt,

If your goal is only to store quotes from a packet, then sheesh, just about anything that takes a packet and stores it to do disk qualifies as a "database." The real meat is in the quering of realtime data with complex queries using a straightforward language on time-series in realtime. Again, I urge you to read the streamDB papers because they go into this very question.

Like I said above, you can do it all in C++ & HDF5 and coding functions that do what you want in C++. But unless you have a language that you have built for doing complex queries, it is half a database imo (let alone ACID, fault tolerant, transactional, etc etc.) HDF5 doesn't have a Query Language on streams AFAIK.

BTW, take a look at db40. Very interesting idea of using objects instead of relations (Object Database), but using classes to do queries. And it's free!!. This may be the best compromise between a stream DB and hand coding all these db/query language in C++:

http://www.db40.com

nitro

 

[ktmexc20]

Quote from nitro:

I will take the time to read the documents. Thank you.

Call me crazy, but I like the exercise of coding up my own function-aggregate interfaces and utilizing pre-formed interfaces sparingly. Must be due to my (faulted?) meticulous nature for granularity, originally coming from my architectural/building/engineering thinking process.

As I gain experience, I find that more and more, I'm appreciating higher level interfaces (or languages) in my code... when needed or efficiently acceptable.

-kt

 

[rosy2]

Quote from dcraig:

Has anybody actually run any benchmark or performance test of HDF5 as compared to say MYSQL or anything else for the matter for the type of data that is being talked about here.

how could you even use mysql for realtime tick capture and analysis. using sql would be near impossible when compared to using simple vectors.

 

[dcraig]

Quote from rosy2:

how could you even use mysql for realtime tick capture and analysis. using sql would be near impossible when compared to using simple vectors.

Not analysis - just capture and storage. Read time series into memory for analysis - into vectors or arrays or whatever.

 

[ktmexc20]

For those interested:


C# Prototype Release
====================

We are pleased to announce the prototype release of a C# API for HDF5.
It consists of C# wrappers for the HDF5 C library for the .NET environment.

In this first prototype, wrappers are implemented for many of the basic C
functions, and a detailed example is included. A future prototype will
provide a comprehensive set of C# wrappers for the HDF5 C library.

This release is available, at:

ftp://ftp.ncsa.uiuc.edu/HDF/pub/outgoing/hdf5/Csharp-prototype/HDF5dotNet.zip

Once you have downloaded and uncompressed the file, you can access
the documentation from the following location:

.\HDF5DotNet\HDF5DotNet\doc\output\HDF5DotNet.chm (double-click on the file)

The list of currently implemented wrappers along with general instructions
for using the wrappers is provided in the release notes at:

ftp://ftp.ncsa.uiuc.edu/HDF/pub/outgoing/hdf5/Csharp-prototype/PrereleaseNotes.txt

 

[txuk]

Quote from ktmexc20: For those interested:

I've been waiting for this to be released before looking into HDF5. Thanks for the notice.

 

[ktmexc20]

Quote from txuk:

I've been waiting for this to be released before looking into HDF5. Thanks for the notice.

Your welcome.
Always glad to help when I can.

 

[nitro]

Another intersting appproach that has many advantages. The question is how efficient is it:

http://www.hibernate.org/

This may be the best compromise to them all..

nitro

 

[txuk]

Quote from nitro: The question is how efficient is it.

That is the key question. I only took a brief look at the site, but if the underlying data store is relational then it seems the primary advantage of this product is ease of coding (i.e. it handles the object to relational mapping for you), but would still have the performance disadvantages of writing ticks directly to a RDB. I did follow your link to db4o and I quite like the sounds of it. They make a compelling argument against object-to-relational tools such as this. Thanks for posting the links.

 

[rosy2]

Quote from nitro:

Another intersting appproach that has many advantages. The question is how efficient is it:

http://www.hibernate.org/

This may be the best compromise to them all..

nitro

hibernate caches objects. specifically java objects. are you proposing to store your ticks or aggregation of ticks in java class and use hibernate? I think the hdf5 implementation is the best for no money and kx if you want to spend money.