Dont invent wheel - use Oracle In-memory data base
http://www.oracle.com/technology/products/timesten/index.html
http://www.oracle.com/technology/products/timesten/index.html
You are using an out of date browser. It may not display this or other websites correctly.
You should upgrade or use an alternative browser.
You should upgrade or use an alternative browser.
Tick Database Implementations
- Thread starter fatrat
- Start date
The best thing is to NOT use an RMDBS when working with time series data. Instead, use a linear database with memory mapping. To locate a record you can triangulate based on the date/time. For example, see RMD Server: <a href="http://www.modulusfe.com/rmdserver/">www.modulusfe.com/rmdserver</a>. It can compress ticks into any bar format and broadcast data to clients in real time. You can set permissions for each user, set entitlements, etc. RMD Server is more for ticker plants like eSignal, DTN, etc. But if you just need something simple for a Desktop solution, I recommend that you write a large linear memory mapped file and cache the file offset using __int64 each time you read/write. Get a fast, cheap sata drive such as the WD Raptor 10k. Nothing is faster.
Richard
Richard
Here's what I use. It's about as reasonably low-level as you can go and still allow data storage to be seamlessly transferable to various arches.
Also check out the new packet-table interface for immediate storage of exceptionally fast streaming data without the need for queuing.
HDF5
It's openSource written in C and it's has several other language api-s.
-kt
Also check out the new packet-table interface for immediate storage of exceptionally fast streaming data without the need for queuing.
HDF5
It's openSource written in C and it's has several other language api-s.
-kt
There are (within database) APIs available for storing large amounts of data quickly to both MSSQL and Oracle. My experience is that with a decent SCSI drive you can easily get MSSQL to store 3000+ ticks per second.
More often than not - when developing tick databases it is worth checking if the speed is being limited by the HD - usually it is.
More often than not - when developing tick databases it is worth checking if the speed is being limited by the HD - usually it is.
Yes but then later try to do a select on a certain span of data or request bar history and you'll see the differences in linear databases such as SunGard's data server, RMD Server, HDF5, etc. and RDBMs like SQL Server, Oracle, Access, etc. The problem is in retrieving the data. RDBMs are not optimized for retrieving time series data.
Richard
Richard
I would say that if it suitable to your app's runtime efficiency needs, then all the power to ya. If not, then you gotta do what you gotta do.Quote from cashcow:
The original question was for "market analysis" - as SQL Server contains analysis tools (especially geared to time-series data) I would still deem it a good choice.
Had the system geared towards streaming data to clients, then I agree, a linear database would be better suited.
-kt
Hi Rosy,Quote from rosy2:
ktmexc20...
are you using HDF5 for storage and backtesting pruposes
Yes, I use it for all of my data storage needs, for the reasons mentioned in my previous post.
or do you use it in realtime?
Yes, that's where it shines. It has, in recent versions, something called "packet tables' for capturing and storing extremely fast streaming data into a formated structure without the need for using queues or the like.
I am looking into using it via python's pytable module. so far it looks good and easy to work with. Have you had any problems so far?
I use to use Pytables when I was working in Python and I think it's excellent for it's seamless integration from the HDF5 backend to Python, Numpy, and Matplotlib.. the combination is very good for crunching numbers considering that HDF5 and Numpy are embedded "C". Pytables does not implement that "packet table" interface yet, but you can safely do what you need to do by using queues when capturing streaming data.
-kt