You are using an out of date browser. It may not display this or other websites correctly.
You should upgrade or use an alternative browser.
You should upgrade or use an alternative browser.
db40 - using it to store historical prices
- Thread starter Batman28
- Start date
May I also suggest looking into the following:
Berkeley DB
Tokyo Cabinet
Redis
MongoDB
AllegroCache
We have tried all of the above and use a combination of them. Performance is obviously much better than what a relational database can provide (Redis is pretty fast!, 110000 SETs/second, 81000 GETs/second in an entry level Linux box.), but you need to layer the query logic into your application.
Berkeley DB
Tokyo Cabinet
Redis
MongoDB
AllegroCache
We have tried all of the above and use a combination of them. Performance is obviously much better than what a relational database can provide (Redis is pretty fast!, 110000 SETs/second, 81000 GETs/second in an entry level Linux box.), but you need to layer the query logic into your application.
Quote from loltrader:
May I also suggest looking into the following:
Berkeley DB
Tokyo Cabinet
Redis
MongoDB
AllegroCache
We have tried all of the above and use a combination of them. Performance is obviously much better than what a relational database can provide (Redis is pretty fast!, 110000 SETs/second, 81000 GETs/second in an entry level Linux box.), but you need to layer the query logic into your application.
thanks alot - I hadn't heard of any of the above will certainly look into it,
but have you used db40 yourself?
when you use a combination of the above - which do you use for back-testing or storing historical data - you don't use all for this purpose (not practical is it?)
many thanks!
Quote from Batman28:
thanks alot - I hadn't heard of any of the above will certainly look into it,
but have you used db40 yourself?
when you use a combination of the above - which do you use for back-testing or storing historical data - you don't use all for this purpose (not practical is it?)
many thanks!
I work at a shop where we don't use java/.net at all, but I have used it for a few personal projects. It will definitely speed up your development time, but the performance isn't much better than a tuned MySQL server, so it fully depends on your use case whether it makes sense to use db4o.
For legacy reasons we use BerkeleyDB for storing all market data and have layered a db query interface on top of it. Most of the projects I listed are fairly new and we're slowly pushing them into the infrastructure and phasing out BerkeleyDB. When we need full ACID compliance, we use PostgreSQL. We have a pretty large investment in Common Lisp code and AllegroCache is just awesome.
I use db4o and love it. But its like everything else, it really depends on what you want to use it to do. I use it as a data store for a pure client app, and it can easily add about 200,000 ticks worth of data in a second or two. Retreiving the data can be just as quick, depending on if you are keeping weak references in memory or not. But again, what do you want to do with it and I may be able to offer better advice.
- jeff
- jeff
Quote from jfilla:
I use db4o and love it. But its like everything else, it really depends on what you want to use it to do. I use it as a data store for a pure client app, and it can easily add about 200,000 ticks worth of data in a second or two. Retreiving the data can be just as quick, depending on if you are keeping weak references in memory or not. But again, what do you want to do with it and I may be able to offer better advice.
- jeff
Thanks alot Jeff,
I simply want to use a database to be used for back-testing so I will be retrieving historical data from it. probably looping through history of multiple instruments, say maybe 50 stocks or more. it will be mainly daily prices but would also like to have something that won't be limited if I choose to store more higher frequency data (say 3 years of 1-min data for example). so don't want to hit a brick in 3 years. I would also like to use this db. to record live data for such frequencies.
at the moment I'm thinking of sql server since I know it best, but I'm looking at others and I'm prepared to put in the effort to learn to use them if the advantages are significant. I'm not interested in crazy commercial dbs like kdb+ etc.
one distinct requirement I have is to be able query the data, say eg. what was price of x at datetime 123. this is simple in sql but do databases like hdf5 or db40 support it? I've given up on hd50 but db40 looks really easy to use.
thanks and appreciate your input
Quote from jfilla:
I use db4o and love it. But its like everything else, it really depends on what you want to use it to do. I use it as a data store for a pure client app, and it can easily add about 200,000 ticks worth of data in a second or two. Retreiving the data can be just as quick, depending on if you are keeping weak references in memory or not. But again, what do you want to do with it and I may be able to offer better advice.
- jeff
You're storing 100k records per second in db4o? How do you manage that?
i just had a look seems to be just a storage tool? you use it?
haven't come across it can't be that adequate for what i need - i must've heard it by now if it was used in this space..
I'm lookin for a good database and can't see anything yet comparable to sql server express (price, functionality, query support etc)
Quote from Batman28:
i just had a look seems to be just a storage tool? you use it?
haven't come across it can't be that adequate for what i need - i must've heard it by now if it was used in this space..
I'm lookin for a good database and can't see anything yet comparable to sql server express (price, functionality, query support etc)
It sits on top of MySQL & is the community version of their commercial software:
http://www.infobright.com/Products/Features/
http://www.infobright.com/Customers_Partners/Customers/
I currently use it for some trade data storage and am considering switching another .Net project over to that the from MS SQL as we're running into limitations with that.