Java - Storing data in memory for post-runtime access

[quatron]

Quote from CalVolibrator:

I need more information from you to judge whether my performance is similar to what you are seeing but you can easily determine that your Redis instance is not as fast as it can be (possibly because of how you setup your config file, persistence can slow things down among a host of other reasons) when you consider that

* Redis : you serialize to binary and write to memory/ deserialize and read from memory through a very thin API layer.

vs

* csv : you parse data to string then you write line by line to a csv file/ you read line by line and then parse from string to whatever type.

I did not run those tests myself I just took the numbers from their site. I was looking for a ballpark number before doing my own testing. Looks like Redis stores data as strings so you will need to parse data anyway to get ints/doubles.
How fast is your Redis? How do you store data? Do you use sorted set where score is the time of a tick?

 

[CalVolibrator]

true, but the strings are binary safe, you can as well handle it as binary with string key look ups. So, there is no per se parsing between strings. A string is just defined as the format of choice to hold data, whether a byte array, string, integer, or image. If you just want to deal with binary data you just store as byte arrays and then deserialize yourself with protocol buffers which are a lot faster than most other serialization libraries.

Here is a basic intro to the data types:
http://redis.io/topics/data-types-intro

If you take a look at Booksleeve, the async C# API for Redis you find that it takes byte arrays as input, so serialization/deserialization is done on the user's side.

But again, I do not want to come across as the one who thinks Redis solves all problems or is the product of choice here, it is not. Binary flat files are the storage of choice here imho. And even if I did that on a server I would not use REST to serve the data. Even the slowest .Net Windows based tcp stack is faster in delivering the data than any REST API could ever accomplish.

Quote from quatron:

I did not run those tests myself I just took the numbers from their site. I was looking for a ballpark number before doing my own testing. Looks like Redis stores data as strings so you will need to parse data anyway to get ints/doubles.
How fast is your Redis? How do you store data? Do you use sorted set where score is the time of a tick?

 

[lwlee]

- "No longer advocate Redis?"

- not criticizing client/server?

Hmmm. Lol.

youporn? nah, hardcore is where it's at. kink.com. Now if they were using Redis, I would be impressed.

Yes, yes. We are all impressed with your incredible knowledge, KDB and such.

In honor of this discussion, Jtrader should christen his new client/server app as "RedisDB 2.0". Lol. Nah, that would be too arrogant.

Quote from CalVolibrator:

* not sure what you mean with data and app together. I wrote my own API that manages a full fledged flat file binary store. Nothing I ever tested, including KDB in memory, is faster than that. My data is on disk (SSD) and my access layer sits within a segregated library that can be referenced in order to access the functionality. That is the fasted way there is.

* Redis is used by the StackExchange network, possibly one of the websites with the most traffic worldwide, and how about Twitter? Ridis in both cases is one of the core components used for lookups. That you conveniently omit those and name a porn website shows your ignorance. (But hey I am sure that porn website also has tons of traffic, maybe by you as well? You sound very emphatic about that site).

* Do your own homework, its just a weblink away from your fingertips to find out what technology they use.

I do not advocate Redis, I merely say it could be used and I know from own usage that it is faster. And please drop out your comical suggestion of REST, really REST has no place in this discussion. Saying REST should be used to load time series data into a testing platform is just bizarre. There is a reason people serialize/deserialize to/from binary format for speed and not serving the data over a REST API.

I bow out of our exchange because I have spent enough time on this stuff, I wrote my own profiling and testing platform, live trading framework, feed handlers, fix gateways, optimizers, and yes, data handling to know what I am talking about. I tested about 10 different technologies and APIs before settling on a combination of binary flat file store and Redis. REST was not even among those 10. I am glad it works for you but it would be awefully slow for my needs. The OP needs to decide himself what works best for him.

Quote from CalVolibrator:

I criticized the use of REST API not that client/server relationships are involved. My opinion took everything into account, (a) lower throughput performance, (b) need for DTOs, (c) possible need for ORM,...REST is just not the right tool for the trade.

If I gave you this wrong impression then my wrong.

As you already checked things out with Redis you should be able to easily pull some performance benchmarks others ran.

 

[CalVolibrator]

I use redis only to do inter-proc communication and, as described, to quickly serve data that my app outputs and which I like to analyze in R.

Other than that I run my own binary data store with query algorithm that can lookup and load a requested data packet between start and end time stamp. The packet location can be determined in a matter of single digit milliseconds regardless of size of file. I store tick data depending on what data we talk about. For example, fx tick based data is stored in 16byte data structures, 8 bytes for the time stamp, 4 bytes each for bid and offer.

Quote from quatron:

I did not run those tests myself I just took the numbers from their site. I was looking for a ballpark number before doing my own testing. Looks like Redis stores data as strings so you will need to parse data anyway to get ints/doubles.
How fast is your Redis? How do you store data? Do you use sorted set where score is the time of a tick?

 

[TigerBalm]

+1 for redis. It's quite easy to get started with redis : http://try.redis.io/

It's quite adequate for your specific use-case.

 

[CalVolibrator]

Very simple to setup and beats the heck out of having to code DTOs, requests, replies to get a simple REST interface going.

Quote from TigerBalm:

+1 for redis. It's quite easy to get started with redis : http://try.redis.io/

It's quite adequate for your specific use-case.

 

[ET180]

Quote from CalVolibrator:

and with that we would be back to square one ;-)

Recommendation in order of preference:

(a) Binary flat file storage and management
(b) In memory database (in case all fits into memory), SUCH AS Redis

I am out of here. Good luck to the OP.

I use this for something similar.

http://code.google.com/p/kryo/

 

[CalVolibrator]

I am not too familiar what exists in Java but I guess your link points to an object serialization library for Java, correct? It can be used to serialize objects regardless of storage type (file, database, memory).

In the .Net world the most efficient libraries I found to be Protocol Buffers (protobuf-net) for binary serialization and the Service Stack serializers (json serializer). Not sure what is best in town in Java world...

Quote from ET180:

I use this for something similar.

http://code.google.com/p/kryo/