Quote from hftvol:
I use ZeroMQ and I transport about 16million 16bit messages per second over tcp ports.
Quote from hftvol:
doing IPC in-memory is a VERY BAD IDEA. You restrict yourself to the processes on the same machine. What is the point of IPC other than just splitting up processes? They still share the same resources: Memory, threads, CPU cores. No point!!! Memory mapping has its applications but I do not see the point in using it for trading framework architectures. Someone with knowledge of how to program concurrently and async will beat the crap out of any app that achieves the same naturally through segregated processes as long as resources are constrained by same-machine-hardware.
Instead make IPC communicate over sockets. TCP is fine but other sockets are also used. UDP is a bad idea because its not reliable albeit faster. I use ZeroMQ and I transport about 16million 16bit messages per second over tcp ports. The advantage here is that you can fire to another machine and forget. For example you can have a dedicated machine that handles all logging of quote, trade, order data for later analysis without impacting the actual machine on which your execution module runs. I wrote a wrapper around ZeroMQ which can pub/sub, request/reply, filter messages, all that in a brokerless environment.
Quote from vincegata:
So it was you who brought up ZeroMQ then I got it wrong about 2rosy.
I found an article where someone compared the performance of OpenDDS vs ZeroMQ vs Boost. He claims ZeroMQ latency 170 microseconds, which is ~5882 messages per second. Am I reading it wrong or why do you think there is such a difference between his and your results?
http://mnb.ociweb.com/mnb/MiddlewareNewsBrief-201004.html
Quote from hft_boy:
Thanks for the feedback. The words you mentioned -- task processing, async programming and actor model -- to my mind are all just frameworks or ways of setting up your APIs. And of course it's important to set up APIs in a nice way. But that has nothing to do with their implementation, which is what I'm dealing with here.
As I said, sockets are one way to go, and they are very elegant. ZeroMQ uses TCP and Unix domain sockets underneath, as far as I can tell. Which can be plenty fast. I am skeptical of 16M msg / s over TCP unless you have TCP_NODELAY turned off, in which case latency might be pretty high.
At a high level, shared memory mapping isn't really any different from TCP or Unix domain sockets, except that it's all handled in user space (and it's faster). I also said that the hard part is writing a nice API to this thing so it looks like a socket, and so the calls are all agnostic as to whether it's another process on the same machine or on another machine.
Cheers,
Quote from hftvol:
At a high level, shared memory mapping isn't really any different from TCP or Unix domain sockets???
Excuse me? The two could not be more different. What are you talking about?
By the way, in case you did not get my point: IPC messaging is utterly useless when being used on the same machine. Then memory mapping is preferable. IPC really makes sense when you send data between different machines over the network sockets.
Not sure what was so confusing in my previous post.
And I stand by my 16m/second performance numbers using ZeroMQ .Net libraries and C# on .Net 4.0. The performance is based on sending messages IPC over tcp albeit on the same machine (which can immediately be changed by registering the ZMQ sockets on different ports and hostname. By the way this performance includes the serialization and deserialization of 16byte messages.
Quote from hftvol:
it was me and ZeroMq also supports UDP and multiplex though its not the recommended way in trading application where you are cannot afford to lose messages.
Quote from SeventhCereal:
who loses messages? maybe on your $150 netgear router at target. lol. Get the infiniband switch and colo on every exchange. Otherwise don't call yourself HFT. if you still get dropped packets, learn to program.
Quote from hftvol:
just 30 seconds skimming over your past posts shows you try to pee on every pole there is. I have no interest in wasting my time to converse with people like you. Make your case and start making sense with facts otherwise consider yourself on my ignore list.
Quote from jtrader33:
Would one of you guys mind giving an explanation of what ZeroMq does and how it's used in an automated system that a fourth grader (me) can understand?