How to build an automated system

[hft_boy]

Quote from hftvol:

At a high level, shared memory mapping isn't really any different from TCP or Unix domain sockets???

Excuse me? The two could not be more different. What are you talking about?

By the way, in case you did not get my point: IPC messaging is utterly useless when being used on the same machine. Then memory mapping is preferable. IPC really makes sense when you send data between different machines over the network sockets.
Not sure what was so confusing in my previous post.

And I stand by my 16m/second performance numbers using ZeroMQ .Net libraries and C# on .Net 4.0. The performance is based on sending messages IPC over tcp albeit on the same machine (which can immediately be changed by registering the ZMQ sockets on different ports and hostname. By the way this performance includes the serialization and deserialization of 16byte messages.

Okay, I got confused. Obviously shared memory doesn't work between different computers because it's impossible. I meant that when they are on the same machine they are not that different; eventually there is going to be some shared memory somewhere whether it's in the kernel or in userland. In the Linux kernel they use these sk_buff things which are the shared memory.

Also I'm interested in the code backing this benchmark of yours; do you want to share the code here or send via PM? I just can't believe you're not sacrificing latency for throughput; on my machine going through the networking stack is about 10 microseconds as measured by half of round trip ping time.

 

[hft_boy]

Quote from jtrader33:

Would one of you guys mind giving an explanation of what ZeroMq does and how it's used in an automated system that a fourth grader (me) can understand?

I haven't used it, but what I gather is that it's an abstraction layer over sending messages via sockets.

 

[vincegata]

Quote from hftvol:

you are confusing latency with throughput. Latency is about how long it takes to get data from A->B. Throughput is about how many items you can send from A-> B per second. I stand by my throughput claims. You can actually squeeze even higher numbers out of ZeroMQ if your messages are <16 bytes (or if you come up with a better approach than I have implemented ;-)

Throughput is measured in MB/s that's why I did not get you, that is 16 million messages by 16 bytes per second equals 256MB/s.

Do you know your latency?

 

[Jack_Larkin]

I wish we saw more fights and standoffs like this around ET, not only are they hella interesting but I'm actually learning something by reading through them.

 

[hft_boy]

Quote from SeventhCereal:

who loses messages? maybe on your $150 netgear router at target. lol. Get the infiniband switch and colo on every exchange. Otherwise don't call yourself HFT. if you still get dropped packets, learn to program.

Stop trolling. Your advice is not valuable; from what I've seen you post things with no content whatsoever. Although it does sound like you have a chip on your shoulder.

 

[jtrader33]

Quote from NetTecture:

It is a message bus / pping system. The idea core being that you can not do everything in one process / machine, so passing messages between a communication / dispather system (that talks upstream to the exchange) and machines running the processing loops (possibly 1 core per algo, hot - i.e. in a spinning loop) is needed, and that must be FAST.

ZeroMQ - MQ standing for Message Queue - is a middleware for passing messages.

Thanks for that. Still not sure I'm quite grasping its value, however. Perhaps using my caveman ATS as an example, someone could point out what ZeroMQ does differently and/or better.

I have a thread that does nothing but read sockets messages coming in from my data feed and pass them to one of 8 queues (say one queue per algo). Each of those queues has its own consumer thread which will pull a quote/trade/etc message off the queue and execute the algo logic -- if warranted, it sends an order messages to the single "order processing queue". This queue also has its own consumer thread which will run some risk checks and send an order to my broker if appropriate.

Which, if any, of these elements would ZeroMQ replace?

Again, appreciate any input...just trying to learn some stuff.

 

[hft_boy]

Quote from jtrader33:

Which, if any, of these elements would ZeroMQ replace?

Again, appreciate any input...just trying to learn some stuff.

Check out some samples here

 

[jtrader33]

Quote from hft_boy:

it's an abstraction layer over sending messages via sockets.

Thanks for trying...but what fourth grader knows what an abstraction layer is? Certainly not this one...ha

 

[NetTecture]

Quote from jtrader33:

Thanks for that. Still not sure I'm quite grasping its value, however. Perhaps using my caveman ATS as an example, someone could point out what ZeroMQ does differently and/or better.

I have a thread that does nothing but read sockets messages coming in from my data feed and pass them to one of 8 queues (say one queue per algo). Each of those queues has its own consumer thread which will pull a quote/trade/etc message off the queue and execute the algo logic -- if warranted, it sends an order messages to the single "order processing queue". This queue also has its own consumer thread which will run some risk checks and send an order to my broker if appropriate.

Which, if any, of these elements would ZeroMQ replace?

Again, appreciate any input...just trying to learn some stuff.

The whole queue. Your dispather would get messages from data feed / exhcnage, and put hem into MQ.

What MQ does is abstract the whole comunication - between languages, between machines. A lot of handcoding, when to be done properly. On your one machine system that gives little, try handcoding a robust messge bus for 30 machines.

 

[hftvol]

Lol, let me CITE your posts that you claim were "generous", valuable advise in this thread:

(1)
02-07-13 07:36 PM
welcome to reality, where things are complex.

(2)
02-07-13 07:38 PM
o rly? stop firing off buzzwords like its an exam. boost has been around for over 5 years now. fail.

(3)
02-05-13 09:21 PM
was entertaining listening to the ramblings of a crazy man

This is all you had to contribute in this thread. Moron!!!

Quote from SeventhCereal:

go ahead, run and put me on ignore, you are of little significance to me. I only state facts so you are only running from the truth. frankly, its not my job to go into details and give out any secrets to the business which may affect the firm I work for's profitability or my own future profitability.

I think everything I've said has been extremely generous, general but valuable advice that if pondered on can save years of time going in circles.