Any nice, simple message queue/event dispatcher for Python?

globalarbtrader said:
No, but I have considered changing my setup to a message based system (currently objects interact via databases), and when I evaluated the various options out there zero mq came out top.

GAT
More...

I would warn against this change because event based systems quickly becoming spaghetti becomes a concern. I recently changed an event based utility to one that was slightly less brittle. I'll post example code later, not at the desk rn.

For me this question is purely about having multiple, independent processes communicate for the sake of not only providing pricing data but also placing orders. The rest of it is generally covered by the above paradigm which is still event based at its core.
 
djames said:
Reids Streams https://redis.io/topics/streams-intro look nice and light.
Also this asyncio stream parser looks great https://github.com/robinhood/faust although it currently relies on Kafka as the message queue which seems a bit heavy. Let us know how you get on.
More...

Nice. I considered redis but last I recall there weren't any guarantees on data not being lost. 0mq doesn't address it either but not deliberately which means the intention is still to not lose data afaik.
 
ZeroMQ between processes is fast, easy to use.

Within one process with many threads, CSP (communicating sequential processes) works very well. They're built-in to the Go language and it's a library for Clojure.

I imagine someone has built the same type of thing for Python. In searching, you may see a reference to "channels" which can either block or buffer. It's like conveyor belt sharing among multiple threads: the threads putting info on "the belt" just do it and leave, doesn't have to care about "who" picks it off.

nooby, you've inspired me to build out my own trading software. My target language is a Lisp though. Racket (scheme) has a nice programming IDE (DrRacket), GUI library, great support group, documentation and can run on Windows, Linux and Mac OS. It's very fun to program in. I can always port it to Clojure or Common Lisp (e.g. sbcl) down the road if I need to.

I never had "fun" programming in Python. For me, after programming in C/C++/Java/C# for years on-end, the return to a Lisp language has become pure enjoyment.
 
Last edited:
SteveH said:
ZeroMQ between processes is fast, easy to use.

Within one process with many threads, CSP (communicating sequential processes) works very well. They're built-in to the Go language and it's a library for Clojure.

I imagine someone has built the same type of thing for Python. In searching, you may see a reference to "channels" which can either block or buffer. It's like conveyor belt sharing among multiple threads: the threads putting info on "the belt" just do it and leave, doesn't have to care about "who" picks it off.

nooby, you've inspired me to build out my own trading software. My target language is a Lisp though. Racket (scheme) has a nice programming IDE (DrRacket), GUI library, great support group, documentation and can run on Windows, Linux and Mac OS. It's very fun to program in. I can always port it to Clojure or Common Lisp down the road if I need to.

I never had "fun" programming in Python. For me, after programming in C/C++/Java/C# for years on end, the return to a Lisp language has become pure enjoyment.
More...

I am a Lisp (CL) fanatic as well but for me, the problem is having enough of the foundation available and rock solid so you can have the fun. Last I checked, even with Quicklisp, it's pretty difficult to get solid foundations. This makes compounding difficult. Racket does seem to be solid though I do not like Lisp-1s for some reason even though conceptually seems more elegant.

Using Python, the fun has translated from "having fun coding" to "having fun using coding to mine $$$". I wish I could do that with Lisp, but then I suspect it would change back to "having fun coding"!
 
globalarbtrader said:
No, but I have considered changing my setup to a message based system (currently objects interact via databases), and when I evaluated the various options out there zero mq came out top.

GAT
More...

Here is the event/message-based -> state based change I made for a tick-based trading system I want to develop. Each member in TradingState was previously represented event that was spread out all over the place, becoming spaghetti:


You can see that it obviously still uses events/messages under the hood but at the core of the (in memory) operation, there is a single source of truth (TradingState) which everything manipulates. The code doesn't show it but you can also subscribe to all levels of changes, so something like:

Code: 
model.onChange(lambda x: x.bar.volume,callback...)  # call call back when volume changes

is possible.
 
Last edited:
nooby_mcnoob said:
I am a Lisp (CL) fanatic as well but for me, the problem is having enough of the foundation available and rock solid so you can have the fun. Last I checked, even with Quicklisp, it's pretty difficult to get solid foundations. This makes compounding difficult. Racket does seem to be solid though I do not like Lisp-1s for some reason even though conceptually seems more elegant.

Using Python, the fun has translated from "having fun coding" to "having fun using coding to mine $$$". I wish I could do that with Lisp, but then I suspect it would change back to "having fun coding"!
More...

I'll keep you posted. The goal is always making $$$ from the market or I wouldn't fool with it. My stuff is more GUI-oriented than yours, running a trading strategy off of an intraday futures chart where the chart is highly interactive b/c I want fully-automated, semi-automated and discretionary choices. When you can code to the very core of just what you want to do, a lot of code bloat / edge cases that pro software is loaded with is not there anymore, getting in the way to slow things down (duh).

I don't need backtesting capabilities. I have yet to see trading strong trends not work (at least, the way I trade them). I know the animal I'm dealing with and it never changes in a way to lose a positive expectancy.

[I really like Clojure. But it's more geared towards backend dev and I'm not doing the GAT type stuff, need easier GUI lib to work with, no more than trading 4 symbols at once. Think of NT7 on an aggressive keto diet w/plenty of intermittent fasting: fast chart interaction, ability to code a strategy, etc. but it only has exactly what you want to trade your systems.]
 
Last edited:
SteveH said:
I'll keep you posted. The goal is always making $$$ from the market or I wouldn't fool with it. My stuff is more GUI-oriented than yours, running a trading strategy off of an intraday futures chart where the chart is highly interactive b/c I want fully-automated, semi-automated and discretionary choices. When you can code to the very core of just what you want to do, a lot of code bloat / edge cases that pro software is loaded with is not there anymore, getting in the way to slow things down (duh).

I don't need backtesting capabilities. I have yet to see trading strong trends not work (at least, the way I trade them). I know the animal I'm dealing with and it never changes in a way to lose a positive expectancy.

[I really like Clojure. But it's more geared towards backend dev and I'm not doing the GAT type stuff, need easier GUI lib to work with, no more than trading 4 symbols at once. Think of NT7 on an aggressive keto diet w/plenty of intermittent fasting: fast chart interaction, ability to code a strategy, etc. but it only has exactly what you want to trade your systems.]
More...

How to choose what to write your system in:

1. What do you know well?

That's it.

By the way, I am (currently) more GUI oriented as well but I use a lot of probabilities to decide which trades to take. Once I feel comfortable enough, I will automate the last part and it won't be a huge project because it'll just be replacing me reading some numbers and clicking a button.
 
nooby_mcnoob said:
I would warn against this change because event based systems quickly becoming spaghetti becomes a concern. I recently changed an event based utility to one that was slightly less brittle. I'll post example code later, not at the desk rn.

For me this question is purely about having multiple, independent processes communicate for the sake of not only providing pricing data but also placing orders. The rest of it is generally covered by the above paradigm which is still event based at its core.
More...

Oh yes I completely agree. I considered but did not go ahead with shifting to event driven. As yet another alternative I quite like the idea of FRP but when I last looked at this the python implementations were not very mature.

GAT
 
ZeroMQ is great and I use it extensively for PUB/SUB but, ironically, it's not technically a "queue". Whatever you publish with zmq is "essentially" gone if there aren't any active subscribers. This is the case within the context of other patterns as well, not just PUB/SUB. If you really need a persistent queue where one process can add to a list and other worker processes can pluck from the list, I'd suggest checking out Redis. It's really fast and has a simple blocking list so one process adds to it with something like r.lpush("SPY", 303.21) to put the last price of SPY into the left side of the queue and another process can use r.brpop("SPY") to get it. The process pushing into the queue doesn't block so it can keep adding as fast as it gets the data while the processing popping from the end will block if the queue is empty which is what you usually want.

There are many benchmarks to be found with a Google search but I've found Redis' performance to be top notch and that's running scanners against a dozen crypto exchanges and the IEX stock market feed so something like 28,000 different assets updating ohlcv in real time all on a single desktop computer. Never had a problem.

In short if I don't require persistence, I go with ZMQ, but if I need an actual job queue with multiple workers, I use Redis.
 
You must log in or register to reply here.
Back
Top