You are using an out of date browser. It may not display this or other websites correctly.
You should upgrade or use an alternative browser.
You should upgrade or use an alternative browser.
Rithmic API for Python
- Thread starter IAS_LLC
- Start date
I’m wrestling with a similar thing conceptually so I’m really interested in your implementation. Apologies if my answers lie somewhere above.
What I want: high level APIs that abstract away the particulars of any single vendors API in my language of choice.
What I’ve come up with: Using my limited/developing engineering toolset the best approach I have found is wrapping each vendor SDK in a web server that exposes their API via REST. So the vendor APIs still have different method signatures but at least there’s some uniform way I can call them from outside of that particular C#/C++ process. Note that I haven’t even made it to the part where I need to forward the streaming data back to the calling context. I just have mocks used for testing that simulate authenticating and instantiating. So we may yet complicate further by using a message bus to handle the actual delivery of data from vendor to the calling context because I’m not sure REST is gonna work there. Trying not to put the horse before the carriage...may end up going entirely message based if half of it is already on a message bus but I’m hoping to not need to do any of this so let’s table it.
Why this approach is bad: I haven’t even made it far enough to determine if it’s my own incompetence or me putting myself in a microservice hell, but this has been extremely difficult to dockerize and deploy. Too many containers being built too many different ways. Even while using gradle to abstract away the different build tool chains I’m still in a place where integration testing is tough.
Would really like to find some way to just magically embed the vendors SDK calls into my calling language of choices runtime, e.g. use JNI from java for example to call the C++ thus freeing me from this microservice debacle, but will that work? I don’t really know how these embedded calls work, but you can’t just call the C++ statically, obviously, right? Your vendor SDK process has some state—needs to know you’re authenticated, what you’re subscribed to, etc. Is there a way I can somehow do these embedded calls and have the vendor SDKs state preserved between calls within one container? Thus I don’t need to have a microservice deployment model that spans multiple languages.
Apologies if that’s unclear, feels very much like I’m attacking the problem incorrectly...
Also would be very interested in hearing more about this R | Protocol. Sounds like it solves the same problem.
EDIT: also, great thread, thanks @IAS_LLC!
What I want: high level APIs that abstract away the particulars of any single vendors API in my language of choice.
What I’ve come up with: Using my limited/developing engineering toolset the best approach I have found is wrapping each vendor SDK in a web server that exposes their API via REST. So the vendor APIs still have different method signatures but at least there’s some uniform way I can call them from outside of that particular C#/C++ process. Note that I haven’t even made it to the part where I need to forward the streaming data back to the calling context. I just have mocks used for testing that simulate authenticating and instantiating. So we may yet complicate further by using a message bus to handle the actual delivery of data from vendor to the calling context because I’m not sure REST is gonna work there. Trying not to put the horse before the carriage...may end up going entirely message based if half of it is already on a message bus but I’m hoping to not need to do any of this so let’s table it.
Why this approach is bad: I haven’t even made it far enough to determine if it’s my own incompetence or me putting myself in a microservice hell, but this has been extremely difficult to dockerize and deploy. Too many containers being built too many different ways. Even while using gradle to abstract away the different build tool chains I’m still in a place where integration testing is tough.
Would really like to find some way to just magically embed the vendors SDK calls into my calling language of choices runtime, e.g. use JNI from java for example to call the C++ thus freeing me from this microservice debacle, but will that work? I don’t really know how these embedded calls work, but you can’t just call the C++ statically, obviously, right? Your vendor SDK process has some state—needs to know you’re authenticated, what you’re subscribed to, etc. Is there a way I can somehow do these embedded calls and have the vendor SDKs state preserved between calls within one container? Thus I don’t need to have a microservice deployment model that spans multiple languages.
Apologies if that’s unclear, feels very much like I’m attacking the problem incorrectly...
Also would be very interested in hearing more about this R | Protocol. Sounds like it solves the same problem.
EDIT: also, great thread, thanks @IAS_LLC!
Trying not to put the horse before the carriage...may end up going entirely message based if half of it is already on a message bus but I’m hoping to not need to do any of this so let’s table it.
You'll have to do both since you've already started w/ REST. I avoided REST and went straight to using ZMQ (not a message bus) and I think it'll work just fine once I get more comfortable with it.
I believe taking the REST out will be reasonably easy. I was already pretty certain I was going to because of another pain point of the aforementioned integration testing struggle: Message based gives you service discovery/location for free in my particular use case. Not a big deal when you’re in local but as soon as you try to run it on a CI server you run into all these “is the other service up? Where is it?” issues.You'll have to do both since you've already started w/ REST. I avoided REST and went straight to using ZMQ (not a message bus) and I think it'll work just fine once I get more comfortable with it.
I believe taking the REST out will be reasonably easy. I was already pretty certain I was going to because of another pain point of the aforementioned integration testing struggle: Message based gives you service discovery/location for free in my particular use case. Not a big deal when you’re in local but as soon as you try to run it on a CI server you run into all these “is the other service up? Where is it?” issues.
You pay a lot for an Enterprise Message Bus. I didn't do an incredibly thorough investigation but 0mq was great for efficiency in my tests. Like millions of messages per second and the messaging is embedded in the application.
The other thing is that you can do is separate the discovery from the messaging which is what 0mq settled on:
http://czmq.zeromq.org/manual:zgossip
Haven't tried it, not a prediction, whatever other crazy things tom and murrayt say.
I believe taking the REST out will be reasonably easy. I was already pretty certain I was going to because of another pain point of the aforementioned integration testing struggle: Message based gives you service discovery/location for free in my particular use case. Not a big deal when you’re in local but as soon as you try to run it on a CI server you run into all these “is the other service up? Where is it?” issues.
Also, with respect to your CI problem, you could try and use terraform.
I believe taking the REST out will be reasonably easy. I was already pretty certain I was going to because of another pain point of the aforementioned integration testing struggle: Message based gives you service discovery/location for free in my particular use case. Not a big deal when you’re in local but as soon as you try to run it on a CI server you run into all these “is the other service up? Where is it?” issues.
1. I have played with HTTP, Websocket & Rest from native C++ apps; they are too heavy for the type of work you have in mind. You can use them to have some of http interface to toggle a few trading config flags or for something like "exit the market now", but you don't want to use them for the communication between your trading server and your OMS.
2. I am particularly not comfortable with distributed process (microservice ) approach unless there is no other way; debugging can become a nightmare. Stay away from dockers & containers if you can... they are great for internal build farms or cloud services where an army of dev ops are at a standby 24x7 to jump in and fix things if something goes wrong. you don't want to use them for production systems that are supposed to be running unattended for long periods of time. These docker guys keep changing code often; there is no guarantee that todays docker version is less stable than the one you get tomorrow ... its not production quality imho...
3. Keep things simple. Try to aim to have all the components in one process if you can. Debugging will be relatively simple.... otherwise, you need some kind of record/replay type of infrastructure to isolate each process which can be done.... but a matter of choice
my 2 cents... hope this helps.
Last edited:
1. I have played with HTTP, Websocket & Rest from native C++ apps; they are too heavy for the type of work you have in mind. You can use them to have some of http interface to toggle a few trading config flags or for something like "exit the market now", but you don't want to use them for the communication between your trading server and your OMS.
2. I am particularly not comfortable with distributed process (microservice ) approach unless there is no other way; debugging can become a nightmare. Stay away from dockers & containers if you can... they are great for internal build farms or cloud services where an army of dev ops are at a standby 24x7 to jump in and fix things if something goes wrong. you don't want to use them for production systems that are supposed to be running unattended for long periods of time. These docker guys keep changing code often; there is no guarantee that todays docker version is less stable than the one you get tomorrow ... its not production quality imho...
3. Keep things simple. Try to aim to have all the components in one process if you can. Debugging will be relatively simple.... otherwise, you need some kind of record/replay type of infrastructure to isolate each process which is a lot of work.
my 2 cents... hope this helps.
I disagree that docker is not worth it though I run three docker container. The best thing is that it's predictable and 100% of my setup is in source control.
Will do some screen grabs tommorow. What was your failure point?
Figured it out. Really happy with it, thanks for telling me that you were able to make it work!
Figured it out. Really happy with it, thanks for telling me that you were able to make it work!
Sorry, dropped the ball on that one. When I'm in front of my workstation, I'm writing code and/or trading...try not to be elite-tradering. Had a lot of other stuff going on.
In a nut shell. I just show account balances,margins, open/closed P&L , Open Orders w some pedigree info, and open positions. And have a button that closes all open orders, flattens all positions, and blocks any new positions from being taken. All using IPyWidgets...pain in the ass to style, but easy enough to just get the information up there as they work via callback functions.