ZMAPI - open source vendor agnostic financial API

[sjlnk]

Hello everyone,

I have been thinking of building open source vendor agnostic financial API for market data and order execution for about a year or two now. I finally decided to do it. The result is ZMAPI. It has fully functional market data (MD) support as of this moment. I have programmed three different connector modules to connect with three different vendors: Interactive Brokers TWS API, CTS T4 API and Bitstamp API. New connector can be added as required and I encourage people to contribute more connectors as the demand for them arise.

Account and execution side (AC) will be built in the future. I haven't started working with that yet and I'm now trying to come up with a good architecture. Any help with the architecture design will be very welcome.

Features:

- uniform API
- combining the best sides of all the vendors, abstracting away poor architecture
- modular design, middleware modules are used to provide uniformity
- vendor agnostic, no need to marry one vendor
- cross platform
- free
- open-source
- ØMQ is used for messaging
- language agnostic, works with every programming language

ZMAPI GitHub Page

ZMAPI Wiki

Bitstamp Tutorial

More tutorials coming soon...

----------

Pull requests, new ideas and comments are very much appreciated. Architecture can still be changed if there is a need as there are no real users yet, it's a brand new API. I'm open to suggestions.

Also looking to have some dedicated developers in the ZMAPI organization that will form the core team. Let me know if you have programming skills and interest in being part of this.

 

[globalarbtrader]

Hello everyone,

I have been thinking of building open source vendor agnostic financial API for market data and order execution for about a year or two now. I finally decided to do it. The result is ZMAPI. It has fully functional market data (MD) support as of this moment. I have programmed three different connector modules to connect with three different vendors: Interactive Brokers TWS API, CTS T4 API and Bitstamp API. New connector can be added as required and I encourage people to contribute more connectors as the demand for them arise.

Account and execution side (AC) will be built in the future. I haven't started working with that yet and I'm now trying to come up with a good architecture. Any help with the architecture design will be very welcome.

Features:

- uniform API
- combining the best sides of all the vendors, abstracting away poor architecture
- modular design, middleware modules are used to provide uniformity
- vendor agnostic, no need to marry one vendor
- cross platform
- free
- open-source
- ØMQ is used for messaging

ZMAPI GitHub Page

ZMAPI Wiki

Bitstamp Tutorial

More tutorials coming soon...

----------

Pull requests, new ideas and comments are very much appreciated. Architecture can still be changed if there is a need as there are no real users yet, it's a brand new API. I'm open to suggestions.

Also looking to have some dedicated developers in the ZMAPI organization that will form the core team. Let me know if you have programming skills and interest in being part of this.

Which language is this?

GAT

 

[sjlnk]

Which language is this?

GAT

Hi globalarbtrader,

Good question! I forgot to mention specifically, it's language agnostic. You can use it with any language. You just need ZMQ bindings, which are available for pretty much every language.

 

[DaveV]

First, congrats on your first release of a very ambitious project.
However, I cannot see any serious trader using it. Unless you plan to eventually charge for the API, I suspect that eventually you will tire of maintaining the code, and anyone who coded to the API will be stuck with no future updates.

 

[sjlnk]

DaveV, I'm a serious trader and I see this as the best option for any serious trader with programming background. More non-programmer friendly trader applications will be available later. This API can be used to build any kind of tools that needs access to real time financial data or execution -- not only trading tools.

Open source projects don't necessarily need funding to be able to work. I'm publishing this work in an attempt to create a community around this so that eventually it may continue evolving with or without my work contribution. That being said, I don't plan on moving on to another projects any time soon.

Closed source proprietary modules can be built on top of ZMAPI and those can be charged for those wishing to do so.

 

[DaveV]

DaveV, I'm a serious trader and I see this as the best option for any serious trader with programming background ... Closed source proprietary modules can be built on top of ZMAPI and those can be charged for those wishing to do so.

My background is very similar to yours. Years of programming experience, including writing part of the Nasdaq platform. Now I exclusively trade, using a system that I wrote myself. I currently use the IB API, but when I started trading 8 years ago I used an open source .Net API that connected to IB. Sure enough after a about year, the open source API updates became less and less frequent then stopped altogether. Thank goodness about that time IB started supporting .Net and C#, so I switched to the IB API. Took me a week to rewrite my software.

Yes I could have made the changes to the open source API myself, but I am in the business of trading, not maintaining APIs.

I truly wish you good luck with your API, but as we both know, there is a vast difference in effort between building software that you use for yourself vs producing software that other people will use.

 

[sjlnk]

My background is very similar to yours. Years of programming experience, including writing part of the Nasdaq platform. Now I exclusively trade, using a system that I wrote myself. I currently use the IB API, but when I started trading 8 years ago I used an open source .Net API that connected to IB. Sure enough after a about year, the open source API updates became less and less frequent then stopped altogether. Thank goodness about that time IB started supporting .Net and C#, so I switched to the IB API. Took me a week to rewrite my software.

Yes I could have made the changes to the open source API myself, but I am in the business of trading, not maintaining APIs.

I truly wish you good luck with your API, but as we both know, there is a vast difference in effort between building software that you use for yourself vs producing software that other people will use.

Your experience in the industry could prove valuable insight to ZMAPI development. I think I know the open source C# API for IB you mentioned about. I used it before as well. It was later replaced by IBs own API.

Frankly, the quality of IB TWS API libraries are quite bad. Threaded programming model with lots of locking is used when single-threaded coroutine based solution would make much more sense. That was the style of doing asynchronous programming 10 years ago but things have developed since then, IB API has not.

These official IB API libraries are open source so the socket communication protocol to communicate with IB TWS/Gateway is possible to deduce from the code. This is exactly what I did when I developed IBTWS connector for ZMAPI. The connector seems to be working very nicely with snapshot functionality available for free from ZMAPI middleware infrastructure. No locking is used, instead a single-threaded coroutine based programming model is utilized.

Now IB TWS API is accessible directly from any language using ZMAPI.

As for the maintenance, it's quite easy to maintain this code base because every module has very clearly defined, limited responsibility. Middleware modules don't really need to be updated virtually ever once they are working well and ZMAPI specs are not altered. They can be sealed in Docker containers. Connector modules may need updates from time to time since vendor APIs will change over time. Talking about IB TWS API socket communication specs, I don't think it will change much over time as it's already a very mature API. So maintaining that should be very easy. I'll be maintaining it personally for the foreseeable future as I'm dependent on it.

If you can afford the time, please have a look at the documentation. Perhaps you can try running the ibtws-acmd connector chain (with sub2snap and optionally ctl-frontend attached). I'll make a tutorial about that soon. I can make it sooner if interested.

 

[DaveV]

I agree that the IB API is not the most optimal. But it works, mostly, and the bugs that I know of, I have coded around. The bugs that I have reported to them they address for a couple weeks, ask for all sorts of screen shots and logs, then drop the matter without the bug being fixed. But they do produce regular updates and enhancements, and they have enough of a user base that the truly bad bugs get fixed.

Your API has piqued my interest, so I would love to download it and take a peek.
Probably in a couple weeks after this crazy market settles down. If you are a trader like me that that needs volatility, now is your moment.

 

[T0pH4t]

First off congrats on getting that much done You have definitely identified a common problem among data/broker apis. They all are different and some of them not very good. I do question how useful others will find it though. I'm going to guess that most people that would use this are writing their own platform/automation, but in that case they most certainly have already identified the same problems you have and have their own solution (tailored to their needs). The main concern I see with trying to create something general purpose like this is the extra overhead it could add to the whole process. My own platform also handles normalization, but it does so after normalizing it to a database so that the overhead of consuming that data is no different then if I went directly to the source. And maybe I missed it, but do you support direct streaming of data directly through the API? Also how do you handle the different amounts of information available between sources (ie the amount of info available for cryto is way more compared to equities)?

 

[sjlnk]

I agree that the IB API is not the most optimal. But it works, mostly, and the bugs that I know of, I have coded around. The bugs that I have reported to them they address for a couple weeks, ask for all sorts of screen shots and logs, then drop the matter without the bug being fixed. But they do produce regular updates and enhancements, and they have enough of a user base that the truly bad bugs get fixed.

Your API has piqued my interest, so I would love to download it and take a peek.
Probably in a couple weeks after this crazy market settles down. If you are a trader like me that that needs volatility, now is your moment.

Indeed often easier to roll your own solution than to work with what you get from IB out of the box. I guess they are just too big to care about individual customer's needs.

I'm happy to hear you are considering giving ZMAPI a go! I'll make the IB specific tutorial as promised ASAP.