Custom Trading Platform

[amazingIndustry]

I think TPL Dataflow and whatever messaging framework you use are unrelated. You can use TPL in combination with Rabbit or Zero. RabbitMQ and Zero MQ are just the messaging frameworks which supply different transport layers, I only rely on them for sending messages between my different out-of-process system modules.

TPL Dataflow is a framework that structures the flow and transformation of data in a larger architecture in very efficient and concurrent ways. I use TPL mostly for the merge/sort (the part where historical data feeds of different symbols are merged and then sorted in order to generate a time-stamp-ordered data stream) and of course each of my strategies runs on top of data blocks. But I also use it right behind the ZeroMQ message retrieval process: Messages are passed to the data block responsible for the particular message type so that ZeroMQ can get back at doing its job, listening to new incoming messages while the incoming message is being processed. I run on a fully concurrent framework.

Quote from januson:

Hi amazingIndustry

I'm in the process of building my own trading application as well, I've developed a lot of different small applications exactly as you did, but unfortunately my work has stalled the last couple of month ... I just don't feel for it at the moment, but I can feel my mojo is returning

anyway.... I did see your comments on TPL Dataflow, where did you implement that in your trading application? I was thinking about a local message queue in each application, because ZeroMQ is brokerless? is that correct?
My initial thought was to use RabbitMQ, but ZeroMQ looks more promising when utilizing TPL Dataflow, what do you think?

 

[naifwonder]

Quote from amazingIndustry:

* you argue in favor of those retail platforms now for some time but each time you say "they are ideal for certain types of projects" you fail to elaborate. Which projects really?

* I was never debunking those packages for lack of advanced features. I am debunking them for not even performing at the absolute basic level: Correct back test statistics, not able to run a reliable broker connection to one single broker, the broker they all love to advertise with, IB. What is any test rig good for if the strategy cannot be traded live? It cannot, because none of the mentioned packages seem to be able to keep one single broker connection alive for any reasonable amount of time. Please google or search on this forum and show me one retail product where users are not constantly reporting bugs and errors when it comes to fills, cancellations, trade closures. Please, just one...

Please keep in mind that I am not exclusively arguing in favor of retail platforms. I am actually the one that started this thread about building a custom platform in the first place. I am pointing out that for certain applications, retail trading platforms are sufficient. I have had excellent experiences working with Ninjatrader to work as the glue between my custom projects and brokers. It gives me a secondary, third-party verified interface to verify my activity with brokers.I have had few problems with connection stability. Also, in the rare event that I use a chart, Ninjatrader has also been nice. You are right though about back-testing - that was a mediocre experience with all the pre-built solutions I used. However, even if a retail platform could support the level of back testing a custom platform could (for instance, tick by tick data for years of data for multiple instruments), chances are that VERY few people would have the hardware to run this software. Heck, I am running on a cluster with over 90 gigs of ram and solid state drives in RAID 10.

Chances are, your custom platform has more bugs than many of these retail platforms. However, the bug-scenarios do not arise in live-use because you debug the platform itself as you test the system and bugs arise. Retail platforms have to be adaptive to wider use-cases. Now, you might be saying that a good custom platform is more robust then a retail platform, and yes, it probably is. However, the "realized robustness", is probably significantly less. More unambiguously speaking, your custom platform can probably be extended to do everything that a retail platform does, but does not actually do it. Instead, it was built to accomplish certain tasks very well, and future upgrades can be made in a modular fashion. As you introduce those future modules - applied to as wide a range of use-cases as a retail platform - there will likely be just as many bugs if not more compared to a retail platform. But, those bugs will be addressed ahead of launch of tested properly because you are both the system and platform developer (another reason why I chose to build my own platform). It is likely that your custom platform is not more stable due to a sounder architecture, but rather, the capacity to work on your architecture alongside the system development process. This is the case with nearly any custom solution across nearly any project when compared to premade commercial software - not just for finance.

Let's not get this mistaken: At the end of the day, I prefer a custom platform which is why I use one. Retail platforms certainly have a ton of room for improvement, but I have found cases (more specifically stated before) where they were sufficient.

 

[syswizard]

Quote from naifwonder:


Chances are, your custom platform has more bugs than many of these retail platforms.

Retail platforms certainly have a ton of room for improvement, but I have found cases (more specifically stated before) where they were sufficient.

Exactly...and for these 2 reasons above, I'd rather be trading with my system, applying new trading systems, and backtesting versus constantly debugging and tweaking my custom platform.

 

[januson]

Quote from naifwonder:

Chances are, your custom platform has more bugs than many of these retail platforms. However, the bug-scenarios do not arise in live-use because you debug the platform itself as you test the system and bugs arise.

It depends a lot on the developer, for me with 15 years of experience I would always develop my own trading software. I've tried 6-7 different platforms and each one had it's own oddities.

But for a beginner that totally underestimate such a big project and don't understand the basic design patterns + ignores the fine art of unit-testing, I would definitely argue to go after a retail platform.

 

[naifwonder]

Quote from januson:

It depends a lot on the developer, for me with 15 years of experience I would always develop my own trading software. I've tried 6-7 different platforms and each one had it's own oddities.

But for a beginner that totally underestimate such a big project and don't understand the basic design patterns + ignores the fine art of unit-testing, I would definitely argue to go after a retail platform.

I think this sums it up well. Taking the custom route is a decision to be weighted carefully in terms of what benefits it poses - both short term and long term. Like other aspects of trading, it is an investment, and it must be assessed if it is the one with the highest return. For me, custom yielded the highest return. Heck, just the knowledge and insight I gained from building the platform was a great return on my time investment.

 

[Fxguru07]

What are the best Open SOurce platforms out there?

 

[vincegata]

Which message layer are you using? I decided to go out-of-process for each module. A central controller with Dashboard in WPF, a strategy module, a data source module, an OMS/PMS/RISK module, and an execution gateway. All independent applications communicate through a brokerless message library, ZeroMQ. What are you using? In that way I have a free choice of language (GUI in WPF/Silverlight), execution related code in C++, most other code in C#.

I know it's an old thread.

Could the poster, or maybe someone else, tell me in which of those modules the quotes are being accumulated?

The strategy module is certainly needs an access to the accumulated quotes to analyse them and to generate a buy/sell signal. If the quotes accumulation is done in other module, say in data source, then how strategy module accesses the accumulated quotes? ZeroMQ is adamant about not sharing state between the modules.

Thank you.