Building an automated, efficient, highly concurrent and resilient trading operation
- Thread starter wave
- Start date
Is this a joke? An absolute basic, nothing special, simplistic, not even well done summary by someone who signs responsible for "managing developer experiences". I flipped though the presentation but could not find anything that stood out.
What are your specific questions? Perhaps we can talk about a specific issue and dive in rather than talking about distributed system, or "ai", or databases.
I can't stand generalists whose job title is project xxx or those who work in xxx experience. There is a reason there is so little available content from bleeding edge traders and developers, they are too busy not giving talks and speeches. Shoot me down if you like but at least I offered to engage in a discussion on a specific issue regarding building an algorithmic trading architecture.
For example, I just finished testing a very powerful script engine that I wrote and added onto my trading architectural platform. The user can stack functions in any order in a simple string format and the system builds out a node tree structure of parent nodes on which this script depends, re-use and share existing nodes, and display point value results as well as chart time series that can contain millions of data points. Recalculation of the entire tree takes a few milliseconds for a medium complex tree while in the background data feed handlers keep on updating dozens if not hundreds of subscribed symbol series. I will use this for my own discretionary trading book, it allows me to quickly visualize, for example the rolling correlation between the moving average series which are in turn based on the return series of a tech stock, each, for example. I can express all that in a simple string that is less than 100 characters long. The outcome can be viewed in a table format as numeric value or on a chart that is updating in real time.
Let's talk specific issues not "distributed systems are great, and concurrency is an issue, and availability is important, bla bla". My only question is, do you actually want to participate in a detailed discussion or are you just excited about those glossy lunch break speeches?
What are your specific questions? Perhaps we can talk about a specific issue and dive in rather than talking about distributed system, or "ai", or databases.
I can't stand generalists whose job title is project xxx or those who work in xxx experience. There is a reason there is so little available content from bleeding edge traders and developers, they are too busy not giving talks and speeches. Shoot me down if you like but at least I offered to engage in a discussion on a specific issue regarding building an algorithmic trading architecture.
For example, I just finished testing a very powerful script engine that I wrote and added onto my trading architectural platform. The user can stack functions in any order in a simple string format and the system builds out a node tree structure of parent nodes on which this script depends, re-use and share existing nodes, and display point value results as well as chart time series that can contain millions of data points. Recalculation of the entire tree takes a few milliseconds for a medium complex tree while in the background data feed handlers keep on updating dozens if not hundreds of subscribed symbol series. I will use this for my own discretionary trading book, it allows me to quickly visualize, for example the rolling correlation between the moving average series which are in turn based on the return series of a tech stock, each, for example. I can express all that in a simple string that is less than 100 characters long. The outcome can be viewed in a table format as numeric value or on a chart that is updating in real time.
Let's talk specific issues not "distributed systems are great, and concurrency is an issue, and availability is important, bla bla". My only question is, do you actually want to participate in a detailed discussion or are you just excited about those glossy lunch break speeches?
Last edited:
Is there anything you can contribute as well? What is your domain of expertise? What is/are your specific question(s)?
I worked on a team back in 2016 as a technical business analyst for a very well known hedge fund. The project was building out the next generation platform for trade processing
applications using a microservices architecture. I now want to take what I learned from that experience scale my strategies from a few tickers to the bigger universe of futures, equities, options, etfs, etc.
• Open Source
• Cloud First
• Reactive
• CI/CD pipelines
• Elastic Scale
• Resilient
• Evolvable
• Secure
• Everything is streaming
• Everything is distributed
• Big things from small things
• Test Driven Development
• Move code not data
• Anything can fail
applications using a microservices architecture. I now want to take what I learned from that experience scale my strategies from a few tickers to the bigger universe of futures, equities, options, etfs, etc.
• Open Source
• Cloud First
• Reactive
• CI/CD pipelines
• Elastic Scale
• Resilient
• Evolvable
• Secure
• Everything is streaming
• Everything is distributed
• Big things from small things
• Test Driven Development
• Move code not data
• Anything can fail
Those are all just buzzwords. If you can provide more specifics of your requirements or plans then someone may be able to better help you. Obviously you do need to start with requirements, specs and a overall architectural design.
* what do you actually want to achieve? Is the platform supposed to be a trading frontend only or a research and development platform as well?
* is a backtest engine required?
* is it for self use or to sell/lease it to future clients?
* what is your time span you can invest in this? financial resources?
* fully automated? Semi? Discretionary only?
* how latency sensitive are your algorithms?
* is all the computing and serving done on a local machine? The same machine?
* does the frontend need to be web served or how else do you want inputs and return values to be shown?
* which asset classes?
* which symbol universe? Have you thought of how to represent symbology? The symbol database?
* which mechanics need to be concurrent?
* what do you actually want to achieve? Is the platform supposed to be a trading frontend only or a research and development platform as well?
* is a backtest engine required?
* is it for self use or to sell/lease it to future clients?
* what is your time span you can invest in this? financial resources?
* fully automated? Semi? Discretionary only?
* how latency sensitive are your algorithms?
* is all the computing and serving done on a local machine? The same machine?
* does the frontend need to be web served or how else do you want inputs and return values to be shown?
* which asset classes?
* which symbol universe? Have you thought of how to represent symbology? The symbol database?
* which mechanics need to be concurrent?
Agreed. @wave, resources in terms of manpower is also a major factor. My previous corporate gig involved working on a SAAS product with a micro-service architecture, advanced CI/CD pipeline, massive test coverage, etc...all the things you would expect from a modern software shop. For my recent project which was completed by one person (me)...did I have a CI/CD pipeline? No. Test coverage? Bare minimum. For anyone wanting to role their own automated trading solution, I strongly suggest checking out something like AlgoTerminal first. I am not saying that is what you should settle on, but I think it can serve as a good baseline in terms of it can do W and costs X per month. Then you may say you need Y which AlgoTerminal does not have and it will take you Z months to build you own solution. Of course, replace AlgoTerminal with another available commercial or open source ATS...I just used that as an example as that is what I use for my own trading.
What do you use as datafeed? Anything aside IB? It's extremely hard to find quality data feeds. Bloomberg bpipe or reuters rmds is out of my budget but solutions like IB or IQFeed leave a lot to be desired. Never really looked in detail at cqg. Any experience with them? Always on the lookout for high quality data feeds at prices that a medium shop can afford.
Thanks
Thanks