Sharing Automated Trading System Design or Code?

[igum]

Anyone have sample code for an automated system? Or technical design details? I checked some open source but it seems many of those projects have died and didn't get very far or aren't what I'm looking for. I'd like to develop my own preferably in java but if I could get some help with java code or just some detailed design that'd be great. I'm sure many people have the same need, but would probably develop their own trading strategy that could be plugged in to such a framework. Its really the trading strategy that would be proprietary. Wondering if somehow .NET, Java, C++ developers could start some open source project for the design of a really cool architecture that everyone can leverage, but then folks can code in whatever language they want, and also code their own trading strategies to plug in.

Is such sharing possible or just not realistic?

 

[stephencrowley]

Great question.. of course this is possible.

I've been involved in opensource development for 7+ years so I really do believe it can be a great thing.

What is the incentive for the ATS developer to share the countless hours of work he has put into it?

For people with lesser ability they definitely want to share to gain the benefits of more developed code.. (not saying you are inexperienced by any means)

For someone with a high level of ability, the benefits are less clear.. perhaps gaining more insight or perhaps looking at something in a new way.

Also, why help the competition? We really are fighting over the same trades a lot of time.

This is not an trivial task, off hand I can think of the major components of my system which does not need much babysitting.

1) Feed monitoring.. should handle fees going down, reconnecting, if its a major problem it should page you.. or email.. whatever
2) failsafes.. if the performance degrades, feeds get stuck, it should close trades out in an efficient way..
3) Data storage/retrieval: for my system this is huge.. realtime archival of every single quote and trade made across many different markets and systems with the ability to play these back, export to matlab for analysis, etc
4) Job scheduling.. how does this thing know when to connect, disconnect, begin winding down trades, etc
5) interface: how is the system controller? For my, GUIs are bad because the system needs to focus all its resources on trading.. not presentation, and needs to be able to be controller over a terminal connection


My experience in opensource has been that a successful project usually has only a few people (many times a single person) doing the heavy lifting.

Quote from igum:

Anyone have sample code for an automated system? Or technical design details? I checked some open source but it seems many of those projects have died and didn't get very far or aren't what I'm looking for. I'd like to develop my own preferably in java but if I could get some help with java code or just some detailed design that'd be great. I'm sure many people have the same need, but would probably develop their own trading strategy that could be plugged in to such a framework. Its really the trading strategy that would be proprietary. Wondering if somehow .NET, Java, C++ developers could start some open source project for the design of a really cool architecture that everyone can leverage, but then folks can code in whatever language they want, and also code their own trading strategies to plug in.

Is such sharing possible or just not realistic?

 

[rufus_4000]

I believe that sharing design (for in the beginning) is probably a decent way to go. The incentive question is very important. For instance, Jody Goldberg, the author of Gnumeric, works on a number of quant analytical systems (using pieces of Gnumeric) as his RealJob for a while, but even he has eventually take a job with OOo to support himself.

As for me, time is a major factor, for instance. While I would love to take a vacation and seperate out from my code what is proprietary and what can be shared (and most of the framework is probably sharable), but I don't have the time to do so, continuous updates and new product research is quite time time consuming.

Also, there is a question of what kind of ATS the framework is designed to support. I believe my system and Stephen's are both high frequency trading systems, ATS doesn't have to be high frequency at all.

From my perspective, the modules that can be shared include:

1) Feed handler, echoing what Stephen outlined
2) Messaging layer, fault tolerance, fail-over, outage detection, etc, and most importantly, performance and scalability
3) Presentation / Trading separation, I was lazy, so I wrote a wrapper for DSM (distributed shared memory), so every update is done via a DSM call, pretty easy. And my GUI was terrible, a simple Tcl/Tk wrapper.
4) memory caching and data storage, unlike Stephen's system, my system uses pre-fetching extensively (basically fetch in anticipation that the memory cache would require it. So I have very little data storage problem to speak of. I hate disk I/O, for any data, so even the small data is requested via a pub/sub mechanism.
5) Simulation and back testing, this is critical, in my opinion, for any automated system to succeed. And writing a good market simulator is very hard (and need extensive data to support), so the shell framework is probably open, but the actual limit book shuffling, simulated fills, etc, are proprietary I guess.
6) External handlers, such as exporting to analytical environment, etc. Again, I was lazy, so I wrote a file adapter for the DSM, so I can pump data into my analytical environment very quickly.
7) Order / Broker adapter, this is pretty much self explanatory
8) Extensibility API hooks, so any user defined modules can be loaded into the environmnt during run time
9) Risk Modules, this is probably highly custom proprietary, since it is dependent on the strategy being traded.
10) Administrative modules, start up, shutdowns, emergency shutdowns, drop copies, notifications (I have the system write to Jabber, another DSM wrapper, heh. So when I am on the road, I can get instant updates in real-time on my phone via IM, I am writing a wrapper so I can *send* commands to my system via IM ...).

I may have forgotten a few ...

Rufus

Quote from stephencrowley:

Great question.. of course this is possible.

I've been involved in opensource development for 7+ years so I really do believe it can be a great thing.

What is the incentive for the ATS developer to share the countless hours of work he has put into it?

... good summary deleted ...

 

[stephencrowley]

Quote from rufus_4000:

As for me, time is a major factor, for instance. While I would love to take a vacation and separate out from my code what is proprietary and what can be shared (and most of the framework is probably sharable), but I don't have the time to do so, continuous updates and new product research is quite time time consuming.

Time is also a huge factor for me.. automated trading is not my real job, writing software is for now until my system is fully functional and I feel safe enough to take the full plunge. The time it would take to extract my broker specific api from my code would be considerable.

Also, there is a question of what kind of ATS the framework is designed to support. I believe my system and Stephen's are both high frequency trading systems, ATS doesn't have to be high frequency at all.

Also true, the requirements would be very different depending on your timeframe.. if I was to open my code, and people modified it to support longer timeframes it would simply complicate my code and increase the complexity.

From my perspective, the modules that can be shared include:

4) memory caching and data storage, unlike Stephen's system, my system uses pre-fetching extensively (basically fetch in anticipation that the memory cache would require it. So I have very little data storage problem to speak of. I hate disk I/O, for any data, so even the small data is requested via a pub/sub mechanism.

I'd be interested to know how you get around this.. even archiving a few symbols for a full day takes about 2GB on disk for me using native-java wrapper to sleepycat database.. very fast access.. simulation and backtesting is not a problem for me.

5) Simulation and back testing, this is critical, in my opinion, for any automated system to succeed. And writing a good market simulator is very hard (and need extensive data to support), so the shell framework is probably open, but the actual limit book shuffling, simulated fills, etc, are proprietary I guess.

Limit book shuffling isn't all that bad.. simulating orders might be though.. I have stuff that shows the executed price, including commission, of any market order aggregated against all exchanges feeding into the system. Again, this part is not rocket science.. it'd just take time to open it up.

7) Order / Broker adapter, this is pretty much self explanatory

Right.. I'd have to create yet another abstraction layer to support multiple brokers, thus increasing complexity and latency.

10) Administrative modules, start up, shutdowns, emergency shutdowns, drop copies, notifications (I have the system write to Jabber, another dsm wrapper, heh. So when I am on the road, I can get instant updates in real-time on my phone via IM, I am writing a wrapper so I can *send* commands to my system via IM ...).

Great stuff.. I'm actually doing the same thing with AIM instead of jabber.. just don't have a jabber client for my sidekick, would be better idea though.

 

[rwk]

You might have a look at TradeMagic. I don't use it because I don't work with [dot]net, but it looks interesting.
http://www.trademagic.net/

 

[rufus_4000]

Quote from stephencrowley:


I'd be interested to know how you get around this.. even archiving a few symbols for a full day takes about 2GB on disk for me using native-java wrapper to sleepycat database.. very fast access.. simulation and backtesting is not a problem for me.

Argh, we are working on different product universes. I can see how equities would mean large data storage. I trade about 10-12 futures and options products, and even with all different instruments (expiration dates), my daily data barely break 1GB.

Basically, I wrote a separate data "processor" (for lack of a better name), that runs a custom classifier (I did research in AI for a while, so bear with me) on the data, so the data would be cut into index-ed chunks, and kept in memory. I didn't use an in-memory DB (I was lazy, and I am not crazy about KDB), so I wrote a memory based master storage, a nasty looking weekend hack.

The Calculation engine would analyze the current position (or potential open orders about to be filled), and then when the job-scheduler (yeah, yeah, I wrote a simple job-scheduler too) is a bit less hectic (priority), it would fetch the relevant data chunks from the data processor(s) (both today and historical references), ready for the calculation engine to use. The data request and message response is handled through messaging (pub/sub channels), and the actual data set is access DSM (just a bit slower a multicast message).

However, since I am an Unix OldFart, I wrote pretty much the entire system in C (no, not even C++), with a bit of assembler tossed in for good measure (thread synchronization, stack traversal, etc). It is not that I don't like OOP (I have taught classes in C++), but since I know Stroustrup in a past live (I was against templates since ANSI C++ V2), so I am a bit prejudiced, heh.

 

[stephencrowley]

Wow, C, you truly are a madman. I'm a reformed unix-addict.. more like a java madman and my 'real job' has made me somewhat of a design freak.. I'm glad for it though.. I've saved so much time pulling off the shelf java components and plugging them togeher and only working on the truly useful stuff. Just beautiful. So far i've used the following components

1) Sleepycat DB
2) JLine - readline clone for java.. plugged the tab-compleation into beanshells completion interface
3) Beanshell - great scripting/java bridge for the console
4) Quartz - very nice job scheduler (wrote my own storage backend to use sleepycat for persistence)
5) Jakarta Commons Logging
6) JMail api..
7) ATLAS (automatically tuned linear algebra).. had to create some extensive matrix packages to plug into this natively..

Just goes to show you, there is more than one way to do this stuff.

Quote from rufus_4000:

Argh, we are working on different product universes. I can see how equities would mean large data storage. I trade about 10-12 futures and options products, and even with all different instruments (expiration dates), my daily data barely break 1GB.

Basically, I wrote a separate data "processor" (for lack of a better name), that runs a custom classifier (I did research in AI for a while, so bear with me) on the data, so the data would be cut into index-ed chunks, and kept in memory. I didn't use an in-memory DB (I was lazy, and I am not crazy about KDB), so I wrote a memory based master storage, a nasty looking weekend hack.

The Calculation engine would analyze the current position (or potential open orders about to be filled), and then when the job-scheduler (yeah, yeah, I wrote a simple job-scheduler too) is a bit less hectic (priority), it would fetch the relevant data chunks from the data processor(s) (both today and historical references), ready for the calculation engine to use. The data request and message response is handled through messaging (pub/sub channels), and the actual data set is access DSM (just a bit slower a multicast message).

However, since I am an Unix OldFart, I wrote pretty much the entire system in C (no, not even C++), with a bit of assembler tossed in for good measure (thread synchronization, stack traversal, etc). It is not that I don't like OOP (I have taught classes in C++), but since I know Stroustrup in a past live (I was against templates since ANSI C++ V2), so I am a bit prejudiced, heh.

 

[SideShowBob]

Check out Wealthlab, www.wealth-lab.com. Plenty of systems on their website. Also if you just look at the list of functions in the online help you see the kinds of things involved in an automated trading system. You might also check out tradestation.

SSB

 

[stephencrowley]

The entire point of this thread was opensource.. wealth lab is not open.

Quote from SideShowBob:

Check out Wealthlab, www.wealth-lab.com. Plenty of systems on their website. Also if you just look at the list of functions in the online help you see the kinds of things involved in an automated trading system. You might also check out tradestation.

SSB

 

[rufus_4000]

Quote from stephencrowley:


Just goes to show you, there is more than one way to do this stuff.

Oh absolutely! In fact, if I were to do it all over again, I wouldn't have chose to write it in raw C. I am sure that everyone who had to it all over again it would be "better" in one way or another.

My problem is that I became a tech manager (I was pretty senior before I switched over), so I really didn't write any production code for 6-7 years. So when the opportunity came, I did the "what the heck, let's do it in the rawest way I can, so to practice". It is weird, I would sit in final design approval meetings (it was my sign-off), and I think I understand the concept, but I "know" there is a better way, but my actual coding skill was so foggy that I couldn't put my finger on it ... bad bad.