Developing a Trading Framework from Scratch

F

fatrat

I think, for the benefit of myself and others, I'm going to detail the development of a trading system from scratch in C++. I've only been working on implementing a blackbox for Genesis only recently. My past experience was as a contractor writing software to manage and distribute quote data from feeds, and Genesis was the cheapest alternative to getting data similar to what I had already seen with ITCH and ARCA.

I'll post screenshots as I progress and possibly UML diagrams detailing the architecture of the system. Since I've already begun, I'll try and keep the reader upto date on where I'm trying to go. I'm encourage readers to interact with me so I take this to completion with a valid system. Dare I dream, I'd like to be a one-man market-maker someday.

If you have strategies you'd like to see run on this framework, feel free to suggest. Obviously, I don't expect to get very many good ones and believe I'll be going it alone for the most part; however, I am open to ideas.

Here is my rough, ultra-beta application's screenshot, along with Genesis Laser and E-Signal in the background to help me understand the progress:

<img src="http://www.losingtrader.com/images/incisor1.jpg">

The "Incisor Trade Runner" is the front-end plain alpha-blended window that reports during the course of the day on what's going on in the market. Initially, I didn't even want a GUI; however, the GTAPIB API that Genesis provides requires an HWND parameter. If that parameter is a console window, the application fails. The application does a lot more than its plain white window suggest, but it's still primitive in comparison to the applications we were running in the former hedge fund I was employed at.

The menus allow you to subscribe to various products, control the degree of alpha-blending (so you can compare the bot's internal variables to what's going on in the market on the same screen), and let you manage your connection state. Right now, the main window is nothing more than an edit control onto which the application reports information. Most of the critical information right now is sloppily reported with OutputDebugStringW() [yes, performance penalties and all] into a WinDBG window pane on the side -- I will show that as well in the future.

The features so far:

- Can process and record Level-1 and Level-2 quotes from NASDAQ and the NYSE
- Has a hacked-together cheap database [that could use work] to store this information
- Is capable of maintaining several NASDAQ books at the same time, though I have decided to focus on just one product in the time being for trading. [NTAP]

Since I'm not much of a DBA, my database is garbage. However, I'll detail that as well. Right now, the C++ application uses ADO with worker threads to commit data from write queues into the database.

I am currently in the process of writing a "product manager" that allows me to track and maintain state variables for multiple products [i.e., stocks] that are traded in the system. When I come back, I will detail the architecture I envision for this.

Tonight, I intend to clean up the child-windows inside the main application window and try to add something to reflect analysis of quote data and level-2 data.

I may move this thread to a personal blog as soon as I clean the C# ASP.NET code for presenting information in that blog correctly. :)

I want to take the application from start to finish with an actual trading strategy.
 
Interesting. A couple of questions:

1. Why C++. It's always going to be more work writing in C++ than say Java or C# and will take longer to shake down the code. I really don't think performance is an issue. A lot is made of "real time" and the garbage collection issue, but IMHO it is rather overblown especially considering these things aren't running on "hard real time" operating systems anyway and the world it interfaces to is anything but "hard real time".

2. Why not use MySQL for the database ? Free, fast, stable and a little bit of SQL puts things in and gets things out. I know RDBs are probably not the best thing for time series, but in the interest of getting things done and concerning oneself with more pressing issues, it's not a bad choice. I log tick data from IB for about 50 stocks, most of the major stock index futures, and the Level 2 for 3 stock index futures into MySQL database on Linux box using Java/JDBC. It's really simple minded code - no FIFOs for buffering - and works fine. On an old Socket A Athlon 2800 CPU utilization is a few percent. Database is not on the root disk and uses the XFS filesystem.

3. How computationally intensive is this alpha blending stuff ? I know in Java it is significant, though may be better in 1.6.

A further comment on the API. Compelling the user of the Genesis API to incorporate a GUI into their application is just wrong and plain stupid. When will these people learn.
 
Quote from dcraig:

Interesting. A couple of questions:

1. Why C++. It's always going to be more work writing in C++ than say Java or C# and will take longer to shake down the code. I really don't think performance is an issue. A lot is made of "real time" and the garbage collection issue, but IMHO it is rather overblown especially considering these things aren't running on "hard real time" operating systems anyway and the world it interfaces to is anything but "hard real time".

2. Why not use MySQL for the database ? Free, fast, stable and a little bit of SQL puts things in and gets things out. I know RDBs are probably not the best thing for time series, but in the interest of getting things done and concerning oneself with more pressing issues, it's not a bad choice. I log tick data from IB for about 50 stocks, most of the major stock index futures, and the Level 2 for 3 stock index futures into MySQL database on Linux box using Java/JDBC. It's really simple minded code - no FIFOs for buffering - and works fine. On an old Socket A Athlon 2800 CPU utilization is a few percent. Database is not on the root disk and uses the XFS filesystem.

3. How computationally intensive is this alpha blending stuff ? I know in Java it is significant, though may be better in 1.6.

A further comment on the API. Compelling the user of the Genesis API to incorporate a GUI into their application is just wrong and plain stupid. When will these people learn.
More...

1)

First, let me point out that C++ is not the end-all, be-all of my trading framework. C++ makes it especially easy to write COM/DCOM objects, so wiring my software for distributed use, free-threading models, and connecting it to C# and .NET can be especially easy.

I don't think C++ takes a lot longer to shakedown the code. I view the primary benefit of C#/Java as garbage collection, but using boost::shared_ptr<> and std::auto_ptr<> make my life reasonably simple. A lot of C++ programmers don't make effective use of smart pointers so memory management becomes a head-ache. Using boost::shared_ptr<> in cases where heap memory allocated objects are passed around removes some of the complexity, because a destructor will get invoked at some point when a copy of the shared_ptr<> no longer exists.

Without auto_ptr<> and shared_ptr<>, I imagine my life would be much more difficult. Exceptions would not have made things easier. However, smart pointers, in my opinion, take more of the hassle out of the mix.

Java and C# come with great class libraries. However, in addition to auto_ptr and shared_ptr, I have the STL. The STL provides me with fantastic templatized code that can make use of custom allocators right off the bat. The flexibility of STL algorithms makes programming equally as easy as using a .NET class library. Furthermore, with ATL and WTL, I get the GUI functionality I need to round out the playing field. The Java/C# edges are largely diminished when I make effective use of template libraries.

Additionally, who's to say the C++ code will run on Windows in the future? Using C# would restrict me to one operating system (Mono is not a viable option yet.) Java requires that I have a VM, so if I do formulate a model that requires a real-time response and want to run it on QNX, what will I do?

There's more benefits, such as fast processing of market data using SSE2 compiler intrinsics, having Intel VTune, simpler debugging interfaces with WinDBG, as well as the option of taking my software into the kernel if performance is a requirement.

Ultimately, C++ gives me the most flexibility in terms of design and interfacing to other languages. It gives me the most flexibility in terms of designing for performance should the need for performance exist. Because it can work as a bridge language, once core trading components are developed I can develop everything else overtop of my framework.

2) For the database choice of SQL Server 2005, I like it because I can author extended stored procs in .NET languages. The other reason I like SQL Server 2005, and it's just a person reason, is because I am familiar with COM and ADO. In C++, using ADO, it isn't that much more difficult than using ADO.NET with C#. The fact that MS provides smart pointer wrappers around a lot of the components used to interact with ADO makes my life easier, and I like that.

The write queue + stored procs for the database were simple enough to write and are already completed. I think I could streamline the process a bit further, but the database is currently not a critical part of the trading system aside from serving as a reference for developing trading systems. I have yet to find an edge, and FrostEngine is ahead of me by a long shot.

SQL Server 2005 also has an intuitive graphical management interface that I like.

3) Alpha-Blending costs me very little. The Win32 API exports alpha blending functionality, and I think (but am not sure) the hardware handles alpha blending once the parameters are set for a window. In order to turn off and re-do alpha blending, you have to reset the state of the Window with regard to it being layered and alpha blended.

4) I agree with that API thing. I don't like that HWND requirement at all. But at the same time, I don't care to reimplement all of the functionality they're offering me. As far as I know, they aren't providing any details for their protocols. I know how a good bit of their protocol works from looking at the wire; however, I don't want to deviate from the API in the event of them making changes and me not knowing it.

I'm not worried about Genesis' software at the moment. Their framework is good enough to build the basics off of. If they start to fail me, my software is flexible enough to allow for other brokerage interfaces to be integrated in easily.

I'm tired, so if any flaws are in my arguments above, allow me the opportunity to clarify and defend later.
 
Thanks, nice reply.

I agree the STL probably does a lot of what you might want. Also agree about C# being windows only and I wouldn't trust mono yet. With Java becoming GPL I don't know if Mono has a great future in the open source world.
 
Quote from itmediaco:

do you intend to share the source code or is this solely a personal project? I look forward to seeing the diagrams.
More...

I haven't decided yet, because I don't intend to make a profit from this framework aside from trading.

Open source is definitely a possibility, but if I go that route I will likely bring aboard a small pool of developers to look over the code, inspect it, and make sure it's reasonable before release.
 
I wrote some components tonight to assist me in managing the products managed by the trading system. While the intention is to manage the trading of NTAP, I added code to dynamically subscribe and unsubscribe from multiple data-streams based on user input. This in itself was not difficult because of the Genesis API functionality for subscribing to streams, but making the tie-ins to the Win32 GUI required some work.

The software construct for managing products in my software is called the "Product Manager". The product manager is an object which manages a collection of products which correspond to traded instruments on the NYSE and NASDAQ. The product manager supports the addition and removal of products and supports product lookup and removal in O(1) time. The product manager also implements the "Visitor" design pattern to allow for external agents interested in gathering information to submit a functor that receives non-modifiable references to products contained within. The product manager is a stand-alone class that needs no GUI.

I also authored two other pieces. The "Product Window", which is a graphical representation of the information contained within the product manager. This product window is a child window of the main window, and in the future will be a dockable window of the primary application. The Product Window was written using ATL classes and ties into the rest of the code framework for the use of ATL.

In addition to the product manager, I created the machinery to maintain level-2 books. This code is essentially similar to software previously written for ITCH. The goals for tomorrow are to improve the error reporting functionality and to create tie-ins such that Level-2 windows can be inspected. The Level-1, Level-2, and Print information will be combined into a framework that will be fed into computation engines. If time permits, a paper-trading interface and report collection framework will be added.

I have not fully designed these engines and am mulling over some design decisions for this.

Screenshots will follow after some more pieces are completed.
 
Rough version of product manager in place. Not the most elegant looking thing, but makes it easy to index and move to the product in question and request that that product's strategy gain control of the reporting interface.

<img src="http://www.losingtrader.com/images/incisor2.jpg">

The images are pulled directly from the NASDAQ site via an http get. If the image isn't available, it just renders large [but ugly] text.

I'd like to add a management panel similar to the AIM Triton user bar. That would enable a nice strategy display interface in a small display area. For now though, the GUI is not important. It just has to be sane enough to work with when I'm running strategies and want to know what information is coming in off of the wire, whether I've been disconnected or not, etc.

[There is no performance consequence, because 1) the images get cached, and 2) I don't plan on adding and removing products all day like a maniac.]

Picture rendering is done via the IPicture COM interface.
 
I will be focusing on the strategy scheduler tonight instead of the GUI aspect of this project. I think the GUI is good enough to organize what I need to know for the time being, so now it's critical to report signal information as well as trade information.

I'll return to the GUI later. I'll be documenting the UML diagrams of the project shortly. On Friday, I will review the code itself and check for leaks, exception safety, and resource management information. There are a lot of TODO's and FIXME's inside the code, and the components of the project are scattered.

Don Bright suggested a strategy, and I intend to implement it using this framework. Once I hit this milestone, I will clean it up and archive a release. I will review where I will take the project at that point, and consider more decision decisions.
 
Looks nice. I assume order management and risk controls come next... Can you implement strategies at runtime or is it compiled into your app?
 
You must log in or register to reply here.
Back
Top