How to Build an Automated Trading System

[greaterreturn]

Folks this is discussion only for 2 groups of people. Those who either already use, are currently building or will be creating an ATS (automated trading system) or others who are actively and successfully trading discretionary.

Specifically, I'll discuss some aspects of automating methods Jack Hershey uses but only by way of example. Others may be included.

A Moderator, Tums, has offered and agreed to quickly delete any posts that I or you, the reader, complain about especially from those who only want to insult or use negative language.

Now, why the mixture of ATS and discretionary invitation?

It has to do with how to build trading systems that emulate what human beings normally do.

My specialty (in the real world) is B.P.A. and B.P.R. as it relates to software engineering. B.P.A. stands for Business Process Analysis and B.P.R. means Business Process Re-engineering.

It's basically the science of automating previously human activities using computers.

It works like this, you get someone very skilled at doing a business function (in this case trading) to walk you through the steps of what they do and how they do it.

You diagram the steps using arrows for decisions that get made (various standard diagram formats can be used) and write down the details (called requirements documents).

That describes business process analysis which ends with the software developers thoroughly understanding what the people (traders) do and what they want automated. So it's not necessary to know or use formal methods. It's simply necessary to thoroughly understand and get the details in writing so you can review and make sure you didn't skip or forget anything while building the software.

The next post will briefly describe B.P.R.

Sincerely,
Wayne

 

[greaterreturn]

B.P.R. also starts happening right way during B.P.A. Basically, as you begin to ask a lot of questions about how people do things and write them do so they can see it in diagrams, they immediately realize themselves all the inefficiencies and redundancies in their regular business process.

Invariably it changes and improves how they do business, making their jobs, easier while waiting for the automated software to get completed.

After everyone agrees that the details of B.P.A. are written down and clearly understood, a more active B.P.R. consists of actively thinking of how to use computers to automated and change the process.

In other words, non-software programs often have misconceptions about what computers can and cannot do. B.P.R. has the them working with the programmers using their business jargon to discuss how to fit computers in most efficiently.

Okay, the point about B.P.R. is that computers can never do things exactly the same way as humans. In some cases they can do certain things better which are highly repetitive and others they can never compete, yet anyway.

Next, I want to discuss efforts to automated Jack Hershey's methods in an ATS. I hope to get feedback on my ideas, synergistic ally, and I hope it helps you in your efforts at the same time.

Sincerely,
Wayne

 

[greaterreturn]

First a little background on the ATS platform.

Skip this post and come back later, if you like. The next one delves into more specifically how to proceed in building an ATS to emulate a human trader.

Over the last few years, I've been working on an ATS after spending many years prior to that learning discretionary trading. I started the ATS working with commercial products that allowed me to build custom code but found them to all be lacking in one way or another. Always tradeoffs.

So I took that code and a lot of new code to create a complete platform for trading myself (a gargantuan effort) while using the Zedgraph library for charting. It's in C# .NET 3.5.

It covers a quote server to pull tick data including DOM data for historical test and a live order processing service that integrates with MB Trading. I focus on Forex trading at this point.

It also includes a historical back testing system that entirely and only runs using tick data. It's that way from the ground up in order to have screaming fast performance using both bid and ask data and DOM market depth. Unfortunately, historical data only has bid. But I'm steadily building my own historical Forex historical data.

The other major feature or reason I built it myself is total flexibility of time frames so that the Chart, the strategies, indicators, money managers, statistics components and every part of the system can operate on any number of different time frames simultaneously.

Of course, it does not have a plethora of features found in many commercial products that are built to please the masses. It focuses on the essential items to build an ATS that emulates human behavior.

Enough said, it has reached critical mass and been thoroughly tested even with a simple trading system and a real account so that it's now ready as a platform to tackle a real job.

Okay, next post, I will lay out my plan for approaching the automation of Jack Hershey's methods. Please offer all suggestions or ideas.

Sincerely,
Wayne

 

[greaterreturn]

I've been working on B.P.A. in the last week. I'd say I'm about 50% complete. Here's my progress so far. Some of it required adding some base features into the platform.

The most help come from the website at traderuniverse.info and, of course, other posts by Spydertrader and Jack himself here on ET.

In the process of studying the material, I drug up some old code I wrote to create channels and got that working in the platform also so I have a framework for the essence of his system.

Additionally, he talks about the importance of volume and use of the DOM.

As far as I knew, there was no volume in Forex but there is Depth of Market data. So I began collecting that over a week ago. (Never found any historical DOM data.)

That took a lot of work to automate and test and optimize to reduce the data size and processing speed for storage and playback in historical testing.

One nice discovery while building my own DOM engine was noticing the many transactions that happens quicker than a blink of the eye and can't be seen on a human terminal.

From that, I found a way to produce true volume. Basically, every trade leaves an extremely brief footprint on the DOM, through experimentation, I got a consistent and very working set of Time and Sales data that is useful as volume data.

So it seems that I'm now armed with volume, DOM, and tools to automate Channels, so I studied deeper.

Next post, next steps and plan for where to go next in the process.

Sincerely,
Wayne

 

[greaterreturn]

What's fascinating to me, is that Jack Hershey's information covers virtually every aspect or the full spectrum of directional trading in every fractal down to the micro and use of the DOM itself.

So far, each thing I've learned I've been able to verify, many of the fundamentals I has learned elsewhere and already use to profit in that past.

That overreaching completeness of how to handle different situation and his level of skill presents an interesting challenge.

Should a developer such as myself attempt to implement every aspect of that from the beginning?

Upon consideration, it must be remembered that the goal of building an ATS is to make money--not to build cool software.

So the sooner it can create revenue the better to feed to process.

Jack even suggests the same thing for learning discretionary traders. He suggests trading when the market is behaving in a way you understand and sideline if you get confused or misunderstand what's happening.

That idea got my attention. It's not necessary for a human student NOT a computer system to master every aspect out of the gate.

So, iterative refinement of the ATS will work best.

In other words, it seems smart to start with several most common market conditions, refine and back test the ability to handle those specific types of trades with an entry and exit.

There is one caveat to attempting this. It has to do with market memory and what Jack calls Market Operating Points.

Let's me address that in the next post.

Wayne

 

[greaterreturn]

In standard BPA and BPR we think of every business activity as having a process.

What you do to start? How decisions are made.

What's very nice is that Jack as spelled out his decision making process which he calls sweeps or MADA.

Just like in normal BPA, it is customary to be totally lost with the business jargon humans develop as they become very skilled and specialized at what they do.

That, if you try to do the same, will be significant undertaking with Jack because he knows so much about so many trading aspects. However, the more I learn the jargon, the more I make sense of everything he says.

Specifically to building an ATS, the market has memory meaning the humans to work everyday in trading the markets remember recent price levels and that impacts how the future unfolds.

But a good analogy that helps me is traffic on he roads. You can never predict exactly how many cars will go by or precisely when they'll pass. But since it's a human activity involve a large number of people, you eventually learn traffic patterns when more people tend to drive and where the most popular routes are. We're all habit forming after all.

Such is the same with the markets.

In business and software systems, we usually solve these problems with a work flow. Which starts as a diagram but becomes a computer program.

This type of program can recognize that the market has a finite number of "states" (computer speak) and certain conditions lead to it moving to another state. Each state leads to decisions the trader makes.

For example a big volume bar is a state than can only lead to a few subsequent outcomes and force certain decisions.

Similar to a DU (dried up) volume bar. It has consequences that must be dealt with.

That must be remembered in the following bars.

So I plan to build (or find open source code) for some type of work flow component.

Basically, I want to start with a few states, program the rules of must happen when that state occurs and how it branches into the the next state.

Again, Jack calls these market operating points.

For example, here's my rough idea.

Of course, I could just code a quick set of rules to deal with one on the states and run that.

But I already wrote some code to handle 3 or 4 states and found it complex without a standard workflow tool.

My guess is that there's dozens of states and their variations so it can become very complex code.

It's better handled by an engine to process so I can simply list the states, the "flow" of which conditions lead to which states and then program the code for each of the measures.

Since it'll take a lot of work and time to build ALL the states and flows.

My thought is start with those that are "beginning" points. Like they say, the best place to begin is the beginning.

I can think of 3 beginnings.

1. The start of the market day or opening. That would be 8 am est for forex due to the highest volume from 8 to 5 est time. Jack walked through step-by-step on how to start the trading day.

2. Large range high volume bars. Jack talks about and my experience confirms that those bars tend to start everything over again and wipe out any previous trend lines or channels you have drawn. You start over.

3. DU or dried up volume bars. Jack states (I've seen that stated elsewhere too) that the market can expand in volume in either direction at that point and be ready with a bracket order.

So if we take those beginning points, follow the steps to make the channels and follow the rules for making decision, we can cover when to enter the market (the first state), and then the beginning of drawing and trading traverses.

Somehow we need to program the computer to do what a human would do. If the market does it anything it doesn't expect or not yet programmed to handle, just exit immediately.

Now that I think about it, it can log the exact occurrence and details that confused it.

Then it can learn just like a human would do during what Jack calls "debriefing". The programmer reviews the failure, the charts and realizes how to handle that situation, adds those rules, perhaps a new states in the work flow, backtests than enhancement and rolls out the fix.

In that way, and also by proactively studying more to find and add more states (operating points) the knowledge and ability of the ATS can grow much like a human would do.

Okay, what are your thoughts? Have you tried anything like this?

As an aside my first attempt at a few rules to test the channel code I wrote resulting an 695 pips profit in one day on the USD/JPY. It has several trades right at the highs and lows due to channel bounces.

However it's very limited and loses money on the following day.

I feel that's for 2 reasons. One is, it lacks any logic to exit when it gets "confused". So it makes good trades some of the day and goes haywire the rest since it wasn't programmed to handle those periods yet.

That partly came from a lame effort to get it all the way to expert level and always in the market either long or short. As Jack points out, it easier to start with what you understand (or the ATS can understand) and build on it.

Again, any thoughts will be warmly appreciate. Especially if you have better ideas on how to do it.

Sincerely,
Wayne

 

[greaterreturn]

Clearly, if you go to this link. The solution involves a Finite state machine like used in video games or robotics.

This article briefly describes.

http://www.odetocode.com/Articles/460.aspx

Imagine trying to build an ATS without a state machine. It would eventually get so complex as new rules and states are added that it become impossible to maintain or improve.

In contrast a finite state machine can make the job almost easy by comparison.

I'll be looking into using the state machine framework built into the .NET C# framework. Cool it already has it and even a graphics designer to make it easier.


Wayne

 

[greaterreturn]

Okay, the Windows Workflow is way over complicated and generated different classes of every state. That will proliferate so many classes that it's unbearable.

A better design I found is the State Machine Compiler. Open source and free.

http://smc.sourceforge.net/

It builds a state machine for one class so it can have the states, receive events, call actions, and then optionally change state.

Now my thought process is to break down the different parts of trading JH methods into individual components.

For example, the bases of everything is using tapes or small bar by bar channels.

I will build one state machine that can handle drawing and taping one bar at a time.

But it won't make any trading decisions.

Instead, I will model work either Jack recommends. For example, this taping component will be called a "housekeeper". Jack calls it that. It's always easier to make changes later if you build software using domain specific terminology.

Later additional housekeepers can be added like MACD monitor, DOM Monitor, etc. Intermediate channels. That's all housekeeping to have the data so the software can "see" the market.

Next come M.A.D.A (monitor, analysis, decision, and action). Jack calls the MAD a sweep for short.

It means the human quickly reviews all the channels, indicators, and information. Makes a quick analysis mentally of what that means. Makes a decision of hold/continue (enter or exit) and then takes action.

In the case of hold, the action is to log the decision. In the case of a reverse or trading decision, you first trade and then log.

So we'll have a monitor component which will proactively and periodically look at the housekeeping components to get their latest stats, at least the high level, pertinent ones for making an analysis.

It will have an internal state machine because what it monitors when varies. As Jack describes, at times your monitor the on coarse grained controls but then switch to monitoring fine grained information like the DOM.

Next comes analysis. That component will take the information complete information from the last sweep by the monitor and analyze it each time to generate a single conclusion or "event to pass to the decision component.

Decision component will take the event information along with its own internal state and decide what action to take and whether to change states.

Each of these have their own state machine and handle changes in events concurrently.

At the high level, these will combine into a "trader robot" that will monitor, analyze, decide, and take action.

The advantage of state machines is that they are designed to deal with the back that events may arise that are unforeseen. So the state machines have the ability to "get confused", handle confusion with certain actions, recognize events and finally reset to a known state to continue.

That's all very critical in trading to avoid the software "crashing" or failing in trading conditions that confuse it.

This system would simply sideline until it can figure out what's going on.

The it can be "debriefed" that is, the logs of any confusion and charts can be analyzed (sorry this requires a human being!!!) to add new states or events to account for it's confusion so it will know what to do the following time.

The beauty of this is that we can build in a few states and conditions. See them work and then iteratively add more and possible discover others.

I think this describes in a nutshell what Jack refers to as a multidimensional matrix of market operating points that the market moves between.

If not, we'll refine it. Any suggestions?

Sincerely,
Wayne

 

[greaterreturn]

Seems noone is posting.

Anyway, it doesn't seem, after more study, that state machine will do the job appropriately.

State machines always start each decision with your previous state and see where you're allowed to go from there. If events that occur don't fit that state's expectations, well it ignores or gives up entirely.

This seems to be more a call for perhaps a "decision tree" design.

I'm looking at a few decision support system patterns to figure out what will be the quickest to start using and be easy to extend and improve over time.

We'll see. Any other ideas?

Wayne

 

[ehorn]

Nice thread. Thanks for starting it. We have common goals and collaboration is a great approach.

Jack speaks about how the PE algorithm can be described as modules which stack on top of each other. These modules accept inputs and create outputs for storage and/or inputs into other modules.

I have attached an artifact which captures some of his recent posts describing this module based approach. It has become the baseline for my development efforts.