This is an interesting article: In 2 parts due to length.
he Manual Work in Mechanical Trading: Evaluating a Trading System
by: Nigel Bahadur
Not everything in mechanical system trading is automatic. Before you begin trading a system, you must evaluate it cafefully.
On May 6, 2106 Rebecca Adams woke up at 6:30 a.m. and started preparing for her day. She showered, dressed, gathered her beach equipment and then went downstairs for breakfast. At 7:45 a.m. she stepped into her office. Displayed on the industry standard, OLED, wall-sized monitor were the results of her overnight trading. Her profits from trading the Shanghai 1000 e-mini futures contracts were offset by her losses from the Bombay 50 options. But overall she had booked a nice profit while she was sleeping. The computer informed her that there had been a communications glitch but that the backup lines had kicked in so her positions were never in jeopardy. Had there been a problem that couldnât be rectified, the computer would have woken her up with a phone call or sounded a shrill alarm guaranteed to wake the dead. She smiled with satisfaction, turned around and went to spend her day on the beach where she would kick her friendsâ butts in beach volleyball. The computer would continue trading while she was gone...
Many beginning system traders have been seduced by this fantasy, and as technology races ahead, portions of this fantasy edge tantalizingly closer to reality. However, a self-correcting, self-adjusting, automated-trading cash machine that will fulfill tradersâ fantasies remains as far off in the future as the stories of Isaac Asimov. For now, system traders will have to develop (or purchase) one or more trading systems and create an infrastructure and routine for implementing the trading signals. So given the time-consuming tasks system traders must undertake, this article will focus on developing and evaluating a trading system â either one that is being written by the end user or one that is being considered for purchase or lease.
Reality Check
Every day thousands of extremely intelligent folks (think Ph.D. types) try to find new ways to wring money out of market inefficiencies but have yet to develop a system that consistently makes money over a long period of time. Experienced system traders know this. Instead, they will trade a basket of systems on a basket of commodities on multiple time frames. Over the long run this is the only way to consistently make money with mechanical systemsâat least until we can program a machine to do the bulk of the work for us.
Of course many readers may disagree, but that depends on how a system is defined. After all, many traders have a single âsystemâ that they use day in and day out, and they make a good living trading it. However, these âsystemsâ allow some wiggle room for the trader. For the purposes of this article the term âtrading systemâ or âsystemâ will mean a mechanical trading system that has strict entry and exit rules with no room for discretion.
It is extremely hard, if not almost impossible, to consistently make money over a long period trading a single mechanical system. A single robust system generally has only a small edge, which will dull over time, and returns only a small profit on a risk-adjusted basis. Systems must be combined into a portfolio in order to maximize returns and reduce risk.
Take a Good, Long Look at a System
There are hundreds of statistical measures that any trading system can generate. Regardless of what numerical attributes of a trading system are used for evaluation, all of those factors are based on historical data. Without the luxury of peeking into the future, the best you can do is monitor the system and compare the performance to historical norms. When the system starts to deviate â and it will â the trader will have to evaluate the reasons for the deviation and determine if the underlying assumptions on which the system is based still hold water.
Over time I have arrived at my own favorite measures for evaluating systems. Out of the hundreds (maybe thousands) of measures, Iâve narrowed mine down to 16, listed here. Although this list may not necessarily be the best set of factors to use, they work for me and can be a good starting point for newer traders.
1. Underlying premise. The first question that should be asked when looking at a system is this: Whatâs the underlying assumption or driver of the system? Then a trader should ask if that underlying assumption makes sense. Every system has an underlying driver that should make intuitive sense to a trader. In my experience most robust trading systems are usually based on very simple underlying drivers.
2. Profitability. Many of you reading this will say, âDuh! Of course Iâll look at profitability.â But what I mean by this is to look at whether or not the system is profitable on a single contract. Many systems use money management that varies the number of contracts on each trade. But remove that and the profitability on a single contract is not so enticing â in some cases the system may even be a losing system. Varying the number of contracts is used to control risk and leverage the system; it should not be used to turn a losing system into a winning one.
3. Profit factor. This is just another way of looking at profitability. It is calculated by dividing the gross wins by the gross losses. A profit factor of more than 1.0 is a winning system. Short-term trading systems tend to have profit factors less than 2.0 with a large number of trades.
4. Drawdown. What is the maximum drawdown as a percentage of the profits made? A drawdown of $100,000 in order to make $100,000 over five years is not attractive. Where did the max drawdown occur? Was it at the start, end or middle of the equity curve? If itâs at the end (i.e. recently), itâs a warning sign that the system is starting to fail â unless itâs due to unusual circumstances like 9/11.
5. Percent winners and average profit per trade. What is the percentage of winning trades? A high percentage of winning trades will be accompanied by a lower profit per trade and a higher average loss per trade. If itâs not, you are, more than likely, looking at a system that will not hold up over time. Does the underlying premise correspond with these numbers? For example, breakout trend following systems tend to have a lower percentage of winning trades, but the average winning trade will be larger than the average losing trade.
6. Testing period(s). What was the length of the testing period? What types of markets did that period cover? For systems based on daily charts, there should be, ideally, ten years of data. The market data should show multiple uptrends, multiple downtrends and multiple sideways price action, so you can see how the system handled each type of action. Did the system catch the majority of the action that it was designed to catch? For example, if the system is a trend following system, it should have caught most of the trends.
Next, look at how the system was developed and tested. Ideally, the developer would have taken the data and broken it into three sets. The middle 60 percent would be the development set (the set on which the initial testing was conducted). The first 20 percent and last 20 percent would be the out-of-sample set. Out-of-sample data is data on which the system is tested that was not used in the initial system development. At best, testing on this data set is done once and only once â it either works or doesnât. Some development shops will only allow their developers access to the development set. They arenât even allowed to see the out-of-sample data. Instead, the system is handed off to others for the out-of-sample data testing â this goes a long way towards preventing curve-fitting.
7. Sample size. The biggest and most difficult factor that system developers have to guard against is accidental curve-fitting of data. A large sample size goes a long way towards mitigating that risk. So the larger the sample size, the better. For short-term swing and day trading systems I want to see 200+ trades. For long-term trend following systems I like to see 50+ trades. The sample size can be on one market or across multiple markets.
8. Consecutive losers. Iâm simply looking at one thing here â can I psychologically handle the number of consecutive losers that the system has had in the past? If the system had ten consecutive losers, can I handle that? Will I stop trading it in frustration after losing five times?
9. Equity curve. You do not want to see a 45-degree straight line on the equity curve. If there is one, then chances are it is curve-fitted and will not hold up over time. Of course there should be a steady trend up, but a robust system will be punctuated by drawdowns. Another item to look at here is the length of time it took to recover from the maximum drawdown.