As the results of the system Iâm trading here have varied over time, I want to introduce a measure of system quality that will allow me to monitor the systemâs performance and detect when it is not performing well.
The most obvious and easiest approach is just to monitor the amount of profit (in either points or dollars) being made per trade. However, that has a serious disadvantage, particularly these days. The problem is that as volatility increases, the amount of profit/trade will increase, and conversely as volatility drops. Therefore, changes in volatility can mask the results of model performance.
I believe a better solution is to use the percentage of perfect that the system is achieving. 100% percent of perfect is defined as the amount (in dollars or points, it doesnât matter) that a system would make if it was long every day the market went up, and short every day the market went down. Because this system is trading at the close, the daily calculation of 100% perfect is measured as the absolute value of the change from yesterdayâs close to todayâs close. If a system was perfect, it would always generate that amount of profit.
But since real systems arenât perfect, the profit they produce will be less. So to calculate percent of perfect achieved by a system, just take the profits it generated over a given period of time, and divide that by the amount that a perfect system would have generated. This measure then cancels out the effects of volatility, and allows us to see how the system is doing over time.
What kind of performance levels should we expect? In his book âDesign, Testing, and Optimization of Trading Systemsâ, Robert Pardo talks about this same concept, which he calls âmodel efficiencyâ. (I actually like that term better, but will stick with âpercent perfectâ, as that is how the BioComp programs report this metric) According to Pardo, âExperience has shown that trading models with model efficiencies of 5% and better are good based on closing prices.â My experience is that 5% may have been good back in 1992 when Pardo wrote his book, but I think it is possible with modern modeling technologies to have reliable systems that produce over 10% perfect for long periods of time.
And that brings up another key concept, which Iâve discussed in earlier posts in this thread. It is quite possible to achieve much higher model efficiencies, if one is willing to tune the model to a shorter window of time. The tradeoff in doing this, of course, is that the effective life of the model is reduced. Itâs somewhat like a highly tuned dragster, which will perform well in a short quarter-mile sprint, but which would fall apart trying to complete the 24 hours of Daytona.
Turning now to the system Iâm trading, it definitely leans towards the dragster end of the performance spectrum. It has high efficiency, but probably a relatively short life. (just how long remains to be seen)
I built the model on 8/28/08. Since then it has performed at a level of 31.6% perfect, neglecting commissions and slippage, which I consider to be trader-dependent. This is a very high level of performance, unsustainable over the long run, but Iâve been happy to have it so far. But while the aggregate performance level is interesting, Iâm really more interested in how the system is doing lately, because that will tell me if its nearing the end of its life. In order to give more visibility to recent performance, Iâve started using a sliding window that measures model performance in terms of percent perfect over the last six trades. Thereâs nothing magical about the number six, it just seemed to feel right, but I might change it as we go on.
Using that 6-trade sliding window, performance has measured as high as 76% of perfect (in October), and as low as -20% of perfect a couple of weeks ago. That last number included a couple of large losses, which have now dropped off outside the 6-trade window, so the current figure is 33.44%. That is a very nice number, and I will happily take that, or even a number in the twenties, and continue trading.
The harder question is what to do when you get a low or negative number. For one thing, I think it is a signal to reduce the number of contracts you are trading, and keep your position size down until the performance rises again. But it is important not to jump to premature conclusions about the systemâs performance, because systems often hit rough patches and bounce back. (As I write, the current system seems to be a good example of that). Keep in mind that a system that has a win rate such as this one can expect to run into 5 consecutive losses while operating within its normal parameters. That would be very rare, but it can happen, and doesnât signal that thereâs anything wrong with the system.
With that explanation out of the way, I will add this new performance metric to the trade reports going forward, and it will be interesting to see what we can learn from it.
