Why Optimization Won't Kill My System and Will It Work In The Future?

I'm curious why you expect that a system must work across multiple markets to be valid?

I didn't say that the system has to work across multiple markets to be valid, but that if the system did work well across multiple markets, it increases the likelihood that the system has successfully captured some underlying characteristics relating to what drives markets in general (fear, greed, etc.).

Obviously there are many systems that try to exploit certain characteristics that only exist in a specific market or instrument (such as economic reports for indexes, or crop reports for some commodities). You would not expect a system designed to exploit these specifics to work well across all markets.

However, if a system did indeed perform well across numerous markets, then it is more likely that system will continue to perform in the future because it is more likely that system has captured a general underlying characteristics of all markets (fear, greed, etc.) than just specific characteristics (economic reports, crop reports, etc.).

This was the case for the Turtle system and Aberration, which was capturing the underlying nature of trends that were prevalent during the 70's, 80's, and somewhat in the 90's.
 
Here are some things I have observed as a system trader and user of TradeStation:

1) Optimization works well. In fact, to not optimize can be a deadly mistake. By not optimizing you might have inadvertently chosen the worst possible parameters and would not know it until after you have begun trading the system and lost money. With the right tool you can even view your parameter choices as a 3D visualization. Surely this should tell you something.

2) The larger the number of trades and the years backtested, the more reliable the system. Quite frequently we read that 30 trades is the minimum. I have never heard where that seemingly arbitrary number originated, but it is not always an accurate indicator. You have to examine each system on a case by case basis to determine whether or not it is reliable. Systems that have 5 years or more of data and 200 or more trades are quite reliable. You must still examine their annual performance, however. I have at least one volatility expansion system that stopped working in the last year, and it has a couple hundred trades backtested. It makes perfect sense, because the stock price fell to around 5.

3) A system should be looked upon as an individual, specialized unit. Period. If you don't understand this, then you simply have not developed enough systems--and the quantity of them does not have to be huge. While it is nice if a given system works on a different stock, futures contract, or whatever, it should not be expected to do this.

Example: you wrote a system to trade Compaq stock. Do you now try it on the Japanese Yen? No!! You might try it on a stock in the same sector, such as Dell or Gateway. Don't expect it to work, but it probably will to at least some degree. However, when you take a system and try it on something with vastly different characteristics the best you can hope for is lowering the risk-to-reward ratio, eventually reaching 1-to-1. This is undesirable.

Unfortunately, there is so little published information on the topic of using systems on different entities that a lot of traders confuse the issue and get lost. I think a lot of them get the wrong idea from having read Market Wizards and holding that as the standard. By contrast, system developers Charlie Wright and David Stendahl tell their trader audience not to try using a system on something greatly different. When you force a well designed system on something it was not designed for you are trying to fit a round peg in a square hole.
 
I hate to be a wet blanket or sound negative but I don't want a bunch of people throwing their money away on something I know doesn't work. Take this to the bank: no intraday moving average system will be profitable on the stock indexes. I don't care how much you optimize or how many times you redo the parameters. The stock market has way too much backing and filling to use moving average crossover systems on.

You may have produced some backtesting profits during one unusual period of market activity, but that does not validate your system, no matter how many trades you did.
 
A system should be looked upon as an individual, specialized unit. Period. If you don't understand this, then you simply have not developed enough systems--and the quantity of them does not have to be huge. While it is nice if a given system works on a different stock, futures contract, or whatever, it should not be expected to do this.

Like I said before, there is nothing wrong with developing a system for just one instrument, but you run the risk of having found an over-optimized system that just happens to work on historical data and some period of walk-forward testing. That probability is less likely if that system works on other instruments or markets. Take a look at the subject of this thread - this is the issue I am addressing.

Let's take an example. Suppose you developed a 100% mechanical way to trade bull & bear flags by identifying specific price, volume, and volatility chart patterns. Ask yourself why bull & bear flags occur, and if they occur in Compaq should they also occur in Dell? What about an unrelated stock or commodity altogether?

My point is do not arbitrarily optimize on various indicators and moving averages to hope you find some mystical correlation that happens to back test and perform well on out-of-sample data. Instead, define what elements of the market/stock/index/sector your system is trying to capture and start from there.

And yes, I trade the same 100% mechanical system on multiple indexes, commodities, and stocks. The only change I make is to adjust for tick size.
 
Systems trading is a difficult subject and is made more difficult by a lack of robust software to test an opinion.

A few observations, fwiw:

1. All markets I've researched have the following common traits: non-linearity, non-stationary means and stochastic volatility. Because of this, I believe that it's nearly impossible to generate consistent profits with a system that utilizes canned analysis techniques (ADX, EMA, RSI, etc). I've had some success with expert systems, pattern analysis / recognition and non-linear methods.

2. Anyone who relies on calculations of return, drawdown, run-up or largest win/loss in dollars is heading for a blowout. A $5,000 drawdown when the NDX is 3000 is very different from a $5,000 drawdown when the NDX is 1538. These must be calculated in percentage terms. Only percentage measures allow a trader to gauge the real risk of a market position. Export those tradestation files to excel and figure out what your real risk is!

3. I don't think anyone has touched on survivorship bias on this thread. Go ahead and run 5700 optimizations on your data. Who wouldn't be surprised to find a pair of indicators (or two) that "performed" well. Unfortunately, these "winning" systems will ultimately fail in the real world - because they have been selected on the basis of their performance on historical data and were likely curve-fitted. In addition, Tradestation can only optimize for gross profit and nothing else. One interesting exercise: take the data from an optimization run and plot on a 3D graph of P/L, Parameter1 and Parameter2. If you see sharp peaks and valleys around your "winning" systems values, it'll probably have unstable returns in the real world. A nice flat table-top would be the preference here.

4. Newbies (no offence) tend to present all their data to the model in optimization. It's better to divide the data into test, validation and out of sample subsets. This way, you'll see what happens when the tested and validated system is presented with new information. It's usually not pretty, but you'll preserve your capital.

5. Calculate a realistic commission and slippage figure and then double it.

6. There is no holy grail, so nobody is going to sell it to you. Do your own work, read everything you can get your hands on. Be prepared to be disappointed in the first systems you test. Be aware that all backtesting software has some pretty serious weaknesses, from when stops are active to calculating return on account...

7. Trading is one of the hardest things to do well, mechanically or otherwise.

Cheers,
 
Originally posted by dvegadvol


5. Calculate a realistic commission and slippage figure and then double it.
More...

Why we should double realistic com&slipp ? Maybe triple it?
DT-waw
 
It's always worse than you think it is. Commish is stable but slippage estimates (by people new to this stuff) are usually very low. To be conservative, sure triple it; worst case you'll have better results! Not profits necessarily, just better results...
 
agree on x2/3 the slippage.... though I doubt if you do that on an intraday system (anywhere upwards of 3trades/day type of system) you will still have anything reasonably profitable.

on longer term swing trades, x3 the slippage shouldn't hurt the performance much for a good system.
 
I like to perform a "severe slippage" scenario by buying at the high and selling at the low, rather than the stop/limit/market price determined by the system. I won't consider trading any system that is not reasonably profitable after a "severe slippage" test (e.g. various scalping methods perform well on backtesting but when traded real-time are marginally profitable or losers after slippage).

The longer your time frame, the less impact that slippage has, but it is interesting to see how well a system that trades using EOD or 60m bars would have performed if you had gotten the worse possible fill for entry & exit. "Severe slippage" can be murder for short-term intraday systems -- which is why I perform it to determine if that system is "tradeable" or not.

This is actually more reasonable than using a slippage multiplier. If you entered on a narrow range bar, the multiplier may add slippage that does not exist.
 
You must log in or register to reply here.
Back
Top