In laymans terms, what is backtesting?

Taking a theory today and apply it yesterday. Test its outcome


Software streamlines the process...google it

if you are a fundemental trader....test what happens 5 minutes before Greenspans/Bernancke statements and 5 minutes after...
 
In the most simple of terms:

Looking to see how a trading system/model/algorithm performed on historical data.

However, don't be fooled into the thinking that all because a trading model trades the past well it will work in the future.

Many of the blackbox trading systems, not all though, that you see for sale is just over optimized junk.

Simple ways to avoid over optimization:

Out of sample walk forward approaches.

Also, a good system, especially trend following models should be able to display profitability in a number of markers with the same parama

For example, if a model utilizes a 10 day and 20 day moving crossover for signals that approach should show profitability in a number of markets.

Keep in mind that financial markets equities, futures, forex etc. for the most part are randomness with trend components.

For example, if a model utilizes a 10 day and 20 day moving crossover for signals that approach should show profitability in a number of markets.

Keep in mind that financial markets equities, futures, forex etc. for the most part are randomness with trend components.

A simple distribution of financial markets adheres to a bell shape curve. With 2/3 of all outcomes within +1 and –1 deviations from the mean.

The other 1/3 is > than +1/-1 or simple put the market trends. That trend is your statistical advantage.

I know this is a "little" bit more that what you asked for but crucial nonetheless.
 
A great 2002 post from as guy who unfortunately spent little time on ET, Rogue Trader. Not to be confused with the troll Zzzzzzzzzzz who was once A Rogue Trader. Two different guys. Obviously.

Quote from Rogue Trader:

Of all the paradoxes that exist in trading, the notion of “Back Testing” could quite possibly use some “Back Testing” of its own. The literal meaning of “Back Testing” is rather self explanatory and actually defines itself with little explanation. “Back” is referring to past occurrences and “Test”, is a trial of a model.

Regardless of the level of sophistication, rigidity, flexibility, or time frame of your model, you help to define it and ultimately accept it. You are simply testing a method against past occurrences and analyzing the results. Conventional wisdom supports that the result of such testing can be optimized, and
when applied to the future, produce a positive expectancy. None the less, one obvious fact still remains. The market, which is what your modeling into, remains a variable. Upon further scrutiny though, a less obvious change also occurs. Your perspective. Unknowingly, you have begun to reinforce in your mind what should happen as a result of your positive experience in literal backtesting.


Once satisfied with your results, your back tested model will still require one more trial. A trial by "The Here and Now.” You have learned the literal definition of back testing above. It is now time for “You the Trader” to learn the traders definition of
back testing. In the traders definition of “Back Testing”, “Testing” simply means to put on the trade, as dictated by the system of your invention. "Testing" remains constant and mechanical as a function of your system. “Back”, on the other hand, is actually a variable.


Now with everything to your satisfaction and your methods in place, your back tested system triggers a trade and your day of discovery begins. This is what you can expect to discover. Everything you thought your system to be, wished it to be, hoped it to be, ever dreamed or believed it could be is absolutely irrelevant under a trial by “The Here And Now”. Go back and read that again.


Enter “Back”, the variable of the traders definition of “Back Testing”. Under a trial by “The Here And Now”, once entered into a trade, you are given but 3 choices. But wait a minute. This is strange. None of choices are related to the literal definition of back testing, or the positive "perspective" from which you saw your trades resulting in. “Back” (traders definition) becomes to mean.....

1) Back Off 2) Back Down 3) Back The Hell Out.

Back Off: This refers to reduction of trading frequency.

Back Down: This refers to reducing your share or contract size.

Back The Hell Out: It means just that. Neither you nor your system are in the flow of the market. Get out and re-enter when
conditions improve. Both yours and the markets.


There in lies a paradox of back testing. From the traders definition of "Back Testing" you derive a finite and constant set of prepared responses on every trade, every time. From the literal definition of "Back Testing" you derive a deceptively skewed departure from "The Here and Now". You only see what should happen, not what could happen.


Now ask yourself this..... If the correct response to any trade is a constant, regardless of all the variable methods, signal generators, systems, and market conditions, where does one find a constant? The answer is quite simple. You become the constant, constant "WITH" the market.


It is one thing to say you trade the markets. It is something completely different to say you trade "WITH" the markets. If you have a bias, a belief, or behavioral response from the past, regardless of origin, of what the market will do, should do, or could do, you are not trading "WITH" the markets, you are trading your past perspective of the markets. The day you decide to trade the markets perspective in the moment, and not yours, is the day you will be trading "WITH" the markets.


Oh, by the way. There is one more definition of “Back Testing.” This one is provided to us courtesy of the markets. It is for all those traders who choose to ignore the traders definition of back testing. The markets actually re-structure “Back Testing” to “Backer Testing”. Backer testing occurs when you, the trader, have to go out and find new ”backers” to fund your account, the one you just blew up, as you chose to “test” your trading account for the sake of “Back Testing.” Literally, that is.
More...
 
In laymens terms....one of the best marketing tools the "system sellers" have in their arsenal. That said, it can be a useful tool for establishing a point of referance. Beyond that, it's a fools paradise.
 
Well obviously the first place you need to start with is software that enables backtesting.

Some Performance Validation Software Platforms:

1. Trade Station (Can use Trade Bolt to trade signal via another firm)
2. Genesis FT
3. Trader Studio (Haven’t tried this yet but the portfolio testing capabilities look very good. They also do a good job of answering questions here on ET.)
4. Amibroker (More for the experienced)
5. (FREE in the Sense) - If you into foreign exchange trading the firm I’m at is rolling out a new platform within the next month or so for currency trading that will enable back testing capabilities. (See attached JPG - AaronFX) You will be able to auto enable a trading model on the platform so that all orders generated by the system are automatically executed.

There are others but these should keep you satisfied for a long time.

I’m told that for higher end traders an option will be available to host trading systems generated from this platform with a server based solution that will execute via their Currenex platform. (I have attached a JPG from the demo version).

If you need programming done they have guys who are all certified and can program in popular high level programming languages such as VBNet, C, C++ etc.

Next you need data in which feed your platform. This depends on what you are trading and the time frame you want to trade. For example, if you don’t plan on day trading you don’t need tick data (the most expensive). Therefore, you will only need daily data which can built daily, weekly and monthly charts. They are many places out there but they do vary in quality so check on the ET boards first.

Keep in mind that for historical backtesting of commodity futures markets you will need what is called back-adjusted data. More on this at –

http://www.aarontrade.com/html/futures_trading_prices.html

This one if you are new can be a little confusing. The reason for it though is to remove the rollover gap. That way your trading system is not witnessing a profit or loss on the artificial gaps created by futures contract rollovers.

Testing. This is where things get hairy.

It can be very easy to create a bunch of rules and curve fit them over past data. That is where the out of sample walk forward approach comes in.

You will very easily see the difference.

For example,

I create a system and back test it on data from 6/15/06 - 6/15/86. Then just keep changing the parameters around until I find something that works. This is the method you want to avoid. If you are purchasing a trading system this is also something to look out for.

The Out of Sample Walk Forward Approach:

I have 20 years of data in my files from 6/15/06 - 6/15/86

I build a system around data from 6/15/90 - 6/15/96

Then I backtest from 6/15/96 - 6/15/86

Then, keep in mind at no point have I seen any data from 1996 - 2006, I then “walk the system forward from 1996 to 2006. You greatly avoid the dangers of over optimization and curve fitting this method because you can’t fit rules around data that you haven’t seen!

Additionally, another very simple method of avoiding curve fitting and over optimization is by testing you system over various markets. You should be able to demonstrate profitability in a number of markets with the same rules. Keep in mind that there are exceptions to this. For example, if you have built a swing trading method for the SP futures.

More on out of sample building - http://www.aarontrade.com/html/out_of_sample.html

If you need more help I recommend giving the guys at Aaron Trading a call. Well if you want to trade futures or forex. Just tell him John the Drunk Irish Man sent ya. They have been of great help to me. The info above they taught me save me a bunch.

He recommended a primer book way back initially called The Dow Jones-Irwin Guide to Trading Systems which really helped built a solid foundation. Don’t know all who is who but the guys running the foreign exchange desk are former VP’s of FX at Speers, Leeds & Kellogg (SLK) which is a division of Goldman Sachs.

That should give you an idea of their mentality. I don’t think some of these guys every sleep! But when it comes to system building and trading in general I highly recommend them. Just don’t go off the commission schedule on their website as I’m paying the same as in my IB account on the discount side. On the foreign exchange side I’m paying $30 per million via Currenex. But I’m also paying on the higher side for a bunch of other services that are considered non-discount.

If you trade futures and forex they will be bringing on a lot of bunch of new platforms in the months ahead outside of Ran Order and J-Trader on the futures side.

One other item is make sure to double your drawdown figure. If you trading commodity futures you would add the margins together then add the drawdown figure x 2 to get the approximate capital needed to trade that portfolio.

Once you capital is large enough you will want to smooth you equity curve with a short-term or day trading system. Rule of thumb is the longer the time frame the greater the profitability but the larger the drawdown. Although the shorter the time frame the more “noise” in the data.

Use common sense. For example, I once seen a trading system for bellies. It applied some version of the Martangle strategy. At one point the thing was long like 75 bellies. While it looked great on paper you are not going to dump 75 bellies plus 10 more for a net short at one price. You will become your own slippage in this instance. Once again theoretically it appeared to be very profitable but in reality this would never work.

And that is the end of the longest post I have ever made!!!!
 

Attachments

You must log in or register to reply here.
Back
Top