Verification of systems using Long/Short success

C

cipherscribe

I'm designing some automated systems, and would like to know what others use to verify the validity of the backtest results. Clearly there is nothing better than trading your system in realtime, but prior to this step, what do people use as go/no go tests?

I use walk-forward tests and different timeframes. Then I test on different instruments in the same categories: ie different stocks for a stock system, different futures for a futures system. I generally test that the system works similarly long and short, however I have found something that works quite well long, but fails dismally going short. Would this fail your go/no go criteria?

What other methods are used to validate backtest results?

Adrian
 
Quote from cipherscribe:

I'm designing some automated systems, and would like to know what others use to verify the validity of the backtest results. Clearly there is nothing better than trading your system in realtime, but prior to this step, what do people use as go/no go tests?

I use walk-forward tests and different timeframes. Then I test on different instruments in the same categories: ie different stocks for a stock system, different futures for a futures system. I generally test that the system works similarly long and short, however I have found something that works quite well long, but fails dismally going short. Would this fail your go/no go criteria?

What other methods are used to validate backtest results?

Adrian
More...

Those are some difficult questions that have fuzzy answers.

Some things to keep in mind though; foremost, the fundamental concept of the system must be sound. There are concepts that are tried and true and work no matter the product or timeframe. Does your system capitalize on a fundamental concept, or, does it use patterns or combinations of optimized indicators?

Usually, a system with a sound fundamental concept works equally well on both long and short trades but tends to have a long bias (i.e. not as good on the short side due to the upward drift inherent with most stocks) - one should be able to flip the logic and tweak the short side a bit to get a tradeable model.

Poorer (but not bad) performance in the short side is usually found in mean reversion based systems but should not be prevalent in volatility based systems.

In terms of validation, what exactly have you tested? What is the systems frequency and what type of data and how much of it are you using?
 
Quote from cipherscribe:

I'm designing some automated systems, and would like to know what others use to verify the validity of the backtest results. Clearly there is nothing better than trading your system in realtime, but prior to this step, what do people use as go/no go tests?

I use walk-forward tests and different timeframes. Then I test on different instruments in the same categories: ie different stocks for a stock system, different futures for a futures system. I generally test that the system works similarly long and short, however I have found something that works quite well long, but fails dismally going short. Would this fail your go/no go criteria?

What other methods are used to validate backtest results?

Adrian
More...

Just about any system that is tested over a relatively long time frame will do better with a long bias, because of the market's upward drift (as mentioned by Mike). That being said, you could try to filter the system to operate with more of a long/short bias, depending on the type of regime you are in. Identifying that is a challenge: but it's been mentioned on the boards here, and is fairly intuitive.
 
I'm testing an intraday futures system that takes a maximum of 1 trade per day. Testing over 90 day/1 year/3 year timeframes I get an annualised return on the longs of about 100%, however the shorts are around -40%. Way too big a difference to be the long bias. The last 2 years are a great test, given the massive decline in 2008 and the appreciation in 2009. However my long side shows a nice equity curve, relatively unaffected by the shocks. The short side is just increasing losers and drawdown.

My post was primarily driven by my lack of indentification of the short trade failures. Generally this is an alarm bell that my coding is wrong somewhere, but it's a simple system, and the code is correct.

I guess I'll just start trading it in sim and test it out. Thanks for your input.

Adrian
 
Quote from cipherscribe:

I generally test that the system works similarly long and short, however I have found something that works quite well long, but fails dismally going short.
More...

What good reason is there to assume that market behavior is the same in updrafts as in downdrafts? Volatility is higher in bear markets for starters.
 
Quote from cipherscribe:

I use walk-forward tests and different timeframes. Then I test on different instruments in the same categories: ie different stocks for a stock system, different futures for a futures system.
More...

There is no rigorous basis as to why a system that works for one stock should work well for another stock. If it works well for the stock you want, why should you care if it works well for another stock? And if it works well in backtesting frt hundreds of stocks, this tells you nothing about the stock you will trade.

Quote from cipherscribe:

I generally test that the system works similarly long and short, however I have found something that works quite well long, but fails dismally going short.
More...

Again, there is no rigorous basis for such assumptions. Some systems work well long-only and some others short-only. IMO. maybe you have been cheated by some authors who claim that systems should work well in both directions. Remember, most trading book authors have never traded anything.

Quote from cipherscribe:

What other methods are used to validate backtest results?
More...

EASY. Just trade your system with a reduced position size for 3 to 6 months. There is NO cheap way to validatate systems. All methods that rely on historical data or paper trading suffer from the same shortcomings. Trading is an expensive business.
 
Quote from cipherscribe:

I'm testing an intraday futures system that takes a maximum of 1 trade per day. Testing over 90 day/1 year/3 year timeframes I get an annualised return on the longs of about 100%, however the shorts are around -40%. Way too big a difference to be the long bias. The last 2 years are a great test, given the massive decline in 2008 and the appreciation in 2009. However my long side shows a nice equity curve, relatively unaffected by the shocks. The short side is just increasing losers and drawdown.

My post was primarily driven by my lack of indentification of the short trade failures. Generally this is an alarm bell that my coding is wrong somewhere, but it's a simple system, and the code is correct.

I guess I'll just start trading it in sim and test it out. Thanks for your input.

Adrian
More...

How many years of data total and how many trades? Walk forward testing does have merit but should not be used on its own.

What does the return distribution look like for the entire sample space?

Are you trading this on a portfolio? If so, how did you select the portfolio? Do you have another portfolio you can test it on?

Are you looking for a specific behaviour in a specific set of products, or , are you looking for behaviour across the entire futures universe? I'd be careful of the first point since your portfolio selection can introduce bias.
 
Thanks for the information. It all helps. The system is futures only, so there is no surviorship bias.

After having spent more time on the system, it further indicates that good assymetrical systems exist (well in terms of my backtest results that is!).

Next step is test in in sim for a few weeks to validate the backtest results, then implement live with a smaller contract size.

Thanks all for the feedback. Attached are the backtest stats.

Adrian
 
You must log in or register to reply here.
Back
Top