backtest for 3 years, blow up in 3 days,

Quote from jacksmith:

One common story I had heard about trading system is that,

Successfully backtest for 3 year's data, yet the system blows up in 3 days,

what causes this ?

Thanks.
More...

The cause is confusing a closed system with an open system.
 
I disagree with the statement that you *must* do out-of-sample backtesting - not that it will hurt.

From hard-earned experience, here is my own advice:

- Use a simple set of trading rules (limited number of parameters).

- Apply these rules to enough historic data that you generate THOUSANDS of trades during backtesting. This is something that Keith Fitschen stresses in his seminars and one of the main reasons he has such a good track record as a system developer. You can't draw statistically significant conclusions about your trading system until you have a large population of trades to look at.

- Live trading will demonstrate whether you have made realistic assumptions about slippage and liquidity. Often, systems test out really well on thin markets that cannot actually be traded. Keep trading size small at first.

- Once you've acquired an edge, emotions and discipline will be your biggest battles.

- There will be many times that you will want to override your system. Sometimes this will actually be the right thing to do. These occasions should be rare. How good a discretionary trader are you, really? And remember that there were no overrides in your backtests.

- Things go wrong. You will make mistakes. Your broker will make mistakes. Orders you think you've placed will not get executed. The exchange will bust your trades. A storm will cause a weeklong power outage in your area. Your broker will go out of business (think Refco). The exchange will change the rules in a way that breaks your system. Plan ahead but expect that some things will be out of your control. Stay on top of everything, and check everything.

- Diversification (different systems, markets, and timeframes) is a valuable and worthwhile goal. To reach it, you will need to automate as much as possible.

- Don't assume that a profitable system will remain so. Markets change.

- You can do it. You actually can succeed and become a profitable trader. And it will be more work than you think.
 
Most people are testing absolutely random stuff. They will get some good results in-sample, some not quite as good results out of sample and then lose money when it goes live... and they will study money management along the way. That will not take a random strategy and make it profitable, in fact it's best use is to make a marginally profitable system somewhat more profitable... that is the most it can do for anybody really...
 
Quote from Diode:

I disagree with the statement that you *must* do out-of-sample backtesting - not that it will hurt.
More...
Diode –

What you and many others have been describing is not just a back testing procedure but your trading method. Most of these I have read seem to be a combination of rules of thumb and simple procedures. Many of them do not require out-of-sample testing because:
- They are too few parameters or degrees of freedom to be curve fit.
- Your strategy is general purpose and works in any price directional movement and volatility.

Many of us, however, have strategies that trade specific directional or volatility movements. These strategies require a trading method that must do out-of-sample back testing. If it is not performed results will not be profitable.

In this forum and others “newbie’ traders try to digest all of our rules of thumb as gospel. They do not have a clue that they must build a trading method for their style of trading. In this process they must discard 60 to 80% of our rules of thumb and write rules that are their own hard earned trading method that describe the envelope in which their strategies must perform.
 
Quote from JimDay:

I back tested my system and every time my RSI went below 29.431 and my 23.6 bar moving aveage turned higher in the past year if I would have bought I would have made money, I am ready to trade the SP500.
More...
I've seen this happen. Tradestation's optimization routine comes to mind !
 
Good thread, the already mentioned importance of out-of-sample data and sample amount cannot be stressed enough.
Also I find it important to have as few parameters as possible to minimize to minimize curve fitting, while this may result in a higher profit factor it will also most probably be a curve fit.
A question for all - what sort of annual % returns, average monthly drawdowns, profit factors do you get in your backtests and to what degree does it agree with live trading.
 
Quote from d08:

Good thread, the already mentioned importance of out-of-sample data and sample amount cannot be stressed enough.
Also I find it important to have as few parameters as possible to minimize to minimize curve fitting, while this may result in a higher profit factor it will also most probably be a curve fit.
A question for all - what sort of annual % returns, average monthly drawdowns, profit factors do you get in your backtests and to what degree does it agree with live trading.
More...

I have a more fundemental question. What premise is the system based on. That the most important factor. If the premise is not sound it does not matter how good the results look.

Another factor which needs to be studied is the 3D surface of the optimization space.
 
You must log in or register to reply here.
Back
Top