backtest for 3 years, blow up in 3 days,

Eight · Sep 16, 2009

Quote from JaiSreeram:

Eight, you have indeed done an interesting study. Is it possible to post the results of your study? Thanks!

I'm looking at SierraCharts for charting and Openquant for testing and perhaps for implementing... with an eye towards TickZoom for later on when I'm profiting like a proper business...

Regarding Openquant I'm going to be checking it's backtest results rigorously until I either prove it's great or know how to avoid pitfalls.. if it has any... I have a good bit of experience with it and I don't really find any problems yet much at all... but a bad backtester can be a real disaster...

..... eternal vigilance my friends, eternal vigilance shall be our watchwords and guideposts... [man I get carried away, I can type as fast as I think nearly and I love seeing my work up in pixels

]

SuperCruz · Sep 17, 2009

Quote from ronblack:

............Someone should contact this author and ask him to provide the full details on a nondisclosure basis. Then, a team of experts should go ahead and try to replicate the results of his study. If the results are confirmed, there may be valid grounds for a class action against the vendors.

ronblack , you have made a good suggestion

However, if the author provides the results on âa nondisclosure basisâ, then how can we ever forward the same to experts in order to check out their correctness or validity?

JaiSreeram · Sep 23, 2009

Quote from Eight:

I'm looking at SierraCharts for charting and Openquant for testing andperhaps for implementing... with an eye towards TickZoom for later on when I'mprofiting like a proper business...

Regarding Openquant I'm going to be checking it's backtest results rigorouslyuntil I either prove it's great or know how to avoid pitfalls.. if it hasany... I have a good bit of experience with it and I don't really find anyproblems yet much at all... but a bad backtester can be a real disaster...

..... eternal vigilance my friends, eternal vigilance shall be our watchwordsand guideposts... [man I get carried away, I can type as fast as I think nearlyand I love seeing my work up in pixels ]

Thanks, Eight for the interesting reply.

The last paragraph was simply out ofthe usual replies.

limitdown · Oct 7, 2009

Quote from d08:

Good thread, the already mentioned importance of out-of-sample data and sample amount cannot be stressed enough.
Also I find it important to have as few parameters as possible to minimize to minimize curve fitting, while this may result in a higher profit factor it will also most probably be a curve fit.
A question for all - what sort of annual % returns, average monthly drawdowns, profit factors do you get in your backtests and to what degree does it agree with live trading.

the basic premise of backtesting is when going live, even in simulated mode, that the events are sufficiently dispersed to cover a broad range of randomness, so as to cover all possible or most possible outcomes in the futures.

the reality is, these are singular events and news issues we're running through in our lives now, and nothing is repetitive or predictive,

the best that can happen is to strip the system of its methods, usually called trade entries and see if those repeat and then try like the Dickens to manage your trade positions based on reality and news events as they occur

always remember to cover often and not fret over the lost runaway trade that did better, because protecting the principle is the objective...not curve fitting or predictive analysis through exhaustive reverse testing.

and yes, I am available for consulting...

trader_david · Oct 16, 2009

people backtest, they trade in history chart or data. trading is a forward activity: need vision in the near term or a little bit far away from current moment, the distance depends on what timeframe you trade.

for a scalper, maybe next several ticks.
for a swing day trade, maybe next half hours or next remaing trading session..

big mistake people made in backtesting is they assume the history will repeat. that is wrong. every moment is unique. do not obsessed with backtesting analysis. son sim trading is useless, backtesting is useless too. actually they do lots of harm to you, not do good to you

that will not help. you should focus on next monent (what will happen), then you will be in right track to success.

iuykcif · Oct 16, 2009

Quote from jacksmith:

One common story I had heard about trading system is that,

Successfully backtest for 3 year's data, yet the system blows up in 3 days,

what causes this ?

Thanks.

There can be various reasons.

The 2 most remarkable are:

- Overfitting
- Not proper data filtering

The first one is the most insidious and hard to fight. Also because people tends to wanting to believe the "too good" results that overfitting yields.

Also depends on strategies, strategies based on signals are more affected by this problem.

Cheers,

Tom

-------------------------------
Browse my daily diary:
<a href="http://www.datatime.eu/public/gbot/2009Oct16/default.htm" target="_blank"> (IB forward trading on <b>Futures</b>) </a>
_

d08 · Oct 19, 2009

Quote from trader_david:

big mistake people made in backtesting is they assume the history will repeat. that is wrong. every moment is unique. do not obsessed with backtesting analysis. son sim trading is useless, backtesting is useless too. actually they do lots of harm to you, not do good to you

that will not help. you should focus on next monent (what will happen), then you will be in right track to success.

There are patterns of behaviour, that's just what a backtest is (if done properly). Scalping, swing trading all rely on price action (mental unconscious backtest if you will). Saying backtesting doesn't work at all is just simply childish...

CET · Oct 19, 2009

Some somebody totaled their car because they thought they could drive while only looking in the rear view mirror. Sweet.

enochbenjamin · Oct 19, 2009

Backtesting does not, cannot take into consideration how your emotions will effect you with real money on the line.

It is easy to watch price oscillate from profit to loss and back when using monopoly money. When it's real money emotions combined with - click <flatten all> will change your "automatic" systems results.

d08 · Oct 19, 2009

Quote from enochbenjamin:

Backtesting does not, cannot take into consideration how your emotions will effect you with real money on the line.

It is easy to watch price oscillate from profit to loss and back when using monopoly money. When it's real money emotions combined with - click <flatten all> will change your "automatic" systems results.

That's a given, you need to be comfortable with the risk level so you manage to control emotions, way easier to say than actually do. In general I agree that the execution side is underestimated, negative slippage needs to be realistic but positive slippage can occur as well. <flatten all> is the same with discretionary trading, there really isn't as big of a difference between mechanical and discretionary as people seem to think - one is 80% preparation, 20% execution and the other 30% preparation, 70% execution. Just my take on things...