backtest for 3 years, blow up in 3 days,

[KCalhoun]

Quote from total_keops:

. . .

Not doing a forward test.

This is refered to as curve fitting.

Quote from ProfLogic:

Great simple answer that will be ignored for it's deadly accurate simplicity.

absolutely true (I used to be a statistician)....

Back testing is thoroughly worthless. That concept is sold by vendors who can't prove they're profitable.

What counts is, the RIGHT edge of the chart, because volatility patterns and markets change.

What's even more important, is money mgmt/position scaling/dynamic position sizes.

-k

 

[intradaybill]

Quote from KCalhoun:

Back testing is thoroughly worthless. That concept is sold by vendors who can't prove they're profitable.

You may be right in some respects but there are some objections. I use backtesting not as a confirmation that a strategy will work in the future but to determine whether it has worked in the past. I think in that respect backtesting is not worthless. It becomes worthless when traders rely on it for strategy future performance rather than past evaluation.

Would you trade a startegy that had a 90% DD in recent years based on backtesting?

Another question:

Would you feel comfortable trading a strategy that is not backtested when you know it could be?

 

[Eight]

Backtesting with optimization can't prove that something is not random... thus you can backtest for three years and blow up in three hours even...

And backtesting with forward testing can't prove that anything is not random either...

 

[slickpick]

Quote from total_keops:

1- Not doing out of sample tests.
2- Not doing a forward test.
This is refered to as curve fitting.

Curve fitting has been notoriously inaccurate, a stochastic model would be much better. However, basic stochastic models are only so good until you get into the tails.

 

[TraderSystem]

Quote from intradaybill:

I think part of the conclusion can be found in this paper from another thread. The author of the paper, a well-known expert in this field, claims that two widely used backtesters have significant defects and produce wrong results because of basic errors in their algorithms

Backtesting Flaws in two Popular Programs

Thanks a lot for the link, intradaybill. The paper was interesting. Any idea about which software it was about?

 

[heech]

Quote from TraderSystem:

Thanks a lot for the link, intradaybill. The paper was interesting. Any idea about which software it was about?

I looked at the first section of the paper. I don't understand the author's point.

He claims that these backtesting tools skipped certain entries/exits by mistake... what if it wasn't a mistake, but rather design? Why is it fundamentally wrong to implement a strategy that doesn't set stop/target losses until the *following* bar after an entry? I do the same in my code.

 

[intradaybill]

Quote from heech:

I looked at the first section of the paper. I don't understand the author's point.

He claims that these backtesting tools skipped certain entries/exits by mistake... what if it wasn't a mistake, but rather design? Why is it fundamentally wrong to implement a strategy that doesn't set stop/target losses until the *following* bar after an entry? I do the same in my code.

It doesn't make any sense to skip a bar when checking for a stop. Actually, this is not good trading practice. You may end up losing multiples of your stop-loss amount.

 

[heech]

Quote from intradaybill:

It doesn't make any sense to skip a bar when checking for a stop. Actually, this is not good trading practice. You may end up losing multiples of your stop-loss amount.

Well, there is still some debate as to whether stops themselves make sense. I don't see why anyone would take as dogma that it's simply fundamentally wrong to wait a bar before setting stops/targets.

Especially if 10 years of back-testing results suggest it's a significantly better solution.

 

[intradaybill]

Quote from TraderSystem:

Thanks a lot for the link, intradaybill. The paper was interesting. Any idea about which software it was about?

I have no idea since the author was careful to hide their identity. I can only guess but that is not good enough.

It boils down that many people have been taken for a ride by companies that did not do basic testing before selling their software.

 

[intradaybill]

Quote from heech:

Well, there is still some debate as to whether stops themselves make sense. I don't see why anyone would take as dogma that it's simply fundamentally wrong to wait a bar before setting stops/targets.

Especially if 10 years of back-testing results suggest it's a significantly better solution.

It is bad if the trading system designer was not intending of doing that but the software did it behind his back. That is the point. I think it is explained in that paper very well.

I also think the difference is important between what you personally design and what you think you are designing because of some faults in the programs used.

I think the difference is clear, to me at least.

I believe stops not only make sense but they are mandatory for trading success. Traders who do not place stops (mental at least, you don't have to place the order in advance) sooner or later blow up.