Backtesting is useless

[Real Money]

I used to think it was useless, but now there are a few things I would like to backtest. Most of it kinda requires more advanced stuff than I am able to do.

Few ideas....

Distribution of deviations from VWAP during the cash open for index spreads.
Volume distributions of aggregated futures data at significant deviation.
Sector correlation rank type analysis for synthetic indexes.

(which sectors does my synthetic index track most closely?)​

Volatility distribution for the basis spread (index basis).
Some kind of ATR distributions for rate spreads to help me analyze the treasury fly.

A lot of this stuff is highly non trivial to get good analytics on. Many issues here, and there is latency sensitive stuff that could be a hurdle as well. There is much more to do with this type of stuff (maybe more than I even care to do).

 

[Buy1Sell2]

I think you may be taking an overly simplistic view! While the past does not infer the future, parts do rhyme. One can learn from history. IMHO, an important role in backtesting is to observe how your detailed trading strategy would have performed if applied to periods of history. This is valuable. The notion of backtesting to develop a trading strategy is one that should instill doubt and fear. (Attributes of curve fitting apply and relate to your "useless" comment)

you mean "imply" not "infer".

 

[heispark]

Visual backtesting using lagging indicators (most indicators are) is a perfect way to blow your account......

 

[Slope Trader]

JCDST1979,

I respect your opinion, but if back testing is useless how else would you know if what you are doing will make money?

I like back/forward testing because it allows me to practice my trading skills, day to day process, thinking, testing my discipline and patience, testing my risk management per minute, per 5 minutes, per trade, per day. Back testing test ME. Back testing keeps money in my pocket and keeps me form making future mistakes. It let's take what is on paper or idea and put it to test.

Back testing improves my odds of winning. Backtesting is free practice.

Well said... best answer in the thread from my perspective.

 

[777]

In the book Fooled by Randomness, Nassim talks that the history that unfolds is just one of the multiple possible ones, this makes backtesting completely useless, don't you think?

Billions have been made back testing paired with other meaningful things.

See: Medallion fund from.Renaissance Technologies.

Of course few people reading ET will be.able to successfully make anything meaningful from backresting except maybe being scared away from some really bad ideas

 

[SimpleMeLike]

Well said... best answer in the thread from my perspective.

Thank you Slope Trader. Back testing make clicking Buy and Sell easier.

 

[Heydrrich]

He's tacitly admitting to trading without any edge, my guess anyway.

Some confusion can stem regarding backtesting of a strict mathematical/"mechanical" rule versus manual backtesting by a discretionary or semi-discretionary trader on events as they happen in a recorded stream. I didn't realize the latter existed for many years. I am not sure what Al is talking about here, suffice to say the smart people he's mentioning are doing the former. If you're doing the latter and it's not comparable to your discretionary live trading, it's probably because your environment wildly differs (for instance, knowledge about the current market regime which you won't know about if randomly dropping into some 10 min bars somewhere, or you're not replaying events in real time, or with different software than your live trading, etc.).

If I had to guess, he probably has been trying some very basic bar patterns or a few indicator combination rules, rules that machines have tested trillions of times (or more) and that generally has no edge (from what I've seen, YMMV). In his belief system (as is rather common) this stuff should have an edge, so he's disappointed.

Tangentially, I own a couple of his books that I bought very early on in a batch. He has incredibly low signal to noise ratio in his writing.

So if I understand it correctly there are 3 types of backtests that you are mentioning

1) backtesting of a strict mathematical/"mechanical" rule
2) manual backtesting of a (semi)discretionary trader
3) backtesting of bar patterns/indicators which has been tested trillions of times and has no edge (the stuff you see on Youtube).

What types of tests would encompass the first point ? As you mentioned that that is what the smart people are doing which makes me curious as I never believed in point 2 (too many variables), and point 3.

Thanks

 

[JCDST1979]

There are plenty of hedge funds that have made money using backtesting.

But as markets become and more random, the effectiveness of backtesting reduces.

But I dont think the markets have become completely random yet.

How can you prove that the hedge funds made money using backtesting?

 

[JCDST1979]

Only if you believe the markets are random and as a result unpredictable.

And back-test what exactly?

A trading strategy.

 

[JCDST1979]

An experienced trader knows that strategies developed with backtesting are not timeless.

Markets change, and the clock keeps ticking.

But that does not at all negate the value of backtesting in the short to medium term. To call it useless is to denigrate an important tool that can save a lot of time and capital.

How can you be sure that backtesting can save a lot of time and capital? Can you make a control experiment? How?