Backtesting is useless

Sekiyo said:
No it doesn’t.
It can tell you what doesn’t work.
No matter how small is the sample size.
More...

This is the most distilled (and hence best) argument. If a strategy doesn't work in backtesting (when represented properly), it sure as hell won't work in forward testing or live trading. You can rule out a lot of BS this way.

The reason backtesting usually fails is A) because people test more than once on (until then) unseen data, aka curve fitting, B) bugs, or C) the strategy has no sound reason for why it would work and it just accidentally looked good even when tested on unseen data.

Similarly, unless you know exactly how a backtest was produced, it is worth absolutely nothing.
 
Last edited:
Snuskpelle said:
The reason backtesting usually fails is A) because people test more than once on (until then) unseen data, aka curve fitting, B) bugs, or C) the strategy has no sound reason for why it would work and it just accidentally looked good even when tested on unseen data.
More...

I'll add D)

Not even having assembled the relevant data to back-test/check to begin with.

Takes time.
 
RedDuke said:
tell it to Jim Simmons
More...

The thing is Jim Simons struggled for years and years and even had a huge team to help him before they finally had their breakthrough. And that's a brilliant mathematician who made major contributions in his field and were considered by his peers to be wasting his talent in pursuit of money.

Contrast that with the amateur retail trader or your average college professor which concludes the market is random after an afternoon of studies.
 
Laissez Faire said:
The thing is Jim Simons struggled for years and years and even had a huge team to help him before they finally had their breakthrough. And that's a brilliant mathematician who made major contributions in his field and were considered by his peers to be wasting his talent in pursuit of money.

Contrast that with the amateur retail trader or your average college professor which concludes the market is random after an afternoon of studies.
More...

Exactly this, ppl who believe that TA will bring them riches, should reexamine who they are up against
 
Laissez Faire said:
The thing is Jim Simons struggled for years and years and even had a huge team to help him before they finally had their breakthrough. And that's a brilliant mathematician who made major contributions in his field and were considered by his peers to be wasting his talent in pursuit of money.

Contrast that with the amateur retail trader or your average college professor which concludes the market is random after an afternoon of studies.
More...

no doubt. 100 percent spot on. But main point is that backtest is not useless. It is just very hard to use it correctly. Plus RenTech earlier issue was the fact that they had huge issues with accurate data. This alone took a long time to fix. We do not have this issue today.
 
Atikon said:
Exactly this, ppl who believe that TA will bring them riches, should reexamine who they are up against
More...

As long as you can buy and sell - are you really up against anyone...? :)

And I do believe that Simons used technical analysis:

upload_2020-6-30_22-6-48.png


IMO - as long as you're using the past to predict a future outcome, you're using TA. There are of course various subsets of TA.

Regardless of all that and semantics aside - establishing and assembling the relevant data/parameters, building a predictive model, algorithm or a good trend following system which works through a wide array of market conditions is beyond most people and takes a lot of time. And even with time - most people don't get there.
 
Laissez Faire said:
As long as you can buy and sell - are you really up against anyone...? :)

And I do believe that Simons used technical analysis:

View attachment 232076

IMO - as long as you're using the past to predict a future outcome, you're using TA. There are of course various subsets of TA.

Regardless of all that and semantics aside - establishing and assembling the relevant data/parameters, building a predictive model, algorithm or a good trend following system which works through a wide array of market conditions is beyond most people and takes a lot of time. And even with time - most people don't get there.
More...

very well said. Takes years, a great team and tons of hard work.
 
Heydrrich said:
What about this article from Al Brooks.

https://www.brookstradingcourse.com/ask-al/backtesting/
More...

He's tacitly admitting to trading without any edge, my guess anyway.

Some confusion can stem regarding backtesting of a strict mathematical/"mechanical" rule versus manual backtesting by a discretionary or semi-discretionary trader on events as they happen in a recorded stream. I didn't realize the latter existed for many years. I am not sure what Al is talking about here, suffice to say the smart people he's mentioning are doing the former. If you're doing the latter and it's not comparable to your discretionary live trading, it's probably because your environment wildly differs (for instance, knowledge about the current market regime which you won't know about if randomly dropping into some 10 min bars somewhere, or you're not replaying events in real time, or with different software than your live trading, etc.).

If I had to guess, he probably has been trying some very basic bar patterns or a few indicator combination rules, rules that machines have tested trillions of times (or more) and that generally has no edge (from what I've seen, YMMV). In his belief system (as is rather common) this stuff should have an edge, so he's disappointed.

Tangentially, I own a couple of his books that I bought very early on in a batch. He has incredibly low signal to noise ratio in his writing.
 
Last edited:
You must log in or register to reply here.
Back
Top