Backtesting is useless

[.sigma]

Have you ever heard of Earnings Management? Lookup how price pinning works with regards to street earnings. Most companies hit EPS down to the Dollar +1 Cent. Makes Earnings Upmoves somewhat predictable. Idc about Technical Analysis, but I respect the fact that ppl like Simons have figured it out. Also Backtesting is only one Part of developing a solid strategy, your game plan has to work going through robustness test as well.

not sure what you mean here At?

earnings management? Price pinning?

are you saying because companies hit EPS down to the dollar and cent this makes the earnings announcement market maker move (ATMF straddle) “predictable”? I’ve traded many earnings but to say its predictable do you mean the magnitude? Or the delta move? Or both? You said upmoves are somewhat predictable, why not down moves, what’s the difference?

 

[.sigma]

JCDST1979,

I respect your opinion, but if back testing is useless how else would you know if what you are doing will make money?

simple, look at your P/L

of course that’s after the fact, but might as well allocate a small sum of percentage of capital to test a new strategy you have hope in.. for those that don’t know how to backtest yet (like me)

 

[Atikon]

not sure what you mean here At?

earnings management? Price pinning?

are you saying because companies hit EPS down to the dollar and cent this makes the earnings announcement market maker move (ATMF straddle) “predictable”? I’ve traded many earnings but to say its predictable do you mean the magnitude? Or the delta move? Or both? You said upmoves are somewhat predictable, why not down moves, what’s the difference?

Because when companies know they will miss, they tend to show all the dirt (big bath accounting) and miss big. EPS will mostly be far away from estimates. Also Investors know about Earnings Management, if a company can't handle the Street Earnings EPS, they assume something must be really wrong. Especially when you consider the Principal-Agent Relationship Top Analyst have with the Board/CEO (Analyst who write unfavourably about a company don't get the same LVL of access/Earnings Guidance). All this is part of the reason we get that insane IV Move before Earnings. You rarely see companies go above and beyond EPS, if they don't intend to prove a point/send a signal, there is a study out there by Graham documenting this effect. Most companies either hit EPS or tend to beat it by just a couple of Cents.

 

[Laissez Faire]

How did you find proof that the markets are not random?

Can you give an example of one of those patterns?

Trends. Seasonality. Weekday effect. Timings (HOD/LOD). Etc.

And yes - I know there can be runs in random data, too.

There's of course a lot more to it than this, but I've spent many years and $$$ to develop the framework I use to understand/read the market on a daily basis, so I won't be sharing any details.

But let's just say that on any given day there are things which are more likely to happen than not by a wide margin.

Even still - I do think there's randomness in the market and there are periods of times where it's more predictable than others.

 

[SimpleMeLike]

Money management rules.

Well JCDST1979,

I wish all I needed was money management rules to be a consistent profitable day trader. That would sure assist with making money easier.

If that is the case, just use only risk 1% of capital per trade and we all good then.

Go for it man. Try it out. If you do not like backtesting, then don't back test.

 

[Same Lazy Element]

Money management rules.

If you do not have a positive expectation, no amount of money management will help you. In fact, even with perfect risk management you are going to be bleeding transaction costs every time you trade.

Maybe, but if you adjust your parameters to make it more profitable, isn't that a biased test?

The general idea is to first find a statistically significant effect that has reasonable expected value. Once you have the effect you are trying to exploit, thats when you start tweaking the strategy parameters, sometimes to reduce trading, sometimes to prevent positions in certain scenarios. Most experienced systematic traders understand that these changes can introduce overfitting and watch for it vigilantly.

 

[Steve Alexander]

In the book Fooled by Randomness, Nassim talks that the history that unfolds is just one of the multiple possible ones, this makes backtesting completely useless, don't you think?

Back testing tells you what would have happened if you has done so-and-so. It does not predict what will happen if you do such-and such.

 

[Windlesham1]

Back testing is not useless but you will miss trades, make erroneous entries and exit too soon/late. I have backtested and forward tested over 8 years - yes my strategy works but no it doesn't work when the market drops 30%,and then causes me serious doubts

 

[Heydrrich]

I think it is very difficult to write good backtests. Let's take backtesting chart structures for example: You need to be forward thinking in different future scenario's simultaneously and assign probabilities to them in real time based on reference and adjust bet sizing accordingly, in a constant evolving environment as the price chart. This encompasses so many variables, you cannot make a program for that. And if you can, I salute you.

A sci-fi neurocomputer could do it is what I imagine.

Maybe there are other methods to make money in the markets; i' m sure there will be many, but for the particular case of coming up with backtests for chart structures and automate a strategy around that; i cannot imagine it.

A trading system is robust if it can provide a reliable probability distribution around each price level instantaneously for every scenario in realtime. If your backtests only work under such and such circumstances, then adding backtests will not work, because it (n+1) will have the same problems as the first (n). That's the way I see it.

 

[CharlesS]

Yesterday, more Brooks critique of backtesting [my blue-lighting] --

Strength usually leads to more strength
CNBC has an interesting article that compares the current market to prior markets.

The 3 month rally was exceptionally strong, up 20%.There have been 9 other times since World War II when the market rallied 15% or more in a quarter. In every instance, the market was higher at the end of the next quarter. The average gain was 9%.

Also, Ned Davis Research says on May 26, 90% of stocks in the S&P500 were above their 50 day moving averages. That has happened 19 times since 1967 and the market was up 1 year later in every instance. The average gain was 17%.

While traders don’t like to fight the tape, there are always countless variables that can affect these results. It is impossible to know if the current variables are significant enough to make traders question these statistics.

For example, how many of the quarterly rallies came from a crash, and how many ended just below the top of a trading range? If you were to only look at prior times that included these 2 additional variables, the sample size would shrink considerably. It might even become so small to eliminate the expected benefit.

There is one other historical trend that is important. When an incumbent president loses re-election, the stock market on average sells off a little in September and October. Since most pundits believe Trump will lose, that increases the chance of a pullback before November 3.

Things like this are entertaining, but not tradable. Traders trade what is in front of them, even though they might give some consideration to historical tendencies. If the market is selling off, but a study says it should be going up, smart traders will sell and ignore the research.​