A real edge hardly requires any testing

[TheBigShort]

In the end, it's all about judgement. I've deployed plenty of strategies without any backtesting and plenty of strategies with rigorous backtesting - it would be hard to say which one is preferable without caveats.


Here is a concrete example. You're a UHF market maker and planning to deploy a new order book strategy, let's say if you see a certain pattern you penny the bid. My prior would be that it's impossible to backtest because of the feedback (i.e. by trading you modify the order book and thus change the market you're working in). So if you have a good model and have done the studies, trying to create a backtest would be a waste of time.

There are other cases when backtesting does not make sense. If you are dealing with an arbitrage strategy (cause you know it works and most of the problem is in implementation), if you are dealing with a highly asymmetric distribution of returns (e.g. backtesting many risk premium strategies is a waste of time).

What I have found in the vol space is, biases need to be back tested while in-efficiencies dont. For example, the implied var premium should be back tested yet a mis pricing in an option does not need to be backtested (SPY Ivol trading higher than QQQ).

I would appreciate your take on this (from a retail perspective).

P.s."Same Lazy Elf" would have made more sense! Especially after Secret Santa

 

[Same Lazy Element]

What I have found in the vol space is, biases need to be back tested while in-efficiencies dont. For example, the implied var premium should be back tested yet a mis pricing in an option does not need to be backtested (SPY Ivol trading higher than QQQ).

Yeah, that makes a lot of sense. Biases are also known as "risk premia", they are persistent and can be tracked historically in one way or another. Inefficiencies, on the other hand, are fleeting and your ability to capture them is primarily driven by your process (models, execution etc).

P.s."Same Lazy Elf" would have made more sense! Especially after Secret Santa

Shadow Moon: Who are you?
Mad Sweeney: I'm a leprechaun.
Shadow Moon: You're a little tall for a leprechaun.
Mad Sweeney: That's a stereotype. It represents a very narrow view of the world.
Shadow Moon: So you're from Ireland?
Mad Sweeney: I told you I'm a leprechaun. They don't come from Moscow, Russia. Or Moscow, Idaho, for that matter.

 

[eternaldelight]

Yeah, that makes a lot of sense. Biases are also known as "risk premia", they are persistent and can be tracked historically in one way or another. Inefficiencies, on the other hand, are fleeting and your ability to capture them is primarily driven by your process (models, execution etc).



Shadow Moon: Who are you?
Mad Sweeney: I'm a leprechaun.
Shadow Moon: You're a little tall for a leprechaun.
Mad Sweeney: That's a stereotype. It represents a very narrow view of the world.
Shadow Moon: So you're from Ireland?
Mad Sweeney: I told you I'm a leprechaun. They don't come from Moscow, Russia. Or Moscow, Idaho, for that matter.

Nice quote, American Gods is fantastic. Was going through your posts from your old account the other day, thanks for sharing some of your insights! Interesting stuff.

 

[Heydrrich]

Al Brooks on backtesting

"If you test enough ideas or enough inputs in any system, you will discover that some have worked in eight out of 10 tests and others have failed in eight out of 10. That is just the distribution of outcomes under a bell curve and has nothing to do with actual likelihood, and that is why so many traders who design great back-tested systems lose money when they trade them in real time."