A real edge hardly requires any testing

[darthtraderbeta]

The problem is most people I see are violating De Prado's First Law of backtesting in that backtesting is not a research tool. Backtesting is a sanity check on a strategy.

Of course the actual backtest is trivial to perform. All the hay should be in the barn at that point.

 

[easymon1]

DedicatioN - "...takes a lickin' and keeps on twerkin."
https://www.tmz.com/2020/02/11/stripper-falls-from-pole-strip-club-claims-not-responsible-accident/

 

[ma_trader]

The problem is most people I see are violating De Prado's First Law of backtesting in that backtesting is not a research tool. Backtesting is a sanity check on a strategy.

Of course the actual backtest is trivial to perform. All the hay should be in the barn at that point.

Well said..!! Thats the core

 

[ma_trader]

The problem is most people I see are violating De Prado's First Law of backtesting in that backtesting is not a research tool. Backtesting is a sanity check on a strategy.

Of course the actual backtest is trivial to perform. All the hay should be in the barn at that point.

Well said..! Backtesting is NOT a research tool,

 

[Same Lazy Element]

The problem is most people I see are violating De Prado's First Law of backtesting in that backtesting is not a research tool. Backtesting is a sanity check on a strategy.

I'd not read too much into his presentations/books, as his recommend process is very specific to ML-based strategies. Stuff based on ML has a lot of parameters flapping in the wind and is very much at risk of overfitting. Hence his idea of limiting the number of backtests and never using backtests as a development tool.

Alternatively, your strategy research could be inductive, where you start with a very specific hypothesis about flows/positioning/actions of other market participants. Once you go down that road, you totally can use backtesting as a research tool. Further, a lot of the usual stuff about "realistic backtest" (e.g. use bid/offer spread in your backtest etc) should be ignored in that case. You work cycle is idea/hypothesis - strategy model - proxy backtest - actionable backtest.

It's totally ok to iterate a few times on the last step, especially if the type of parameters you're tweaking is unrelated to the actual source of alpha (stuff lookback windows, thresholds for executions, various ways of juicing up the signal etc). In fact, it's very likely you gonna have to play with some parameters to get to a desired combination of strategy metrics, like getting to a reasonable pnl/tradeval while keeping most of the Sharpe.

 

[dtrader98]

Backtesting is absolutely a research tool. If it helps, remove the word back, and just think about testing. You should run tests over data to draw any conclusions about it.

I still have yet to see someone describe a (legal) edge that would not benefit from some sort of quantitative analysis or testing. If anything, it could help to determine if your qualitative edge wasn't really any kind of edge to begin with, rather luck due to chance.

Could those proponents of no testing give some kind of concrete examples to help us see your reasoning?

 

[Same Lazy Element]

I still have yet to see someone describe a (legal) edge that would not benefit from some sort of quantitative analysis or testing.

In the end, it's all about judgement. I've deployed plenty of strategies without any backtesting and plenty of strategies with rigorous backtesting - it would be hard to say which one is preferable without caveats.

Could those proponents of no testing give some kind of concrete examples to help us see your reasoning?

Here is a concrete example. You're a UHF market maker and planning to deploy a new order book strategy, let's say if you see a certain pattern you penny the bid. My prior would be that it's impossible to backtest because of the feedback (i.e. by trading you modify the order book and thus change the market you're working in). So if you have a good model and have done the studies, trying to create a backtest would be a waste of time.

There are other cases when backtesting does not make sense. If you are dealing with an arbitrage strategy (cause you know it works and most of the problem is in implementation), if you are dealing with a highly asymmetric distribution of returns (e.g. backtesting many risk premium strategies is a waste of time).

 

[d08]

Well said..! Backtesting is NOT a research tool,

The traders using ML methods to trade and making millions would be laughing at you.

 

[darthtraderbeta]

Well I don't know how you can disagree with something you haven't read and what you said about ML is just not correct. What you are describing is actually what people are doing who do not use ML because you can't measure what is actually contributing to the prediction like you can with ML. The real hard problem with ML is data leakage IMO. Most bad ML stuff I see online is like predict the closing price of ES while already knowing the closing price of YM.

His idea applies across the board and not just to ML. The basic idea is if you run enough backtest you will find something:

"Backtesting is one of the most essential, and yet least understood, techniques in the
quant arsenal. A common misunderstanding is to think of backtesting as a research
tool. Researching and backtesting is like drinking and driving. Do not research under
the influence of a backtest. Most backtests published in journals are flawed, as the
result of selection bias on multiple tests."

He points out that in any other domain, running tests until you find something is considered a type of scientific fraud.

If you run enough tests you will find something and you don't have to run that many tests to find something.
What you find though will be useless.

He also adds "ignore this fact at your own peril."

I'd not read too much into his presentations/books, as his recommend process is very specific to ML-based strategies. Stuff based on ML has a lot of parameters flapping in the wind and is very much at risk of overfitting. Hence his idea of limiting the number of backtests and never using backtests as a development tool.

 

[darthtraderbeta]

I think you are confusing hardly requires any testing with don't test at all.

IMO the process should be:
1. you have a model that everything about it says it should work.
2. you test it on data
a. it works and gives reasonable results inline with expectations before the test
b. it fails, you scrap the model and start over

B is the problem. You can't do tweak/test/fail, tweak/test/fail, tweak/test/fail, tweak/test/success!

The hysterical thing to me is that is exactly what I thought the point of backtesting was when I started.

Could those proponents of no testing give some kind of concrete examples to help us see your reasoning?