Strategy validation

[fan27]

When constructing a strategy/algo, the first thing I am interested in is finding a signal/ pattern that has some predictive qualities. To do that, I will backtest the strategy with a stop and a limit order that is a multiple of the ATR and where the stop and limit are equal distance from price. For example:

entry price: 10
stop: 8
limit: 12

Let's say I have a strategy that has 100 trades and a 75% win rate (I only care about win rate at this point because I am only testing if the signal has predictive qualities). Not being a stats expert, my approach to validate this strategy would be to run a simulation over the same data set (maybe 10,000 iterations) where I randomly select 100 entry points with the same exit criteria and see what percentage of runs beats a 75% win rate. For example, if only 10% of the runs beat my strategy, then I can say the strategy beats random entry 90% of the time. If this approach is valid, what are meaningful percentages (i.e. strategy beats random entry 10%, 20%, 30%). Would be interested on feedback regarding this methodology for strategy validation or some alternatives. All this being said, ultimately, the strategy will have to be validated via live trading.

 

[Scataphagos]

what percentage of runs beats a 75% win rate.

There are a few trades like that, but they only occur a couple of times per year... and maybe not even that often. What do you do about the rest of the time??

 

[fan27]

There are a few trades like that, but they only occur a couple of times per year. What do you do about the rest of the time??

Yes. Through my testing strategies that have very high win rates do not have many trades. The key (for me) is to have many high probability strategies so that trade frequency goes up and capital is optimally deployed. That being said, I am at a point where I am trying to best validate the strategies I have.

 

[Scataphagos]

The best validation you can have... is that your plays are in tune with the market's tendancy.

 

[bpr]

When constructing a strategy/algo, the first thing I am interested in is finding a signal/ pattern that has some predictive qualities. To do that, I will backtest the strategy with a stop and a limit order that is a multiple of the ATR and where the stop and limit are equal distance from price. For example:

entry price: 10
stop: 8
limit: 12

Let's say I have a strategy that has 100 trades and a 75% win rate (I only care about win rate at this point because I am only testing if the signal has predictive qualities). Not being a stats expert, my approach to validate this strategy would be to run a simulation over the same data set (maybe 10,000 iterations) where I randomly select 100 entry points with the same exit criteria and see what percentage of runs beats a 75% win rate. For example, if only 10% of the runs beat my strategy, then I can say the strategy beats random entry 90% of the time. If this approach is valid, what are meaningful percentages (i.e. strategy beats random entry 10%, 20%, 30%). Would be interested on feedback regarding this methodology for strategy validation or some alternatives. All this being said, ultimately, the strategy will have to be validated via live trading.

interesting thought process I would say the strategy should beat random entry by 75% time at least to give me confidence in the strategy
what are your results ?

 

[Simples]

Do you have strategies that avoids what data mining biases?

 

[fan27]

interesting thought process I would say the strategy should beat random entry by 75% time at least to give me confidence in the strategy
what are your results ?

Not sure yet as I have not written the code to run this simulation. It will be easy to do though. Just need to get the random points in the record set, create a collection of Signal objects based on those data points and then inject those into my backtesting engine which will override the need to look for entry criteria. I will have this done and post results in the next week or two. The thing I like about this approach is you should be able to disqualify most systems out there. I look forward to seeing vendors posting systems with spectacular results on a small slice of data and then being able to say that random entries beats that system 90% of the time

 

[traider]

Comparing to random entries is useful if you have not done any optimization or data mining.

 

[fan27]

Do you have strategies that avoids what data mining biases?

Here is my approach. Let's say I looking for a bull market long only strategy in the ES (S&P).

- Generate/test/filter strategies on a slice of data that encompasses a bull market (i.e. 2009 - present).
- Next, test/filter the strategies that passed the filter on another slice of bull market data (i.e. 2003-2007).
- Next, test/filter the strategies that passed the filter on the entire available data set (i.e. 2000-2018)

The strategies that are left over are strategy candidates. Are there data mining biases? Yes, but I think this approach minimizes the biases. Some people say you need to walk forward test the model on live data, but I don't see how that is any different than walk forward testing on past data because by the time you are done walk forward testing on live data, it is past data.

 

[Simples]

Here is my approach. Let's say I looking for a bull market long only strategy in the ES (S&P).

- Generate/test/filter strategies on a slice of data that encompasses a bull market (i.e. 2009 - present).
- Next, test/filter the strategies that passed the filter on another slice of bull market data (i.e. 2003-2007).
- Next, test/filter the strategies that passed the filter on the entire available data set (i.e. 2000-2018)

The strategies that are left over are strategy candidates. Are there data mining biases? Yes, but I think this approach minimizes the biases. Some people say you need to walk forward test the model on live data, but I don't see how that is any different than walk forward testing on past data because by the time you are done walk forward testing on live data, it is past data.

Sounds exactly like my newest system, currently in dd

Few trades means low statistical significance and higher probability of bias. Could work for longer term, but searching too much also pose problems.

All your current data is tainted by your efforts. That's what walk forward helps with, but is hard to pass.

So there's much to overcome
And haven't even looked into monte carlo simulations yet!