Here is something to consider. What if all testing is done minus the previous year of data. Then, any strategies are "walk forward" tested on the previous year of data. How is this different then testing on all data and then walk forward test for the next year except for the fact you have more data to test with?
Strategy validation
- Thread starter fan27
- Start date
You’re overthinking this. All such theories and conclusions about how to identify, filter and select strategies will change over time as you work on them. You’ll be changing your assumptions hundreds of times along the way. But even making wrong assumptions will allow you to discover new things and edges, so don’t worry about going into wrong direction as that may become your main edge afterwards (things that others dismiss outright).
Selecting strategies that work better than random is a given, while at the same time unnecessary because you won’t find almost any random strategies that will give you high chance of winning like 75%+. While when you do find such seemingly random strategies then you’d focus on finding out why it works, because this could mean a bug or a bias that you must identify first, before basing your system off of it.
So then assuming that random strategies won’t make you rich, you can select any strategies that win more often than 75% of the time as having decent potential.
But later you’ll discover that strategies that win 50% of the time may work just as well if they generate 5% profit on average vs losing 2%.
That’s when you’ll also realize that your stop loss could be at a much different level than your profit target.
Later you’ll also find tons of other criteria that will be unique to you, as otherwise you cannot be different from everyone else that is trying to sell you their strategies.
Though using last year of data for walk forward can be important.
You’ll also discover that drawdowns are unavoidable and therefore you’ll have to deal with risk and can’t just use stop losses for everything or otherwise you may kill good strategies.
Finally, depending on what you plan to trade, your biggest issue may be getting clean data. There are plenty of trades reported minutes of hours late by dark pools that cause false spikes in the data that your system will learn and base strategies on.
I’ve been working at this for 3 years and most of the time when I thought I discovered something great, it was because of bad data. So at the same time I’ve spent 3 years cleaning the data, looking for new data sources, and even then still cleaning their data. There is no data provider on this planet that I haven’t contacted dozens of times reporting bad data to them, until I finally decided to purchase tick data and implement my own custom filtering when summarizing to minutes.
Though now I’m adding back some fake data spikes to make my system resilient to random events. While with clean(er) data, creating strategies is much easier. A bigger problem now is selecting strategies among millions that all have downsides vs benefits.
Selecting strategies that work better than random is a given, while at the same time unnecessary because you won’t find almost any random strategies that will give you high chance of winning like 75%+. While when you do find such seemingly random strategies then you’d focus on finding out why it works, because this could mean a bug or a bias that you must identify first, before basing your system off of it.
So then assuming that random strategies won’t make you rich, you can select any strategies that win more often than 75% of the time as having decent potential.
But later you’ll discover that strategies that win 50% of the time may work just as well if they generate 5% profit on average vs losing 2%.
That’s when you’ll also realize that your stop loss could be at a much different level than your profit target.
Later you’ll also find tons of other criteria that will be unique to you, as otherwise you cannot be different from everyone else that is trying to sell you their strategies.
Though using last year of data for walk forward can be important.
You’ll also discover that drawdowns are unavoidable and therefore you’ll have to deal with risk and can’t just use stop losses for everything or otherwise you may kill good strategies.
Finally, depending on what you plan to trade, your biggest issue may be getting clean data. There are plenty of trades reported minutes of hours late by dark pools that cause false spikes in the data that your system will learn and base strategies on.
I’ve been working at this for 3 years and most of the time when I thought I discovered something great, it was because of bad data. So at the same time I’ve spent 3 years cleaning the data, looking for new data sources, and even then still cleaning their data. There is no data provider on this planet that I haven’t contacted dozens of times reporting bad data to them, until I finally decided to purchase tick data and implement my own custom filtering when summarizing to minutes.
Though now I’m adding back some fake data spikes to make my system resilient to random events. While with clean(er) data, creating strategies is much easier. A bigger problem now is selecting strategies among millions that all have downsides vs benefits.
Good info. One point I would disagree with is "selecting profitable strategies that beat random entry is a given". For example, I am sure I could easily construct strategies that would have performed well with very high win rates during the bitcoin bull market. I would also be willing to bet that random entry would do just as well as most of those strategies.
Good point. I use minimum 7-12 years of data (all I can get at tick- or minute- level), so I don't find anything that works well through all that period, maybe except for bitcoin and SVXY until they crashed
Though even then, when "random strategy" works then coming up with a strategy that works better than that random strategy can be just as dangerous, if not more. Because your system could automatically come up with strategies that made more money on bitcoin or SVXY, but then it could crash even harder. Even stop losses may not help, as some seemingly well-managed multi-million dollar funds crashed overnight when trading volatility/SVXY.
I guess my point is that when a "random strategy" works then rather than trying to create new strategies that beat it, I try to identify why it works and what's not random about it. Then I may work on custom strategies for this situation. For example strategies for trading crypto or volatility will be much different from strategies that trade unrelated stocks, futures or ETFs.
Can you give an example of random strategy?
All strategies have entry and exit criteria. The idea I propose is to test a strategies entry criteria by comparing it against random entry with the same exit criteria. For example, say the strategy has 200 trades over a data set and has a win rate, profit factor or whatever you want to measure with a value of X. I would be interested in performing a simulation randomly selecting 200 entry points over the data set and using the same exit criteria. If I were to do this 10,000 times, what percentage of the time would I meet or exceed the results of the original strategy? If I can beat the results 50% of the time, my entry criteria has little predictive value. If I can only beat the original strategy 10% of the time, there is a good chance the original strategy's entry criteria does have predictive value.
Finally got around to implementing this today. So glad I did because it is invalidating some of my previous assumptions. Previously, I thought a 75% win rate where the stop and limit exit orders are equal distance from the entry price (based on an ATR calculation...i.e. 4 x ATR value) would be a good minimum performance metric for considering the strategy and its entry criteria. That is way to high. Here is why:
One strategy I have trades ES and NQ on the hourly chart has 50 total trades and a 76% win rate. Not a big sample set but here is where my tool comes in. I can run a simulation on NQ and ES with hourly data and randomly select entry points between the two symbols but keep the exit criteria of my original strategy. Out of 5000 test runs, none have a win rate above 74%. Only 115 test runs have a win rate above 65% which is 2.3% of the test runs. If I run the same simulation but with only 10 trades per test run, then 911 of the test runs have a 65% win rate which is 18.22% of the test runs. This shows why larger sample sets are better as they are more likely to outperform random entry.
So my new process will be to first determine the win rate for the top 2.5% (or some other low number) of the random trade entry simulation. In my ES and NQ hourly strategy example, rather than filter on a 75% win rate, I will now filter on a 65% win rate and can say that the strategy entry beats random entry 97.7% of the time.
One strategy I have trades ES and NQ on the hourly chart has 50 total trades and a 76% win rate. Not a big sample set but here is where my tool comes in. I can run a simulation on NQ and ES with hourly data and randomly select entry points between the two symbols but keep the exit criteria of my original strategy. Out of 5000 test runs, none have a win rate above 74%. Only 115 test runs have a win rate above 65% which is 2.3% of the test runs. If I run the same simulation but with only 10 trades per test run, then 911 of the test runs have a 65% win rate which is 18.22% of the test runs. This shows why larger sample sets are better as they are more likely to outperform random entry.
So my new process will be to first determine the win rate for the top 2.5% (or some other low number) of the random trade entry simulation. In my ES and NQ hourly strategy example, rather than filter on a 75% win rate, I will now filter on a 65% win rate and can say that the strategy entry beats random entry 97.7% of the time.
Suppose you train an algo on the 'training data' while also validating on the 'validation data.'
Then you test on the 'out-of-sample' data.
Suppose the test results are no-go.
So then you repeat many many times with different (or modified) algos.
In the above scenario (because all of the data is being used over and over), the OOS data essentially becomes the validation data. And the validation data essentially becomes the training data. Essentially, all of the data becomes the training data.
Many don't understand this concept.
Strict, real-time walk-forward testing prevents this from happening. The costs? It's slower. The benefits? It's more reliable and more robust.