You are using an out of date browser. It may not display this or other websites correctly.
You should upgrade or use an alternative browser.
You should upgrade or use an alternative browser.
How reliable is the backtesting result?
- Thread starter TAM
- Start date
Thanks for the comment. The same strategy performs similarly on other JPY pairs such as GBPJPY, EURJPY, and USDJPY. But for different pairs the optimized parameters are different. It is a simple strategy, using the same parameters on other pairs can lead to profit but not as good as optimized. Do I have to use same parameters?
Below is another testing example on GBPJPY 15min:
Below is another testing example on GBPJPY 15min:
Attachments
It's just very easy to "over-optimise" so that the resulting backtest is a result of pure datamining.
The reason to run the system with the same parameters on different instruments is to see if the underlying idea has merit.
The earthquake disaster produced powerful trends. It explains why your method worked so well recently.
Does the system work on any instruments not involving JPY? The currency often move quite strongly against all other currencies. So, if you base trades on, for example, GBPJPY but trade EURJPY, I expect you will see very similar result anyway. So, testing the system on another JPY pair probably doesn't add much insight into whether the system actually works.
The reason to run the system with the same parameters on different instruments is to see if the underlying idea has merit.
The earthquake disaster produced powerful trends. It explains why your method worked so well recently.
Does the system work on any instruments not involving JPY? The currency often move quite strongly against all other currencies. So, if you base trades on, for example, GBPJPY but trade EURJPY, I expect you will see very similar result anyway. So, testing the system on another JPY pair probably doesn't add much insight into whether the system actually works.
Based on my study, different pairs show different patterns / cycles, mine is a simple strategy that only use a few parameter to determine trades. It is hard to find a simple algo that fits all. For example, the strategy that works well on AUDJPY generates zero trades on AUDNZD. I have another strategy that generates good result (see attachment) on AUDUSD 60min but generates zero trades on AUDJPY.
If there is one strategy that can fit all, I guess it might be a complex algo that determine trades based on different patterns, but this way the complex algo is not much different from combining all the simple strategies together into a portfolio of strategies, which is the direction prefer. What do you think?
If there is one strategy that can fit all, I guess it might be a complex algo that determine trades based on different patterns, but this way the complex algo is not much different from combining all the simple strategies together into a portfolio of strategies, which is the direction prefer. What do you think?
Attachments
What I like to do with strategies that only have 1 or 2 parameters is do a plot with the parameters as independent variables and the PF from all the optimization runs as the dependent variable. Then you can visually see if the optimized point is a "spike" of supposed profitability in a sea of marginal or bad strategies, or if any similar set of parameters would have also been highly profitable. If the resulting plot is very un-smooth, that generally means you have too few points in your backtest or your strategy is too prone to over-fitting.
If you've got more than 2 parameters, hold the less important ones fixed at the optimized values and look at the most important ones.
If you've got more than 2 parameters, hold the less important ones fixed at the optimized values and look at the most important ones.
You are using 15 min data, yet your system is not an intraday one but keeps open positions for an average of about 3 days or so, if I am not mistaken.
The win rate is too low IMO and when conditions change this system will deteriorate. It is probably an over optimized system with several filters.
The win rate is too low IMO and when conditions change this system will deteriorate. It is probably an over optimized system with several filters.
To Big D: Thanks for the advice. Good point. Although there could be possibility of over-optimization, I remember for most optimization runs the optimized point is not a spike. There are a fair amount of points close to the optimized one. Actually in many cases it is hard for me to pick the optimized point because each point can be considered as "best" by certain criteria (such as profit factor, sharp ratio, total net profit etc)...