Gents, thanks for your replies. There are lots of authors that only perform rolling walk forward; however, I am leaning toward to anchored walk forward. Do you guys have a preference of one method over another? Is it possible that you can post a successful WFO result of R or Amibroker? Like many forum readers, we would like to see how it is done correctly.
Anchored Walk Forward Optimization to Avoid Curve Fitting
- Thread starter fiverr
- Start date
Personally, I would do both...and more. I like to have multiple confirmations that a strategy is likely to work, both statistical and logical/financial. I want to have a sound reason for doing something (ie there's some financial cause and effect that explains (at least partially) why the strategy should work) and also have statistically significant evidence that it worked in the past.
There are advantages and disadvantages to each of these approaches to optimizing a time series (anchored and rolling). Personally I would insist my out of sample performance is good both ways I analyze it. If it's good on one and not the other, I would carefully examine the data to see if I can tell why, and whether I can save the strategy or whether it's junk. Depending on the situation I would likely perform other statistical analyses as well. That said, I take a highly statistical approach to the markets -- there are others who would take other approaches.
As for posting a result, I'm not sure what you're after. Syntax? Concepts? The best way to learn syntax is to read the documentation. The best way to learn the concepts is a basic textbook or web search.
There are advantages and disadvantages to each of these approaches to optimizing a time series (anchored and rolling). Personally I would insist my out of sample performance is good both ways I analyze it. If it's good on one and not the other, I would carefully examine the data to see if I can tell why, and whether I can save the strategy or whether it's junk. Depending on the situation I would likely perform other statistical analyses as well. That said, I take a highly statistical approach to the markets -- there are others who would take other approaches.
As for posting a result, I'm not sure what you're after. Syntax? Concepts? The best way to learn syntax is to read the documentation. The best way to learn the concepts is a basic textbook or web search.
One other thing: I want my optimizations to be robust in the sense that I can change the optimized parameter values I get from my in-sample data and still get fairly good results in my out-of-sample data.
As an example, if you have two parameters, p1 and p2, that each range from 0 to 1 then you have a [0,1]x[0,1] square of parameter values. If the optimum value (of whatever you're measuring...you'd be advised to be maximizing some risk-adjusted measure of return, not return outright) obtains when p1=0.35 and p2=0.75 then I would be making sure I still get decent results in the neighbourhood of (0.35,0.75), perhaps [0.30,0.40]x[0.70,0.80] as an example.
Also, personally I would be optimizing over more than one objective function. I happen to be doing an analysis right now where I'm "optimizing" (sort of...not exactly) over annual return, annual Sharpe, correlation with a couple other strategies of interest to me, and a couple other proprietary metrics. That is, I'm doing the optimization process several times, once for each objective function of interest.
As an example, if you have two parameters, p1 and p2, that each range from 0 to 1 then you have a [0,1]x[0,1] square of parameter values. If the optimum value (of whatever you're measuring...you'd be advised to be maximizing some risk-adjusted measure of return, not return outright) obtains when p1=0.35 and p2=0.75 then I would be making sure I still get decent results in the neighbourhood of (0.35,0.75), perhaps [0.30,0.40]x[0.70,0.80] as an example.
Also, personally I would be optimizing over more than one objective function. I happen to be doing an analysis right now where I'm "optimizing" (sort of...not exactly) over annual return, annual Sharpe, correlation with a couple other strategies of interest to me, and a couple other proprietary metrics. That is, I'm doing the optimization process several times, once for each objective function of interest.
Virtually nothing is certain in life. You can, however, take steps to improve your odds, sometimes remarkably so. There is no method to avoid curve-fitting with certainty; there are many methods to increase your likelihood of avoiding curve-fitting when used judiciously. Even when done properly, though, you are right that you can never know with certainty whether you have successfully avoided curve-fitting or not.
Statistical Trader - I am looking for concepts. Have you come across any optimization books that you can recommend? Let's say that I optimize from the start of 2010 to the end of 2014 and then I perform the out of sample test from 2015 to 2016. Let's assumed that I obtain a handful of parameters with more than 50% WFO efficiency. I then run these parameters from 2007 to 2009 to find parameters that would work. Am I still curve fitting?
Can elaborate this part further? For example ,my P1 range is -30 to 30 and my P2 range is -30 to 30. After optimization, I find that P1 is best at 15 and P2 is best at -3. So in this case, I will need to test [12,18]x[-6,0]
Call it luck then.
If we are depending on luck then we are gambling. Are Quants gamblers?
This is the right general idea, however as Metamega has already said, there is an art to statistical analysis. There's no hard and fast rule that that is the precise region of interest.
I don't understand the last part of this. You generated parameters (p) that worked fairly well out of sample. Great. What do you mean by running these parameters from 2007 to 2009 to find parameters that would work? You already have the parameters.
I think that what you are interested in learning more about is "model selection." I can't really refer you to a specific book...most of what I've learned on the topic comes from lectures at university and journal articles that wouldn't be appropriate for a beginner in this area. There has been plenty written about model selection, however. There should be lots of statistics (and engineering, etc) textbooks out there with a couple chapters devoted to it. You might also want to look into "cross validation."
Note that the "best" model or the "most predictive" model isn't always the "optimized" model.
Also, I'm assuming there is a reason rooted in finance which underlies your strategy?