Is backtesting necessary before Walk Forward Analysis?

[volpunter]

None of the other posters here backed up their claim with their background, that was the reason why asked for it. And for some reason I cannot view your posts&threads because "This member limits who may view their full profile." I will take that offer after reading your posts.

 

[Murray Ruggiero]

I am not a native English speaker, either but I am not sure which part of parameterization is incorrect:

Murray Ruggiero in another thread openly admitted that his software lacks some of the most basic features that I asked him about (tick data backtesting capabilities, correct currency conversions of non base currency performance data, among others). Let me know if you need a link. Some have a strong incentive to make backtesting sound its complicated and complex and only their software feature take a "right" approach, difference here is that I have no incentive whatsoever other than helping newbies as yourself to cut down on needless complexity and just see the forest for what it really is, a bunch of trees.

First of all TradersStudio backtests down to 1 minute bars. I really don't believe that most traders should be using Tick data, too much noise, too short of a timeframe for most people to trade. TradersStudio does handle correct currency conversion for futures back to USD, so yes some traders that might not be enough, but the dollar is the world reserve currency so for a lot of people that all they need. Where did I say it did not handle currency conversion correctly ?.
TradersStudio WFA is also very powerful and in fact is used by Robert Pardo who is created with inventing walk forward analysis.

If your a scalper then then TradersStudio is not for you. I believe it is of value for all other traders.

 

[volpunter]

So yes, your platform does not handle tick based data, that is all I said. And no, your platform does not handle currency conversions correctly for currency products even currency conversions to USD. (and pretty much anyone who does not use USD as broker account currency is more interested in choosing a different base currency than USD, even for futures currency conversion purposes; unless of course you limit the marketing of your product to US citizens, only of course). We discussed this at length in one of your threads and you openly stated that this was the case. I am happy to dig up a link as reminder. By the way I never claimed your software does not provide value.

So, Murray Ruggiero, could you please explain then what the difference is between an out-of-sample backtest and walk forward analysis and paper trading? What is special about WFA that is not exactly applied in an identical way in an out-of-sample backtest or the acquisition of live data and the feeding of such into a strategy algorithm?

First of all TradersStudio backtests down to 1 minute bars. I really don't believe that most traders should be using Tick data, too much noise, too short of a timeframe for most people to trade. TradersStudio does handle correct currency conversion for futures back to USD, so yes some traders that might not be enough, but the dollar is the world reserve currency so for a lot of people that all they need. Where did I say it did not handle currency conversion correctly ?.
TradersStudio WFA is also very powerful and in fact is used by Robert Pardo who is created with inventing walk forward analysis.

If your a scalper then then TradersStudio is not for you. I believe it is of value for all other traders.

 

[Murray Ruggiero]

View attachment 150733

He has a point, you are not disclosing who you are , we don't know if that is really your thesis because you whited out the name , date ect. See it's easy to attack me because my login is my name. It's always been my name. You can look up my articles, see my linkedin ect. If you are going to use your background in putting people down, just tell us your name so we can "Google it"

 

[Murray Ruggiero]

Walk forward testing and my WF analyzer , compiles a report for only out of sample results. We select the best parameters on a training set and then trade for N bars, After N bars we drop off the oldest N bars from the training set and add the newer data, then walk it forward. We compile the Walk forward results without using them to gauge the test. Yes if you run it multiple times and modify the system you do get some bias, but my wf analyzer gives you a true walk forward analysis. Now , my reports give you a deep analysis of the walk forward results, in sample, out of sample, how boundary trades are handled over the analysis period.

 

[volpunter]

Your explanation is 100% identical to several in sample and out of sample tests. Fancy terminology for a very simple and basic concept.

Walk forward testing and my WF analyzer , compiles a report for only out of sample results. We select the best parameters on a training set and then trade for N bars, After N bars we drop off the oldest N bars from the training set and add the newer data, then walk it forward. We compile the Walk forward results without using them to gauge the test. Yes if you run it multiple times and modify the system you do get some bias, but my wf analyzer gives you a true walk forward analysis. Now , my reports give you a deep analysis of the walk forward results, in sample, out of sample, how boundary trades are handled over the analysis period.

 

[Murray Ruggiero]

See all WF tests are out of sample tests , but not all out of sample tests are WF. For example you could train on 80% of the data and then test on out of sample 20%. and only accept the system if out of sample results are good. That is classic out of sample testing and not what I do. Walk forward uses multiple windows and regardless of performance on out of sample windows selects best set of parameters from training.

See you are correct, terms in trading are different than strict statistical definitions, but when in Rome do as the Romans do. I need to call WF testing WF. If I call it out of sample testing then people will ask why don't I have walk forward analysis in my product.

 

[Murray Ruggiero]

Here a link to my WF tutorial

 

[volpunter]

... it is the exact same thing as optimizing on in-sample data and testing out-of-sample. No difference. Nothing new in town. What is new, and imho a horrible way is how the data is sliced into in- and out-of-sample segments. And here is why: First of all people are deluded into thinking that you end up with a lof ot out-of-sample testing periods. You dont. You end up with a very small data window for out-of-sample testing. Basically the data-fitted strategy is tested on a short period without ever evaluating whether it will hold up during and after changes in market dynamics. Unless of course you even re-optimize/re-calibrate during live trading in production it is a horrible way to design and fit a strategy. Furthermore, how will you interpret your n-number out-of-sample periods? You can draw any number of conclusions but certainly not that the strategy is robust or not robust overall. All you learn is about the performance of the period right after you optimize your strategy to in-sample-data that directly precede the out-of-sample section. You potentially optimize n-times, something you won't do in live-trading, hence the question begs how much confidence one can put into such optimized strategy to hold up during market dynamics you never actually tested your strategy on.

Example:

You test in the following manner:

Jan-Mar 2008 in-sample
Apr 2008 out of sample

Feb - Apr 2008 in-sample
May 2008 out of sample

and so on. When you arrive at 2009 you basically constantly re-optimized your strategy and all you get is a bag of different out-of sample test results over very short data windows that nobody actually knows how to interpret. So, lets say your results look like this

Period / Period Returns / Period Drawdowns / Period ....
Apr 2008 / 5% / 2% / ....
May 2008 / -1% / 3.1% / ...
Jun 2008 3.8% / 2% / ....
...
Dec 2008 -4.4% / 6%/ ...

What conclusions can you draw for your expected strategy performance going forward? Hardly any. Why? Because you never allowed one strategy (you essentially end up with 10 different strategies (strategies with different parameter values) to do its work throughout various market cycles. Even if you continued doing so for 2009 (an environment with starkly divergent market performance and dynamics than 2008) you still cannot draw any conclusions whatsoever because you continuously changed your entire strategy properties.

And an optimization approach that is described as follows (by its inventor, Pardo) pretty much lets me lose any remaining confidence: "Think of it as an ‘out-of-sample’ testing on steroids".
Enough said.

He has a point, you are not disclosing who you are , we don't know if that is really your thesis because you whited out the name , date ect. See it's easy to attack me because my login is my name. It's always been my name. You can look up my articles, see my linkedin ect. If you are going to use your background in putting people down, just tell us your name so we can "Google it"

See all WF tests are out of sample tests , but not all out of sample tests are WF. For example you could train on 80% of the data and then test on out of sample 20%. and only accept the system if out of sample results are good. That is classic out of sample testing and not what I do. Walk forward uses multiple windows and regardless of performance on out of sample windows selects best set of parameters from training.

See you are correct, terms in trading are different than strict statistical definitions, but when in Rome do as the Romans do. I need to call WF testing WF. If I call it out of sample testing then people will ask why don't I have walk forward analysis in my product.

 

[Jerry030]

... it is the exact same thing as optimizing on in-sample data and testing out-of-sample. No difference. Nothing new in town. What is new, and imho a horrible way is how the data is sliced into in- and out-of-sample segments. And here is why: First of all people are deluded into thinking that you end up with a lof ot out-of-sample testing periods. You dont. You end up with a very small data window for out-of-sample.......................08 in-sample

Well we might not know the name behind volpunter but he or she is one of the few people on this thread who understands something about data analytics and data science. My suggestion for 74.3438% of you is to stop debating this topic and sign up for some college statistics courses at your local junior college.