Please properly define what you understand under Walk Forward Analysis because either you or a lot of other users here seem to base their understanding on a divergent definition. Are you talking about using historical data to test a strategy that has been parameterized with previous data or are you talking about a strategy that has been parameterized today and you now want to wait a year to run the strategy over live incoming data before you deploy? Both apply new time series data after the parameterization process.
Is backtesting necessary before Walk Forward Analysis?
- Thread starter kojinakata
- Start date
you seem to be the only one who has a solid understanding of the testing and optimization cycle during strategy development.
Its scary (for their own future losses) how some naive people jump into systematized trading without even the most basic of understanding.
Its scary (for their own future losses) how some naive people jump into systematized trading without even the most basic of understanding.
I think most people backtest wrong to start, so what do I know, or I am just too much picky, I don't know cause I never have asked how other people back test. I worked three years of back testing 3 separate "reasons", reasons of not taking a trade or reason of getting out or reason to reverse, so when I got reasonable happy, and it is ongoing, I use these segments that I eventually drop into any intraday method I am testing, but not in the beginning as I want to see how the entries work out.
So the question of doing forward testing first, for me it can't happen cause the back testing will define what to test into the future. But back testing is so much easier for me to do cause all my "reasons" don't change, most of my systems are of one of two ways to enter, most changes are delays of some kind.
So the question of doing forward testing first, for me it can't happen cause the back testing will define what to test into the future. But back testing is so much easier for me to do cause all my "reasons" don't change, most of my systems are of one of two ways to enter, most changes are delays of some kind.
"back testing will define what to test into the future"??? What do you mean by that? You should have a firm concept and strategy in place before you even start testing it on any data.
I have always gone into back testing to see if any idea had any soundness to it, it certainly does not have to be firm or even a strategy at all, I certainly did not do it this way at all developing my "reasons" for money management that way. I test at times to find the worst way to trade as this leads me onto reasons of what I should not take certain signals my systems might generate. I always have back tested to see if an idea was warranted to spend time on developing more time into it. And especially when many are only starting into back testing, they have no clear history of knowing what works and what doesn't. Are you telling me the very first time you back tested your very first attempt you had entire method well defined? We all learn how to back test better as our experience increases. When I am ready to forward test, I do have completed method done and then start doing multiple forward tests.
Yes, I generally only back test well-defined strategies, with well defined entries, exits, stops, and money management. Everything else will fit parameters to past data and that by definition will lead to overfitting a strategy. You will actually witness huge performance degradation if you optimized strategies during backtest in sample and then move out of sample. Hence generally a strategy should make sense intuitively and should be backed by experience of what works and what does not, not the other way around.
I believe this is the single biggest reason 90-95% of people and strategies fail because people work on such design that have no clue about market intrinsics, micro market dynamics, and trends and pullbacks. Most people who enter this space actually have never made consistent money with discretionary (aka manual) strategy execution and that spells disaster. No wonder so many people blow out even when they systematize strategies.
I believe this is the single biggest reason 90-95% of people and strategies fail because people work on such design that have no clue about market intrinsics, micro market dynamics, and trends and pullbacks. Most people who enter this space actually have never made consistent money with discretionary (aka manual) strategy execution and that spells disaster. No wonder so many people blow out even when they systematize strategies.
I agree with you, people overlook even when they optimize their methods to make sure there are profitable occurrences on both sides of what they want to end up. And too often they take what gives largest profits instead of looking at drawdown and equity curve smoothness.
My Money management reasons seldom change as it comes as segments and comes down to want it is I am trying to identify, not all back testing am I trying to design methods but often times of when not to trade or when to increase size. I been cutting back on time I am manually trading last couple of years, enough to keep on top of lifetime of nuances but just want to be less in front of screen. Cut back from monitors from sixteen to one, we all get old after awhile.
I still do not really understand what you are trying to say.
From your last sentence I get you spend 1 hour per day now on this endeavor? Well, I spend about 12 hours every single day and still outsource tons of projects and work to programmers and quants because I cannot get all the work done myself. 1 hour sounds more like you take this as a hobby rather than a serious and competitive career.
From your last sentence I get you spend 1 hour per day now on this endeavor? Well, I spend about 12 hours every single day and still outsource tons of projects and work to programmers and quants because I cannot get all the work done myself. 1 hour sounds more like you take this as a hobby rather than a serious and competitive career.
Suppose you backtest and don't like what you see. You're probably going to drop the rule, or modify it
You shouldn't really take any action at all, since you have forward information at this point.
But if you're going to backtest and ignore the results, then what's the point of doing it?
To be honest I find the dichotomy artificial, and it's not a distinction I'd seen before, so some of the argument might be that I am not understanding the nomenclature. I don't really understand what you mean by 'forwardtesting'....?
To me WFA is a form of backtesting*, and in my opinion the only valid one.
* other time domains are fully in sample, half out of sample (fit on first period A, test on second period B), half and half (fit on A test on B, fit on B test on A), knock one out (fit on all years except 2002, test on 2002; fit on all years except 2003, test on 2003...)
which is why mentioned that first WFA needs to be clearly defined. Some think it means out-of-sample backtesting, others run live data on the strategy that has been parameterized via historical time series. Both, imho, is exactly one and the same. I actually think the term "forward-testing" or WFA is nonsense. You have a data set on which you parameterize the structure and you have out-of-sample data on which you test the performance. Whether those out-of-sample data are historical data or data that is fed into the system on a live data-feed basis makes zero difference, the strategy could not care less.