BackTesting vs WalkForward

[asynchronous]

Quote from iagainsti:
As for me, I use price data. Indicators are too subjective...

Pardon my ignorance iagainsti,

but indicators, lagging, leading or otherwise, would be components in your trading system - and you would always test against real historical price data.

tell me this is always the case, or tell me when this would not be the case - I am curios?

 

[redwraith]

Buy when price goes up, sell when price goes down? Are you using price patterns or something?

 

[iagainsti]

Quote from asynchronous:

Pardon my ignorance iagainsti,

but indicators, lagging, leading or otherwise, would be components in your trading system - and you would always test against real historical price data.

tell me this is always the case, or tell me when this would not be the case - I am curios?

The only indicator I use is price. Then optimize around buy points and stop losses, etc.

In other words, I don't use oscillators, MA's, Fib's, etc. I'm not smart enough to make use of them, I guess. Whenever I try, I end up chasing my tail or see horrendous whipsaws, etc.

 

[asynchronous]

Quote from iagainsti:

The only indicator I use is price. Then optimize around buy points and stop losses, etc.

In other words, I don't use oscillators, MA's, Fib's, etc. I'm not smart enough to make use of them, I guess. Whenever I try, I end up chasing my tail or see horrendous whipsaws, etc.

Thank You iagainsti!

 

[asynchronous]

Quote from iagainsti:

The only indicator I use is price. Then optimize around buy points and stop losses, etc.

In other words, I don't use oscillators, MA's, Fib's, etc. I'm not smart enough to make use of them, I guess. Whenever I try, I end up chasing my tail or see horrendous whipsaws, etc.

iagainsti,...sure your smart enough to know how to these indicators, but like most of use (myself included) it takes a little coaching.

If you haven't done so already, consider these two web sites - I particularly like FM Labs, consider purchasing their code - they did a good job.

http://stockcharts.com/school/doku.php?id=chart_school

http://www.fmlabs.com/reference/default.htm

 

[walterjennings]

Quote from ButterMilk:

I hear allot about backtesting systems, including TV commercial from retail brokers. My experience, however, is that walkforward testing (with varying test periods) is significantly more effective than backtesting, when it come to consistence and profitability.

All comments and experiences would be appreciated.

Backtesting and forward testing are pretty much exactly the same if you are simulating a set system (the forward data will yield the same results when it becomes backtest data). The reaction to the data doesn't change just because it hasn't happened yet or is coming in slower (or least it shouldn't if your backtest is coded accurately).

The only time you really need to worry about backtesting vs forward testing is if your system is changing, learning or optimizing during your backtest, then an additional forward testing or out of sample period without optimization is suggested to guard against over fitting.

Though if by forward testing you are referring to 'live testing' which actually executes small lot orders on the market, in my experience thats extremely different from simulated testing (often the killer of new systems), and will give you the best predictor on how your system will perform in the long run.

 

[walterjennings]

Quote from iagainsti:
Your ignorance is showing, my friend. Your statement shows clearly that you do NOT understand out of sample testing, robust parameters, etc.

If you are curvefitting, then you're doing it wrong.

This is a common misconception. Curve fitting leads to a great predictor if you are trying to predict the 'curve' you are fitting to.

Example: Predicting whether a fruit is an apple or orange based on color. System if color == orange => then not apple. Would be considered a highly curve fitted system to the given sample set (since it is technically possible to get an orange apple). Though in reality it offers a very good predictor.

I think what you are worrying about is known as over fitting. Which is where you optimize or fit to a sample set beyond the point of useful predictive knowledge, like in the above example, because in the first sample, say (A,A,O,O,O,A,A), your system believes that all future sample sets will be palindromes, which does not offer useful predictive knowledge of the solution space.

Quote from januson:

Curvefitting = Optimization !!

In my early days I actually believed in such crap, now I'm building on a system that simply just works without lagging indicators.

Wouldn't any system using any element of knowledge outside the current state of the market (ie. knowledge of the past), be considered to be using 'lagging indicators' in a sense, since past knowledge can not exactly predict the future? I'm curious how you can design a system which only uses current market state (outside of stat arb, but even then you will want to model past arb opportunities). Even if you had current price and price triggers to trade, those price triggers will have some element of knowledge of the past (range, offsets, direction).

 

[asynchronous]

Quote from walterjennings:

This is a common misconception. Curve fitting leads to a great predictor if you are trying to predict the 'curve' you are fitting to.

Example: Predicting whether a fruit is an apple or orange based on color. System if color == orange => then not apple. Would be considered a highly curve fitted system to the given sample set (since it is technically possible to get an orange apple). Though in reality it offers a very good predictor.

I think what you are worrying about is known as over fitting. Which is where you optimize or fit to a sample set beyond the point of useful predictive knowledge, like in the above example, because in the first sample, say (A,A,O,O,O,A,A), your system believes that all future sample sets will be palindromes, which does not offer useful predictive knowledge of the solution space.

walterjennings,...your knowledge of trading optimization seems excellent. Did you conclude this through your own studies, or did you derive it from folks like Robert Pardo? I am curious about your source!

 

[walterjennings]

Quote from asynchronous:

walterjennings,...your knowledge of trading optimization seems excellent. Did you conclude this through your own studies, or did you derive it from folks like Robert Pardo? I am curious about your source!

Mostly schooling, my main base of knowledge coming from a statistical machine learning course taught by Shai Ben-David(if you are interested in looking up some of his publications), along with various other math/stat/a.i courses. Though my specific implementation have obviously been tweaked away from common theory over years of practice, I wouldn't consider myself anything close to an expert compared to some of the people I know.

 

[asynchronous]

Quote from walterjennings:

Mostly schooling, my main base of knowledge coming from a statistical machine learning course taught by Shai Ben-David(if you are interested in looking up some of his publications), along with various other math/stat/a.i courses. Though my specific implementation have obviously been tweaked away from common theory over years of practice, I wouldn't consider myself anything close to an expert compared to some of the people I know.

Thank You "walterjennings",...This was useful.