How far back is enough for a backtest?

[nyc-hotshot]

Quote from nitro:

I don't backtest. I forward test.

nitro

As I was saying in a different thread...
For many strategies I don't think that a programmed back test translates well to real trading. It's just not that simple. Your subjective mind plays into real trading. I think its good to have a baseline strategy and stick to it for the most part, but you will constantly be tweaking it (for various reasons like current market conditions, etc...) to some degree IMO.

 

[TriPack]

Quote from nitro:

I don't backtest. I forward test.

nitro

So I take it you're not system trading any time soon?

 

[progers82]

thats why i say the more trades you have in your test (over 450) the better chance it will work into the future. but i absolutely agree you have to always observe the system behavior for tweeking.

 

[murray t turtle]

Quote from gms:

in addition to that, I would like to ask why it is that in testing two systems (for stocks, not futures), system A performs moderately well in a backtest, system B performs wonderfully, yet in forward testing system A performs very well and system B is horrible, even after accounting for slippage and commissions.


Knowing that back-tests are not predicative of future performance (which would argue against backtesting wouldn't it?) what does this disparity of performance imply of either system? And what steps are further needed to explore? Thanks.

%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%

Like occasional derivatives & selected stocks;
but I havent tested much ''mostly exit on close'' yet.

Agree back-tests dont promise or predict;
but backtesting still helps as does lifelong learning.



If system ''z'' tests & trades noticebly better on stocks in bull uptrend;
and if system ''d'' didnt pertorm as satisfactory in bear downtrend .

I would rather work/ explore too much than to little;
discretion helps.

 

[harrytrader]

The expression they used is probably meant to impress : it's just kind of monte carlo using statistical parameters of a distribution of their own (imagination is endless for that, usely it is more or less inspired from fat tailed distribution like Levy law), you can then generate thousands or even trillions of synthetic datas if you want. The problem is that such statistical distribution cannot really reproduce all properties of market like long term memory effect, persistency-anti persistency, nevertheless such technique I won't say is justified theorically because empirical distribution is never sure thing but at least it has some sense.

Quote from DT-waw:

How about 5,000 years of data? Yes, it's not a mistake. Data is synthetic, created from real history. Read here http://www.rodia.co.uk/Downloads/Swisspulse_Pres.pdf
pages 17-20. IMO, very interesting concept.

 

[mmillar]

IMHO the most important thing you can do with any system is run it against multiple markets. We can't be certain of any outcome - all we can do is try and increase our odds of success. If you run against multiple markets then your odds are increased that a) you will make money, b) your drawdown will be smaller.

 

[Troglodyte]

I don't backtest. I forward test.

nitro

Can you tell me Nitro, specifically what you mean by "forward test" as distinct from "backtest"?

Why do you think this is better?

 

[markc]

This strategy only works in reasonably volatile markets, as the equity curve posted in the initial post shows. My concern was what to do if we return to a low-volatility environment.

The answer turned out to be fairly logical. I simply put a filter in the system that enables trades to only be taken when the atr of the previous 60 days is > 10. At least if the ranges drop too far the system will just switch off and I won't lose money.

I was not able to identify why it had 2 or 3 drawdown periods in the backtest. I guess I would be more worried if there were no losing periods. So, I will just live with it - the drawdowns are small and should not cause any concerns.

My continuous floor session daily S&P contract that tested this system ran out in October 2003. When I get some more data I will be able to see how this thing performed in an out-of-sample test.

Mark

 

[nitro]

Quote from Troglodyte:

Can you tell me Nitro, specifically what you mean by "forward test" as distinct from "backtest"?

Why do you think this is better?

Forward test means coming up with good ideas and testing them in realtime without optimizing any parameters.

I have no objection with backtesting per se. But unless you are fantastically good at it and do it VERY carefully, it is a time bomb waiting to happen when you go live because real trading is nothing like the perfect world you see in your computer.

It is like in Architecture - A building that was put up and you show up to see the finished product. In fact, there was a bunch of scaffolding all over the place while building it.

Now you try to build a building, but you have never seen the scaffolding and attempt the same feat without one. I will leave it to you as an exercise what the likely outcome is.

nitro

 

[markc]

Quote from nitro:

Forward test means coming up with good ideas and testing them in realtime without optimizing any parameters.

I have no objection with backtesting per se. But unless you are fantastically good at it and do it VERY carefully, it is a time bomb waiting to happen when you go live.

nitro

What is your opinion on an out-of-sample test? I have no idea how this system performed over the last 6 months, as I don't have the data. I couldn't possibly have curve fit it to the data. Is that similar or the same as the forward test you refer to?

Mark