What do these results mean?

[ImamicPH]

Alright for the past month I've been trying to form a decent strategy. I learned about backtesting and developed a VB.net program to sift through the data. I tried a bunch of short-term systems I found in trading books, you know buy-and-sell given a bunch of of indicators such as volatility, consec. number of up days etc.

After endless hours of tests I finally found one that was at least profitable on a well known etf an corresponding index.( The trade results turned out to be the same).

This is the first test that even showed profits. After you see the chart you see that it goes thru one big period of losses and then an long string of relative wins.

The question is does this show promise? or should I throw it out with the other losers?

 

[ImamicPH]

This shows points one one contract.

I guess the first chart was too big sorry.

 

[cashonly]

The first question to ask yourself is can you go 17 months, after being up a little and then now down and stick with your strategy? That's what you would have had to do starting July 15, 2001.

Cash

 

[5yrtrader]

I would also test the same rules on many different etfs, if you show vastly different results between products, scrap the system, if not you should consider trading it. Also dial up slippage and commisions and see if it can perform. Basically test the shit out of it, beat it up, see how it performs over the worst time periods, 87 and 9/11 are two that stick out, and if it is still a winner you have a good system. You will also be confident in it to sit through long losing periods. I wouldn't but a ton of money in someing that makes money half the time.

 

[Dantheman]

you are falling into the trap that 90% of people fall into:

substituting a lack of skill and experience in trading with a rote backtested overlay.


don't fall for it. don't make that mistake. The fundamental way you are approaching trading won't work.


lesson number one: if your 'system' can have a 10,20, or higher percent drawdown over a given time, then it can have any number of consecutive similar drawdown %'s.

to eliminate the above possibility, at some point you'll either realize that you just have to buckle down and deal with what the market is telling you, or you won't.

good luck, but "this is not the trading paradigm you are looking for...move along".

 

[ImamicPH]

Quote from Dantheman:

you are falling into the trap that 90% of people fall into:

substituting a lack of skill and experience in trading with a rote backtested overlay.


don't fall for it. don't make that mistake. The fundamental way you are approaching trading won't work.


lesson number one: if your 'system' can have a 10,20, or higher percent drawdown over a given time, then it can have any number of consecutive similar drawdown %'s.

to eliminate the above possibility, at some point you'll either realize that you just have to buckle down and deal with what the market is telling you, or you won't.

good luck, but "this is not the trading paradigm you are looking for...move along".

Any mechanical strategy that has potential to make a profit, an exposure to risk, has the the potential for drawdown.

A drawdown of 10% over a five period isn't terrible. Sure it "can" happen.

But what's the probablity?

 

[Dantheman]

Quote from ImamicPH:

Any mechanical strategy that has potential to make a profit, an exposure to risk, has the the potential for drawdown.

A drawdown of 10% over a five period isn't terrible. Sure it "can" happen.

But what's the probablity?

you are making the assumption that drawdowns of 10% or more are necessary...

i do not share your assumption.

nevertheless, my commentary still stands. If your testing shows a 10% drawdown in the past, then there is NOTHING preventing forward results from netting back to back 10% drawdowns in the future. or worse.

 

[ImamicPH]

Quote from Dantheman:

you are making the assumption that drawdowns of 10% or more are necessary...

i do not share your assumption.

nevertheless, my commentary still stands. If your testing shows a 10% drawdown in the past, then there is NOTHING preventing forward results from netting back to back 10% drawdowns in the future. or worse.

This is where I'm at:

If you throughly tested a strategy . . and it has no historical drawdown . . This DOES NOT mean you won't be exposed to a drawdown in the future. All it shows is that the setup worked very well in the past.

It's impossible to say with CERTAINTY that any tested strategy will not suffer losses, therefore drawdowns aren't necessary, but as I said POSSIBLE.

At the same time, if similar market conditions do persist going forward(no way to tell the likeliness), a strategy with historically smaller DD's will perfom better.

 

[toe]

If you're only testing this on one symbol then you definately need a measure for curve fitting. Otherwise how can you be sure that these results didn't happen by accident.

If I were testing on only one symbol I'd want to see zero losing months, because a fund may change the way it makes stock selections, or the market may change in behaviour, and your whole system may be history. If there are no losing months then you have an early warning system.

 

[bolter]

imamicph,
It's impossible to comment on whether you're results are the result of over-fitting or not, without understanding the process you went through to obtain them.

I will say this however, it is virtually impossible to develop and test a system on a single instrument without over-fitting, even for a very experienced system trader. This is especially true for an instrument that has limited history such as an ETF. You simply don't have enough datapoints to produce anything approaching statistically significant results. The fact that the system doesn't work on other ETF's is corroborating evidence that your system is far from robust.

But don't just throw it away, learn something from it. The system performed poorly during 2001-2002 when volatility was high and the primary trend was down. Why is that?

Your test data only goes up to early 2005. Keep the same parameter set and test it (walk-foward) with current data. How does it perform?

Good luck,
bolter