Backtesting and Strategy Validation - How Much is Enough?

[Algo_Design_Kid]

Quote from SnakeEYE:

So what was wrong with all of your designed algos,Kid?

Can you summarize it please in a few lines?

1 ) Slippage was misrepresented

- Let's take this a step further. When you trade 5-10x a day and your estimation of slippage is off by one tick I can almost guarantee you will be a net loser and see the same upside down equity curve I did. All it takes is to be off a half of a tick. Doesn't sound like much, but it will ruin you.

2 ) Wrong hypothesis by backtesting engine for entries and exits

- I think IDBill may have covered this. In the live market I came to find out that it did not get into trades like it did in the simulation and it also SKIPPED trades that it did in the simulation. Which were correct? Hell, I don't know, but I assumed the backtesting engine was correct with its hypothesis. It wasn't.

3 ) Ego.

- Check your ego at the door. It has no place in this environment. Perhaps I was too proud of myself at the time to realize what a slippery slope I was really on.

There are probably a million more. I could write a book on it, but I rather forget about all of it to be honest and just try to look ahead. I'll be done with school soon so hopefully I can get out of here.

Have a good one

 

[SnakeEYE]

Quote from electroniclocal:

I don't do that. It's not in your best interest. I'll teach you how to create your own and I'll show you mine.

You need to "own" your trades and I strongly believe that unless you intimately know and understand the trading logic behind and algo, you cannot have the discipline to trade it and won't know how to manage it. I really didn't post to get biz but I have strong views about the rubbish being sold to people when what really should be taking place is people learning how to create them.

Creating an algo that can trade profitably is a LOT easier than trading as a discretionary trader. You only have to be clever infrequently, not every day.

I know exactly what i need.And to create my own i have to have some programming skills.What,you will show me ho to become a programmer?I have some algos i want to be automated but have no skills to do that and doubt you can show me how to do that unless you are a programmer.

 

[SnakeEYE]

Quote from Algo_Design_Kid:

1 ) Slippage was misrepresented

- Let's take this a step further. When you trade 5-10x a day and your estimation of slippage is off by one tick I can almost guarantee you will be a net loser and see the same upside down equity curve I did. All it takes is to be off a half of a tick. Doesn't sound like much, but it will ruin you.

2 ) Wrong hypothesis by backtesting engine for entries and exits

- I think IDBill may have covered this. In the live market I came to find out that it did not get into trades like it did in the simulation and it also SKIPPED trades that it did in the simulation. Which were correct? Hell, I don't know, but I assumed the backtesting engine was correct with its hypothesis. It wasn't.

3 ) Ego.

- Check your ego at the door. It has no place in this environment. Perhaps I was too proud of myself at the time to realize what a slippery slope I was really on.

There are probably a million more. I could write a book on it, but I rather forget about all of it to be honest and just try to look ahead. I'll be done with school soon so hopefully I can get out of here.

Have a good one

and your conclusions? backtesting is useless or daytrading is impossible?

'a million more' doesn`t count,only the first two!

 

[bhause]

Wow - I go away for 2 days and 9 pages worth of replies - you guys are great.

Thanks for the advice - its encouraging to hear some people say backtesting is reliable and credible, and good to hear the admonishments of others who have had bad experiences with it. Given what I've read, I think I am going to keep working on my model, get more data, and then decide if I want to actually trade it. A few notes based on your comments-

1) I am not using any program for the backtesting. I am doing it manually based on intraday charts and tracking everything in an Excel sheet. While this is extremely time consuming, I do find it beneficial as I get to study price behavior within the charts in addition to simply getting the results of the backtest. I am early in my trading career and price charts still fascinate me.

2) Regarding slippage - my system works on 5 minute bars with the assumption that I will enter at the immediate beginning of the next bar after the signal. That being said, for the backtest I am assuming the worst price of the bar in order to make the test as hard on the system as possible. E.g. If the signal is to short and the bar opens at 95, but goes as low as 94.8, in reality I would be shooting to get in at 95, but for the sake of the test I am inputing 95 as the entry. I think giving myself the worst possible execution within a 5 minute time frame should account well for slippage.

Again - thanks for the help, I will be collecting more data and continuing to paper trade before this goes live for real.

Thanks!

 

[electroniclocal]

Quote from SnakeEYE:

I know exactly what i need.And to create my own i have to have some programming skills.What,you will show me ho to become a programmer?I have some algos i want to be automated but have no skills to do that and doubt you can show me how to do that unless you are a programmer.

SnakeEYE, there are a number of platforms out there that use a psudo code for implimenting strategies. Does require a bit of work but then you have something you know intimately. I'm no programmer but create algos in easylanguage (multicharts) and RTL )MarketDelta). I can't get to first base with Ninja as its a real computer language but when I need Ninja code I go to a couple of guys who implement MY ideas.

I believe that the trading logic is what is key. Lots of it can be worked out just using your charts.The piano playing of coding up the algo is a much lower level skill and can be bought easily if you don't want to do that part. But then using a psudo language is not that hard.

 

[Algo_Design_Kid]

Quote from SnakeEYE:

and your conclusions? backtesting is useless or daytrading is impossible?

'a million more' doesn`t count,only the first two!

95% of backtesting is probably useless IMO. That is unless you are doing something very unique and you probably will not get to that stage until after years of work. If you are using "time interval" charts you might as well give up now.

Most of you are really marginalizing this effort. It isn't simple and the time and money you must invest to actually be legit is too much for the majority to handle.

Best advice would be not to even waste your time. I'm not going to go through every pitfall. Do a google search, you will find plenty.

But please, feel free to prove me wrong. It isn't impossible. Best of luck.

 

[SnakeEYE]

Quote from Algo_Design_Kid:

95% of backtesting is probably useless IMO. That is unless you are doing something very unique and you probably will not get to that stage until after years of work. If you are using "time interval" charts you might as well give up now.

Most of you are really marginalizing this effort. It isn't simple and the time and money you must invest to actually be legit is too much for the majority to handle.

Best advice would be not to even waste your time. I'm not going to go through every pitfall. Do a google search, you will find plenty.

But please, feel free to prove me wrong. It isn't impossible. Best of luck.

No,man i don`t want to prove you wrong at all.I`ve never backtested anything myself.I opened a small account and tried with real money what worked and what was not.

 

[intradaybill]

Quote from SnakeEYE:

No,man i don`t want to prove you wrong at all.I`ve never backtested anything myself.I opened a small account and tried with real money what worked and what was not.

I again will say that Kid is correct. Most people shouldn't even attempt to backtest anything. Too many pitfalls and the whole idea is ill-conceived at its root:

http://www.elitetrader.com/vb/showthread.php?s=&postid=3271049#post3271049

Trying to find a system via backtesting is like trying to go from London to New York swimming accross the ocean. Some may make it, I agree.

 

[Argent]

This whole thread reminds me of a friend who once told me "I tried for years to learn to play guitar and could never get any good at it." One is amazed therefore that there are virtuoso guitarists.

 

[spindr0]

Quote from bhause:

2) Have any of you ever run into a situation where you have an extremely well performing technical indicator that one day simply breaks down? I mean, has the market every changed in a basic way such that you needed to reconsider your entire trading system (I realize this happens all the time from a fundamental standpoint, but I am coming from a technical standpoint)?

I'm going to ditto the content of the 1st reply that you got. I trade intraday based on a very simple system. In '08 and '09, its return was nothing anyone here would turn their back on. With the exception of the past week, now it provides just a bit more than pin money.

Did I mention that the market dropped 50% in '08 and '09 ? :eek: