Back-testing software, my progression to freedom

[chaostheory]

Quote from lolatency:

I'm trying to break-free from the man and have been working with strategies in R. We have similar testing software at work, but I wanted a clean break from stuff at work. I'm scared to test my ideas at work in the event that they want to own my ideas.

I've got two options:

1) Use a retail software package to backtest, and/or
2) Write my own framework

I have no interest in rewriting my own framework. TradeStation's platform is too expensive as a standalone and their execution is too slow.

What I did this weekend was try to automate my strategy in E-Signal. I used their EFS package, but the interface for backtesting seems really broken and doesn't generate those equity graphs over time. When I used this software about 5 years ago, it seemed to work then. Now, it doesn't seem like it works.

What are you guys using? I don't care about speed -- I just want back-testing results quick. When I go to market, I intend to write my own execution framework and trade managers.

As warning from those who have been there before, there is theory to what you want to do and there is practice.

In theory what you plan above is great.

In practice (because I did it) it fails. Why?

Well if you back test on one system (whatever it is) and then run the strategy on a different one (whatever it is) you're doomed before you start.

Why? Well, unlike what most people say, there is still psychology involved in automated trading or even if you will manually trade your back tested strategy.

It's because when your automated system takes a few losses, you begin to wonder if it is something wrong with your "strategy" or is the code on your execution server doing something wrong.

So, invariably, you are forced to compare the results from your back tested strategy to your live running trades.

If they're different, what do you do? Well, that's always what happens when you use different platforms.

Instead, you must have a fundamental sameness about everything possible between your back testing system and your live trading system.

That way, if you run comparison from live trade data and data through the back test system, they match.

And that will give you more confidence to stick with the system through the draw downs.

Another related tip, which you probably already know, make sure you examine your back tested strategy on the same granularity you will watch your live auto system. Why? When you trade an auto system with real money, you will definately be "watching" it.

If you watch it daily, like I do, then make sure you can stomach any daily draw downs in your strategy. Other traders are comfortable only "watching" there auto trade systems on a weekly basis. If so use that.

I hope this helps.

Unfortunately, most of the platforms out there (I tried them) force you to do something differently in your code between historical testing and live trading which forces different results on the trades.

I wish you every success!

 

[lolatency]

Quote from TraderZones:

Why do you think backtesting progresses you to freedom? It simply helps you find multiple trading methodologies that are unlikely to work in the future.

This is what many do and this is one reason many fail. It does not matter if the MACD was at point Y while RSI was in range W to X and the ADX was doing Z. This is almost always coincidence, not discovery...

Just one step along the way.

 

[chaostheory]

Yes, there are lots of reasons that back testing creates strategies that don't work in real life.

Other than the brainless issues of curve fitting the real problems relate back to using the wrong kind data.

All historical testing must be done using tick data for entries, exits, stops, targets, etc. Why?

That's because you can have a strategy that looks good in historical testing using only minute data or 10 second data ...

But if you test that exact same strategy using tick data for entries, exits, stop, target, etc. then...

You find that the simulated the stop prices or targets were incorrect in enough cases to make the strategy unprofitable when using real tick data.

One example is that during a strategy based on minute data, the strategy exits you at your stop exactly in the middle of a bar.

However, using real tick data, you exit you at a much worse price because there were zero trades at your stop price due to a gap during that minute.

In other words, all those minute only systems attempt to simulate your prices and assume that big, long minute bars (or any other kind of bar) were continuous.

Other strange things can happen with large outside bars. Those simulators try to guess the activity intrabar but they get it wrong more often than right.

In short, there's zero chance to develop a winning strategy using anything less that tick data with bid/ask and last prices for an intraday strategy.

When you do, make sure the system gives you real entry and exit prices based on Bid/ Ask spreads.

Why? Many even tick based systems only use the bid. That's also erroneous because in fast market conditions the spread between bid and ask can get large. So the price entries and exits you get a erroneous.

It seems that if you're only trading a daily strategy, maybe you can get away with 10 second data or 1 minute data as granular enough.

Frankly, switching to the use of tick data with accurate bid/ask data finally set me on the right track to developing strategies that reproduced the results from the testing system to real life trading on automation.

Once you have this kind of setup, you can really progress rapidly at developing winning strategies.

 

[SideShowBob]

Quote from chaostheory:

Yes, there's lot's of reasons that back testing creates strategies that don't work in real life.

Other than the dufus issues of curve fitting the real problems related back to using the wrong data.

All historical testing must be done using tick data for entries, exits, stops, targets, etc. Why?

That's because you can have a strategy that looks good in historical testing using only minute data...

But if you test that exact same strategy using tick data for calculating entries, exits, stop, target, etc. then...

You find that the minute system simulated the stop prices or targets incorrect in enough cases to make the strategy unprofitable.

One example is that during a minute bar the original strategy exited you at your stop exactly.

However, using real tick data, it exits you at a much worse exit because the were zero trades due to a gap during that minute.

In other words, all those minute only systems attempt to simulate your prices and assume that big, long minute bars (or any other kind of bar) were continuous.

Not true. Most of the time, the big bars mean that during that bar some news or other info caused the market to JUMP and DROP so quickly that your stop would have exited much worse.

In short, there's zero chance to develop a winning strategy using anything less that tick data for an intraday strategy.

It seems that if you're only trading a daily strategy, you could get away with 10 second data or 1 minute data as granular enough.

Frankly, switching to the use of tick data finally set me on the right track to developing winning strategy that reproduced the results from the testing system to real live trading.

While some of what you say is true, using words like "must" and "zero chance" make it easy to disprove you, and make your comments seem less like a good insight and more like a rant.

In many cases what you say is true, however depending on the strategy and item being traded there are plenty of cases where daily data is fine. I trade several of them....

 

[heech]

Quote from chaostheory:

That's because you can have a strategy that looks good in historical testing using only minute data or 10 second data ...

Does anyone have practical experience, whether other types of bars (like volume bars) addresses this kind of problem?

Anyone aware of any data providers that provide volume bars?

 

[chaostheory]

Quote from SideShowBob:

While some of what you say is true, using words like "must" and "zero chance" make it easy to disprove you, and make your comments seem less like a good insight and more like a rant.

In many cases what you say is true, however depending on the strategy and item being traded there are plenty of cases where daily data is fine. I trade several of them....

Yes, near the end of the message it mentions that perhaps with a daily strategy you can get away with minute data as your basis.

And I will further concede that i never trade daily strategies, only intraday.

So my comments should apply mostly in that context.

When trading daily strategies you have entirely different parameters.

Thanks!

 

[chaostheory]

Quote from heech:

Does anyone have practical experience, whether other types of bars (like volume bars) addresses this kind of problem?

Anyone aware of any data providers that provide volume bars?

Well, I build volume bars from tick data.

Volume bars do not solve the problems I discuss if that were the lowest level of data you use to base the entry and exit prices.

Why? It's because the only difference between volume and time bars is the duration of the bar. So they're the same in that price activity during the bar can vary in many different ways which software can never guess or similute to any reliable way for you to base a strategy on that will work reliable when it is trading on real, live tick data.

Think of flying an airplane remotely, purely off controls (no visual at all). How safely could you fly if your controls were only updated every 1 minute?

During that time, the plane could already be nose diving before you get notice of it on your controls.

So volume bars can be somewhat better than time bars during high volume periods because they complete faster. In fact, a volume bar might complete even few seconds in high volume.

But when volume pace slows down, a volume bar could last longer than a minute bar and price variation could be much wider and worse than minute bars.

I use volume and range bars in my trading. But as for stops, targets, and other entries and exits. They all get calculated from real bid/ask spread from tick data.

 

[Tim2000]

It seems like so many of the backtesting systems out there focus on pricing data. Is there one that is suitable for fundamental data?

For instance, perhaps I'd like to test a strategy of buying a portfolio of stocks with consistent earnings over 5 years whose return on assets was very high and whose price to earnings was very low. I'd like to see how this strategy would play out if I selected a portfolio at the end of 1996 and held it for 1 year, 2 years, 3 years, etc. I'd do the same for 1997 and so on.

Thoughts?

 

[lolatency]

Quote from Tim2000:

It seems like so many of the backtesting systems out there focus on pricing data. Is there one that is suitable for fundamental data?

For instance, perhaps I'd like to test a strategy of buying a portfolio of stocks with consistent earnings over 5 years whose return on assets was very high and whose price to earnings was very low. I'd like to see how this strategy would play out if I selected a portfolio at the end of 1996 and held it for 1 year, 2 years, 3 years, etc. I'd do the same for 1997 and so on.

Thoughts?

There aren't that many datapoints in what you are trying to do, so I imagine some basic programming in your statistical package of choice would be adequate for this task, particularly once you have pulled the data from a reliable data source.

 

[bwolinsky]

Quote from lolatency:

I'm trying to break-free from the man and have been working with strategies in R. We have similar testing software at work, but I wanted a clean break from stuff at work. I'm scared to test my ideas at work in the event that they want to own my ideas.

I've got two options:

1) Use a retail software package to backtest, and/or
2) Write my own framework

I have no interest in rewriting my own framework. TradeStation's platform is too expensive as a standalone and their execution is too slow.

What I did this weekend was try to automate my strategy in E-Signal. I used their EFS package, but the interface for backtesting seems really broken and doesn't generate those equity graphs over time. When I used this software about 5 years ago, it seemed to work then. Now, it doesn't seem like it works.

What are you guys using? I don't care about speed -- I just want back-testing results quick. When I go to market, I intend to write my own execution framework and trade managers.

I believe I've mentioned several times in threads on similar topics you have to visit http://wl4.wealth-lab.com to see every single possible system known to man. There's over 8,000 systems using every single indicator. That alone should be your starting line. You'll quickly find what has worked, and continues to work. You'll also find even the most profitable or highest risk adjusted systems require a bit of modification before going live with them.

And, I guarantee wealth-lab is really the only development platform you should be backtesting in, because it is the only portfolio level simulator available on the market. TS obviously is too stupid to write the portfolio simulation, or they would have done it already.

I saw you were working in R, and there's a discussion page on that site that's entirely devoted to R, mostly written by Dr. Koch for use with wl, and he is one of a very select group including myself that migrated to C2 after wl4.wealth-lab.com was closed.