Proper back and forward testing

[Allistah]

Quote from Wide Tailz:

I love Tradestation and will probably never leave, unless they change in some major way.

I played with the WFO tools last year to confirm that I had not made any major mistakes, and my backtests didn't change much, so it came as a good sign. I also had Portfolio Maestro for about a month and that confirmed some other ideas (I started a thread about it somewhere, you may find it entertaining).

One thing I did find to be extremely important is the overall trend of the stock, etf, or futures contract (for long - only algos).

The attached algo is long & short so it may be immune, but I had a couple small cap algos break down when the stock's personality changed after the back test was finished.

Another surprise was how different SPY was from ES..... totally different solutions were found due to the different price action

:eek:

Very interesting. Good to know that you like Tradestation. It is going to take me some time to get used to the platform.

One thing that I'm trying to learn is.. How do you go from idea to live trading? From the little time I've had with Tradestation, it seems like you need to backtest it on a small amount of time like a month or something for intraday bars. Then once you get it to work there, expand it a bit then keep adding until you get to 5 years. What I have found is that when you're working on ideas, you don't want to start with 5 years because it would take forever to optimize and backtest. Start with smaller times at first to see if it even works. Then go to larger times. Does this sound like something you guys do when making a new strategy?

 

[ofthomas]

they can't handle FOB... but they are pretty good though, I find it a bit hard to use which is why I got my money back after I tried it for 30 days...

my only experience with tools handling FOB (or the whole data stream as you call it) is with in-house custom apps or CEP tools like APAMA or OneTick, I have deployed architecture for both and I have used OT... but now you are talking about a subscription feel per core for PROD $$$$ (DR/UAT/Dev are free) ... but it enables you to do real time analytics and queries, come up with a model, and implement those within "minutes" per say... so TTM for models is reduced significantly...

Quote from NetTecture:

Yes, that is the only one. Intresting enough it is not a "major" one - as in: not one you see people talking all the time. Likely due to them not stuffing it down any beginner's throat with a free version.

I am not sure sure they do backtest with full order book, though

It was the only tool standing when we did our evaluation - at the end we decided to go with an inhouse solution (thanks a partner now had developped a VERY good start) due to (a) C#, (b) Visual Studio and (C) the ability to just update it we do regular updates at the moment, rolling out ike 2 new versions a week - even more of the UI when I get a request for something one of my dudes needs

 

[Wide Tailz]

Quote from Allistah:

It is going to take me some time to get used to the platform.

What I have found is that when you're working on ideas, you don't want to start with 5 years because it would take forever to optimize and backtest.

The built-in optimizer is definitely slow, but I don't use it with new ideas. I just manually play with the parameters for about an hour, doing backtests over 5 years (which take about 2 seconds on my laptop). This gives you great insight on the stability of the strategy. If its equity fluctuates all over the place as you tweak the settingz, it's unstable and should be discarded. However, if you can get it to work with many different parameters the likelihood is high that it will survive in the wild.

And my profit factors are coming in at about 1.5-1.8, just like the strat that dom993 posted. I can hit around 2.5 with manual trading, using "purely subjective" (Quant-repulsive) methodz

You want to build an old fashioned pushrod V8 engine, not a turbocharged rotary that will blow if the boost controller fails, or Ferrari flat-12 that will lose compression if it ingests a bug :eek:

 

[Allistah]

Quote from Wide Tailz:

The built-in optimizer is definitely slow, but I don't use it with new ideas. I just manually play with the parameters for about an hour, doing backtests over 5 years (which take about 2 seconds on my laptop). This gives you great insight on the stability of the strategy. If its equity fluctuates all over the place as you tweak the settingz, it's unstable and should be discarded. However, if you can get it to work with many different parameters the likelihood is high that it will survive in the wild.

And my profit factors are coming in at about 1.5-1.8, just like the strat that dom993 posted. I can hit around 2.5 with manual trading, using "purely subjective" (Quant-repulsive) methodz

You want to build an old fashioned pushrod V8 engine, not a turbocharged rotary that will blow if the boost controller fails, or Ferrari flat-12 that will lose compression if it ingests a bug :eek:

Thanks for the reply, this is helpful. Can you tell me what sort of things you've had the best luck with? I hear such a mixed bag of opinions from people like what works and what doesn't. Can you give me an overall type of opinion on what sort of things you've had the best experience with? If you were starting over right now what would you want to tell yourself to get yourself going in the right direction?

 

[Wide Tailz]

Trend following has been my main approach, and some range algos have worked as well. One entry, no averaging in.

Edit- what would I do differently? I'd get tradestation sooner in my career, trading stocks only (no options) until I really had an edge, but the overall approach would be the same. The Market will always have a signal for you to harness, because there will always be hedgers, suckers, and freaked out traders who create tails.

 

[Allistah]

Quote from Wide Tailz:

Trend following has been my main approach, and some range algos have worked as well. One entry, no averaging in.

Edit- what would I do differently? I'd get tradestation sooner in my career, trading stocks only (no options) until I really had an edge, but the overall approach would be the same. The Market will always have a signal for you to harness, because there will always be hedgers, suckers, and freaked out traders who create tails.

Since you're doing trend following mainly, do you have items that you need to optimize for that? Or is it just using price action and not any indicators to follow the trend?

I also heard someone else say that your targets and stops need to be dynamic.

Trying to get something that works for me is pretty overwhelming.

 

[Wide Tailz]

Knowing what to use and what to optimize will come with experience. If anyone shows you their secret recipe, it is probably already obsolete or soon to be, depending on how many beginners try to use it in The Real Market.



I just finished polishing this one off. This was based on something I thought I saw as I watched the index every day, while trading stox.

 

[Allistah]

I've had a number of strategies that have worked for a while and then they stop working. Should you try to make it continue to work so that it keeps going or do you just scrap it and start over with something else?

I would think that you would want to develop a system that has worked for at least 5 years in the past and also does well with walk forward testing. You'll never know but I would think it would continue to work for a while after that.

You also touched on an interesting point. If multiple people start using a system will it cause it not work anymore? Is this why when people have a working system that they guard it with their life (so to speak)?

 

[Wide Tailz]

Quote from Allistah:I've had a number of strategies that have worked for a while and then they stop working. Should you try to make it continue to work so that it keeps going or do you just scrap it and start over with something else?

Exact thing happened to me when I optimized the "price beetle" on one year of data. It instantly failed the forward test. :eek:

Quote from Allistah:I would think that you would want to develop a system that has worked for at least 5 years in the past and also does well with walk forward testing. You'll never know but I would think it would continue to work for a while after that.

Quote from Allistah:
You also touched on an interesting point. If multiple people start using a system will it cause it not work anymore? Is this why when people have a working system that they guard it with their life (so to speak)?

If too many night clubz open in a small town, what would cause their business model to stop working?

 

[GeorgeClueney]

Quote from Allistah:

Hi there..

I hope this is the right forum for this question.

Lets say that my strategy runs on a 5m chart and it only trades during regular trading session hours.

How do you properly back and forward test this? Optimize on 4 months and then forward test the 5th month? The move up to the next 4 months that include the prior forward gest and optimize that. Then forward again the next month forward.

How do I know what the right thing to start off with is? Is there any good books that can teach you about proper back and forward testing?

Thanks,

-Alli

AmiBroker is able to do that.