Backtesting Question

S

Shreddog

Let's say you've developed an intraday system for ES that relies heavily on optimization of the variables used in it. You only have 3 years of data. It was optimized over 2 years of data and then tested out of sample on the subsequent 3rd year and the results looked quite good. So now it's time to put it to work.

Would you:
a) leave the variables unchanged and go with the originals since they work
b) re-optimize over all 3 years of data to use as large of a sample size as possible
c) re-optimize over the 2 most recent years since the original backtest consisted of optimization over 2 years of data followed by 1 year of "forward testing".

Note that all 3 scenarios give you different results for your variables. They are appreciable, though not drastic differences.

Is there such a thing as too much data? After all, markets do change and many systems will quit working eventually. At what point would you throw out old data because it's "stale"?

Thanks!

Shreddog
 
What do you mean exactly by optimization?

For example, if you have looked after the perfect set of MA's cross, your system will certainly fail.

For example, if what you have optimized is the stop or profit level of a basically sound system, your system could still perform in the future.
 
Optimize optimize optimize...

You won't find treasure with optimization. Gurantee

You need a map to find one first. Then a boat to get there. Then a crew to support your trip to the destination... etc. etc.

Optimization is part of how you get to the destination... has to be and is(at least for me when I do optimize) the smallest part of creating a system.
 
I agree with what you're saying, but at some point you still have to pick some numbers to go with. What would you choose?
 
2 or 3 years are way too much in my opinion to optimize stops and targets.

I would use a shorter term volatility indicator.
 
You haven't forward tested it yet, you have out of sample tested it. Forward test it for a month or a week or until you are comfortable in RT using real $$$ or simulated fills / simulated brokerage and then compare how your system is performing to how it had been performing on the out of sample year. If the system starts deterioriating fast in simulation / live trading take some time to reevaluate it.
 
You are correct about forward testing vs out of sample testing.

Actually, I've been trading it since May with good results.... until a drawdown in August. Drawdowns are always unnerving and it was while I was in the process of making sure the drawdown was normal (it was) that I got myself sidetracked into this philosophical puzzle that I presented.

And to further update: After spending some more time with the data, I've decided that the differences between the 3 scenarios are just not significant enough to worry about.

So for me this discussion is now somewhat moot.

But I'm still interested in hearing people's thoughts.

There's a fine line for over-optimizing that I haven't been able to define. I just seem to know it when I see it. It's intuitive. Does anyone have a rule or definition for deciding when there's too much going on?
 
I heard from one guy on this same question who optimized every couple of months on the last 3 months's data. That resulted in minor tweaks to his system and ongoing profitability. I don't know how he did in the last couple of years however. If he was smart he probably changed over to a shorting system to follow the markets down at some point and did the same optimization routine.
 
You must log in or register to reply here.
Back
Top