When are you optimizing too much?

[Joe Doaks]

Quote from earlyexit:

Thanks Joe. I was thinking of doing something similar. Optimize each weekend using the previous month of data. But, I wasn't sure if a month was enough, or if I should stick with something like what you are doing (6 months).

Thanks again.

Are you optimizing systems for trading intraday?

 

[Algo_Design_Kid]

Quote from earlyexit:

At what point is optimizing a strategy become too much, and you are just curve fitting?

I have an automated strategy that I wrote that performs pretty well (profitable 76% of the time, 3.24 profit factor). I've been using it for a while, but the optimal values change as time goes on.

For instance, if I optimize over the past 3 months, I get one set of ideal settings, but if I optimize over the past month, I get slightly better results from different settings.


Just wanted to hear what others are doing and where you draw the line.



Thank you.

IN regards to optimization you really need a large enough sample size to be significant.

Unless I missed the other posts I just see 3 months.

How many trades would this have produced / what time frames are we talking about here?

BTW, profit factor really doesn't mean shit. I can have a .34 Profit Factor and still have an excellent program

 

[dloyer]

Read about "Walk forward testing"

1 month of in sample is way to short. At least for my systems, I need at least 6 months to a year of data before optimiztion results have a positive corrolation with out of sample results. At very short in sample periods, the corrolation is actually negitive.

 

[earlyexit]

To answer some questions...

This is an intraday strategy that uses 1min bars. It produces an average of 1 trade a day on ES. I also run a slightly modified version on TF. Will probably adapt for others as well.


Algo Kid,

I don't doubt that a lower PF can still do very well. I'm still relatively new to automating strategies, so I wasn't sure what information to put in my original post. Come to think of it, I guess I didn't need to provide any result info. (I'm new to this forum, so I'm not sure of what to do or not to do).


Thanks.

 

[earlyexit]

Quote from dloyer:

Read about "Walk forward testing"

1 month of in sample is way to short. At least for my systems, I need at least 6 months to a year of data before optimiztion results have a positive corrolation with out of sample results. At very short in sample periods, the corrolation is actually negitive.

dloyer,

While I didn't do any walk forward testing. Those are not back test results. But live results over the last few months.

Thanks

 

[Algo_Design_Kid]

Quote from earlyexit:

To answer some questions...

This is an intraday strategy that uses 1min bars. It produces an average of 1 trade a day on ES. I also run a slightly modified version on TF. Will probably adapt for others as well.


Algo Kid,

I don't doubt that a lower PF can still do very well. I'm still relatively new to automating strategies, so I wasn't sure what information to put in my original post. Come to think of it, I guess I didn't need to provide any result info. (I'm new to this forum, so I'm not sure of what to do or not to do).


Thanks.

Don't worry -

What software are you using?

And more importantly what are you optimizing for?

I have many 1 min strats for ES. After optimization for 1 full year you should expect over 500- 1000++ trades at least.

For 5 minutes over 100-200. Anything lower for each one IMO is too much optimization, or I should say too small of a sample.

 

[jack hershey]

Quote from Joe Doaks:

I optimize with the previous six months' data at the end of each week and use that result for the following week. So it is kind of like a moving average. It is mostly old data, but it has enough new to start factoring in optimization creep.

I do bar by bar delete add on a 1600 bar sample.

 

[endsongs]

Quote from Algo_Design_Kid:

IN regards to optimization you really need a large enough sample size to be significant.

Unless I missed the other posts I just see 3 months.

How many trades would this have produced / what time frames are we talking about here?

BTW, profit factor really doesn't mean shit. I can have a .34 Profit Factor and still have an excellent program

You mean 1.34 right?

 

[Algo_Design_Kid]

Quote from endsongs:

You mean 1.34 right?

no

 

[earlyexit]

Quote from Algo_Design_Kid:

Don't worry -

What software are you using?

And more importantly what are you optimizing for?

I have many 1 min strats for ES. After optimization for 1 full year you should expect over 500- 1000++ trades at least.

For 5 minutes over 100-200. Anything lower for each one IMO is too much optimization, or I should say too small of a sample.

I've recently started using Ninja Trader.

If you optimize a 1-min strategy over a year.. Do you wonder if that is too much of a sample? I mean, things change so much over a years time. Should you really sample for that long of a period (be easy on me, I'm a self admitted automated newbie ).

My initial thought was, you should really be focusing on the not so distant past. Like the last 3-6 months for a 1-min strategy. Perhaps I was incorrect assuming such.

For an example. I've been trading for several years now. Strategies (while not automated, but in my head) came and went. I wouldn't back test them. I just used them until I noted they stopped working and I moved on.

Thanks