You are using an out of date browser. It may not display this or other websites correctly.
You should upgrade or use an alternative browser.
You should upgrade or use an alternative browser.
better way to do Out-of-sample test?
- Thread starter mizhael
- Start date
the best method of avoiding curve fitting is not having inputs at all.
if you want to optimize then you should do it properly. make a plan with your available data and split it up according for in and out of sample runs.
1. Optimize over X period
2. Run setting over out of sample
3. Repeat
there are no shortcuts.
if you want to optimize then you should do it properly. make a plan with your available data and split it up according for in and out of sample runs.
1. Optimize over X period
2. Run setting over out of sample
3. Repeat
there are no shortcuts.
Unless your strategy is dynamic adjusting to market changes of course your going to have different results.
ES trading at 1200 - 1500 is very different from ES trading 600 - 900.
Option strikes are still 5 points but the contract notational value is $30K vs $75K.
How well does your strategy adjust, manage risk and perform trading random numbers? After all that is really what your after.
ES trading at 1200 - 1500 is very different from ES trading 600 - 900.
Option strikes are still 5 points but the contract notational value is $30K vs $75K.
How well does your strategy adjust, manage risk and perform trading random numbers? After all that is really what your after.
Quote from Craig66:
So in your findings, it is not always the case that a larger training sample produces better out of sample results?
It all has to do with whether you found something, that given the current market structure, has a REAL
99.9+% of things backtested will prove to be spurious. And most of the things that are walkforward test will still fail.
The size of the sample is to give SOME statistical credence to the idea. Too small a number or too short a time means you have less confidence in it.
Quote from PocketChange:
Unless your strategy is dynamic adjusting to market changes of course your going to have different results.
ES trading at 1200 - 1500 is very different from ES trading 600 - 900.
Option strikes are still 5 points but the contract notational value is $30K vs $75K.
How well does your strategy adjust, manage risk and perform trading random numbers? After all that is really what your after.
for sure ...thats the reason for walkforward.. if done correctly it will catch the contracts cyclical (or other ) changes . I use 6 years of tick data (and percentages) to check the robustness of my ES parameters.
Careful... last trade time & sales tick data will lead you down a false path. Test against bid/ask/last trade data or better yet level II tick data if you plan to trade large lots.
Quote from ButterMilk:
for sure ...thats the reason for walkforward.. if done correctly it will catch the contracts cyclical (or other ) changes . I use 6 years of tick data (and percentages) to check the robustness of my ES parameters.
I eventually settled on a different way to optimize. In my opinion the point is not to "optimize" but to "stabilize" and I therefore approach the problem as follows:
I break down the data in e.g. 1 year blocks and optimize each of those blocks separately. I am looking for consistency across the blocks. If a parameter drifts, i look for a different way to express the relationship I'm optimizing so that I can get the stability I'm looking for. I have had very limited success with systems where the parameters needed frequent optimization (but that's just my experience).
I break down the data in e.g. 1 year blocks and optimize each of those blocks separately. I am looking for consistency across the blocks. If a parameter drifts, i look for a different way to express the relationship I'm optimizing so that I can get the stability I'm looking for. I have had very limited success with systems where the parameters needed frequent optimization (but that's just my experience).
Quote from PocketChange:
Careful... last trade time & sales tick data will lead you down a false path. Test against bid/ask/last trade data or better yet level II tick data if you plan to trade large lots.
Not true...you can account for slippage. You can not get (that I know) 6 years of bid/ask/last data. I trade large lots and monitor bid/ask sizes.