Forward Optimization: How to optimize multiple parameters

[SebR]

Just look up Darvinex on YouTube. He explains it all. Probably the best videos I have ever seen on the subject.

Hi RedDuke!

Thanks for that recommendation; in fact I have studied their content a lot and I agree that it is the best Youtube content I ever found for Trading. But I did not find an answer to that particular question: although Martyn also recommends to keep the amount of optimization parameters low in each optimization and rather stack them, he does not really specify how do to that in the forward optimization. Or did you find something that I missed?

Thanks,

Seb

 

[SebR]

You may want to look at doing backtesting with in-sample and out-of-sample data. During testing you want to make sure that the strategy performs well on historical data but it's very important to remove periods from the backtest and see how the strategy performed during these out-of-sample periods to confirm that you're not over-optimizing or over-fitting. A common way to do this is to run the backtest over 80% of the historical data and leave the most recent 20% for out-of-sample. Another possible more robust method but more labour intensive is to divide the backtest into several bins and define an in-sample and out-of-sample range for each bin.

Thanks GaryBtrader!

Well actually this is what I am coming from, and what should be improved further by the walk-forward-optimization: For each window, in-sample is what I called BWD and out-of-sample is what I called FWD.

BR

Seb

 

[Clark Bruno]

It helps understanding and suggesting by sharing common terminology. Training data, validation data, test data.

Thanks GaryBtrader!

Well actually this is what I am coming from, and what should be improved further by the walk-forward-optimization: For each window, in-sample is what I called BWD and out-of-sample is what I called FWD.

BR

Seb

 

[Q.E.D.]

You may want to look at doing backtesting with in-sample and out-of-sample data. During testing you want to make sure that the strategy performs well on historical data but it's very important to remove periods from the backtest and see how the strategy performed during these out-of-sample periods to confirm that you're not over-optimizing or over-fitting. A common way to do this is to run the backtest over 80% of the historical data and leave the most recent 20% for out-of-sample. Another possible more robust method but more labour intensive is to divide the backtest. into several bins and define an in-sample and out-of-sample range for each bin.

I know this will not be well accepted, but testing "out of sample data" is a waste of time. I will neither identify, confirm, or disprove that the particular system has been data-fitted. Sounds plausable that it might, but no evidence whatsoever. Why would arbitrarily declaring some data range, more impt than other data. And the person that tests out-of-sample starting say with Jan data, would just be included in the in sample historical data, for the trader who does not start his testing 'til a few months later. The plausible argument is that testing a system on "new" data is somehow unique. However, if you have already tested on say 5 years of daily / intra day data, the one month of out-of-sample, is no more unique than any other period. More significant, is that those who use out-of-sample don't just stop there, they go back, reconfigure their system, & then repeat the process. So it could just be data-fitting in 2 segments, rather than one. The best out-of-sample data is real time trading. And the truism is that your largest draw-down period is always ahead of you. Regards,

 

[ph1l]

I know this will not be well accepted, but testing "out of sample data" is a waste of time.

This contradicts

More significant, is that those who use out-of-sample don't just stop there, they go back, reconfigure their system, & then repeat the process. So it could just be data-fitting in 2 segments, rather than one.

Using out-of-sample validation data is useful because when the system fails on that data, it's unlikely to be successful in the future. When the system succeeds on the out-of-sample data, it has a chance of working in the future.

 

[Q.E.D.]

This contradicts

Using out-of-sample validation data is useful because when the system fails on that data, it's unlikely to be successful in the future. When the system succeeds on the out-of-sample data, it has a chance of working in the future.

Why? What makes 1 or 2 months out-of-data so important? The system presumably worked on 5 years of data, but not on 2 months? System are typically profitable about 1/3 of the trades, which means any 1-2 month data period more likely to be losing than profitable. Makes no difference if that is in or out-of-sample. Sorry, my last comment on this subject Just wanted to provide some contra to the popular methodology popular with most newbies re out-of-sample testing. Regards,

 

[Clark Bruno]

It is not a method chosen by some or most newbies but by the world's leading data scientists and the entire science community. You may want to read up on the concept to better understand it. There is tons of content on the internet that explain it.

Why? What makes 1 or 2 months out-of-data so important? The system presumably worked on 5 years of data, but not on 2 months? System are typically profitable about 1/3 of the trades, which means any 1-2 month data period more likely to be losing than profitable. Makes no difference if that is in or out-of-sample. Sorry, my last comment on this subject Just wanted to provide some contra to the popular methodology popular with most newbies re out-of-sample testing. Regards,

 

[userque]

Hi!

I am struggling to transfer the strategy development process from a backtesting approach (with a single forward phase to confirm the result) to a classic forward optimization process when there is more than a couple of parameters involved:

General consens seems to be that optimizing too many parameters at once produces overoptimization, and according to my own experience (correlation BWD>FWD) as well as literature it seems better to optimize only a limited amount of parameters at the same time, then fix them and optimize the next.

But how can this be applied to a Forward Optimization approach?
Here my thoughts so far:

"Stage1":
If I start an Optimization only with the "core" of a strategy and limited number of parameters, I can check the predictivity of the result using different optimization targets, based on this decide for a target and finally receive the "best setup" for each backward window. This "best setups" are obviously not exactly the same for each window.

"Stage2":
If now I want to optimize the next group of parameters in (e.g. adding an indicator with another 3 parameters to set it up), I need to fix the previous parameters to not optimize too many at once. Up to now, I see 3 different approaches, but I am not really happy with either:

1. If I am lucky, the "best setups" from "stage 1" only vary in one of the optimized parameters, so I can fix all others to the "common best" value, leave this one flexible and add the new ones. But what can I do if this is not the case? Plus, latest in "Stage 3" I would anyway end up with too many parameters.
2. Instead, I could fix for each window its best setup from the previous optimization, and then apply the next part of the strategy to only that setup. I am worried that the results would become very unique for each time step, so that finally I am developing a quite different strategy in each window. This would reduce my confidence in the robustness and predictivity.
3. Alternatively, I could try to find a relatively good common setup from all windows of the previous optimization. But this basically makes a backtest with a control phase out of the previous forward optimization, so the only against a BWD/FWD Test is that I have more chance to find an optimization target that produces a robust sytem (as I can proof this with a good correlation in each window).

With the first and last scenario I am also aware that I break the principle of forward optimization, that in each backward phase I should not know anything about what happens after this backwards phase, because my final choice of "stage 1 parameters" for that window is based on the outcome of the previous optimizations of later windows.

For this reason, I would if anything tend to choose the 2nd scenario, but wonder if forward optimization makes sense at all for strategies with > 5 parameters?

Any experience or recommendations on this?


Thank you very much!

Best Regards,

Seb

None of the above.

I recommend boosting.

https://en.wikipedia.org/wiki/Boosting_(machine_learning)

 

[ph1l]

Why? What makes 1 or 2 months out-of-data so important? The system presumably worked on 5 years of data, but not on 2 months? System are typically profitable about 1/3 of the trades, which means any 1-2 month data period more likely to be losing than profitable. Makes no difference if that is in or out-of-sample. Sorry, my last comment on this subject Just wanted to provide some contra to the popular methodology popular with most newbies re out-of-sample testing. Regards,

What you write is true for a small out-of-sample data set. But if you made the validation set larger, such as 30% of all the data, failure of a trained system on the larger validation set means that system probably won't work in the future.

 

[HolyGrailSeeker]

Code all your parameters into 1 strategy then use the optimized parameters for the best returns.