Is Walk-Forward (out of sample) testing simply an illusion?

[pursuit]

I'm not HFT...at most two trades per day. The algo will hold if it thinks the instrument is in a trend.

Not as secretive as I probably should be. It's not that I believe I've found the holy grail, but I certainly won't entertain trolls/manipulators/asholes--even if it were concerning information I have no problem discussing.

I actually wish I could talk about it completely...but of course, that wouldn't be smart.

Oh, so you lied because you thought I'm "manipulating" you. LOL you need a psychiatrist, bud.

 

[sle]

I actually wish I could talk about it completely...

I find that there are two separate degrees of discussion.

There is theoretical “what if and how about” which most professionals engage in all the time without holding back much. A chat along the lines of "in my experience, cost of risk is a better way of smothing out your transaction frequency than hysteresis, especially in strategies with multiple legs".

Then there are detailed "how do I make money" type of conversations that you could only have with the members of your team. Anything that involves your specific factors, instruments you trade or anything that can really dilute your alpha falls in that category.

 

[userque]

There is theoretical “what if and how about” which most professionals engage in all the time without holding back much. A chat along the lines of "in my experience, cost of risk is a better way of smothing out your transaction frequency than hysteresis, especially in strategies with multiple legs".

Yes, I understand what you are saying, but even with this example...generally speaking...it still just depends, imo. Just for hypothetical...

You quote, "in my experience..." in your first example.

If that experience was the result of lots of research, computer time, coding, testing, thinking, money, blood, sweat, tears, lost family time, lost social time, and led to some supposedly unique edge (or edges); then I doubt "most professionals" would willy-nilly reveal the results of 'their experience.'

So, I see what you're saying, but 'it depends' upon the exact facts/circumstances, imo.

 

[sle]

If that experience was the result of lots of research, computer time, coding, testing, thinking, money, blood, sweat, tears, lost family time, lost social time, and lead to some supposedly unique edge; then I doubt "most professionals" would willy-nilly reveal the results of 'their experience.'

Well, I have not and would never bleed or even cry for any of my employers However, I can speak from decades of professional experience in quant trading, both as a book runner on a dealer side and a PM at a couple funds. If there is one thing you learn is that there is nothing new under the sun. My portfolio has over 30 active strategies at the moment and I can’t name anything “unique”.

By no means I would share the actual details of my alpha generation, but I have done in-depth conversations with coworkers or friends regarding more general ideas like risk metrics, measures of significance or various “how-tos” that are not specific to my own strategies. That my experience.

 

[userque]

Well, I have not and would never bleed or even cry for any of my employers

Neither would I. I never referred to an employer.?

However, I can speak from decades of professional experience in quant trading, both as a book runner on a dealer side and a PM at a couple funds. If there is one thing you learn is that there is nothing new under the sun.

There're always new things...or new ways to combine old things.
https://en.wikipedia.org/wiki/Timeline_of_mathematics

My portfolio has over 30 active strategies at the moment and I can’t name anything “unique”.

By no means I would share the actual details of my alpha generation, but I have done in-depth conversations with coworkers or friends regarding more general ideas like risk metrics, measures of significance or various “how-tos” that are not specific to my own strategies. That my experience.

I agree. I never said I wouldn't discuss anything at all. I said "it depends"...same thing you seem to be saying.

 

[maler]

The out of sample thing is a fallacy and great for marketing, especially to retail traders.

Your skepticism is understandable.
However, if done correctly, out of sample testing is useful.
Sometimes it is easier to get a feel for things if they are exagerated.
Say two people come to you and claim they have a method to play the lottery.
The first guy shows you his correct predictions for last week lottery.
The second guy predicts the numbers for tomorrow and they turn out to be true.
What model will you use to play the lottery?

 

[MoneyMatthew]

I think that out of sample/forward testing guarantees nothing. It can provide an added level of confidence to those who are about to deploy a new trading system but it cannot guarantee that a future sample will generate greater alpha than a historic sample. One subject that does not get talked about nearly enough here is how to define edge death/erosion. I know plenty of people here might just say "historic max draw down peak to valley will decide if my system is active or not".

Step #1. Finding a profitable historic "edge" and hope that some sembalance of that persists into the future.
Step #2. MAKE MONEY or at least break-even. Don't blow up!
Step #3. Did you exit this system gracefully and with some money or did it blow up and take all of your profits with it? Yeah I know any quant reading this would just say "run a bunch of non correlated systems at the same time and who cares if any one individual system blows up? I writing this post with the retail trader in mind, someone who probably runs one or two trading systems max.

 

[ironchef]

Can someone kindly explain to me what is walk forward and out of sample testing? I don't understand the terminology.

Back testing segment 1 and 2 vs back testing 1+2 combined segment, so what is out of sample, same data set if they are consecutive?

Thanks.

 

[ironchef]

Well, I have not and would never bleed or even cry for any of my employers However, I can speak from decades of professional experience in quant trading, both as a book runner on a dealer side and a PM at a couple funds. If there is one thing you learn is that there is nothing new under the sun. My portfolio has over 30 active strategies at the moment and I can’t name anything “unique”.

So, there is really no hope for us small retail traders? If we cannot find new strategies how can we beat the professionals with vast resources with the same strategy they use?

For all the professionals, if all employs similar strategies, how can you make money? Unless you take them from us? With this logic, we retail traders should give up trading?

 

[sle]

So, there is really no hope for us small retail traders? If we cannot find new strategies how can we beat the professionals with vast resources with the same strategy they use?

I think this is something that deserves a separate thread (or even a whole book).