Stability/Curve-Fit Analysis For Low Trade Count Strategies

Quote from Kevin Schmit:

Statistically assess the validity and robustness of the underlying vol and heuristic models, preferably in combination...

Why in combination? Do you mean "test separately AND in combination" or "best to test in combination"?
 
Too much jargon for me to follow it all ... but perhaps that's your objective; to weed out those who can't understand your point!

Quote from sle:
...
- bootstrap the results to asses the statistical significance of the performance (usually vs some ex-condition tests)
...

What's an example of an "ex condition" test?

Quote from sle:
...
- if the strategy uses multiple filters/models, asses the performance of the strategy with all possible combinations of the filters...

Test with all possible combinations of filters? Hmmm ... Now, THAT sounds like curve-fitting to me.

Quote from sle:
...
- if the strategy uses stop-losses, only overlay stop loss performance from ex-condition testing
...

How would this work? Please explain.
 
From the context, I suppose "ex-condition" means a separate test of trade exit parameters.

Quote from sle:

I'll try that, interesting idea. Would you use random bar length x=R(1,2,3) or simply generate all possible combinations and use random sample of trades as "out-of-sample"?
I use fixed bar lengths. Random lengths might have a negative effect on time dependent parameters. I've tried random trade samples for OOS, but found that the results are then still biased, probably because the trades cover the same market situation as the in sample data. Therefore, I think the conventional method of separating a part of the price curve for OOS is still the best.
 
Quote from abattia:
Too much jargon for me to follow it all ... but perhaps that's your objective; to weed out those who can't understand your point!
Nope, the more the merrier.

Quote from abattia:
What's an example of an "ex condition" test?
For example, lets say you are buying S&P futures when 50 day MA crosses 250 day MA. In this case, your "condition" is the cross between two MAs, that is your "signal". Your ex-condition test would be - what is the return of S&P futures on all days, not only when your condition occurs.

Quote from abattia:
Test with all possible combinations of filters? Hmmm ... Now, THAT sounds like curve-fitting to me.
No, if I am using say 3 filters and the strategy shows good performance, I would remove one or remove two of them and see how much the strategy suffers. An ideal situation is that each filter (and each combination) has independent alpha.

Quote from abattia:
if the strategy uses stop-losses, only overlay stop loss performance from ex condition testing -- How would this work? Please explain.
You "imagine" that you are doing the trade you are doing, e.g. buying S&P futures without any prior conditions. Then you say that you are going to use stop losses in these trades and find stop loss that makes decreases in returns/decreases in losses acceptable to you. This ex-condition stop performance you can now apply to any of the strategies that uses that very same trade.
 
Quote from DT-waw:
To be honest, nothing can be such indication as future is always unknown.
Yup, that is very true.

Every new trade is up to God. However, as you spend more time building models, testing and thinking about risks, God helps you more.
 
I keep asking myself: "for what reason does this setup (or filter) works ?" ... if I can find a real solid answer to that, then I consider trading it.

Also, imo, the smaller the test sample, the smaller you should bet on a strategy.
 
Quote from sle:

Every new trade is up to God. However, as you spend more time building models, testing and thinking about risks, God helps you more.
:)
The trader's equivalent of Einstein's quote that "God does not play dice".

Also, imo, the smaller the test sample, the smaller you should bet on a strategy.
Suppose you test 2 strategies on the same period. A's trade sample size is half of B's. Now operationally, A's total allocation would already be half of that of B because it generates less trades. So you're automatically betting smaller, right?
 
Quote from NYDreamer:

:) Suppose you test 2 strategies on the same period. A's trade sample size is half of B's. Now operationally, A's total allocation would already be half of that of B because it generates less trades. So you're automatically betting smaller, right?

Wrong - the actual bet on a system, is the max drawdown you will allow the system to get into, before pulling the plug. One component of that is intrinsic to the system (although it is up to you to calculate it, and a key factor is the confidence level you use to that effect), the other part is a linear function of position size.
 
Quote from dom993:

Wrong - the actual bet on a system, is the max drawdown you will allow the system to get into, before pulling the plug. One component of that is intrinsic to the system (although it is up to you to calculate it, and a key factor is the confidence level you use to that effect), the other part is a linear function of position size.

OK, from the viewpoint of retiring systems this makes sense.

I was assuming equal risk allocation and equal expected returns per trade for system A and B.
 
Back
Top