Optimization, curve-fitting and probability

N

NinjaTrader_Dierk

ET Sponsor
Hi,

I'm currently reading the book Katz/McCormick "The Encyclopedia of Trading Strategies" which brought me back
to one of my main concerns:

Trying to find a strongly systematic approach to trading I have to deal which adjusting strategies by
applying specific parameter sets. Selecting specific parameter sets may improve to strategy performance which
is called optimization. My main concern - as of many traders - is:

"How far do I have to optimize to make a strategy profitable, and when does over-optimization or
curve-fitting start?"

There are many approaches out there to solve this problem, but reading this book an idea came to my mind:

Katz/McCormick focused me again on the t-test.
Scenario:
The in-sample test of a strategy in this book resulted in
- #trades: 118
- Avg profit per trade: 740.9664
- StdDev of profit: 3811.355
(comment: I prefer using avgprofit as percent value, but that's personal style ...)

The StdError ("Expected SD of Mean") is 3811.355/sqrt(118) = 350.8637. Executing a t-test yields a probability of 1.84% that
the results are just by luck. This does sound really good, but has to be adjusted by the number of optimization steps
which have to be performed to achieve this result. The results in this examples have been achieved by executing 20
optimization steps.

The statistical significance is adjusted: 1 - power(1-0.00184, 20) = 0.3103 = 31.03%
(Detail: Can anybody explain the concept between the last step? It's somewhat unclear to me.)
This number is far worse than 1,84%. Anyhow: The book states, that due to serial dependencies, the real error may be less.

So far, all this stuff can be found in this book on page 63ff, unfortunately I didn't give birth to those ideas :-).

Now my idea:
Improving statistical significance could be target of an optimization process. I could make sense to not only focus on
good past performance but also (or only!!) look for a maximum statistical significance. This might be a way to overcome
(or make at least less hurting) the problem of over-optimization and curve-fitting.

Is there any edge in it? Has anybody tried to evaluate past performance based on this kind of t-test and tried to execute the results on unseen data ?
Has anybody executed (and probably executed) optimizers based on this principle ?

Any comments are highly appreciated.

Dierk Droth
 
It is a good idea to try to downwardly adjust your in-sample results to account for the optimization. But how far to adjust seems difficult to figure out. Your formula might be correct, but it seems less convoluted to just test your system on an out-of-sample dataset.
 
Hi. I once used a program that would curve fit past performance and then forecast future price series. Very sophisticated and unreliable. Katz and McCormick know there stuff on statistics. Optimization trys to eliminate the noise while still capturing the underlying signal. The problem you seem to be having is one of deciding which is the best signal to follow. Over-optimization just makes the problem you may be facing worse because it tends to pick up everything in the history file. When the system encounters data it is unprepared to handle it fails.

As to your question about optimization steps and statisitical significance...I understand it like this: say you flip a coin in the air and the probability is 50/50. Now, say you unbalance the coin in some way so your chance of one side over the other becomes 75/25. You couldn't honestly say you were playing with a fair coin, could you? With an optimized sytem this is what you are attempting to do in the market. Sometimes your sytem will work and sometimes it won't. But since it is overbalanced in one direction, the times it doesn't work will be much more severe than if it was just an unoptimized system.

It is better to try and understand how the market works from a probabilistic frame of mind rather than trying to capture market nuances in a rule-based system. That is a whole other topic.
 
Quote from Aaron:

It is a good idea to try to downwardly adjust your in-sample results to account for the optimization. But how far to adjust seems difficult to figure out. Your formula might be correct, but it seems less convoluted to just test your system on an out-of-sample dataset.
More...

Thanx for your comment. "Convoluted", yeah right, nice term. Since I'm not english native speacker I had to look it up in the dictionary ...

You're right. The out-of-sample test is essential for any trading system development.

Currently my SW does something like a step-wise rollforward along the time-axis. So I can optimize the past and roll-forward to see how results are. The problem simply is: results are poor - not only due to small sample size - in the past, but - that's what I assume, and have not checked yep - due to low probability or curve-fitting.

So what I could do is:
- forget about sequentielly roll-forward and simply do a in-sample strategy development and one (or a few more) out-of-sample test manually
- try to find some "method" to messure the probability of the in-sample results to do some form of auto-select of the high-probability solution, roll forward and test out-of-sample data and then continue the whole process stepwise

Dierk
 
Droth,

You may want to look into something called the binomial cumulative distribution function (a google search for binom cdf will probably do the trick).

The binom CDF function essentially tells you the risk (percentage wise) that a strategy you are using can be beaten by a random strategy.

You can also call this your 'lucky factor'.

Example:

You try a given strategy. Out of 100 trades, 60 are correct, 40 are incorrect.

You end up having a system with a '60% accuracy' (you think).

BUT, if the binom CDF value is something like 40%, this means that there is a 40% chance that any random system will match or beat your accuracy. In other words, there is a 40% chance that your system, given the number of trials and accuracy, is simply lucky.

Ultimatley, you want the binom CDF value to be as close to 0% as possible.

If you have a system that gives you a 60% accuracy, and the binom CDF value is about 0%, this tells you that your system isn't lucky, and you aren't getting your results by chance - your system genuinely produces a 60% accuracy.

Regards,

Brandon
 
This subject has been discussed here before - use search.

My comment is that the construction of trading systems is IMHO as much an art as it is a science. FYI I have spent many years in mathematical research and application. There are many mathematical criteria that can be applied to a particular model or past trading data. However, performing a real world experiment - read trading the model - is the best test as to its validity. Constructing a real experimental test that is actually traded will help greatly to validate your model and my advice is to not spend too much time on the a-priori analysis but to develop the analysis interatively through a sequence of low risk experiments.
 
Here's a quote from the thread 'Can Strategy Trading Work?'

Quote from metooxx:


Real time testing with real money in small lot sizes is the answer we found.

I spent a decade plus designing and backtesting using textbook theories.

Never made any real money that way

extended backtesting is mental masturbation ...
More...
 
Quote from droth:


Trying to find a strongly systematic approach to trading I have to deal which adjusting strategies by
applying specific parameter sets. Selecting specific parameter sets may improve to strategy performance which
is called optimization.

-Snip-

"How far do I have to optimize to make a strategy profitable, and when does over-optimization or curve-fitting start?"
More...

Only a thought:

Basically the overall dynamics of market prices consists of nine (maybe more) combinations due to three common patterns: Uptrend, Downtrend, and Sideways.

We need to think further each of these common patterns would possibly have its own dynamics in terms of such as time duration, rate of change, etc.

Would it be a very challenging job to derive an optimised system by means of curve-fitting, whether based on historical or randomly generated data set, and selecting an ideal set of parameters (what are they?)?

:confused:
 
Quote from CalTrader:

develop the analysis interatively through a sequence of low risk experiments.
More...

i second that emotion. such an approach will also tell you very quickly whether they system you're playing with is compatible with your trading psyche, which is an oft-overlooked part of the game.
 
Thanx for all your replies. I like "extended backtesting is mental masturbation ..." most :-)

I have to admit that I'm a bloody newbie, so my points could be somewhat academic ...

What distracts me most is, that I don't know when I cross the small border line between proper system setup and
system over-optimization. In fact, I dislike the term "optimization", since it draws the attention to the wrong point.
I mean: Trading systems should reflect market characteristics and "hopefully" they are profitable. I have the feeling, that I would be better off
looking for approaches, that reveal some kind of "persistence" in the markets. That's where the idea of probability fits into the
picture. If I was where able to find high probability systems (systems that yielded their results - negative (!!) or positive - not just by chance),
"rational" decisions could be made to trade them or not (based on the performance they showed in past). It would be interesting to see if there is
an edge in this idea.

I feel uncomfortable to just "select" some kind of parameter set for a strategy and reject others. I asked myself if there are better (or additional!!) ways then checking by
out-of-sample tests -which are still absolutely necessary. Parameter sets may show past profit or not, as long as the results are not significantly different from random,
they are meaningless. Am I wrong ?

@GIG
Do you have the feeling binom CDF shows more than the method Katz/McCormick offer ?

@CalTrader
The idea of incrementally improve analysis based on seen/real results stands. I will try to stick to it.

The thread
http://www.elitetrader.com/vb/showthread.php?s=&threadid=1019&perpage=6&pagenumber=1
is useful. I noticed it before and just am rereading it.

Dierk
 
You must log in or register to reply here.
Back
Top