Avoiding Curve fitting

Quote from Random.Capital:

An all too common error. Magic numbers are magic numbers, regardless of where they come from, or how they are encoded or interpreted. The fact that one might be a constant in one implementation or another simply means the parameter has already been fit to the curve.

Cheers.
More...
great comment IMO.
 
Quote from jack hershey:

This is illustrated in "Putting the Pieces Together".
More...

Jack has mentioned this document a few times in this thread. Would someone be willing to direct me to its location? I have been unable to find it thus far.

Thank you,

river
 
Looks like Jack never made it past the talking stage. Three quick examples:

1) Listen to Jack crow about this trade before it went bad (mp3 file at link):
http://www.mediafire.com/?0ielxnmeiro

<img src=http://www.elitetrader.com/vb/attachment.php?s=&postid=2023032>


2) Jack's failed September 1st prediction:
"As a trader for 53 years, this turning point tomorrow is the most significant I have seen in my life."
http://www.elitetrader.com/vb/showthread.php?s=&postid=2560747#post2560747


3) Yet another great call from wrong way Hershey!!!

<img src=http://www.elitetrader.com/vb/attachment.php?s=&postid=2656562>


Quote from jack hershey:

At this point it maybe possible for some people to see the difference between talking and doing the walk.
More...
 
Your perspective is valid in certain situations and totally wrong in others. I agree for simple linear systems curve fitting is a problem. . However the essence of non-linear predictive analytics is a very sophisticated form of what you tell us to discard. In fact you can't do it without out it. It works as well.

So:

1) Simple linear = avoid curve fitting, etc,
2) Complex non-linear predictive = use curve fitting to discover what is both currently unknown to you and is predictive of future market activity.

In both cases you don't know what you've found until the system is frozen and tested on out of sample data. If out of sample works the same and on going it works the same on live data, then you have discovered something that works. Do not discard that because it was not discovered by the theoretically correct method of discovery.

There are complex non-linear predictive conditions in Foex EUR/USD for example that work exactly the same today as they did 28,000 bars ago. (+ or - a few % in W/L ratio over hundreds of trades regardless of the where tested in that timeline)




Quote from mind:

extensive testing will inevitably subject results to curve fitting. the problem of curve fitting is not merely addressable within code, it must include the human being doing the tests as well.

example: consider you test a system with merely one indicator (assume there is a system with a single variable for simplicity's sake). you decide you run 1ooo variations of the indicator. assume your output criteria is sharpe ratio. assume your minimum value for that criteria is 1. so you are running simulations of the single indicator system with 1ooo variations and you are looking for an equity curve with a sharpe above 1.

in order not to curve fit you do the same thing with random data. thus you create random equity curves and you find out that let's say three out of one hundred have a sharpe ratio of above 1.

now let us say you run your 1ooo sims and 3o of them have a sharpe above 1. you throw them away since this is perfectly in line with pure randomness. with this number of tests you expect this number of outputs without them having any value tradingwise.

now here comes the thing.

most traders move on to the next indicator and start all over again. finding now lets 27 sharpes out of 1ooo tests. again throwing results away.

assume the whole thing is repeated ... a thousand times. you see where i am heading. the outcome of 3o sharpes above 1 is as well subject to randomness. if you do the whole thing often enough you will find a pure random set that has ... 6o sharpes above and ist still ... purely random.

problem is that these 6o might as well be valid, but it is rather unlikely in the context of these 1ooo different indicators. (btw 1ooo different indicators might sound much, but use the word "systems" instead and assume you are using three indicators for each one. now you only need ten indicators to get to one thousand different systems.)

then what can you do? IMO this is the crucial point of modern technical trading.

1. limit your tests. only start testing if you already have an indication that this is a good fertile ground. acrary used correlations to do so. or you use your trader know how. whatever. be careful not to run a machine doing this search for you ... or you start with a possible fit in the first place.

2. use in and out of sample tests to detect the flaws of your avoidCurveFitting method. be careful not to do this too often, otherwise your in and out of sample begin to merge.

3. get tougher with the criteria with each new run. this is not easily done, though it might sound convincing. it depends IMO how homogenous or diversified your different tested systems are. but this is becoming philosophical.

4. test, once you found something, on other markets. if it does well there too ... good indication it was not a fit. but remember not to do this too often. testing it against 1ooo other time series ...

final note.
acrary got famous on the board for his "edge test", which is nothing but a "curve fit test", if you will. i fully trust that alan was the highly successful trader he claimed to be. yet my guess is, it was not the edge test that made him successful, but the combination of him rigorously studying the market quantitatively, statistical tests and his personal judgement of the markets and their behavior (you might call it hindsight).


summing up
the problem of curve fitting IMO is that metalevels must not become optimization levels. consider you are smart and build a system that optimizes itself in a walk forward process. smart move. consider it works immediately. deal done. trade. but if it works only after 1ooo different combinations of the walk forward variables ... your meta level, namely the walk forward process, designed to avoid fitting, shifted on the optimization surface.

whatever ...
More...
 
Regarding the archived videos and illustrated transcriptions, PM me how to get them to you. ET does not have an achive capability that I understand.

Attached is the current approach we use to handle the chart posted. BUT we do not trade SPY. When I made the Friday end of day comment it was related to an instrument called the ES.

To handle the continuation of a trend (on any level) from the close to the open, the approach is to equate the day's close to the next day's open and use the gap value to adjust the "containers on all levels. The H and L of the bar 81 are transposed as well to determine the bar to bar relationship from the 9 cases. This is logged while the bar is forming.

Today the open was an OB and it set an FTT in place and a short followed on several levels. Depending on skill level, people can trade subs, BBT's or Tapes. In any case, the trade is from FTT to FTT.

I email, daily, a synopsis of the day, the annotated charts and, often a textual narrative to give focus on what is ahead and the next steps for drilling. It is like the bootcamp thread a while back. Sometimes, a few bars into the open I email the day's beginning as a support to those who are annotating and trading on various levels.

Enjoy.
 

Attachments

Only you, Jack, would try to defend shorting in a strongly uptrending market.

And to top it off, you've doctored the pic to make it look like the market went down instead of up

Attached is what the S&P 500 did in numbers, which won't be as easy for you to obfuscate:

<img src=http://www.elitetrader.com/vb/attachment.php?s=&postid=2697727>
Quote from jack hershey:

Regarding the archived videos and illustrated transcriptions, PM me how to get them to you. ET does not have an achive capability that I understand.

Attached is the current approach we use to handle the chart posted. BUT we do not trade SPY. When I made the Friday end of day comment it was related to an instrument called the ES.

To handle the continuation of a trend (on any level) from the close to the open, the approach is to equate the day's close to the next day's open and use the gap value to adjust the "containers on all levels. The H and L of the bar 81 are transposed as well to determine the bar to bar relationship from the 9 cases. This is logged while the bar is forming.

Today the open was an OB and it set an FTT in place and a short followed on several levels. Depending on skill level, people can trade subs, BBT's or Tapes. In any case, the trade is from FTT to FTT.

I email, daily, a synopsis of the day, the annotated charts and, often a textual narrative to give focus on what is ahead and the next steps for drilling. It is like the bootcamp thread a while back. Sometimes, a few bars into the open I email the day's beginning as a support to those who are annotating and trading on various levels.

Enjoy.
More...
 

Attachments

More fantasy from the mind of Jack Hershey :p

My original chart of Jack's 20 Nov 09 market call is on the left. Look carefully at how he altered it on the right :eek:

Sorry Jack but mind pretzel won't make you a profit shorting the S&P in an uptrend.

<img src=http://www.elitetrader.com/vb/attachment.php?s=&postid=2697782>

Quote from jack hershey:

Regarding the archived videos and illustrated transcriptions, PM me how to get them to you. ET does not have an achive capability that I understand.

Attached is the current approach we use to handle the chart posted. BUT we do not trade SPY. When I made the Friday end of day comment it was related to an instrument called the ES.

To handle the continuation of a trend (on any level) from the close to the open, the approach is to equate the day's close to the next day's open and use the gap value to adjust the "containers on all levels. The H and L of the bar 81 are transposed as well to determine the bar to bar relationship from the 9 cases. This is logged while the bar is forming.

Today the open was an OB and it set an FTT in place and a short followed on several levels. Depending on skill level, people can trade subs, BBT's or Tapes. In any case, the trade is from FTT to FTT.

I email, daily, a synopsis of the day, the annotated charts and, often a textual narrative to give focus on what is ahead and the next steps for drilling. It is like the bootcamp thread a while back. Sometimes, a few bars into the open I email the day's beginning as a support to those who are annotating and trading on various levels.

Enjoy.
More...
 

Attachments

Quote from Trader666:

Only you, Jack, would try to defend shorting in a strongly uptrending market.

And to top it off, you've doctored the pic to make it look like the market went down instead of up

Attached is what the S&P 500 did in numbers, which won't be as easy for you to obfuscate:

<img src=http://www.elitetrader.com/vb/attachment.php?s=&postid=2697727>
More...

Is there some reason you and Jack have this endless argument over something.....I'm not sure what as the subject isn't interesting enough to do anything except scan qucikly.

Is the goal for one or the other of you two to admit defeat or that the price was such and so on this date or the chart or track records are rigged? If so to what end?

Or is this playing to some unseen audience?
I'm just curious...nice colors though.
 
jerry,

could have been a nice discussion, but i refuse to post
in between this BS that spreads all over my screen here.
some things never change ...

whatever ... take care ...
 
Quote from mind:

jerry,

could have been a nice discussion, but i refuse to post
in between this BS that spreads all over my screen here.
some things never change ...

whatever ... take care ...
More...

Well, let's take action...start a new thread and let Jack and the trend guy here to post very colorful attacks on each other endlessly. The first step would be to farme a topic, question or assumption. I'm open to suggestions. Let me know what you think.
 
You must log in or register to reply here.
Back
Top