did i apply curve fitting to my system

Hey Guys,

Changing any parameter in a model, whether it be OHLC or some percent stop, is still a change in a model parameter and may introduce bias into your model.

In Short, I agree with nephos on this matter.

I'll try to explain with an example:

Every trading model is a collection of rules. Rules can essentially be anything. Some learning algorithms, under supervised learning (such as pattern finders), can be utilized to guess at rules and then determine which rules work best. Selected/guessed Rules can be modified for "wieght" in the model, or they can be eliminated or combined. No matter how one argues it, O(t) > C(t-1) or O(t) > H(t-1) constitutes 2 rules that can be assigned either together, seperately or with a weigthing function. The process is no different from changing your stop loss amount or using a variant on the ATR length.

Changing any rule or any permutation of a rule after the model has been run and results have been interpreted constitutes curve fitting.
 
Quote from TSGannGalt:


In another words, you'll be dealing 2 separate systems if you are going to have one system use Open and the other using Close. Which makes them out of the scope for a relative analysis of whether something is "curve-fitted".
More...

No you won't. They are the "same" system with a rule permutation. Unless you're talking about the sample data set itself, in which the rules aren't incorporated.

Rather than laugh at what is being argued you ought to present clearer contradictory logic.

So far you seem to be saying that changing the rules of a system is not curve fitting, which is wrong (or maybe you're talking about something else).

Mike
 
Quote from Mike805:

No you won't. They are the "same" system with a rule permutation. Unless you're talking about the sample data set itself, in which the rules aren't incorporated.

Rather than laugh at what is being argued you ought to present clearer contradictory logic.

So far you seem to be saying that changing the rules of a system is not curve fitting, which is wrong (or maybe you're talking about something else).

Mike
More...

Define curve-fitting. After that, try defining the difference between "robust" and "curve-fit" as a measure. Then give it a thought... you'll come up with an answer.

I'm not your math teacher.
 
There is far too much semantic quibbling on this thread over irrelevant peripheral points. Too many posters, while having good intentions, are missing the forest from the trees.

When discussing curve fitting or model optimization, the only relevant focus should be on how well that model reflects performance out of sample.

The simple fact of the matter is that however you choose to model some system, there will always be some unobservable bias or variance for the simple reason that you cannot see into the future. In the original post, both systems have bias as again, you cannot see what the system looks like out of sample.
Any boolean or mixed rule set suffers from the same problem.
Stop arguing over the explicit definition of 'curve fitting'; it wastes time. Instead, if it makes you happier, use the term model fitting.

If you can agree with this, then the 'relevant' question becomes, how do you deal with potential bias in something you are modeling? I would argue that some good suggestions have already been made early on.
 
Quote from TSGannGalt:

This thread is actually getting funny.

All this talk can end with a few Wiki links, but then... why end the humour.
More...

Nerds like to quibble over these things. Emacs vs. Vi, Windows vs. Linux, C# vs. Java.

It goes with the territory.

Off-topic, but worthy of discussion:

The real issue with these debates is how they get resolved in the work place. Is it better to standardize on one guy's opinion, or is better to give traders/quants/programmers their own freedom, so long as their product is functional?
 
Quote from TSGannGalt:

Define curve-fitting. After that, try defining the difference between "robust" and "curve-fit" as a measure. Then give it a thought... you'll come up with an answer.

I'm not your math teacher.
More...

Seriously? That's your answer?

This is a waste of time.

Answer this: is changing a rule in your model POST OBSERVATION of model results curve fitting or not?

Mike

P.S. Stop with the math teacher crap, its old. In line with your "my dick is bigger than yours" type comeback, I did my master's thesis on vola. modeling using something called an ASAM under the leading expert in AI and neural networks. What exactly are your credentials?
 
Quote from Mike805:

Seriously? That's your answer?

This is a waste of time.

Answer this: is changing a rule in your model POST OBSERVATION of model results curve fitting or not?

Mike

P.S. Stop with the math teacher crap, its old. In line with your "my dick is bigger than yours" type comeback, I did my master's thesis on vola. modeling using something called an ASAM under the leading expert in AI and neural networks. What exactly are your credentials?
More...


This all depends.

I agree, Mike, anyone with Gann in the name has no clue what he's talking about.

If your close to your original hypothesis, tweaking for an extra 10% APR or more is fine.

If you go from -50% NP to plus 1000%, there's a little too much fitting. If you're going from 70% APR to an extra 5% to 75-80% APR, there's nothing wrong with that.

It will depend on how much of a boost this is to your original results.

On your "if-touched" problem, only consider your order filled if the trade limit order "trades through" the price, or do what Mike suggested and add a tick of slippage plus commission to both sides.
 
Quote from dtrader98:

There is far too much semantic quibbling on this thread over irrelevant peripheral points. Too many posters, while having good intentions, are missing the forest from the trees.

When discussing curve fitting or model optimization, the only relevant focus should be on how well that model reflects performance out of sample.

The simple fact of the matter is that however you choose to model some system, there will always be some unobservable bias or variance for the simple reason that you cannot see into the future. In the original post, both systems have bias as again, you cannot see what the system looks like out of sample.
Stop arguing over the explicit definition of 'curve fitting'; it wastes time. Instead, if it makes you happier, use the term model fitting.

If you can agree with this, then the 'relevant' question becomes, how do you deal with potential bias in something you are modeling? I would argue that some good suggestions have already been made.
More...

Yes. You are correct.

Though, the little quibbling is about whether the set of performance samples are relevant between systems that use OHLC as a parameter. To do so, some people need to understand the nature and gain clarity towards what "curve-fitting" is. The measurement of "curve-fitting" is simply a relative measure between a set of parameters in and/or out-of sample. Though, if the logic used to derive the set of samples for the analysis is irrelevant, the decision of whether something is "curve-fitted" becomes completely irrelevant.

In all, we agree. There are "biases" and "pitfalls" of assessing whether something is curve-fitted and we need to be discussing about how to avoid them, but to do so, the "biases" need to be defined and understood. Then generate a viable set of sample data. And finally, assess the "robustness"-so to speak.

So far, some people are not willing to understand.

PS. Do we really have to make this a flaming fest?
 
Quote from TSGannGalt:

Yes. You are correct.

Though, the little quibbling is about whether the set of performance samples are relevant between systems that use OHLC as a parameter. To do so, some people need to understand the nature and gain clarity towards what "curve-fitting" is. The measurement of "curve-fitting" is simply a relative measure between a set of parameters in and/or out-of sample. Though, if the logic used to derive the set of samples for the analysis is irrelevant, the decision of whether something is "curve-fitted" becomes completely irrelevant.

In all, we agree. There are "biases" and "pitfalls" of assessing whether something is curve-fitted and we need to be discussing about how to avoid them, but to do so, the "biases" need to be defined and understood. Then generate a viable set of sample data. And finally, assess the "robustness"-so to speak.

So far, some people are not willing to understand.
More...

Because you're not saying anything. What he's doing is perfectly fine. It's optimization as opposed to curve fitting, with curve fitting being the case where he purposely hard codes something into his system that handles specific cases. If the data is not restricted by any extremum value he can optimize on this data and will be very representative of results going forward.
 
You must log in or register to reply here.
Back
Top