Degrees of Freedom

[horribilicus]

I prefer 3 or 4 adjustable parameters for the entry+exit portion, and another 1 adjustable parameter for the position sizing portion, in the mechanical systems I trade.

However, there is an interesting philosophical disagreement about "how should you count parameters?"

Let's say your system is

• Reverse to long when MovingAverage(Close, param1) > MovingAverage(Close, param2)
• Reverse to short when MovingAverage(Close, param1) < MovingAverage(Close, param2)

How many parameters is that?

Some people would say that you made a choice, a decision, you picked a parameter value, when you decided to average the Closes. You could have just as easily chosen to average the Opens, or (H+L)/2 or (H+L+C)/3 or (O+H+L+C)/4. So that decision of yours adds one to the parameter count. ... maybe it adds four since there are four moving averages in the system ...

Some people would say that you made a decision, so you picked a parameter value, when you chose to use the SIMPLE moving average. You could have chosen the Exponential moving average or the Jurik moving average or the Weighted moving average or the T3 moving average. So that decision of yours adds one to the parameter count.

Some people would say that you made a decision when you chose not to have a stoploss order as part of your trading system. There is a binary, yes-no parameter named "DoesItTradeWithStops" and you set it to "No." So that decision of yours adds one to the parameter count.

You made a choice, you picked a parameter value, when you decided to trade your system on 60 minute bars. You could have chosen 5 minute bars or 180 minute bars or daily bars or weekly bars. So that decision of yours adds one to the parameter count.

This line of reasoning will quickly lead you to the conclusion that ALL systems have hundreds of parameters dues to the hundreds of choices you made. Most of those choices were to leave something out but they were choices nevertheless.

 

[nononsense]

Quote from horribilicus:

[]
This line of reasoning will quickly lead you to the conclusion that ALL systems have hundreds of parameters dues to the hundreds of choices you made. Most of those choices were to leave something out but they were choices nevertheless.

Let me answer the following - without implying that this is going to make money for you.

In model building - in areas where demonstrable merit exists - a popular but naive misconception is to throw in a liberal bunch of extra parameters for good measure so as "not to miss any". Not much advanced schooling is required to grasp the fallacy of such proposition.

One more thing, model building as described above deals with what you could call a priori known topology - allowing perhaps for some undetermined number of parameters. This is the popular case in everything related to regression type of models.

A much wider vista is offered by what one could call model building with unknown topology where things like functional dependencies are unknown. Parameters could also be involved here, perhaps at a later stage.

What percentage of physical processes can be modelled by the first and what percentage by the second kind? What do you need to make money in the marketplace? These questions often depend on what you want to get out of your model but are difficult to answer a priori.

What do I use? Not much of the above. However, an intense contact with these above fields has been of tremendous help to me in finding out what doesn't work in the marketplace.

nononsense

 

[RoughTrader]

I have found a strong correlation in number of adjustable parameters and number of trades within the optimization period where curve-fitting is concerned. That is, in general, the less trades taken in the observation period and the more parameters used will tend to curve-fit the results more.

That being said, it should be recognized that there is a small mitigating factor. In a given parameter set, if some parameters are relatively uncorrelated from other parameters, then the total number of parameters in the set tends to effect the likelihood of curve fitting less than if all parameters in the system correlated with each other.

I find that, in general, parameters for setup, those for entry, and those for exit tend to be relatively uncorrelated (at least for my strats).

In any case, I believe that the most robust way to ensure that the underlying system logic is correct and results are not simply fitted is to maximize the number of trades in the optimization period.

RoughTrader

 

[stephencrowley]

Quote from nononsense:

Stephen,

Your piece lacks some common sense. All I meant to say is that if you have to worry about the degrees of freedom and overfitting within you brain, I would suggest that not much fruitful thinking is taking place.

I'm not worried about it, and that is precisely my point, and humans in general CAN be good at overfitting if they think about what they are doing. Just look at the graphs showing the relations of # of pirates existing in the world to the rise in global sea temperature, if you make any inferences from that then it is obvious overfitting if you assume a direct casual relationship, the variables do have some relationship, namely a common factor, that of the flow of time and progress of society.

Don't forget that all mathematical concepts, including the in these posts much beloved "overfitting" and "degrees of freedom", are only fictions of human construction. Sometimes mathematics has been helpful in gaining a frail human understanding of the physical world. Such understanding is almost certainly dated and will be considered crude and obsolete in a not too far distant time.

I don't think you meant to use the word fictitious, cause obviously this stuff is all some simplification of reality, and that is what modelling is all about, simulating reality. If it was 'fictitious' then the model would mean absolutely nothing.

Models definitely become outdated which is why strategies should be adapted, either implicitly as part of your strategy, a kind of continuous tracking, or you can do it explicitly at some time, depending on how you deem the model performance to be changing.

This applies of course to your: "overfitting can be avoided by carefully understanding how you are optimizing, using regularization techniques, bayesian methods, reinforcement learning, etc." - your etc& being quite prudent. As we are dealing here with theories of speculation, I can tell you that after having spent a considerable time of my life with the above, that most of these are useless if you ever aspire to make some money in the markets. Especially, don't further mix in pretentions of simplistic understanding about the inner workings of the brain. Stop dreaming and look for something better. In fact, you could put your brain to work on this instead of mimicking control and estimation procedures that indeed appear useful in the case of some physical processes. If you don't, come back in about 10-20 years to tell us about your pipe dreams.

I like how you make assumptions about my understanding of the brain. I'm sorry your control models didn't work out for you, but just because you were incapable of creating successful control models does not mean that it is impossible to create a successful model. It all depends on how you formulate the problem and how you choose your performance criteria and make inferences.

 

[nononsense]

Quote from stephencrowley:

[]
I like how you make assumptions about my understanding of the brain. I'm sorry your control models didn't work out for you, but just because you were incapable of creating successful control models does not mean that it is impossible to create a successful model. It all depends on how you formulate the problem and how you choose your performance criteria and make inferences.

It's one assertion in juxtaposition to another.

 

[stephencrowley]

Great post.. any time you make apriori assumptions about the data, you are introducing what is known as bias into the model, your model will be biased towards your presuppositions.

I try to focus extremely hard on explicitly writing down all assumptions.. assumptions must be made at some level, no matter how abstract or complex that assumption might be.

Quote from horribilicus:
However, there is an interesting philosophical disagreement about "how should you count parameters?"

 

[stephencrowley]

Yes.. your optimization criteria is just as important as the data that goes in to your model. When defining the function, the sky is the limit as to how you want to optimize. My trade executions are optimized (along with all the fee structures on all the various market places) along with everything else so I don't make assumptions as to how many trades I "should" be making, I let that number arise on its own according to the data.

Quote from roughtrader:

In any case, I believe that the most robust way to ensure that the underlying system logic is correct and results are not simply fitted is to maximize the number of trades in the optimization period.

roughtrader [/B]

 

[nononsense]

Quote from stephencrowley:

[]
I try to focus extremely hard on explicitly writing down all assumptions.. assumptions must be made at some level, no matter how abstract or complex that assumption might be.

Ever heard about "Garbage IN, Garbage OUT"?

I quit making assumptions a long time ago. I use my brain instead.

nononsense

 

[stephencrowley]

Quote from nononsense:

What percentage of physical processes can be modelled by the first and what percentage by the second kind? What do you need to make money in the marketplace? These questions often depend on what you want to get out of your model but are difficult to answer a priori.

Have you ever considered that these two "types" are not types at all, and they can be estimated or 'grown' simultaenously? Perhaps your apriori assumption that these two steps are seperate has led you to believe that these methods have no use.

What do I use? Not much of the above. However, an intense contact with these above fields has been of tremendous help to me in finding out what doesn't work in the marketplace.

nononsense [/B]

So, what does work for you?

 

[stephencrowley]

That's a pretty amazing feat nonsense, I believe you are the first person ever to exist without making assumptions.

Maybe you should let everyone know that Gödel was wrong..

Quote from nononsense:
Ever heard about "Garbage IN, Garbage OUT"?

I quit making assumptions a long time ago. I use my brain instead.