Optimization, curve-fitting and probability

[Scientist]

Quote from Wong Lee:


ROFLMAO!!!!!!!!!!!!!!!!!!!

Wong!

LOL! At least Wong gets the joke. He probably knows that exactly this has been said by Confuzius and later re-quoted by Kim Ung-Yong, two major philosophers, both probably a lot smarter than "nononsense".

"nononsense", however, still walks around the lives of the literate completely blind withtout a cane, embarrassing himself by criticizing those who know. How funny can you be LOL?

I'm putting "nononsense" into quotation marks now, because it's such a funny oxymoron, I just have to highlight it.


Best Compliments,
~Scientist

 

[NinjaTrader_Dierk]

@harrytrader

I appreciated your in-depth post much. Let me please ask some questions:

Quote from harrytrader:


Supposing that the premisces of the student test (more generally of a parametric test) are reasonable (which is not evident as said above but let's do as if they were) this says that if "the (true but unknown) mean was equal to 0" (called null hypothesis H0) the probability would be alpha = 0.00184. The contrary of H0 is "the (true but unknown) mean wasn't equal to 0". Then the basic axiom of probability says that Prob(H0) + Prob (non H0) = 1 (since 1 is the probability of certainty ) so that Prob (non H0) = 1 - Prob(H0) = 1-0.00184. This is called the significance of the test.
After that if an experience E is repeated with independancy, etc., the probability of P(E1*E2*E3*..*En)=P(E1)*P(E2)... this explains power(1-0.00184, 20). Substracted from 1 give the significance.

Thanx for your explanation. I understand the basics of statistical calculations. So I know how and how the calculation is performed, but I don't understand the "meaning" (don't have a better term). The calculation return the probability that 20 independent test of the same probability all(!!) show me that there is a Prob(non HO) is true. But what does that mean? What does this imply in the above context? May be, I still have turned my brain off ...

Quote from ButterMilk:

Words from a successful optimizer:
...
3. always walkforward realistically

@ButterMilk
Could you please elaborate on the term "realistically" ?

Quote from nitro:

The reason is obvious. Even when you write a new system, althought the system has never been optimized against the data, every time YOU have written a system against the data set, you are optimizing YOURSELF, which in turn are becoming more and more curvefitted to writing winning systems on that data. The correct way to really test this is not only to hold back data from the system, but hold back data FROM YOURSELF that you will never see. Once you have seen that data, you should have some sort of utility penalty function that subtracts some profit from the system if you write a new system against that data, or optimize against it in any way. The idea is ALWAYS to have at least a year of data that you have NEVER seen.

@Nitro
True words. That's what in fact made me thinking about what I'm doing when I select any parameter set for a given strategy. Any selection of parameter (or even a strategy itself) is an "optimization". Every time I test and validate a strategy, I choose settings parameters which can even hardly be testet "out-of-sample" or "walk-forward". Examples:
- the pool of instruments I focus on
- length of in-sample and out-of-sample periods

Ok, I do not want this issue to become too academic. So I ask myself: do strict roll-forward tests (stepwise through the complete time-frame, verifying virtually "all" selected parameters) make sense, or is this simply not possible ?