validating a trade strategy
- Thread starter larrybf
- Start date
Are you referring to 95% accuracy, or x% accuracy within a 95% confidence level, whatever x may be?
If you are referring to the former, I can assure you that you will never have 95% accuracy in your trading unless you know something that no one else in this world does.
Assuming that you are referring to the latter, the sample size required to draw any meaningful conclusions is in part a function of the variance in your trade results. Even so, such "confidence" can only be tenuous at best for practical purposes because statistics is not very robust in predicting FUTURE human behavior, at least not as it relates to the dynamics of the markets. Many people refer to the market in terms of probabilities. In fact, there is no real probability distribution in the market's future price action from which to draw comfort. This is because you are dealing less with probability and more with uncertainty. Uncertainty does not come with hard numbers.
Judging by your question, it would appear that you are not too familiar with statisics. Please do not make the mistake of taking comfort in something you do not adequately understand. I think I know just enough about stats to realize that it is grossly overused and misused by market participants. Sure it's great to have hard numbers to support your trading strategy, but it's even better when those hard numbers are not illusory. The only hard numbers you'll ever have in trading is in your account balance. Everything else will be decidedly...softer. See it for what it is.
Sometimes common sense and a bit of pessimism are your best tools in arriving at conclusions about your method. As I understand it, the scientific process involves trying to DISPROVE a hypothesis. I think that is a good framework. If you have difficulty disproving the value of a trading plan with repeated testing, then you know you are headed in the right direction. But you will never be sure. At best, you will be less uncertain.
Just my 2 cents (Canadian currency).
Regards,
Thunderdog
...may I be permitted to post? I killed a thread recently by doing so, having become nearly the albatross that Harry is, and I don't want to screw yours up.
Thunderdog has a point. The way I usually put it is that you would have to assume a certain sort of distribution, like for example a normal distribution, in order to get any meaningful numbers. But in reality, we don't even know what kind of distribution would be appropriate, so there is simply no answer to your question. I have asked myself the same thing, and at first I thought 1000 trades would be a good number, but now I just accept the fact that there is not a number high enough for the kind of confidence we were looking for. Even if the market was as simple as roulette, there _are_ winning streaks where a player never dips into negative territory for a stretch of 10,000 games or more. These are not too infrequent. And that's just roulette, which _can_ and should be approximated by a normal distribution.
thunderdog and lobster..... thanks very much for the replies. it is true that i am rather ignorant of statistics. i have a once a day trading system that has been quite profitable in 2003 and i suppose i foolishly thought i could "prove" the system good enough to seriously increase my "bet size" by 3-4 times. thanks for stopping my search before i easted lots of time or even worse came to the wrong conclusions.... thanks again....
Larry,
That's great to hear about your profitable system. If it has served you well, then there is no reason why you should not increase your "bet size," assuming that you are risking a relatively modest amount per trade in relation to your trading capital. However, I think that most people will tell you to do so incrementally rather than in large jumps. That's because you will never have absolute "proof" in advance that your method will work in the future as planned. I think increasing size should be done with the kind of caution one takes when walking barefoot in the dark when you don't know for certain where the furniture is laid out. Slow and steady keeps the screaming to a minimum. Also, keep in mind that increasing size will likely have an effect all its own on your trading.
Regards,
Thunderdog
The distributions of this kind can be very bizzare and not good enough to be treated probablistically. They tend to have fat tails and may not even be normalizable, which in particular means that the standard deviation is infinite and so is your measure of error. People do assume the normal distribution for most practical purposes, but that's only an approximation which is known to be very crude and can in fact be totally wrong.
Just a theoretical perspective ...
... the main danger of ramping up size in a profitable system is increasing the damage of blowout when a "black swan" occurs, or when you hit a "perfect storm".
I would try to understand what are the "perfect storm" conditions (every system has one, IMHO), eg long periods of low vol, steep corrections et al. and do a max damage assessment of what could happen to your solvency if you are at full position and the market does a jump of +/- 20%
You can prevent "black swans" from killing you by always placing a far stop - it's effectively free insurance for all but the most discontinuous jump risks.
You can prevent "perfect storms" by placing screens that get you out of a position when the odds of a perfect storm start to brew.
These should help slice the tails.
And yes, a previous poster mentioned how conventional Y% interval at X% confidence is not very good at capturing trading risk management. There are a lot of good papers to read about the nature of market risk - but the quick answer is there are orders of magnitude more 5-6 sigma monster events roaming the markets than normal distribution would predict, so beware of their tails!
PS : I do not personally have experience with these systems under the conditions described, so I'm just describing how the systems are supposed to function in event of nuclear war
How they actually function is something only experience can teach you I guess
... the main danger of ramping up size in a profitable system is increasing the damage of blowout when a "black swan" occurs, or when you hit a "perfect storm".
I would try to understand what are the "perfect storm" conditions (every system has one, IMHO), eg long periods of low vol, steep corrections et al. and do a max damage assessment of what could happen to your solvency if you are at full position and the market does a jump of +/- 20%
You can prevent "black swans" from killing you by always placing a far stop - it's effectively free insurance for all but the most discontinuous jump risks.
You can prevent "perfect storms" by placing screens that get you out of a position when the odds of a perfect storm start to brew.
These should help slice the tails.
And yes, a previous poster mentioned how conventional Y% interval at X% confidence is not very good at capturing trading risk management. There are a lot of good papers to read about the nature of market risk - but the quick answer is there are orders of magnitude more 5-6 sigma monster events roaming the markets than normal distribution would predict, so beware of their tails!
PS : I do not personally have experience with these systems under the conditions described, so I'm just describing how the systems are supposed to function in event of nuclear war
How they actually function is something only experience can teach you I guess I like this thought. Of course, the most difficult step is the leap from one contract to two. I wonder if I will ever make it. Used to trade 100 share lots, after several months increased to 200, then 300, and then sometimes a lot more if the per share price was low. Now I trade 1-lots (futures), and I am wondering how long it will take before I am ready to increase my size by 100% !