How do you know if it's luck or real edge?

[jonnysharp]

you know your got a real edge because your method captures some sort of market behaviour and you understand why it works and when to change parameters and when to stop trading it. think in terms of poker.

if you don't know if your got a real edge, i suggest you record historical maxDD%, multiply that by 1.5 and use the amount as a trailing stop on your equity curve, for insurance, just in case you were exploiting a temporary nuance in the markets.

 

[martys]

Quote from EricP:

I've seen many systems with confidence levels over 99%. Any system can deteriorate over time, as the market changes and evolves. Personally, I recalculate my confidence levels daily, based upon the new trades (if any) executed each day. If the confidence level ever drops to below ~90%, then I discontinue trading in the system. I will continue to monitor and paper trade the system, and will reactivate the system only after the market is trading in a way that is more in synch with that strategy again.

-Eric

Hi Eric,

Have you done testing on the effects of activate/deactivate using short-term confidence test on the your systems? BTW, I do perform confidence test on the overall history of my systems so I am not questioning that.

The reason I am asking is that there is also the logical idea of trading equity curves (such as activate/deactivate as the equity curve crossover its moving average etc.) . I found various methods of trading equity curves actually hurt the performance of my systems (they might help some other systems out there).

 

[jonnysharp]

equity curve tatics will hurt systems in most cases, but if he doesnt have some sort of equity insurance for a half ass edge, when does he stop trading it, 30%, 40%, 50%, 60% drawdown? so even if you do stop trading a system and it then proceeds to turn profitable again, it doesn't matter because you cut the exposure of ruin, you have to draw the line in the sand somewhere.

 

[illiquid]

Quote from gnome:

It's not "how many trades", but rather "what your trades are about".

I think this hits the nail on the head. Part of developing and implementing an edge is understanding exactly what it is you are exploiting or seeing ahead of the crowd.

It's tempting to say that if you have to ask, then you probably don't have one, but then again, the only way you'll ever find one is to keep on asking.

 

[martys]

Quote from jonnysharp:

equity curve tatics will hurt systems in most cases, but if he doesnt have some sort of equity insurance for a half ass edge, when does he stop trading it, 30%, 40%, 50%, 60% drawdown? so even if you do stop trading a system and it then proceeds to turn profitable again, it doesn't matter because you cut the exposure of ruin, you have to draw the line in the sand somewhere.

Equity curve tactics can hurt the systems is probably because the trades are more or less independent events (losing streak does not necessarily make the next trade more likely a loser at least for system trading) but there is a huge opportunity cost into thinking you are losing your edge prematurely. I am just wondering how good this sliding window confidence test is...

 

[TriPack]

Quote from EricP:

This thread might be a good place to start:
http://www.elitetrader.com/vb/showthread.php?s=&threadid=36083&highlight=deactivate

It will show you how to calculate a statistical confidence level, providing a more rigid and statistical way to determine if the system is good, or whether it might just be in a lucky streak instead.

-Eric

Hello Eric,

Following your instructions I implemented this confidence testing in all my systems. Since doing so I have tested possibly hundreds of systems. Of those systems I have tested hundreds and hundreds of different permutations on several markets. I found out something interesting about this.

While not ideal, the smaller spread of the P&L results increases our confidence that this system MAY be profitable over the long haul.

This is the basic assumption that this type of analysis makes: that a smoother P&L curve will lead to future profits with more certainty than a bumpy P&L curve. One thing I stumbled upon was that I took an always in system, and tested it. It got like 60-70% confidence depending on the particular market it was applied to. Then I took the same system and only added a very tight profit target and stop loss (turning it into a scalping system) and did the tests again. Overall profitability went down significantly but the confidence level shot way up above 90% on virtually every market tested. So the conclusion I came to is that the confidence interval really measures the smoothness of returns, rather than whether that particular system may or may not be profitable over the long haul. This is from forward testing and observing these and other systems side by side. And specifically the confidence interval measures the short term smoothness of returns, which can be affected by a string of wins or losses in the short term, which may or may not continue. There is no guarantee that the system will stay hot or will stay cold, and I haven't found a way to test whether these hot/cold streaks tend to persist or not so I'm not certain that filtering out systems based on a low % is necessarily the best course.

Don't get me wrong, I appreciate your work and continue to seek out systems that as a general rule have smoother equity curves. I think having a systematic measurable approach is far superior to what I was doing previously. It is a good tool and I use it regularly.

But the thing that gives me the most confidence in finding an edge is if I apply the same system to multiple markets without changing the system at all, and see if it achieves similar results in many different markets. If it does, my confidence goes up that there might be something there. If the system is a one market anomaly, then it is probably curve fit to the data.

 

[jonnysharp]

matrys, i agree with what you say but imo historical data is the best thing you've got, trades can be independant but for some systems they profit of certain market cycles, so trading streaks can be explained, so once you get the skill you can discretionary trade systems i.e. asset allocation, but until then i would recommend he use equity curve insurance.

the system will always be there and can always be paper-traded, but if it ain't making coin like she used too, then i would be on the sidelines. thus i dont welcome new extreme data sets kindly. if you have a stable of systems this shouldnt infer with the overall operation.

 

[martys]

Quote from jonnysharp:

matrys, i agree with what you say but imo historical data is the best thing you've got, trades can be independant but for some systems they profit of certain market cycles, so trading streaks can be explained, so once you get the skill you can discretionary trade systems i.e. asset allocation, but until then i would recommend he use equity curve insurance.

the system will always be there and can always be paper-traded, but if it ain't making coin like she used too, then i would be on the sidelines. thus i dont welcome new extreme data sets kindly. if you have a stable of systems this shouldnt infer with the overall operation.

I understand where this is coming from... you have to respect the risks first.

 

[martys]

Quote from TriPack:

Hello Eric,

Following your instructions I implemented this confidence testing in all my systems. Since doing so I have tested possibly hundreds of systems. Of those systems I have tested hundreds and hundreds of different permutations on several markets. I found out something interesting about this.



This is the basic assumption that this type of analysis makes: that a smoother P&L curve will lead to future profits with more certainty than a bumpy P&L curve. One thing I stumbled upon was that I took an always in system, and tested it. It got like 60-70% confidence depending on the particular market it was applied to. Then I took the same system and only added a very tight profit target and stop loss (turning it into a scalping system) and did the tests again. Overall profitability went down significantly but the confidence level shot way up above 90% on virtually every market tested. So the conclusion I came to is that the confidence interval really measures the smoothness of returns, rather than whether that particular system may or may not be profitable over the long haul. This is from forward testing and observing these and other systems side by side. And specifically the confidence interval measures the short term smoothness of returns, which can be affected by a string of wins or losses in the short term, which may or may not continue. There is no guarantee that the system will stay hot or will stay cold, and I haven't found a way to test whether these hot/cold streaks tend to persist or not so I'm not certain that filtering out systems based on a low % is necessarily the best course.

Don't get me wrong, I appreciate your work and continue to seek out systems that as a general rule have smoother equity curves. I think having a systematic measurable approach is far superior to what I was doing previously. It is a good tool and I use it regularly.

But the thing that gives me the most confidence in finding an edge is if I apply the same system to multiple markets without changing the system at all, and see if it achieves similar results in many different markets. If it does, my confidence goes up that there might be something there. If the system is a one market anomaly, then it is probably curve fit to the data.

Thanks for sharing... I do think confidence testing is valuable but as someone else told me as a rule of thumb... you should have at least 95% confidence level for your system's profitability (over many trade sequences).

 

[kiwi_trader]

Tripack,

Having done a lot of statistical testing in the past I support what you say about smoothness not proving that the system will not fall of its edge and fail in future. A change in market volatility, say, can collapse a wonderfully smooth system.

I also partially agree with the view that testing over multiple markets (or even simulated or segmented markets) can give you confidence - but I think that this is confidence in the breadth of the edge and thus the chance of falling off it.

This breadth is only appropriate to some edges ... to me a kind of generic edge like buying retracements or breakouts in a trend. It is not appropriate to edges that relate to particular behaviours that you discover in a specific market. Such edges may be valid and may persist for a long time so they can be well worth trading. It seems to me that key with these edges is understanding them and being able to monitor them in such a way that one stops trading them if they lose their sharpness.