Activate/Deactivate System?
- Thread starter EricP
- Start date
Doesn't look like it. I compared it to some stats software. The mean needs to be constant for the whole equation and denominator should be constant as well, ie. it should be the variance for the whole series.
Just to confuse you, you could look at the partial autocorrelation function as well. You're looking at the system P&L? This could show if there is some sort of similar pattern, say every five days. It removes the muck of the days in between. I tried to include a picture of what it should look like, but my software isn't cooperating.
The top attachment is a PACF of the S&P 500. The bottom is a non-stationary time series of something.
Just to confuse you, you could look at the partial autocorrelation function as well. You're looking at the system P&L? This could show if there is some sort of similar pattern, say every five days. It removes the muck of the days in between. I tried to include a picture of what it should look like, but my software isn't cooperating.
The top attachment is a PACF of the S&P 500. The bottom is a non-stationary time series of something.
A few loose thoughts on "Activate/Deactivate System?"
(1) I myself would never call "System" something to be activated and deactivated. If I 'deactivate', I would throw it in the garbage can.
Of course, I might do things like turning on and off algorithms or strategies, but I always would treat such procedures as an INTEGRAL PART of the SYSTEM. If you believe that you can make things work by activating/deactivating the whole, you simply don't have a system!
(2) As to evaluation and testing. I don't think that there is any fixed 'optimum' mathematical path to do this. You simply don't have (m)any rigorous assumptions to start from. I used to work on computer control of industrial processes. In this environment, your models being derived from physical processes, are often way better understood, although still stochastic. Nevertheless, I learned from this experience that a successful project always depended on combining theoretical (mathematical) know how with common sense. In fact, this common sense part kind of guides you all the way along (I hope that I don't start sounding like Jack on this
)
(3) From the above, I adopted a thoroughly 'common sense' approach to the market system development. I use sufficient mathematics, but never too much! Recently, I came across the following by Fredrico Zeri, the titan fine arts historian and astounding expert: "I don't believe in improvising. In truth, great art is always the result of extraordinary technical ability." This might appear to be contradicting 'common sense' but you can't have 'extraordinary technical ability' without applying some common sense in manipulating mathematical tools in the marketplace.
(4) Coming back to turning on and off systems. I myself have only ONE system. If I would turn it off, I would quit for good. I have tried many things that I dumped along the way. The system I kept coming back to is the only one for me. This is very important in testing. I often use lengthy spread out trials in many dissimilar markets before adopting some new idea. I found out though that developing something new can often be done on the basis of very limited data sets - at least in the case of my habitual system. The lengthy tests simply confirm later what you thought to be valid. I came only to this after having injected sufficient common sense in the whole thing: i.e. "System design is testing and testing is system design.©
"
Be good,
nononsense
(1) I myself would never call "System" something to be activated and deactivated. If I 'deactivate', I would throw it in the garbage can.
Of course, I might do things like turning on and off algorithms or strategies, but I always would treat such procedures as an INTEGRAL PART of the SYSTEM. If you believe that you can make things work by activating/deactivating the whole, you simply don't have a system!
(2) As to evaluation and testing. I don't think that there is any fixed 'optimum' mathematical path to do this. You simply don't have (m)any rigorous assumptions to start from. I used to work on computer control of industrial processes. In this environment, your models being derived from physical processes, are often way better understood, although still stochastic. Nevertheless, I learned from this experience that a successful project always depended on combining theoretical (mathematical) know how with common sense. In fact, this common sense part kind of guides you all the way along (I hope that I don't start sounding like Jack on this
)(3) From the above, I adopted a thoroughly 'common sense' approach to the market system development. I use sufficient mathematics, but never too much! Recently, I came across the following by Fredrico Zeri, the titan fine arts historian and astounding expert: "I don't believe in improvising. In truth, great art is always the result of extraordinary technical ability." This might appear to be contradicting 'common sense' but you can't have 'extraordinary technical ability' without applying some common sense in manipulating mathematical tools in the marketplace.
(4) Coming back to turning on and off systems. I myself have only ONE system. If I would turn it off, I would quit for good. I have tried many things that I dumped along the way. The system I kept coming back to is the only one for me. This is very important in testing. I often use lengthy spread out trials in many dissimilar markets before adopting some new idea. I found out though that developing something new can often be done on the basis of very limited data sets - at least in the case of my habitual system. The lengthy tests simply confirm later what you thought to be valid. I came only to this after having injected sufficient common sense in the whole thing: i.e. "System design is testing and testing is system design.©
"Be good,
nononsense
nononesense,
It sounds like we come from similar backgrounds. I come from a background of automated control of industrial processes, as well (chemical engineer at a manufacturing plant).
Your point is well taken about simplicity. Too many traders are trying to find the 'perfect' system, or the 'perfect' analysis technique to assist them in their trading. Seems like someone in Market Wizards said something to the effect of "you came become rich, by not being perfect." The point is that having a statistically exact answer is impossible in trading, and almost useless if you could get it. The future is not easily predictable, and we need coarse analysis techniques to show us the way, IMO. Simple is good.
As related to your comments on activation/deactivation, you can look at my methods as a structure for when to activate or deactivate a subsystem, and view all trading being done as the complete 'system'. For me, I trade 4+ systems, with some trading 200+ stocks. So, I use the analysis discussed in this thread to determine when to turn OFF live trading of "System 3" for security INTC, for example.
Thanks for the informative post.
-Eric
It sounds like we come from similar backgrounds. I come from a background of automated control of industrial processes, as well (chemical engineer at a manufacturing plant).
Your point is well taken about simplicity. Too many traders are trying to find the 'perfect' system, or the 'perfect' analysis technique to assist them in their trading. Seems like someone in Market Wizards said something to the effect of "you came become rich, by not being perfect." The point is that having a statistically exact answer is impossible in trading, and almost useless if you could get it. The future is not easily predictable, and we need coarse analysis techniques to show us the way, IMO. Simple is good.
As related to your comments on activation/deactivation, you can look at my methods as a structure for when to activate or deactivate a subsystem, and view all trading being done as the complete 'system'. For me, I trade 4+ systems, with some trading 200+ stocks. So, I use the analysis discussed in this thread to determine when to turn OFF live trading of "System 3" for security INTC, for example.
Thanks for the informative post.
-Eric
Quote from nononsense:
A few loose thoughts on "Activate/Deactivate System?"
(1) I myself would never call "System" something to be activated and deactivated. If I 'deactivate', I would throw it in the garbage can.
Of course, I might do things like turning on and off algorithms or strategies, but I always would treat such procedures as an INTEGRAL PART of the SYSTEM. If you believe that you can make things work by activating/deactivating the whole, you simply don't have a system!
(2) As to evaluation and testing. I don't think that there is any fixed 'optimum' mathematical path to do this. You simply don't have (m)any rigorous assumptions to start from. I used to work on computer control of industrial processes. In this environment, your models being derived from physical processes, are often way better understood, although still stochastic. Nevertheless, I learned from this experience that a successful project always depended on combining theoretical (mathematical) know how with common sense. In fact, this common sense part kind of guides you all the way along (I hope that I don't start sounding like Jack on this
(3) From the above, I adopted a thoroughly 'common sense' approach to the market system development. I use sufficient mathematics, but never too much! Recently, I came across the following by Fredrico Zeri, the titan fine arts historian and astounding expert: "I don't believe in improvising. In truth, great art is always the result of extraordinary technical ability." This might appear to be contradicting 'common sense' but you can't have 'extraordinary technical ability' without applying some common sense in manipulating mathematical tools in the marketplace.
(4) Coming back to turning on and off systems. I myself have only ONE system. If I would turn it off, I would quit for good. I have tried many things that I dumped along the way. The system I kept coming back to is the only one for me. This is very important in testing. I often use lengthy spread out trials in many dissimilar markets before adopting some new idea. I found out though that developing something new can often be done on the basis of very limited data sets - at least in the case of my habitual system. The lengthy tests simply confirm later what you thought to be valid. I came only to this after having injected sufficient common sense in the whole thing: i.e. "System design is testing and testing is system design.©
Be good,
nononsense
prophet,
Excellent post. Thanks for the information and the link. I can certainly see how autocorrelation could be helpful in trading system activation/deactivation.
It seems like an advanced, but possibly very useful tool. Now you've got me also wondering whether other traders use this analysis technique in their system development and activation procedures. If so, hopefully, they will come forward to explain how they use this technique and whether it's been useful for them.
Thanks again for the explanation,
-Eric
Excellent post. Thanks for the information and the link. I can certainly see how autocorrelation could be helpful in trading system activation/deactivation.
It seems like an advanced, but possibly very useful tool. Now you've got me also wondering whether other traders use this analysis technique in their system development and activation procedures. If so, hopefully, they will come forward to explain how they use this technique and whether it's been useful for them.
Thanks again for the explanation,
-Eric
The following posts were taken from another thread (http://www.elitetrader.com/vb/showthread.php?s=&postid=746181#post746181), but added here to help consolidate this discussion for reference:
Let me try to address what the confidence level calculations from the linked thread actually measure (more detailed information may already be on that thread, not sure). The confidence level attempts to mathematically determine the likelihood that a system has an expected average profit above zero, based upon the date you give to it. So, for example, the following three series of trade results will have exactly the same confidence level:
a) +$50, -$20, +200, +$10, - $20
b) +$500, -$200, +2000, +$100, - $200
c) +$5000, -$2000, +20000, +$1000, - $2000
In this case, examples b and c are the same P&L results, but multiplied by a factor of 10 and 100, respectively. All three examples will have the exact same statistical confidence, since the statistical confidence does not measure dollar profit or loss, but the likelihood that the average profit is above zero in the entire population (versus this small sample) of trades using this system.
For your systems tested, therefore, you can conclude that the implementation of the stop-loss and profit objective makes it more likely that the system will be profitable over the long run. Note that I am assuming that you have roughly the same number of data points for both tests (with stops/targets, versus without), because simply the increase in the number of datapoints will also increase your confidence level. This makes sense, right? You become more confident about a system's profitability after steadily making net profits in your first 30 trades, versus only your first 10 trades. So, to be fair comparing your systems with and without stops/targets, you should be sure that there is not a huge difference in the number of trades being using for each method in the confidence calculation.
I agree. Using the statistical confidence calculations is merely a tool, and does have shortcomings. I find it to be very valuable as well, and much better that what I used previously (manually making the on/off decision, without any structured methodology).
Another good point. Systems that work across many markets are very nice when you find them. Another thing I look for is a system with very few optimized parameters, or better yet, a system that works well before any optimization is even done.
Best of luck,
-Eric
Let me try to address what the confidence level calculations from the linked thread actually measure (more detailed information may already be on that thread, not sure). The confidence level attempts to mathematically determine the likelihood that a system has an expected average profit above zero, based upon the date you give to it. So, for example, the following three series of trade results will have exactly the same confidence level:
a) +$50, -$20, +200, +$10, - $20
b) +$500, -$200, +2000, +$100, - $200
c) +$5000, -$2000, +20000, +$1000, - $2000
In this case, examples b and c are the same P&L results, but multiplied by a factor of 10 and 100, respectively. All three examples will have the exact same statistical confidence, since the statistical confidence does not measure dollar profit or loss, but the likelihood that the average profit is above zero in the entire population (versus this small sample) of trades using this system.
For your systems tested, therefore, you can conclude that the implementation of the stop-loss and profit objective makes it more likely that the system will be profitable over the long run. Note that I am assuming that you have roughly the same number of data points for both tests (with stops/targets, versus without), because simply the increase in the number of datapoints will also increase your confidence level. This makes sense, right? You become more confident about a system's profitability after steadily making net profits in your first 30 trades, versus only your first 10 trades. So, to be fair comparing your systems with and without stops/targets, you should be sure that there is not a huge difference in the number of trades being using for each method in the confidence calculation.
I agree. Using the statistical confidence calculations is merely a tool, and does have shortcomings. I find it to be very valuable as well, and much better that what I used previously (manually making the on/off decision, without any structured methodology).
Another good point. Systems that work across many markets are very nice when you find them. Another thing I look for is a system with very few optimized parameters, or better yet, a system that works well before any optimization is even done.
Best of luck,
-Eric
Good post, a couple of thoughts about it. I come from an options perspective so this is not totally applicable. Markets can trade a little differently from my perspective. This is true when comparing equity and commodities, but for a system this may be irrelevent abd signals may work the same. I have no idea on this point as I've never traded a system in either.
The main point I do have some idea, that being the applicability of stats. The confidence interval only tells you something about what you put in. As I use these methods more and more, I realized that you need to go far beyond simple critical values. You need to beat the hell out of the data a little(more like a lot) to get at the underlying process. The first thing I would do is remove the outliers and look at the data again. A system that relies on them for profitability is going to give a higher probability of never getting off the ground.
The main point I do have some idea, that being the applicability of stats. The confidence interval only tells you something about what you put in. As I use these methods more and more, I realized that you need to go far beyond simple critical values. You need to beat the hell out of the data a little(more like a lot) to get at the underlying process. The first thing I would do is remove the outliers and look at the data again. A system that relies on them for profitability is going to give a higher probability of never getting off the ground.
Good point. To some extent, taking the confidence level will weed out systems that 'look good' based upon only a very few excellent trades. However, if you are trading systems with only moderately high confidence levels, then you should be especially cautious of that sort of system performance (as they might fall through the cracks and pass your confidence level test).
I'm not certain whether I would do the same thing on the outlier bad trades, though. I always like to error on the conservative side, and would tend to include all of the really bad trades without question. For example, I would imagine that a trader writing options would have mostly profitable trades, with the occasional 'outlier' bad trade. These bad trades are part of the risk of writing options and should be included in the system analysis, IMO. These sorts of systems can be especially hard to determine 'real' confidence levels, since it is hard to know if you have got a 'fair' number of outlier bad trades in your data set. I guess this is sort of the Black Swan issue.
-Eric
Definitely don't throw them out. Rather, analyze the outliers seperately. In a lot data analysis you would just toss em and not worry about it, but to us they are important. You could look at the P&L of the system as a whole, the P&L of the outliers and the P&L of those removed.
I came up with this idea only recently as I was taking a time series class(I'm also a grad student). For my final, I had a high frequency data set that was actually biological in nature. I ended up using a financial model to, well, model it! The key to my conclusion was to remove the jumps, or in this case outliers, to look at how volatile the underlying process was. The patients were actually very similar with the exception of these jumps. The sickest one's volatility were nearly entirely driven by these.
This seems very similar to examining the P&L as we want to see what's driving the numbers. A volatile system with a large number of outliers may still be worth trading but you may need to cut down the size so as not to stop yourself out from trading it. Or a non-volatile system may need to be increased in size. Does that make sense?