Activate/Deactivate System?

my first real ET post...be gentle, please.
(great thread by the way)

regarding non-gaussian p/l distributions:

This is just a random idea -

I remember reading ehlers talking about using a fisher transform to normalize price data so that std. dev. based oscillators would give sharper signals.

Couldn't a similar concept be applied to a p/l that looks like -2, -3, -2.5,, +10?

Maybe you code it up as W,W,W,L? I know there are statistical tests for this kind of categorized data but i don't know them off the top of my head and leave that for people smarter than me.

 
I thought I'd add some additional thoughts to this thread, specifically with respect to paper trading.

Paper trading has a bad reputation among many in the industry as being a worthless exercise, and having no value for the successful trader. I completely disagree with this view, and feel that when done properly, paper trading can be an extremely valuable tool for the successful trader.

First of all, paper trading must be done properly, to ensure that 'impressive' results are not returned for an otherwise worthless system simply due to the over-optimization of testing. There is more on this topic in the posts above.

Let's assume, that a trader uses valid methods to back test his/her system, and then decides that the system is 'worthy' of activation for live trading. Furthermore, let's assume that the system produces an average profit of $100 per trade for this testing with a standard deviation of $200 over 300 trades in the test data.

Once activated, the trader is disappointed to see that Murphy's Law is in effect and the system loses money on it's first 5 consecutive trades, giving a quick loss of $1000. Should this system be immediately deactivated?

On it's own, five trades would certainly not seem to be statistically significant. In addition, adding these five trades to the 300 trades in the prior test results is unlikely to change the overall P&L of the system sufficiently to warrant deactivating the system.

However, the first thing I would do in this situation is to update my paper trading results... Typically, I will simultaneously paper trade all systems while I am also trading the system live, so I already have the information I need. The question here, is:

How did my paper trading perform over the past five trades?

Assuming that my paper trading results match my live trading results (and of course, my back testing was valid in the first place), then I don't have any problem other than bad initial luck with this system and my trading will continue.

However, what if my paper trading shows breakeven, or possibly a small profit for these past five simultaneous trades? => This would indicate that my paper trading execution assumptions are invalid and my entire backtesting needs to be reviewed. The system should be deactivated immediately, in this case, since the activation of the system was based upon incorrect assumptions. We know these assumptions are invalid, since we cannot duplicate our paper trading 'results' with our real performance when 'traded' simultaneously.

This is the way that we 'close the loop' to solidify our trading results. We use 'correctly conducted' paper trading to validate our trading system in order to determine what to activate. Then, once activated, we compare of real results with our simultaneous paper results to validate the execution assumptions of our paper trading.

Assuming that our system passes both of these tests, then we continue to assume that we have a 'good' system, and give the system more time to perform after some initial bad trades.

-Eric
 
I just received a PM regarding this post, and thought I would post the PM (minus name) and my response, to help others that might have the same questions. The PM was:

"My question to you is related to confidence levels you showed in your post. My statistics foundation is very poor and I could not managed to get the whole table of confidence levels with "x" for all % levels. Could you please advice me where I can find table with all confidence levels for a test you suggested for activation/deactivation of a trading system."

My reply was:

You should be able to use the table shown for all %levels as follows:

For any value of x less than zero, you can use the same table. Just make the x value positive, and the table will show you the confidence level that the system is a LOSING system.

For example, if x = -1.04, then we will look up +1.04 in the table.
=> By doing this, we see that the system is a LOSING system at a 85% confidence level... Put another way, the system is a WINNING system at only a 15% confidence level.

I hope this helps. Best of luck,
-Eric


Quote from EricP:

Is my system profitable? At what level of confidence?

Assuming we are using 'future' data (whether this is backtested future data or actual forward tested data doesn't matter), then we can use statistical analysis to our data to determine the likelihood that our system is a profitable system over the long run as follows:

x = abs(Avg-Profit) * (Number of Trades)^0.5 / (Std Dev of Profits)

Once "x" is calculated, you can look up the confidence level on the chart below to see the likelihood that this system will be profitable over the long haul:

x ------------------ Confidence Level

0 ------------------ 50%
0.075 ------------ 53%
0.126 ------------ 55%
0.25 -------------- 60%
0.36 -------------- 64%
0.52 -------------- 70%
0.67 -------------- 75%
0.84 -------------- 80%
1.04 -------------- 85%
1.28 -------------- 90%
1.4 ---------------- 92%
1.645 ------------- 95%
2.05 --------------- 98%
3.0 ----------------- 99%
4.0 ----------------- 99.5%
5.0 ----------------- 99.7%

Note that if the average profit is negative, then this will give you the confidence level that this is a losing system, over the long run.

One quick example to illustrate this equation. Assume that we have 50 trades, yielding an average profit of $40 per trade, with a standard deviation of $150 per trade. Using the equation and table:

x = abs(Avg-Profit)*(Number of Trades)^0.5 / (Std Dev of Profits)
x = abs(40)*(50)^0.5/150
x = 1.88, from table, Confidence Level ~97%

-Eric
More...
 
Eric, sorry for being so blunt but how smooth does all of that make your combined multi-system equity curve? A picture (or maybe even pictures of several system equity curves + cumulative to illustrate benefits of activation/deactivation??!!?) would be worth a thousand words.
 
Quote from sunnyskies:

Eric, sorry for being so blunt but how smooth does all of that make your combined multi-system equity curve? A picture (or maybe even pictures of several system equity curves + cumulative to illustrate benefits of activation/deactivation??!!?) would be worth a thousand words.
More...

Sure.

First, a disclaimer... I don't manage any client funds. I don't want to manage any client funds. I don't have or want clients of any kind, so I am not hyping anything. I am only answering this question because someone asked. In other words, there is no need to flame me for replying with the truth.

My equity curve closely resembles what you would see by putting my performance numbers into the "Random Equity Curve" simulator at http://hquotes.com/tradehard/simulator.html
(cool site, I might add).

My performance stats, over the last 34 months of live trading (~640 daily P&L numbers) are:

Avg Profit: Avg Loss (i.e. Win/Loss) = 1.227
Win Probability = 0.72

Plug those performance numbers into the Equity Curve Generator and you will see the rough equity curve 'picture' for my trading. I'm sure that my results are not nearly as smooth as Acrary's are, based on the risk tolerances that he mentioned for his trading. But, the results are very consistent, with few losing months and a steady uptrend in overall equity.

I should also point out that I believe the reason for my consistent results are not only due to system diversification, but also heavily influenced by the diversification of trading many markets using the same system, while only taking very small risks with any specific trade. For example, I might trade one system with ~150 different stocks during the day and execute 500 trades with that system. So, even if only 55% of my trades are profitable, I stand a very good chance of consistent profits simply due to the diversification of so many trades each day. The losers are offset by the winners, and hopefully, the overall net is positive on most days. My point is that you should not interpret these results as being exclusively due to the diversification among different systems. For my trading, I think diversification within the same systems have been equally important for the smoothing of the daily equity curve.

Best of luck,
-Eric

P.S. For what it's worth, I posted some other performance data recently at the following post, http://www.elitetrader.com/vb/showthread.php?s=&postid=685494#post685494
I got several flames from that, so I'm a little wary of sharing much these days in terms of personal results. However, you might find that that post helps to illustrate things better to answer your question, as well.
 
Are all of acrary's posts preserved on this forum and will be preserved in the future? I believe they're of very high value and want to save them for the future generations so to speak. I believe if someone made an archive of all acrary's posts and posted them on a web page somewhere that would be a tremendous contribution.
 
Eric, what I do is monte-carlo the trade distribution that I get from testing. If I do as poorly as the bottom 10% then I start to question the methodology. The advantage is you don't have to assume anything about the distribution of results this way.
 
Quote from peterfigliozzi:

Eric, what I do is monte-carlo the trade distribution that I get from testing. If I do as poorly as the bottom 10% then I start to question the methodology. The advantage is you don't have to assume anything about the distribution of results this way.
More...

I think that sounds like an excellent method. Perhaps a bit too advanced for me (not sure how I would do a Monte Carlo), but I think it sounds as good or better than what I do.

A couple of questions for you: How many real time data points (i.e. trades) do you require from your system before you make the comparison with the 10 percentile Monte Carlo results? Also, assuming you have been trading the system live for a very long time (500+ trades), then how many real trades do you use for your comparison? In other words, do you always use your full data set of real trades, or only the most recent x trades?

Finally, with your method, activation/deactivation seems to be a two step process. First, the system must pass your backtesting results in order to be activated in the first place, and then it must continue to perform above the 10 percentile Monte Carlo to continue live trading. Assuming I have stated this correctly, how do you judge that a system has passed your backtest criteria?

Thanks,
-Eric
 
Quote from peterfigliozzi:

Eric, what I do is monte-carlo the trade distribution that I get from testing. If I do as poorly as the bottom 10% then I start to question the methodology. The advantage is you don't have to assume anything about the distribution of results this way.
More...

This is exactly what I have begun to do. Since none of my systems have more than 100 trades, I assume that my systems are curve fit and will at some point fail. I wrote a program that will randomly select 20 trades (could be more or less) from my entire sample and create the performance stats for that sample. I repeat the process 1000 times. If results for the next 20 trades going forward underperform the bottom 10 percent of the 1000 random samples the system will be deactivated. I just wrote the program yesterday, so I have not decided on the optimal sample size.

fan27
 
You must log in or register to reply here.
Back
Top