Activate/Deactivate System?

QUOTE]Quote from EricP:

Sorry, just saw your questions.
1) That's a fortunate problem to have. I would trade them both! If I could only trade one of the two, I would trade the first one and consider increasing trade size by 10% if desired to increase profitability.

2) I'm not sure I understand the need to make further "adjustments" for the sample size. Obviously, the larger the sample size, the greater the degree of confidence you can have in the profitability of the system being reviewed. But, that is precisely what the equation already accounts for. Personally, I like to put an upper limit on the sample size (for me, upper limit ~ 150), to ensure that a huge sample size doesn't make any negligible profitable system have a confidence level of 99%+. Otherwise, you could have a system with a very small profit and a very large standard deviation (an otherwise unattractive system) APPEAR to be extremely attractive based upon a sample size of 20,000 trades, for example.

Other way to handle this would be to require a minimum profit per trade of some fixed amount, in order to trade the system. In this way, a system with an average profit of $1 per trade, with a standard deviation of $1500 could be eliminated, regardless of a potentially high confidence level that would be generated based upon 20,000+ trades.

P.S. Post any other questions in the other thread, to keep the discussion more easy to follow. [/QUOTE]

thx for the feedback - i suppose another approach could be to calculate the "implied" minimum sample size for a given level of confidence, e.g. i have avg profit of 1 and stdev of 5; i want a 98% confidence level, t-value of 2.0 - to get to this level, my sample needs to be at least 100 (i.e. 1/5*sqrt(100)=2)

i suppose what you are saying is that you are comfortable when the implied / necessary sample size is 150 or less (i am assuming that actual sample size will always be reasonable, i.e. hundreds, at least).

i guess the question is - what level is one "comfortable" with in terms of the number of trades which will take your average profit to the level of 2 standard deviations above zero?
 
I feel that one danger in paper designing systems is to rely too strongly on assumptions of normal distribution. This is fine in most cases, but I feel that ideally you would want to preserve all characteristics of the underlying distribution, something best achieved by random sampling.

In the case of a system with a historical performance of 1 and std dev of 5 based on sample, I would also consider the following simulation : use the sample to construct randomly sampled simulated results, complete with friction cost and whatever position sizing regime you use, and then take measurements from each simulated sample.

You'll want to measure the outcome of course, but also the max drawdown and the time spent in drawdown and all sorts of other metrics that interest you.

If you make thousands of simulations and collect thousands of these metrics, you can build confidence intervals for all those metrics - eg what is the 90th percentile drawdown, what is the 10th percentile outcome after 100 days .... etc, all of which will give you a much better sense of what kind of experience you will have trading the system.
 
Mean Variance tests - a previous poster had written about a modification of adding tight stops to a system that lowered the expectation but increased the z-stat.

All else being equal, what this result means is that the modification is superior, not inferior.

The fact that it has a lower expectation is more than made up for the fact that it can take on more leverage to produce greater return at the same level of risk.

This is the theoretical argument of course and is subject to close examination of the skewness of the returns.

Increased leverage makes one much more sensitive to outliers.
 
Quote from TheStudent:

I feel that one danger in paper designing systems is to rely too strongly on assumptions of normal distribution. This is fine in most cases, but I feel that ideally you would want to preserve all characteristics of the underlying distribution, something best achieved by random sampling.

In the case of a system with a historical performance of 1 and std dev of 5 based on sample, I would also consider the following simulation : use the sample to construct randomly sampled simulated results, complete with friction cost and whatever position sizing regime you use, and then take measurements from each simulated sample.

You'll want to measure the outcome of course, but also the max drawdown and the time spent in drawdown and all sorts of other metrics that interest you.

If you make thousands of simulations and collect thousands of these metrics, you can build confidence intervals for all those metrics - eg what is the 90th percentile drawdown, what is the 10th percentile outcome after 100 days .... etc, all of which will give you a much better sense of what kind of experience you will have trading the system.
More...

good point, i think this process is called nonparametric bootstrapping, and was discussed in a recent thread here on ET.

i looked up bootstrapping after somebody on ET posted an interview with Eckhardt (sp?) who mentioned it and said he prefers to work with raw data rather than summary statistics - it is making more sense to me now.
 
Quote from EricP:

Is my system profitable? At what level of confidence?

Assuming we are using 'future' data (whether this is backtested future data or actual forward tested data doesn't matter), then we can use statistical analysis to our data to determine the likelihood that our system is a profitable system over the long run as follows:

x = abs(Avg-Profit) * (Number of Trades)^0.5 / (Std Dev of Profits)

Once "x" is calculated, you can look up the confidence level on the chart below to see the likelihood that this system will be profitable over the long haul:

x ------------------ Confidence Level

0 ------------------ 50%
0.075 ------------ 53%
0.126 ------------ 55%
0.25 -------------- 60%
0.36 -------------- 64%
0.52 -------------- 70%
0.67 -------------- 75%
0.84 -------------- 80%
1.04 -------------- 85%
1.28 -------------- 90%
1.4 ---------------- 92%
1.645 ------------- 95%
2.05 --------------- 98%
3.0 ----------------- 99%
4.0 ----------------- 99.5%
5.0 ----------------- 99.7%

Note that if the average profit is negative, then this will give you the confidence level that this is a losing system, over the long run.

One quick example to illustrate this equation. Assume that we have 50 trades, yielding an average profit of $40 per trade, with a standard deviation of $150 per trade. Using the equation and table:

x = abs(Avg-Profit)*(Number of Trades)^0.5 / (Std Dev of Profits)
x = abs(40)*(50)^0.5/150
x = 1.88, from table, Confidence Level ~97%

-Eric
More...

Just wanted to make sure that I understand this right. From my back test over the last 3 months, I got the following sequence of win/losses for 18 trades:

295
283
83
-79
395
-67
208
-192
8
220
-18
120
-30
133
-17
-5
-55
145

This amounts to $79.28 profit per trade, with the standard deviation of $156.

Plugging these into your formula gives me 2.16 which maps to confidence interval of between 98% and 99%. It feels like the formula defies the intuition. I mean, would you really feel that the strategy has greater than 98% chance to succeed over the long run based on the results of 18 trades over a 3-month test period?
 
Quote from nonlinear5:


This amounts to $79.28 profit per trade, with the standard deviation of $156.

Plugging these into your formula gives me 2.16 which maps to confidence interval of between 98% and 99%. It feels like the formula defies the intuition. I mean, would you really feel that the strategy has greater than 98% chance to succeed over the long run based on the results of 18 trades over a 3-month test period?
More...

As I probably mentioned earlier in the thread, I prefer to have at least 30 trades before using the formula for statistical significance. However, to answer your question => Yes, based upon your data, I would certainly be happy to activate the system you describe. Obviously, going forward, the results might improve or deteriorate, but my activation decision must be made with the information we currently have at hand.

Just to be clear what the formula is saying. Let's assume that a certain trading strategy makes 5000 trades over the course of two weeks (we'll call this the entire 'population' of trades that we have). Now, let's write the results of each of the 5000 trades on a piece of paper and put them in a bucket. Next, we will draw out 18 random pieces of paper from the bucket and do the calculation above, based on the results of those 18 trades. The formula tells you that there is a ~98% chance that the entire population of 5000 trades has an average profit above zero (i.e. net profitable). That's all it tells you. But, I find this to be extremely useful in providing an unbiased foundation for making activation/deactivation decisions.

Best of luck,
-Eric

P.S. I've now 'subscribed' to this thread, so that I will be able to see when future posts are made and will hopefully do a better job of replying to questions in the future. (this is the first thread I've subscribed to, so hopefully it will work). Also, if I fail to address anyone's that posts to me on this thread in the future, please feel free to send me a PM and point out the post. Thanks.
 
Hi,

Someone can help to code this formula
x = abs(Avg-Profit) * (Number of Trades)^0.5 / (Std Dev of Profits) in EL tradestation ?

Thanks.
Best Regards.
Ludo.
 
Quote from robinxing:

http://hquotes.com/tradehard/simulator.html

Eric, the equity curve varys when I input the same data

but when w/l is greater than 2 and w prob >0.4

the equity changes little

why
More...

That site provides a random equity curve to be expected for a system with your input system stats. Every time you rechart a new equity curve, it will be slightly different, based upon normal random variation that is to be expected for any system. The point of the site is to give you a rough idea of what an equity curve might look like for a given system (including drawdowns, etc).

If this doesn't answer your question, then you'll need to rephrase it, because I wasn't exactly certain what you were asking.
 
I use mean-variance optimization to determine the weights of the systems in my portfolio. Because I reoptimize regularly, systems whose performance starts to deteriorate are "gradually" stopped out, i.e. their weight will diminish over time, and may reach zero eventually in case they haven't offered anything in terms of reducing the variance of the returns or improving the profitability of the portfolio.

I also like to plot things like Sharpe and Sortino ratios (together with bootstrapped confidence intervals) over time and see if and how they converge as the sample gets larger. Ideally, they settle around some average after 100-200 obs, while the confidence interval narrows over time.
 
You must log in or register to reply here.
Back
Top