How to research and verify trading ideas

[talontrading]

Yikes, made a mistake with what i posted. Correction here in bold:

The rule I posted is fine, but there was a potential issue with entries and exits that you'll see if you use it. You should set up your testing structure, however you do it, so that these things are variables... making a change like this should be extremely easy.

If someone goes with the original rules we'll use them too, but these are more clear to start with.

Buy when today's close is equal to or less than the 25th percentile of the past 10 days' range.

Sell that position when the close is equal to or above the 50th percentile of the past 5 days' range.

Rules for shorts are opposite.

Quote from talontrading:

Ok... cool... but how about this deal? You guys do the work. I'll give ideas and lay out each step, but it's up to the readers of this thread to actually crunch the numbers and come up with answers to each step. As long as we get good answers or good attempts at answers I'll go through the evaluation of each step so you can see the thought process. If the answers we get are wrong I'll correct them as long as there's interest.

This can be done in Excel or whatever people have available? (btw, what do the readers here use? Tradestation? Matlab? R? Excel? VBA? WealthLab?)

First step: Buying pullbacks.
1. Look at this portfolio of stocks. (more or less randomly selected big stocks from different industries)
MSFT
COH
MON
OXY
DE

2. Going back to 1/1/2004, what would have been the results of applying this rule set:

Buy when today's close is equal to or less than the 50th percentile of the past 10 days' range.

Sell that position when the close is equal to or above the 50th percentile of the past 5 days' range.

Rules for shorts are opposite.

This is just a starting point, but start here. Someone post some answers and we'll go from here.

 

[jtrader33]

More than fair. I normally use VBA and Amibroker (can access R through it...still learning though), but will post results in Excel for ease of sharing. Also, I've put the code here (want to make sure I interpreted the percentile of the range correctly).

PositionSize = 1000;

EntryPct = .25;
ExitPct = .5;

Range_10d = (HHV(H,10) - LLV(L,10)) ;
Range_5d = (HHV(H,5) - LLV(L,5));;

BuyTrigger = LLV(L,10) + EntryPct * Range_10d;
ShortTrigger = HHV(H,10) - EntryPct * Range_10d;
ExitTrigger = LLV(L,5) + ExitPct * Range_5d;

Buy = C <= BuyTrigger;
Sell = C >= ExitTrigger;

Short = C >= ShortTrigger;
Cover = C <= ExitTrigger;

Quote from talontrading:

Ok... cool... but how about this deal? You guys do the work. I'll give ideas and lay out each step, but it's up to the readers of this thread to actually crunch the numbers and come up with answers to each step. As long as we get good answers or good attempts at answers I'll go through the evaluation of each step so you can see the thought process. If the answers we get are wrong I'll correct them as long as there's interest.

This can be done in Excel or whatever people have available? (btw, what do the readers here use? Tradestation? Matlab? R? Excel? VBA? WealthLab?)

 

[talontrading]

Ok. Good start. I did not have a chance to go through your trades carefully, but they are slightly different than mine. I did code mine from scratch so there's a chance the mistake is actually mine... haven't had a chance to check carefully but I'm fairly sure what I did is correct.

Attaching a spreadsheet with all the trades for the specified tickers and dates and some basic analysis. Two points about the analysis: I have the code generate a binary variable called "IsWin" that is 1 if the return is greater than 0. Averaging this variable gives you the win % for the system. Cute trick. Second, the return is in basis points (in other words 1% = 100 in the output.) The reason for this is that the human mind deals with a number like 53 much better than .0053. Again, little tricks.

Please check the trade lists and see where the difference is. Also, take a look at the analysis and let me know what you would think at this point if this were a fresh idea and you had no idea whether it was a good one or not. What would be your concerns at this point?

This is open for discussion for the anyone reading. Don't be shy.

Quote from bjs211:

More than fair. I normally use VBA and Amibroker (can access R through it...still learning though), but will post results in Excel for ease of sharing. Also, I've put the code here (want to make sure I interpreted the percentile of the range correctly).

PositionSize = 1000;

EntryPct = .25;
ExitPct = .5;

Range_10d = (HHV(H,10) - LLV(L,10)) ;
Range_5d = (HHV(H,5) - LLV(L,5));;

BuyTrigger = LLV(L,10) + EntryPct * Range_10d;
ShortTrigger = HHV(H,10) - EntryPct * Range_10d;
ExitTrigger = LLV(L,5) + ExitPct * Range_5d;

Buy = C <= BuyTrigger;
Sell = C >= ExitTrigger;

Short = C >= ShortTrigger;
Cover = C <= ExitTrigger;

 

[talontrading]

It's also quite likely to be a difference in the adjustment for dividends. Different data services will do it slightly differently.

Quote from talontrading:

Ok. Good start. I did not have a chance to go through your trades carefully, but they are slightly different than mine. I did code mine from scratch so there's a chance the mistake is actually mine... haven't had a chance to check carefully but I'm fairly sure what I did is correct.

Attaching a spreadsheet with all the trades for the specified tickers and dates and some basic analysis. Two points about the analysis: I have the code generate a binary variable called "IsWin" that is 1 if the return is greater than 0. Averaging this variable gives you the win % for the system. Cute trick. Second, the return is in basis points (in other words 1% = 100 in the output.) The reason for this is that the human mind deals with a number like 53 much better than .0053. Again, little tricks.

Please check the trade lists and see where the difference is. Also, take a look at the analysis and let me know what you would think at this point if this were a fresh idea and you had no idea whether it was a good one or not. What would be your concerns at this point?

This is open for discussion for the anyone reading. Don't be shy.

 

[jtrader33]

Yep, that's exactly what it is...of your 1075 trades, we agree on Symbol, Entry Date, Exit Date on 1021, have differences on 46, and you have 8 that I don't. We're off by a few bucks on actual entry/exit prices but the returns are very close for the trades we agree on, so I think the logic of the trading rules ties out. Big picture, you have a mean of 45 and stdev 464 against my 44 and 469.

I was missing COH and OXY in my database so I just downloaded Yahoo adjusted data (which i think is from CSI) for all 5 symbols. I can attach either text or excel files with those prices if you think it will help with those following along?

My concerns:

1. The most obvious one is that the short trades don't make money. I would guess that it rarely makes sense to simply mirror long rules when developing a system given the different behavioral biases behind buying and selling.

2. The large stdev relative to average profit.

3. Leptokurtic w/ negative skew / losers considerably bigger than winners??
-I say this more because I've heard people preach about it rather than being convinced of it mathematically. Sure its great to have fat tails work in your favor whenever possible, but in my understanding, thats not what mean reversion trades are about. So i'm not really sure whats wrong with a system that consistently has a 70% win rate and 1.5:1 risk/reward. Plus, this could likely be improved with stops (or would you consider that curvefitting?).

4. The .65 ratio of annual % return / max DD isn't great. Although, it was interesting to see that while the shorts barely broke even, their inclusion improved this ratio for the long trades from .54 to .65 - reduced drawdown through diversification. Of course this measure is highly path dependent anyway.

 

[SomeYoungGuy]

First of all, love this thread. I actually fount the conflict between BoWo and Talon to be highly valuable. I learned what to care about what to ignore from their interaction. It reminded me of those old Point/CounterPoint skits on SNL with Jane Curtin and Dan Akroyd where they would talk about some topic, and Dan would always start off his counterpoint with "Jane you ignorant slut" lol good stuff.


I will venture a guess that a good addition to this strategy would be placing a $500 stop loss on each trade. The total losses for losing trades as-is was $144,005.40 and if there was a $500 stop it would only be $98,816.23 which makes for a fantastic bump in overall return.

Now, I understand that I am doing a bit of curve fitting here, but $500 is a nice round number that we could very understandably place on a trade regardless of strategy. If I came to you with a number like $483.12 then yeah that reeks of curvefitting.

I'm not set up with the tools to backtest myself, but the way to test my theory would be to see if any of the winning trades bounced below a $500 loss that would hurt us on the upside.

 

[talontrading]

Ah yes, good old BoWo. Still have a "conversation" with him over in his thread if you want the entertainment. It's basically me pointing out things he could do better and him saying that he, his wife and his mother (I cannot make stuff like this up) are the greatest ever at what they do. You just can't take him seriously and then it's funny.

As for your suggestion on a stop, it's not a bad idea but it may not be a good one. (I don't want to just pontificate and hand out answers here, but want to make people think.) I would suggest that a percentage stop probably makes more sense than a dollar stop. To that same end, I would always run a test like this assuming a fixed DOLLAR amount (not a fixed SHARE number) per trade. That way, your trade p&l is already basically a percentage and it's easier to evaluate. If you just buy 1000 shares it's a very different trade at $5 or $95.

Your thinking of finding a spot where putting a stop would not cut off too many winners is one good way to think. On the other hand, mean reversion systems like this can often work quite well with no stop at all if they are designed properly, but that makes position sizing difficult.

Quote from SomeYoungGuy:

First of all, love this thread. I actually fount the conflict between BoWo and Talon to be highly valuable. I learned what to care about what to ignore from their interaction. It reminded me of those old Point/CounterPoint skits on SNL with Jane Curtin and Dan Akroyd where they would talk about some topic, and Dan would always start off his counterpoint with "Jane you ignorant slut" lol good stuff.


I will venture a guess that a good addition to this strategy would be placing a $500 stop loss on each trade. The total losses for losing trades as-is was $144,005.40 and if there was a $500 stop it would only be $98,816.23 which makes for a fantastic bump in overall return.

Now, I understand that I am doing a bit of curve fitting here, but $500 is a nice round number that we could very understandably place on a trade regardless of strategy. If I came to you with a number like $483.12 then yeah that reeks of curvefitting.

I'm not set up with the tools to backtest myself, but the way to test my theory would be to see if any of the winning trades bounced below a $500 loss that would hurt us on the upside.

 

[talontrading]

It's fine to have slight differences in the database, but people starting work like this are surprised to find that even a simple question like "what price was COH on 4/1/06?" might not be simple to answer. We can proceed with the slightly different trade lists I think.

Short thoughts on your notes:
1. yes, a concern, but not that unusual for a stock idea. it is hard to find stock ideas that make money on the short side. in fact, some or all of the p&l you see for the longs may be the result of that upward bias. how could we address this?

2. does not bother me. i have seen much larger variation for profit on workable ideas.

3. yes and no. i think it's very important to really look at the distribution. look at it as a whole, look at it without outliers then look at only outliers. does it look like there's something real and worthy of further investigation going on here? we should not be too put off by big events in the tails, but you want to see some signs that the whole result is not just the result of random luck.

4. i like your thought process but it's too far at this stage. too path dependent as you said and at this point we just want to drill into the basic building blocks of market behavior.

you have raised good issues... what might the next steps be.

and for everyone reading this thread... this is not a private conversation. the more people adding ideas the better.

(i'll bet someone good money that BoWo will pop his pudgy little head in here and say i'm trying to get all of you to do the work so i can steal your ideas. watch and see lol...)

Quote from bjs211:

Yep, that's exactly what it is...of your 1075 trades, we agree on Symbol, Entry Date, Exit Date on 1021, have differences on 46, and you have 8 that I don't. We're off by a few bucks on actual entry/exit prices but the returns are very close for the trades we agree on, so I think the logic of the trading rules ties out. Big picture, you have a mean of 45 and stdev 464 against my 44 and 469.

I was missing COH and OXY in my database so I just downloaded Yahoo adjusted data (which i think is from CSI) for all 5 symbols. I can attach either text or excel files with those prices if you think it will help with those following along?

My concerns:

1. The most obvious one is that the short trades don't make money. I would guess that it rarely makes sense to simply mirror long rules when developing a system given the different behavioral biases behind buying and selling.

2. The large stdev relative to average profit.

3. Leptokurtic w/ negative skew / losers considerably bigger than winners??
-I say this more because I've heard people preach about it rather than being convinced of it mathematically. Sure its great to have fat tails work in your favor whenever possible, but in my understanding, thats not what mean reversion trades are about. So i'm not really sure whats wrong with a system that consistently has a 70% win rate and 1.5:1 risk/reward. Plus, this could likely be improved with stops (or would you consider that curvefitting?).

4. The .65 ratio of annual % return / max DD isn't great. Although, it was interesting to see that while the shorts barely broke even, their inclusion improved this ratio for the long trades from .54 to .65 - reduced drawdown through diversification. Of course this measure is highly path dependent anyway.

 

[jtrader33]

Quote from talontrading:


1. yes, a concern, but not that unusual for a stock idea. it is hard to find stock ideas that make money on the short side. in fact, some or all of the p&l you see for the longs may be the result of that upward bias. how could we address this?

Two ideas come to mind:
A. Detrend by subtracting the average daily return for the entire testing period from the actual daily returns and then recreating the price series from 1/1/2004 forward. Then apply the trading rules and see if the shorts show similar performance to the longs.
B. Change the short rules to be more selective, effectively acknowledging that the upward bias will continue. Which i guess is not unreasonable with non-negative interest rates, but having a directional bias doesn't seem right.

Quote from talontrading:


3. yes and no. i think it's very important to really look at the distribution. look at it as a whole, look at it without outliers then look at only outliers. does it look like there's something real and worthy of further investigation going on here? we should not be too put off by big events in the tails, but you want to see some signs that the whole result is not just the result of random luck.

I would think that this distribution passes this test since the profits are not the result of a few large positive tails. In fact, the high rate of consistent winners (albeit smaller in size) are more than enough to overcome a number of negative outliers. That is probably a good sign that the system is not in for a big surprise down the road (i.e. this is a fairly realistic representation of performance in live trading).


For next steps I would take a closer look at the losers in the system and see if there is some kind of pattern (low volatility periods, in a strong trend or range bound, etc.). So along with each trade I would record the ATR, some measure of trend and then analyze the winners vs. losers.

 

[TraderZones]

Quote from wutang:

Your point...
Adding "positive value" to another thread I see.

sorry it was so difficult for you to grasp a single sentence...

A user ID is identical to a multi-word website. Do you want to explore the probabilities of this combination?