Ranking indicators across all equities

[Rocko Bonaparte]

I wanted to describe something I have been trying to do in hopes that some others could describe some terms for this that may help me find more information about it. I'm trying to treat indicators as if they report a magnitude, rather than a crisp signal. Chaos ensues.

I am trying to develop a strategy that is mostly automated in picking and ranking equities to trade, leaving for now to execute the orders. For a start, I intended just to run it every night, and queues orders to execute on market open. Since I'm doing this in the evening, I figured I'd have some time to crunch a whole lot of data before some picks would plop out the other end. To that extent, I've written a lot of code.

Everything I've found so far on indicators has talked about solid buy/sell signals, and nothing about the shades in between. And here I am, running these formulas on every stock, with some intention of ranking them. So instead of calculating a solid buy signal, I'm trying to rank them by how much potential gain I could realize. Then after doing all my crunching, I sort by the highest scoring picks and go for them.

I'll contribute something more concrete here since I personally hate it when people keep their cards too close to themselves. I spent a bunch of time implementing something based off of this MACD strategy:
http://stockcharts.com/school/doku.php?id=chart_school:technical_indicators:moving_average_conve

This mentions three strategies for both buying and selling, and implies different magnitudes for each on how reliable they are the gains/losses one could realize with them. To try to rate the triggers, I'd estimate a percentage gain based on the slope of the fast EMA, and multiply it by 3, 2, or 1 based on the purported returns of each rule. The end result is a score, not a projected gain. However, the general idea is that the higher the score, the "better" it is.

Most wouldn't be surprised in backtesting that I'm not doing too well, as that's fairly usual. What would be surprising is if I apply this to individual stocks, I do come out fair; sometimes I lose, but so far I'm winning more than losing. When I open it up to all of NYSE, NASDAQ, AMEX, and ETFs, I get bled out. Most of the time, I trigger a 2% stop.

I noticed that the condition I applied the 3x multipler to wasn't doing well. IMO I don't think I am implementing that rule very well:

A Positive Divergence occurs when MACD begins to advance and the security is still in a downtrend and makes a lower reaction low. MACD can either form as a series of higher Lows or a second Low that is higher than the previous Low. Positive Divergences are probably the least common of the three signals, but are usually the most reliable, and lead to the biggest moves.

Generally if my macd crosses the reference line while the slope of the slow EMA is negative, then I was trying to trigger there. It just didn't look right when I studied individual charts. Anyways, after stripping that, I still don't do well scanning the whole market.

The rules I'm using to apply some magnitude to the indicator as basically coming out of my butt, so I hoped to find help in terms or links to stuff that are trying to do the same thing so I can do this all a lot better.

 

[black diamond]

This is similar to what a lot of quant equity strategies do, and the basic framework is in the public domain. I can post more on it later maybe with some references but the basic idea is you have a regression model with return on the left hand side and a bunch of stock characteristics on the right hand side. In the "classic" quant equity model each stock gets an expected return (usually monthly) based on things like P/E, book-to-market, and lagged returns.

You could probably do something similar with daily or even intraday returns, and use technical signals for each stock (like price- moving average) in place of the ratios. I've worked with some monthly models and I don't think they are too good for picking an individual stock but they are good at picking portfolios - like the top 10% outperform the bottom 10%.

 

[maxpi]

Some testing software comes to mind that is perfect for vetting theories like that. Use Openquant, use the simulated annealing backtesting mode. It's good for testing a strategy with a large number of variables. AMIBroker has a much faster backtester, can test large portfolios and has some kind of intelligent algo backtester but I'm not sure if it handles large numbers of inputs as well...

That is an interesting plan you have there, I never quite thought of approaching like that.. keep us posted if you have time..

There was a fund at T.O. Richardson that used what they called "sector rotation", they had 23% average annual returns for many years leading up to 2000 but after that the fund utterly self destructed. I think they were finding the sector with the strongest RSI and buying the one or two stocks in that sector that had the strongest RSI, something like that... I don't know how often they traded at all...

I'm just day trading a single index future currently but I'm jonesin' to get back into stocks and a longer time horizon, maybe start a pick service...

 

[jack hershey]

Quote from Rocko Bonaparte:

I wanted to describe something I have been trying to do in hopes that some others could describe some terms for this that may help me find more information about it. I'm trying to treat indicators as if they report a magnitude, rather than a crisp signal. Chaos ensues.

I am trying to develop a strategy that is mostly automated in picking and ranking equities to trade, leaving for now to execute the orders. For a start, I intended just to run it every night, and queues orders to execute on market open. Since I'm doing this in the evening, I figured I'd have some time to crunch a whole lot of data before some picks would plop out the other end. To that extent, I've written a lot of code.

Everything I've found so far on indicators has talked about solid buy/sell signals, and nothing about the shades in between. And here I am, running these formulas on every stock, with some intention of ranking them. So instead of calculating a solid buy signal, I'm trying to rank them by how much potential gain I could realize. Then after doing all my crunching, I sort by the highest scoring picks and go for them.

I'll contribute something more concrete here since I personally hate it when people keep their cards too close to themselves. I spent a bunch of time implementing something based off of this MACD strategy:
http://stockcharts.com/school/doku.php?id=chart_school:technical_indicators:moving_average_conve

This mentions three strategies for both buying and selling, and implies different magnitudes for each on how reliable they are the gains/losses one could realize with them. To try to rate the triggers, I'd estimate a percentage gain based on the slope of the fast EMA, and multiply it by 3, 2, or 1 based on the purported returns of each rule. The end result is a score, not a projected gain. However, the general idea is that the higher the score, the "better" it is.

Most wouldn't be surprised in backtesting that I'm not doing too well, as that's fairly usual. What would be surprising is if I apply this to individual stocks, I do come out fair; sometimes I lose, but so far I'm winning more than losing. When I open it up to all of NYSE, NASDAQ, AMEX, and ETFs, I get bled out. Most of the time, I trigger a 2% stop.

I noticed that the condition I applied the 3x multipler to wasn't doing well. IMO I don't think I am implementing that rule very well:



Generally if my macd crosses the reference line while the slope of the slow EMA is negative, then I was trying to trigger there. It just didn't look right when I studied individual charts. Anyways, after stripping that, I still don't do well scanning the whole market.

The rules I'm using to apply some magnitude to the indicator as basically coming out of my butt, so I hoped to find help in terms or links to stuff that are trying to do the same thing so I can do this all a lot better.

 

[jack hershey]

Great beginning.

Magnitude is an excellent measure when using any indicator. for the MACD which can produce a histogram based upon its component parts, there is another advantage as wel. I will pick up on several other advantages below.

Using indicators for timing in stocks is commendable too.

To get the results you are looking for, there are three simple refinements required. I'll stick with the MACD primarily.

The orginal designer used magnitude as well so you are in good company. Thje advent of the PC and software advancd the utility of stock trading using indicators.

1. The key to your success lies with invoking the concept of repeatability for a given stock.

2. assuming 1. then you have the timing down.

3. By having 2, in hand then you get to use the "leading indicator" function of MACD.

You get to do 1. by introducing a strong kurtosis to the distribution of the stocks you use for making money. Quality is the driver to achieve kurtosis. Since you already are using filters, you can code the solution easily.

Once you observe the repeatability aspect of this stock subset you filtered, then you get to use the indicator as originally designed; using a magnitude trigger. This assurance is not where you now expect it to be and it sacrifices a lot of the trending movement. But the original designer did achieve his goal of risk minimization at the sacrifice of the length of individual stock periods of engagement for the money making. As you discovered in 1. you can just process capital form one triggered stock to the next triggered stock by sequencing through your filtered stock subset.

So you have to step out of the box to exact more of the trend. This is where 3. comes into the picture. Understanding the math tells all about the indicator. So you do understand the "sentiment" aspect of indicators and the associated general principle of being on the right side of the market as a money making requirement.

One sophomoric solution is to use a set of charts tuned to different periodicities (See redneck's second post of his experience here).

To reconcil the nature of indicator signals (the conventional wisdom signal myths) and carving trend ends to optimize making money, you have to go to dervatives of non continuous functions. A tranformation does this for you. By understanding the possible conditions of two lines in relation to each other, you see that they may be grouped in specific patterns (there are 6) and the grouping is not symmetric. You can lay out the appearance of these patterns in the order that they occur. From this you get to understand how the order of events of the patterns takes you through the extremes of cycles of trends. This automates your two goals for trading: holding during trending and reversing during the passage of extremes, a period called "change".

That takes care of all the items above which address all the issues you didn't resolve. If you appreciate being able to differentiate trends and their end effects as in 3. above, perhaps you can see how adding an array of different dula lines indicators could be a more staccatto way of interleaving the individual order of events of each contributing indicator.

Another way to assure this and still maintain synchronicity by default adjustments is to skew the triggering of the order of events. T28 did this in his SPM with one caveat he didn't recognize what he was doing because of his failure to properly skew and he did not use the proper signals, either. These were two insurmountable handicaps because of his earnest effort to not take any input from others. Skewing is done by upping the fractal (shrinking the defaults in a compatible manner) and filtering out all but one kind of the signals (keep the first pattern signal in tact and unfiltered). This is an example of not using amplitude but instead taking advatage of an increased rate of change of a first derivative to get an earlier signal. In cycling the maximum first derivative occurs on the neutral of the cycle and as far away from either inflection point (the magnitudes you are focussed upon now) as is possible.

Because there are two lines, you deal with the faster line at all times where generating such signals is important.

Each iterative refinement improves the take. Because there is an advantage related in the compound interest formula in each of its variables, then you get to look to other variables than the profit per cycle that we have looked at. After you get stocks done, consider raising the exponent by a factor of 50 by switching to a market other than stock cycles.

 

[Rocko Bonaparte]

All very good replies so far. Regarding the tools fundamental analysts are using, that would be a "scanner." I had asked heard about that from people looking up stocks based on certain fundamental criteria they had established. Say, they want a certain P/E ratio with a certain market cap or somesuch. It's a good way to make picking companies to research more dry and unemotional. I haven't found anything relating to a scanner for using indicators.

Even if there was such a thing that used indicators, you'd have to figure out how to take all those things together and project something from it. A voice on my shoulder suggested I use an AI once...

I was trying some indicators together with an AI--a neuroevolving network. The indicators stunk by themselves, and it turned out together, they stunk more. The neural network took the "War Games" attitude to training: the only way to win was not to play. It would eventually saturate itself to one extreme or the other. And that was after chasing fools gold for about 4 months due to 4 different bugs in the process. $20k to $2m in a year. Hmm... forward testing didn't do so well.

Now I'll try to handle Jack Hershey's post because that had a lot of stuff in there. Jack, you may have to explain some of this like I was a seven-year-old.

By apply a kurtosis, do you mean to cull the list of stocks I'm scanning? Or do you mean to change the indicator i'm using so it doesn't just shoot off single peaks, but rather has some decay to the peak?

I also wondered what you meant by the term "repeatability." Perhaps is that consistency? Reliability? That would be enough for now, which means I got confused right at the beginning.

The strategy I intend to use--I guess I shouldn't get pigheaded and insist on it--was to hold the position for 24 hours from open to open. It just seemed suitable to the time I had to apply the strategy, and I feared if I held something across multiple days that I wouldn't keep as solid a regime.

So I'm looking for an indicator that suggests when it's time to take that quick, 24-hour dip. When I was looking at these stocks individually, I was doing that round-trip strategy. When the indicator had no opinion, I had coded it to just wait. If it triggered, then I'd do my 24-hour open-open plan. I'm pretty sure I'm not using the indicator the best I could as a consequence.

I had ran the strategy on every equity I had data for over a recent year's span last night, so now I have to go through and see if it was really true that this strategy does work ok generally for individual stocks, but not together. I'll post back eventually, but I've gotten a cold and I don't know if I'll be able to assemble the numbers together tonight.

Update: I did manage to mine the data. I tested roughly from Sept 1st, 2008 to Sept 1st, 2009 for each equity by itself. I looked at the amount of value I had in my portfolio start with $20k on day 1, and then got the percent change based on what I ended up with at the end of the last day. The histogram isn't so clear that I do better looking at only one stock after all, so this might be a wash:

(note that I removed everything where no trades happened at all, which did come up)

Bin	Frequency
<-0.5502472	0
-52.3%	5
-49.6%	11
-46.8%	1
-44.1%	0
-41.4%	0
-38.7%	1
-35.9%	3
-33.2%	0
-30.5%	1
-27.7%	0
-25.0%	2
-22.3%	1
-19.6%	0
-16.8%	0
-14.1%	13
-11.4%	49
-8.6%	173
-5.9%	492
-3.2%	938
-0.5%	1666
2.3%	920
5.0%	259
7.7%	112
10.5%	40
13.2%	22
15.9%	15
18.6%	6
21.4%	6
24.1%	3
26.8%	1
29.5%	1
32.3%	1
35.0%	0
37.7%	0
40.5%	0
43.2%	1
45.9%	0
48.6%	0
51.4%	0
54.1%	3
>0.5410287	0

 

[black diamond]

Quote from Rocko Bonaparte:

All very good replies so far. Regarding the tools fundamental analysts are using, that would be a "scanner." I had asked heard about that from people looking up stocks based on certain fundamental criteria they had established. Say, they want a certain P/E ratio with a certain market cap or somesuch. It's a good way to make picking companies to research more dry and unemotional. I haven't found anything relating to a scanner for using indicators.

Even if there was such a thing that used indicators, you'd have to figure out how to take all those things together and project something from it.

No, I don't mean a scanner. There are different ways to do it but what I am thinking of is called a Fama-Macbeth regression, and it definitely meets your need for putting it together an making a projection. It is especially good in sorting out redundant information from multiple indicators. Many quant funds use models like this to get a projected return for every stock in their universe and put that plus some risk and transaction cost info into an optimizer to come up with a portfolio.

So here's the crib notes on it. Say you are doing it daily. You organize your data with an an observation with the stock's return and the inidicators you want to try. On each day you do a separate linear regression like this:

Return = a + b1xIndicator1 + b2xIndicator2

You might have to play around with transormations on the indicators to get good results, I like replacing them with ranks and Z-scores are another idea. Do this for each day in your data and keep the b1 and b2. b1 and b2 are the fitted returns to the indicators for that day, and a is basically a market return that you don't use again. If an inidicator turns out to be worthless, its regression coefficient history will tell you.

So once you have your history of b1 and b2, you predict your future b1 and b2 and use it along with each stock's inidcator values to get forecast each stock's return. The simplest way to forecast is to just average b1 and b2 over the entire history - this is the basic Fama-MacBeth regression. This assumes the payoffs for each indicator are constant. If you think they are slowly time-varying, just use the recent history. And if you did all this and you want more, you can try to use things like VIX and yesterday's market return to time the b1 and b2 payoffs.

It is not as bad as it sounds - I can do it in about 20 lines of SAS code onse the data is set up so if you have messed around with AI it should be easy enough for you.

I glossed over a bunch of stuff but the post is long enough, especially if you are not going to persue this. If you want to see an example of it google Robert Haugen, he has a monthly fundamental model like this with results on his website and model details in a published paper.

 

[maxpi]

Zack's has a service where you can backtest fundamentals... I've never done that at all... I've spent many many hours with a screener that had lots of fundy data in it and never found a single item that related much to very short term trading.. I found higher gains in the NASDAQ arena and higher gains as the price went lower... that would be my first round of draft picks method if I wanted to trade baskets of stocks.. the OTC market is the bottom end of the price spectrum usually and I found that I couldn't get fills that allowed me to play in that game... conventional wisdom regarding stocks is that you want to be in the same issues that the big funds are trading, probably so you can read the volume clues but I would go there second after I exhausted all my lower priced picks...

There are several softwares that have scanners that base on technical indicators, Tradestation, Multicharts, Neoticker, Ninjatrader, AMIBroker come to mind..

I'd be surprised if any indicator or set of indicators could tell you much of anything.. they seem to work best in the divergence mode really... Toby Crabel tested a lot of volatility based indications and had some successes over the years. He had a book out but it's hard to find nowadays...

 

[PowerST]

Quote from Rocko Bonaparte:
Most wouldn't be surprised in backtesting that I'm not doing too well, as that's fairly usual. What would be surprising is if I apply this to individual stocks, I do come out fair; sometimes I lose, but so far I'm winning more than losing. When I open it up to all of NYSE, NASDAQ, AMEX, and ETFs, I get bled out. Most of the time, I trigger a 2% stop.

Quote from Rocko Bonaparte:
I had ran the strategy on every equity I had data for over a recent year's span last night, so now I have to go through and see if it was really true that this strategy does work ok generally for individual stocks, but not together.

Interesting thread...

But I don't understand what you are saying in the two above quotes. You can get it to work on individual stocks but when you trade too many individual stocks in a portfolio it doesn't work? I don't understand. If it works on individual stocks, shouldn't it work when you add up the returns of a group of stocks in a portfolio?

Or maybe you are saying that you have found some stocks that it backtests well for, but there are more stocks that it doesn't backtest well, with the net sum that trading a large portfolio it doesn't do well?

Or possibly you are saying that when you attempt ranking and trade a portfolio that is a subset (the highest ranked stocks) then the strategy doesn't do well. So if you had the capital to trade all stocks that qualify it would work, but attempting to cherry pick stocks to trade doesn't work?

 

[Rocko Bonaparte]

Quote from PowerST:

Interesting thread...

But I don't understand what you are saying in the two above quotes. You can get it to work on individual stocks but when you trade too many individual stocks in a portfolio it doesn't work?

I think that all turned out to be moot, since some initial work at verifying my claims seemed to contradict me. That is, I'm winning just as much as I'm losing, and it just happened that the handful of stocks I used as a reference when I originally posted were all good ones. But I'll clarify anyways:

On an individual stock, each simulation day looks something like this:

1. If I already own the stock, sell. Doesn't matter what the indicator says.
2. Run the indicator. If it's spitting out something above 0, then I buy. So basically I use the indicators as a buy trigger.

When I'm scanning:
1. Empty out my entire portfolio of whatever I have.
2. Run the indicator on all stocks on which I have enough data to use.
3. Rank the scores from highest to lowest.
4. Pick the highest scores to balance a portfolio.

So in the second set, I'm throwing multiple stocks in to the mix, so I can construct a portfolio from it. That does open a can of worms; maybe for the sake of this post I'll shut that off. But if you were curious, with $100,000 in simulation money, it'll try to balance it across 5 stocks. This was chosen arbitrarily; I wanted some kind of basic diversification when I was testing this out but wasn't trying to get fancy yet.