Can linear regression analysis really predict the future?
- Thread starter tradrejoe
- Start date
Yep, they are percentages, sorry, I should have mentioned this. Currently thinking...
Edit: Feeling kind of dumb as it's late in the day here, but I guess the thing which leaps to mind is that the percentages are consistently below 50%, your rule as I interpret it is 'will there be a reversion to the median in the next step', so the only thing I can conclude is that a lack of reversion is much more likely than a reversion.
Quote from MAESTRO:
Absolutely! Our discussion here is not about "HOW" we spot the patterns but rather "WHY" those patterns exist. The underlying premise of my approach is probabilities, randomness, stochastic, clustering of events and spontaneous synchronization. The arguments from others here are: the markets are deterministic because all the participants behave rationally and act to their advantage. The main disagreement here is that I believe that no matter what actions market participants are taking they all sum up in a very much random process in the price/time space. Following this train of thought I also argue that the traditional TA (head and shoulders, moving averages RSI etc.) do not have any reason to work as they presume that people âMUSTâ act certain way when they see those patterns. My approach is that nothing can work on a single run. The only way to exploit the markets is to implement law of Large Numbers where stable distribution patterns could be played out to our advantage. The arguments from some people here are also based on their personal âsuccessâ story, however, I believe that any personal trading history is a type of a random walk that could well drift into positive territory and prompt those who happened to be âLucky Random Walkersâ to believe that their success is somehow a result of their wisdom, intuition, experience, or proper implementation of the TA tools. That should put some more âoilâ into the fire of this discussion!
Although I believe technical analysis works for me I agree with this statement. What you are saying is that in a truly random world, one guy gets rich with "rorschach" double top patterns, someone else gets rich cause he bought AOL in the beginning. Another, puts his life savings in the pot 10/18/87 never to be heard of again. A truly random world has room for all of these possibilities. Taleb refers to this in survivorship bias, maybe I am the luckiest trader who ever lived throwing 7's every time I trade for a whole lifetime. My question is what is a good sample, a minute, a day, 150 years.
Here is a link to the largest moves in the dow, interesting that the largest percentage daily move up ever was 15.34% a move from 53.84 to 62.10 dated 3/15/1933. Hardly exciting these days but probably terrifying and life changing back then. It is the wild randomness of the markets (fat tails) that is scary or rewarding depending where you are positioned.
http://en.wikipedia.org/wiki/List_of_largest_daily_changes_in_the_Dow_Jones_Industrial_Average
Not had the time to read the thread, been fu**ing around with s/w, adding some histogram calc and auto charting/image gen capabilities...
Looked into the DJIA last week and created this little animation, shows histograms of all ticks captured during market open hours (cash) on each of Nov 09 thru 13, plus last image is a cumul of all ticks for that week.
Not sure if it's of any help here, if not then just skip it. At least it'll have helped me develop and debug my home brewed toolset here. Pretty cool when you can control exactly what yr s/w is doing vs relying on other's...
What's not so cool is the week-end time spent on this stuff! Oh well...
Maybe a next experiment would be to histogram the behaviour against the mean (as suggested in another post), i.e. how often do the Index tend to RTM vs get away from it...
Looked into the DJIA last week and created this little animation, shows histograms of all ticks captured during market open hours (cash) on each of Nov 09 thru 13, plus last image is a cumul of all ticks for that week.
Not sure if it's of any help here, if not then just skip it. At least it'll have helped me develop and debug my home brewed toolset here. Pretty cool when you can control exactly what yr s/w is doing vs relying on other's...
What's not so cool is the week-end time spent on this stuff! Oh well...
Maybe a next experiment would be to histogram the behaviour against the mean (as suggested in another post), i.e. how often do the Index tend to RTM vs get away from it...
Yes, you are correct, underlying market distribution shape is much closer to cloud silhouettes-- just make sure to call the Nobel prize committee and alert them of Merton, Black, and Scholes' asinine premise.
LOL
---------------------------------------------
you cannot discuss the beauty of music with a deaf person A. B.
unfortunately, since I'm dealing with CFDs for now I don't have volume information (not on an intraday per-tick basis anyway) therefore not in a position to verify the amount of truth in that assertion.
Now if someone here is willing to provide me with that data, say 1mn or 1000 ticks volume would be good enough (I guess) I'll be more than happy to code this is show results here.
On another note, I've spent a few mns comparing histograms posted above with histograms yielded via random walk simulations (my posts on p.12), and all I can say after this brief "visual experiment" is:
-> actual DJIA data vs random walk: histograms look pretty much alike, prolly not a surprise to most here, not that one can expect any different (though I thought maybe I'd find higher peaks around pivot points)
-> it's definitely true that the human mind tries to perceive "patterns" in random waveshapes: tops, bottoms, repeating shapes, but also "cycles", it's funny how we inconsciously try to discern repetitions of equal length when presented a picture of s'thing even purely random. I'm sure dtrader98 and MAESTRO will concur.
I wonder if Maestro could help answer a problem in regards to the centre of gravity or central tendancy of a data set , which one is a better measure or are they all valid ?
There are a few ways to measure the centre of a data set from which the standard deviation can be caluated .
1. Using the basic mean(average) of the data set.
2. A Linear regresion line can be used which is basically a least squared average of the data set.
3. The midpoint of the data set or median or the difference between the high and low for that time of day.
There are others but each are a valid measure of the centre or average of the data set and each will give a different standard deviation calculation.
I used your example(Maestro) from a previous thread where you pointed out about making the time series more gausian or normally distributed by calculating the 30 day average true range of the series and then putting a 30 day long linear regresion line on it and then measure the deviations from it.
I myself have along those lines have tried to normalize and create more normally distributed data by experimenting with intraday data with different time series such as a basket of stocks above and below the current day open and then normalizing the data into a percentage and then looking at the 10 20 and 30 day average ranges of the data set for a particular time of day (like a vwap but withot the volume)
I have created "variable period regresion lines" and "mean averages" and "midpoint of the day" of the timeseries and calculated the deviations from it.
It seems from just an emperical observation (not statistically tested) that the analogy of mass behavior or the flock of birds changing direction and the metronome example where they all start moving in sync with each other happens in the stock indexes intraday such as the dow 30 and the S&p 500 when there is a confluence of the timeseries when price is hitting the standard deviation created by the regresion line as well as the deviation created by the mean and the midpoint when all these different central tendancies hit their deviations at the same time , it seems to create meanful or large reversions , its as if all the algorithms that are used for trading stocks for the buy and sell side part of the institutions all see the same benchmark and act in unison to create a kind of "hearding affect of computer models"
There are a few ways to measure the centre of a data set from which the standard deviation can be caluated .
1. Using the basic mean(average) of the data set.
2. A Linear regresion line can be used which is basically a least squared average of the data set.
3. The midpoint of the data set or median or the difference between the high and low for that time of day.
There are others but each are a valid measure of the centre or average of the data set and each will give a different standard deviation calculation.
I used your example(Maestro) from a previous thread where you pointed out about making the time series more gausian or normally distributed by calculating the 30 day average true range of the series and then putting a 30 day long linear regresion line on it and then measure the deviations from it.
I myself have along those lines have tried to normalize and create more normally distributed data by experimenting with intraday data with different time series such as a basket of stocks above and below the current day open and then normalizing the data into a percentage and then looking at the 10 20 and 30 day average ranges of the data set for a particular time of day (like a vwap but withot the volume)
I have created "variable period regresion lines" and "mean averages" and "midpoint of the day" of the timeseries and calculated the deviations from it.
It seems from just an emperical observation (not statistically tested) that the analogy of mass behavior or the flock of birds changing direction and the metronome example where they all start moving in sync with each other happens in the stock indexes intraday such as the dow 30 and the S&p 500 when there is a confluence of the timeseries when price is hitting the standard deviation created by the regresion line as well as the deviation created by the mean and the midpoint when all these different central tendancies hit their deviations at the same time , it seems to create meanful or large reversions , its as if all the algorithms that are used for trading stocks for the buy and sell side part of the institutions all see the same benchmark and act in unison to create a kind of "hearding affect of computer models"