Predicting randomness

[Perseus]

I think I posted this earlier but a study has been done where the autocorrelation of prices has been calculated for an ungodly amount of data. It shows a short lived correlation of prices in the stock market. Other than that the predictability is 0, thats what an autocorrelation measures. This is not saying that trading is impossible and the autocorrelations are not measured with respect to 'events', which is something I would like to see done.

Volatility on the other hand does seem to have a long term memory.

There is a buttload of stuff out there, but I'll quote from an article here:

http://lagash.dft.unipa.it/pdf/physicaa274.pdf

"This communication briefly discusses some of the stylized "universal" facts that are observed in financial markets and are considered robust by several researchers working
in the field. Starting from these results, one can devise studies trying to enrich and expand this knowledge to provide theoreticians and computer scientists the empirical
facts that need to be explained by their models progressively proposed. The ultimate goal is to contribute to the search for the best model describing a financial market, one of the most intriguing complex systems". "



The short-range memory between returns is directly related to the necessity of absence of continuous arbitrage opportunities in efficient financial markets. In other words, if correlation were present between returns (and then between price changes) this would allow one to devise trading strategies that would provide a net gain continuously and without risk. The continuous search for and the exploitation of arbitrage opportunities from traders focused on this kind of activity drastically diminish the redundancy in the time series of price changes. Another mechanism reducing the redundancy of stock price time series is related to the presence of the so-called "noise traders". With their action, noise traders add into the time series of stock price information, which is unrelated to the economic information decreasing the degree of redundancy of the price changes time series.

It is worth pointing out that not all the economic information present in stock price time series disappears due to these mechanisms. Indeed the redundancy that needs to
be eliminated concerns only price change and not any of its nonlinear functions [20]. The absence of time correlation between returns does not mean that returns are identically distributed over time. In fact, different authors have observed that nonlinear functions of return such as the absolute value or the square are correlated over a time scale much longer than a trading day. Moreover, the functional form of this correlation seems to be power-law up to at least 20 trading days approximately [19,21{26].




As i think I said- not random, but also not predictable and highly nonlinear. But also I maintain they have not done a study about event correlations.

The arbitrage comment is simply one that says if a predictable pattern emerges, then sooner or later big money will come along and most likely destroy it.

Too many people here are discounting the fact that most of the other players are doing exactly as you are, trying to beat the market. The model seems to be 'dumb market, smart me'. Before you get mad I know some of you make good money, so 'me' refers to all traders, ok? Take the EMINI, how many players in the EMINI don't care about buying low and selling high? About 0 I reckon, that means everyone is trying outguess each other and outguess the external events, sort of like a knifefight in a closet. I take my confirmation of this the very high percenatge of traders who don't beat the market and thsi includes a lot of the so-called smart money as well. It's just a damn good thing that things like stocks have an overall rising value in time.

 

[jaronimo]

Quote from cnms2:

I don't advocate giving away indiscriminately, au contraire: you can do a lot of good things with your money, i.e. for your family and friends, but also for some deserving causes like education, health, or whatever else makes you tick. Try for a moment to imagine that you don't have to work for a living anymore. What would you do everyday? How much money would you need to support that indefinitely? The rest, you can put it at good use, either yourself if you're so inclined, or through somebody you know and trust. I don't advocate giving away indiscriminately. Just accumulating more money will not make you happier, and proves at least your lack of imagination. I think.

I think you may be confused. People that make it to the billionaire level, and lots in the millionaire level have never worked a day in thier lives. You usually do not become great at what you do if you do it for money. Most of these people do it because its what they love to do. For many reasons. Most of them would probably say they would have done it for free.

Take what I need and give the rest away, basically thats what you are saying? What if I am thinking, not of myself, but of my family, born and unborn? What if I want to provide for the next 20 generations of my family? Who are you to say thats wrong? What percentage of your income do you consistently give away?

You say you dont advocate giving away indescriminately? Maybe these people give away hundreds of millions a year? Maybe they see giving control of their earned money to someone else as "giving away indescriminately"?

Its their money. They earned it. They can do what they please with it. And yes, even if they made billions, they still earned it.

 

[bdixon619]

Quote from Perseus:

Volatility on the other hand does seem to have a long term memory.

Are you sure you don't mean prices themselves? Volatility has the tendency to reverse itself often. I look forward to reading the article you posted, thanks for it.

Edit: From the paper

(iii) a
model where a geometric difusive behavior is superimposed on Poissonian jumps [31];

This is the market view most like my own. Please see: http://mathworld.wolfram.com/GammaDistribution.html

 

[murdog]

Quote from oddiduro:

This ties in with another post made about large time frames being more predictable that shorter ones. You can see the growth of the overall forest, but that single tree may be a difficulty.

Thoughts?

I don't think I can disagree that large time frames are more predictable. I can say that when I make a trade I'm looking for everything to line up. Here's today's example:

Long time frame (forest)

I've been trying to watch the money rotation in the sectors. With the rising interest rates I've been watching the banking sector for strength or weakness.


Long time frame (forest)

Retail trade seems to be picking up.


Middle time frame (a grove)

Holiday season and thanksgiving week.


Short time frame (tree)

FMOC minutes get released. Sounds like there might be a pause in the rate hikes. Could be good news for the banking sector.


Very short time frame (branch)

Jim Cramer sends out an alert that he likes AMTD


Everything seemed all lined up for a good trade on AMTD. I was in and out twice today on it, and three times yesterday. I might have done just as well riding it out, I haven't done the math yet.

But because of the "branch" I don't like to stick around too long. Cramer can create volume and when the volume dies his stocks usually come back down before figuring out where they really want to go.

My decision to buy was based on trying to predict human behavior given all of the above reasoning... as was my decision to get out. Granted, I did not mention any very, very short term events, but I did try to time my entries and exits based on direction, volume and some intraday charts. And I will admit that I'm more confortable picking a general direction than trying to figure out exactly where a stock will head at certain swing points within the general direction. Which is why I usually bail when I question the immediate direction, and wait until the stock picks a direction off any resistance before getting back in.

Maybe we are closer in thought than we think. Any more comments are appreciated.

murdog

 

[kjkent1]

Quote from nononsense:

kent,

you almost sound like Jack now.

OMG! Pass me the Kool Aid(r)!

 

[murdog]

Quote from Perseus:


both you and nik are making the case that human behavior is predictable, and obviously sometimes it is. The markets though are not a typical arena for human behavior to act itself out in. I can predict that people will shop a lot on friday, but can anybody tell me exactly how that will affect INDU consistently?

You are right, but my answer to that would be that the trading masses don't care about how people shop on Friday, so it's not affecting their trading behavior, so it isn't a useful thing for me to try to figure out.

And don't get me wrong. I'm not saying I'm some great social psychologist. My experiences have taught me to pick out certain things and try to put the puzzle together to form a particular trading decision. I'm quite certain I understand a very, very small scope of what moves a market, and I need to constantly remind myself to think through if my conclusion is a hunch or based on past rational observations. I might be able to look at a particular stock and "read" what it is going to do, but on the flip side I might miss an oppurtunity that's completely obvious to everyone else but me, because I don't have the background to understand the forces behind it.

I take your examples as proof you are a good trader, but I just have to ask then why wall street analysts are so poor at this stuff and then why so many traders lose. Don't doubt you make money but your anecdotes don't constitute a study. In order to really address the question we need a long track record of predictions (not trades).

Absolutely not. I can only venture as to why I've had success. It is certainly not a study. It is no where near infallable. My experience is all I have to go on, and it's certainly limited experience... although based on 30 years of trading. I've only been an active trader through certain stretches of my life. But having had success, it's something that I'll keep playing out until I'm proven wrong. On the same note, because I don't know for sure if I will have continued success is why it is very interesting to hear other people's perceptions.

Why are wall street analysts so poor?! Maybe they do suck... maybe they have other motives and are brilliant in their deception (and sometimes maybe not.) Another factor is as a retail trader with $200,000 to work with I can jump in and out of the market on short term opportunities. I probably would do just as bad predicting longer term trends. There is also the strange reality that things have a great tendancy of proving people wrong. You can find that in everything... sports for example.

I agree with your last paragraph somewhat. I think most all trades are predictions of some sort, but some people like taleb simply want big moves, they don't care in what direction.

That's true... I try to find volitile stocks to work with. But I'd rather enter the game on the winning side to start with, and I think it's possible.

Ok, I have a basic question. If human behavior and its effects on the markets is so transparent, why are market forecasts mostly bad and why are there market forecasts that are directly opposed to each other? It would seem that there should be many more successful people around like murdog and they should all be saying about the same thing.

My success is very relative. Some people might look at my p/l sheet and laugh.

But I think the answer lies partly in that people's goals are very different. As I said before, I think the big players have a much tougher job to do. They have a lot more money they need to find a winner with (not to mention over a longer time frame) and can't take advantage of many of the opportunties a small player like myself can. Their very investment can change the whole dynamic.

And frankly there are a lot of people who are just flat out wrong out there, just like in any other business.

back to weather, the one day forecast is usually very uniform among forecasters and fairly accurate. Why can't I get also then get an accurate one day forecast for MSFT everyday in the paper?

Forecasters are accurate to a certain degree. They've learned to read weather patterns and predict with some degree of certainly what the weather will be like. They are working with a relatively small amount of data.

The market is way more complex. You can't just measure air pressure, the jet stream, humidity, and surrounding high and low pressure areas. There are millions of factors that affect the direction of the market, and they can change in an instant.... 9/11.

No one could ever hope to master the market, but I still do believe that one can watch for enough "signs" to line up to find enough opportunities to make a good living. It is certainly imprecise, but I think with patience one can hit significantly more winners than losers.

When I said fast I am referring to simply my own perception, which of course could be wrong, that whenever a situation arises where we have a predictable response to a given event (Freeze in Iowa in July), then most players seem to act very fast- just my own thought and your experience could be different.

sorry so choppy- i'm tired.

btw, I wonder what would happen if I complained about my own post. [/B]

I'm not positive I understand what you are saying, but if it's that the obvious moves happen too fast then I'd agree with you for the most part. It's still predictable but can't be traded. There are more subtle events and are still predictable.

Good news for ya all... fingers are tired. Sorry for the monster post.

 

[oddiduro]

An interesting article found on this subject...enjoy, and happy Turkey Day.

Falcons 27 Lions 7 Go Dirty Birds!!!


Randomness, Risk, and Financial Markets
Ivars Peterson
Pi, the ratio of a circle's circumference to its diameter, is known as an irrational number because it can't be exactly expressed as a ratio of whole numbers. It would take an infinite number of digits to write it out in full as a decimal or, in binary form, as a string of 1s and 0s. The square root of 2, the square root of 3, and the constant e (the base of the natural logarithms) fall into the same category.
The known digits of these numbers appear patternless. According to one novel method of assessing the randomness of a sequence of numbers, however, the digits of pi turn out to be somewhat more irregular than the digits of the other irrational numbers.
The measure used to determine the irregularity or degree of disorder (entropy) of these sequences is called the approximate entropy. Invented by Steve Pincus of Guilford, Conn., and developed in cooperation with Burton H. Singer of Princeton University, this measure characterizes the randomness of a sequence of numbers.
Suppose the data are expressed as a string of binary digits. The idea is to determine how often each of eight blocks of three consecutive digits—000, 001, 010, 011, 100, 101, 110, and 111—comes up in a given string.
Given the first 280,000 binary digits of pi, the most frequently occurring block is 000, which appears 35,035 times, and the least common block is 111, which appears 34,944 times. The maximum possible irregularity occurs when all eight blocks appear equally often.
For the square root of 3, the block 000 occurs most often (35,374 times) and 010 (34,615) least often. The greater divergence from exactly 35,000 occurrences means that the first 280,000 digits of root 3 are farther from maximum irregularity than the digits of pi.
The formula for approximate entropy developed by Pincus takes such data about a sequence of numbers, whatever its source, and assigns a single number to the sequence. Larger values correspond to greater apparent serial randomness or irregularity, and smaller values correspond to more instances of recognizable features in the data. Overall, approximate entropy grades a continuum that ranges from totally ordered to maximally irregular (or completely random).
Putting the four irrationals in order, starting with the most irregular, gives pi, root 2, e, and root 3. That's a curious, unexpected result. Irrational numbers such as root 2 and root 3 are known as algebraic numbers because they are the solution to a polynomial with a finite number of terms. Others, such as pi and e, are known as nonalgebraic, or transcendental, numbers. Mathematicians had regarded algebraic numbers as, in some sense, simpler than transcendental numbers. But, according to approximate entropy, this distinction doesn't show up in the irregularity of the digits. Whether such quirks in the irregularity of irrationals have any implications for number theory remains an open question for mathematicians.
Because the approximate entropy method does not depend on any assumptions about the process involved in generating a sequence of numbers, it can be applied to biological, medical, or financial data and to physical measurements, such as the number of alpha particles emitted by a radioactive element in specified time intervals, as readily as to the digits of irrational numbers.
For example, Pincus has looked at stock market performance, as measured by Standard and Poor's index of 500 stocks. His calculations show that fluctuations in the index's value are generally quite far from being completely irregular, or random.
One striking exception occurred during the 2-week period immediately preceding the stock market crash of 1987, when the approximate entropy indicated nearly complete irregularity. That change flagged the incipient collapse.
Now, Pincus and Rudolf E. Kalman of the Swiss Federal Institute of Technology in Zurich have applied approximate entropy to the analysis of a wide range of other financial data. They describe their findings in the Sept. 21 Proceedings of the National Academy of Sciences.
Approximate entropy "appears to be a potentially useful marker of system stability, with rapid increases possibly foreshadowing significant changes in a financial variable," Pincus and Kalman contend.
To provide another example of such foreshadowing, Pincus and Kalman examined fluctuations in Hong Kong's Hang Seng index from 1992 to 1998. In this case, the approximate entropy value rose sharply to its highest observed value immediately before this market crashed in November 1997.
Pincus and Kalman also show the usefulness of approximate entropy in characterizing volatility. Volatility is normally understood as the size of asset price fluctuations. A market with large swings in price is generally considered highly volatile and, hence, unpredictable. Pincus and Kalman argue that large fluctuations are not necessarily the same thing as unpredictability.
"The point is that the extent of variation is generally not feared; rather, unpredictability is the concern," Pincus and Kalman say. "Recast, if an investor were assured that future prices would follow a precise sinusoidal pattern, even with large amplitude, this perfectly smooth roller coaster ride would not be frightening."
Standard deviation remains the appropriate tool for characterizing deviations from centrality, the researchers say, and approximate entropy might well be the appropriate tool for grading the extent of irregularity (and unpredictability).
Use of approximate entropy to characterize disorder in financial time series data also suggests that random walks and related models don't generally fit the actual behavior of markets. There's often more order or structure in the data than such models allow.
"Independent of whether one chooses technical analysis, fundamental analysis, or model building, a technology to directly quantify subtle changes in serial structure has considerable real-world utility, allowing an edge to be gained," Pincus and Kalman conclude. "And this applies whether the market is driven by earnings or by perceptions, for both short- and long-term investments."

 

[cnms2]

Thanks oddiduro. Very interesting article.

• Quote from oddiduro:...
  Randomness, Risk, and Financial Markets
  Ivars Peterson
  ...
  "Independent of whether one chooses technical analysis, fundamental analysis, or model building, a technology to directly quantify subtle changes in serial structure has considerable real-world utility, allowing an edge to be gained," Pincus and Kalman conclude. "And this applies whether the market is driven by earnings or by perceptions, for both short- and long-term investments."

 

[smilingsynic]

Markets in general may not be predictable in that X market will be trading at X level within X amount of time.

But does that mean that the behavior of market participants is NEVER predictable?

I am not sure.

 

[profturf]

Might I suggest with all due respect that the work showing that information theory or chaos theory to markets is descriptive and not predictive. and that we tend to place too much weight on it because of the subtle use of the transfer method of propaganda. smart people tend to know these subjects so transfering our allegiances and insecurities to them, we tend to greet them as gurus when it would invite disbelief to show the same reverence to Gann or Livermore or their modern followers without some degree of proof. I have been guiden in this "propaganda " analysis of subjects held dear to the hearts of those who say that technical analysis works by a very insightful post by a Mr. Rod Fitzsimmons Frey on the net. He shows how glittering generalities,appeals to authoritiy, seemingly scientific language, and testimonials lead to our love affair with chaos theory but al that he says is applicable to the use of concepts of entropy and the related epicyclic work of Sornette on similarities to show that there is a greater likehood than usual to predict two major declines such at those in 87 or 97. Not shown of course is what the expectated distribution of prices , including means is for various levels of the retrospectively constructed index. I would appeal again to the good, well meaning people contributing to this discussion to take out their pencil and paper, try to quantify a few things, study such things as runs, serial correlation coefficients, and conditional future distributions of price changes given various paths in the past to improve their ability to speculate rather than basking in the aura of some retired engineer , scientist without mojo in his own field unloading stuff on our own field where it cant be properly vetted et al. I believe you get the point. proturf.