I have never seen or touched random, even MAESTRO has trouble defining it. On the other hand nonlinear is everywhere, and this spontaneous sync he speaks about (like the metronomes video, etc) is merely weak coupling amongst nonlinear systems governed by very specific and established nonlinear equations. True random would not couple, if true random existed in the physical world. IMHO. The "noise" would be similar though.
Do you see patterns in Random Walks?
- Thread starter atlTrader666
- Start date
I will take a stab at what Maestro is proposing (although I may be 100% off the mark).
It is perfectly consistent with my earlier notion of flocking behavior and complex systems moving to order during vulnerable events.
Imagine that you represent a time series as a sequence of binary (or ternary) events (b,b,s,b,b,s,s,h,b,b,s,b,b,s,s,b). Each time slice might represent the buy,sell,hold behavior of an individual investor. Now imagine that there are several such investors operating independently, neither sees the exact information as the other. Let's call the investors agents, and label them a-z (for 26 agents in the example). Then we can further imagine 26 agents operating in parallel, and all sequences appearing uncorrelated or random relative to each other. We can look at properties of the independent agents from a statistical perspective. We might take a measure of average frequency of each signal and generate a distribution of such averages (call it fdist). Because they are all mostly uncorrelated, we might have a fairly stable average with a wide dispersion of sample measurements about that average.
Now, it's possible that even though each agent can not see the other agents behavior, they might see just enough influential behavior from common sources to begin to make similar decisions and thus cause the original fdist dispersion to tighten up a bit, as they begin to synchronize. The original mean fdist energy would become larger relative to the freq/phase noise or dispersion and begin to dominate as they start to sync up (not certain whether it would oscillate itself or remain stable, Maestro's comments suggest the former). An omniscient observer might say the signals are spontaneously phase locking (since there is no fixed reference to lock to). One way to measure this property would be to monitor properties like rolling allan variance, and watch for it to tighten up as a harbinger to an avalanche.
Now there are other ways to look for the same properties (such as sequential contagion spanning trees, complexity sand piles, etc). But I think they are all looking for the same phenomena, which is synchronization (in signal parlance) or increasing temporal correlation being evidence of order self-organizing, and thus possible precursors to large scale fat tails on the horizon (sort of like unusual event of many simultaneous smaller earthquakes leading to bigger ones).
Could be way off, but I added it in the hopes that if it is what you are getting at, that maybe it gives you more food for thought on that path.
It is perfectly consistent with my earlier notion of flocking behavior and complex systems moving to order during vulnerable events.
Imagine that you represent a time series as a sequence of binary (or ternary) events (b,b,s,b,b,s,s,h,b,b,s,b,b,s,s,b). Each time slice might represent the buy,sell,hold behavior of an individual investor. Now imagine that there are several such investors operating independently, neither sees the exact information as the other. Let's call the investors agents, and label them a-z (for 26 agents in the example). Then we can further imagine 26 agents operating in parallel, and all sequences appearing uncorrelated or random relative to each other. We can look at properties of the independent agents from a statistical perspective. We might take a measure of average frequency of each signal and generate a distribution of such averages (call it fdist). Because they are all mostly uncorrelated, we might have a fairly stable average with a wide dispersion of sample measurements about that average.
Now, it's possible that even though each agent can not see the other agents behavior, they might see just enough influential behavior from common sources to begin to make similar decisions and thus cause the original fdist dispersion to tighten up a bit, as they begin to synchronize. The original mean fdist energy would become larger relative to the freq/phase noise or dispersion and begin to dominate as they start to sync up (not certain whether it would oscillate itself or remain stable, Maestro's comments suggest the former). An omniscient observer might say the signals are spontaneously phase locking (since there is no fixed reference to lock to). One way to measure this property would be to monitor properties like rolling allan variance, and watch for it to tighten up as a harbinger to an avalanche.
Now there are other ways to look for the same properties (such as sequential contagion spanning trees, complexity sand piles, etc). But I think they are all looking for the same phenomena, which is synchronization (in signal parlance) or increasing temporal correlation being evidence of order self-organizing, and thus possible precursors to large scale fat tails on the horizon (sort of like unusual event of many simultaneous smaller earthquakes leading to bigger ones).
Could be way off, but I added it in the hopes that if it is what you are getting at, that maybe it gives you more food for thought on that path.
An unbelievably accurate interpretation and an incredibly clear path of thought! Are you sure you did not brows through my notebooks?
Thank you for this concise and insightful summary of the approach. I will try to develop this concept further here. Obviously, I am under certain constrains and must avoid sensitive areas that are proprietary to our company. However, I think there is enough meat to discuss at the general (conceptual) level without revealing the actual implementation (which is by far is not trivial).A very solid and accurate interpretation as well! I am now relieved; at least some of the participants of this discussion are getting the right idea of the direction that I am trying to present (the proposed point of view). I just want to state very clearly here I DO NOT INSIST THAT THIS IS THE ONLY RIGHT APPROACH. I do not mean to present it as a Holy Grail or even as an epiphany. I just want to discuss a brand new concept that I am currently developing and to present a few results that came out of it already (they are quite good if I may point it out). The markets are like chess; there are unlimited number of possible winning strategies and it is incredibly difficult to state which one is better or worse. This is what gives me hope and enthusiasm to continue. I do not object or stand against any other fruitful approach that ET members might have developed or learned. I am not even suggesting that this approach has advantages compared to any other winning strategy. I am just simply fascinated by it and I am trying to share this fascination with some of you. In return I hope to hear some criticism, suggestions or fruitful insights that might benefit us all.
Cheers,
MAESTRO
I have a favor to ask. Could you please try to understand that I do not deny existence of stable patterns in the markets at all! In contrary, my whole research is dedicated to finding those patterns! All I am presenting here is my method of identifying them. The random actions of multiple participants very often create very stable and sustainable patterns that are very exploitable! This is the whole point! All I am trying to present here is a consistent approach on how to establish, explain and anticipate those patterns. Some of the participants of this discussion keep on thinking that RANDOM means no patterns. As the matter of fact, pure randomness has the most stable pattern of all - its distribution. Please try to understand that PATTERNS DO EXIST, however the reason why they are there is what I am trying to research. I hope we could move from this point a bit further
I've worked at two national labs and have worked with part time Ph.D. students at both. At my grad school there were no part-timers. But to say part-timer PhDs don't exist at all is wrong. In fact both labs had programs with direct ties to academic institutions to support it's employees/students.
Antisyzygy is right.
Antisyzygy is right.
Argument #4
Letâs construct a simple one dimensional discrete random walk using the following rules. We would flip an unbiased coin and every time it lands on âheadsâ we would draw a step (size 1) up and every time it lands on âtailsâ we would draw a step down. It is well known that on average this random walk would deviate from itâs initial starting point by sqrt ( 2 * N / pi) where N is the number of flips (mean deviation of the walk). This walk will have standard deviation equal to sqrt (N).
The chart below is a weekly chart for SPX. The lines above and below present one standard and one mean deviation of 100 point steps (steps are plotted on the chart in green). It is self evident that this particular chart is extremely close to a pure random walk outcome. It could be shown that if we used not a constant step but a percentage based step the similarity would be even closer. Also, if we take under consideration the reduction of the US dollar buying power over this period of time we would have an exact match between SPX weekly chart and pure random walk. These observations were confirmed by us without a hint of a doubt on hundreds different securities and on thousands different time frames.
Letâs construct a simple one dimensional discrete random walk using the following rules. We would flip an unbiased coin and every time it lands on âheadsâ we would draw a step (size 1) up and every time it lands on âtailsâ we would draw a step down. It is well known that on average this random walk would deviate from itâs initial starting point by sqrt ( 2 * N / pi) where N is the number of flips (mean deviation of the walk). This walk will have standard deviation equal to sqrt (N).
The chart below is a weekly chart for SPX. The lines above and below present one standard and one mean deviation of 100 point steps (steps are plotted on the chart in green). It is self evident that this particular chart is extremely close to a pure random walk outcome. It could be shown that if we used not a constant step but a percentage based step the similarity would be even closer. Also, if we take under consideration the reduction of the US dollar buying power over this period of time we would have an exact match between SPX weekly chart and pure random walk. These observations were confirmed by us without a hint of a doubt on hundreds different securities and on thousands different time frames.
Hi Maestro:
I noticed a key component missing from your analysis - you didn't discuss the temporal correlation of the residuals of each sample. A random variable with no correlation from sample to sample would be impossible to predict motion - regardless of the type of distribution of the data.
The residuals are the delta change from sample-to-sample in the chart. If the residuals are temporally decorrelated (white noise process), then you have a truly random walk. This can easily be seen by either looking at the autocorrelation function (as described by The1 earlier) or alternatively you can look at the covariance matrix of the time samples.
This is the fundamental reason why stocks are often modeled with autoregressive integrated moving average (ARIMA) functions rather than random walks in time-series analysis:
http://en.wikipedia.org/wiki/Autoregressive_integrated_moving_average
ARIMA functions and chart movements ARE temporally correlated, unlike random walks, which is why they can be exploited, unlike a random walk. The important point is that the autocorrelation shows you randomness, not the distribution.
Hope this helps!
-ax
I noticed a key component missing from your analysis - you didn't discuss the temporal correlation of the residuals of each sample. A random variable with no correlation from sample to sample would be impossible to predict motion - regardless of the type of distribution of the data.
The residuals are the delta change from sample-to-sample in the chart. If the residuals are temporally decorrelated (white noise process), then you have a truly random walk. This can easily be seen by either looking at the autocorrelation function (as described by The1 earlier) or alternatively you can look at the covariance matrix of the time samples.
This is the fundamental reason why stocks are often modeled with autoregressive integrated moving average (ARIMA) functions rather than random walks in time-series analysis:
http://en.wikipedia.org/wiki/Autoregressive_integrated_moving_average
ARIMA functions and chart movements ARE temporally correlated, unlike random walks, which is why they can be exploited, unlike a random walk. The important point is that the autocorrelation shows you randomness, not the distribution.
Hope this helps!
-ax
Thank you, this I can understand, so for an alternative, I'll look into coupling in nonlinear systems. Brings to mind an article in Science Review (I think that's the name) about the Eureqa software app. (It's an awesome app that takes data and tries to fit equations to it, it's a lot of fun.) Using it, they were able to find an equation to describe the motion of 2 hinged pendulums swinging together, previously believed to be unpredictable and no equation could describe it. Perhaps that's similar to 2 nonlinear systems, but not sure about "coupling" like the metronomes, I think the 2 pendulums were independent or did not affect each other.