Can linear regression analysis really predict the future?
- Thread starter tradrejoe
- Start date
Disagree. If that would be true we could never drive cars or fly airplanes. Our eyes do regression analysis (not necessarily linear, of course) all the time. When we follow the perceived middle of the lane while in traffic we use other carsâ positions and their relative movements to assess the center of gravity of their combined motion and adjust our position accordingly. Our ability to PREDICT is what makes us survive! Natural evolution is all about the survival of the best predictor or anticipator. Adaptation to the environment could not be achieved without the development of the prediction and anticipation reflexes that we all possess. .
Flocking Behavior
Flock of birds, school of fish, herd of cows and stock market traders surprisingly exhibit the same type of behavior. This type of behavior called âFlockingâ could be simplistically described by just a few rules that each member of a flock must obey. Each entity of a flock must:
⢠Move in a generally random pattern
⢠Move in a general direction of a flock
⢠Keep relatively constant distance to their immediate neighbors
⢠Follow the center of gravity of the flock
The last rule in the above list was recognized just recently, but it plays the most crucial role in understanding of the flock behavior. Following the center of gravity of the flock is the essential survivor skill that enables each member of a flock to minimize the risk of being spotted as a âblack sheepâ thus efficiently blending in and reducing the chances of being attacked by a predator.
Think of driving to work in the morning. Hundreds of other cars on a highway are trying to maintain similar speed and the distance between each other even if that speed is above the allowed limit. Moving with the center of gravity of a block of cars that are spread over multiple lanes reduces the chance that your car will be stopped by a highway ranger. Moving in such a pack also reduces the chance of an accident and allows us to share a highway more efficiently. We also fairly accurately can position our car in the middle of the lane and quit efficiently maintain the distance between the cars that surround us. This ability is very natural to many animals and has been genetically transferred from generation to generation ensuring our very survival.
Knowing the location of the âcenter of gravityâ is essential for all the members of a flock as well as for the predator who is trying to attack the flock and catch one of its members. If the flock is rapidly moving it is virtually impossible to trace individual motions of its members, however, it is much easier to follow the flockâs center of gravity. More so, an ability to accurately anticipate where this center of gravity might be in the future allows most of the species plan their actions accordingly and implement them more efficiently. Many experienced market traders, for example, have developed their unique intuitive methods of anticipation what other traders might do in certain conditions and where majority of other traders might place their bets. This enables experienced traders to make their decisions more accurately thus creating a positive expectation for their trading activities. But what makes the anticipation of the center of gravity more reliable than anticipation of individual flock member movement? The answer is -INERTIA. By definition, Inertia is the resistance of an object to a change in its state of motion. This classical understanding of inertia that we all learned in school apparently could be applied not to just material objects but to the collective behavior of multiple beings. The only difference is that in describing behavioral inertia of a flock the âmassâ could be substituted by a number of members of that flock. In other words, the more members a flock have the more difficult it is to change the position of its center of gravity. This very phenomenon allows our eyes to follow a small cloud of mosquitoes without registering individual positions of each mosquito in this cloud. This handy ability could be very useful in determining where the center of gravity is in the stock market. But what would allow us to âseeâ it?
Flock of birds, school of fish, herd of cows and stock market traders surprisingly exhibit the same type of behavior. This type of behavior called âFlockingâ could be simplistically described by just a few rules that each member of a flock must obey. Each entity of a flock must:
⢠Move in a generally random pattern
⢠Move in a general direction of a flock
⢠Keep relatively constant distance to their immediate neighbors
⢠Follow the center of gravity of the flock
The last rule in the above list was recognized just recently, but it plays the most crucial role in understanding of the flock behavior. Following the center of gravity of the flock is the essential survivor skill that enables each member of a flock to minimize the risk of being spotted as a âblack sheepâ thus efficiently blending in and reducing the chances of being attacked by a predator.
Think of driving to work in the morning. Hundreds of other cars on a highway are trying to maintain similar speed and the distance between each other even if that speed is above the allowed limit. Moving with the center of gravity of a block of cars that are spread over multiple lanes reduces the chance that your car will be stopped by a highway ranger. Moving in such a pack also reduces the chance of an accident and allows us to share a highway more efficiently. We also fairly accurately can position our car in the middle of the lane and quit efficiently maintain the distance between the cars that surround us. This ability is very natural to many animals and has been genetically transferred from generation to generation ensuring our very survival.
Knowing the location of the âcenter of gravityâ is essential for all the members of a flock as well as for the predator who is trying to attack the flock and catch one of its members. If the flock is rapidly moving it is virtually impossible to trace individual motions of its members, however, it is much easier to follow the flockâs center of gravity. More so, an ability to accurately anticipate where this center of gravity might be in the future allows most of the species plan their actions accordingly and implement them more efficiently. Many experienced market traders, for example, have developed their unique intuitive methods of anticipation what other traders might do in certain conditions and where majority of other traders might place their bets. This enables experienced traders to make their decisions more accurately thus creating a positive expectation for their trading activities. But what makes the anticipation of the center of gravity more reliable than anticipation of individual flock member movement? The answer is -INERTIA. By definition, Inertia is the resistance of an object to a change in its state of motion. This classical understanding of inertia that we all learned in school apparently could be applied not to just material objects but to the collective behavior of multiple beings. The only difference is that in describing behavioral inertia of a flock the âmassâ could be substituted by a number of members of that flock. In other words, the more members a flock have the more difficult it is to change the position of its center of gravity. This very phenomenon allows our eyes to follow a small cloud of mosquitoes without registering individual positions of each mosquito in this cloud. This handy ability could be very useful in determining where the center of gravity is in the stock market. But what would allow us to âseeâ it?
Spontaneous Synchronization
Time plays a key role for all living beings. Their activity is governed by cycles of different duration which determine their individual and social behavior. Some of these cycles are crucial for their survival. There are biological processes and specific actions which require a precise timing. Some of these actions demand a level of expertise that only can be acquired after a long period of training but others take place spontaneously. How do these actions occur? Could that possibly be through synchronization of individual actions in a population? Here is an example. Suppose we attend a concert. Each member of the orchestra plays a sequence of notes that, properly combined according to a musical composition, elicit a deep feeling in our senses. The effect can be astonishing or a fiasco (apart from other technical details) simply depending on the exact moment when the sound was emitted. In the meantime, our heart is beating rhythmically because thousands of cells synchronize their activity. The emotional character of the music can accelerate or decelerate our heartbeat. We are not aware of the process, but the cells themselves manage to change coherently, almost in unison. How? We see the conductor moving harmoniously his arms. Musicians know perfectly how to interpret these movements and respond with the appropriate action. Thousands of neurons in the visual cortex, sensitive to specific space orientations, synchronize their activity almost immediately when the baton describes a trajectory in space. This information is transmitted and processed through some outstandingly fast mechanisms.
Just a few seconds after the last bar, the crowds filling completely the auditorium start to applaud. At the beginning the rhythm may be incoherent, but the wish to get an encore can transform incoherent applause in a perfectly synchronized one, despite the different strength in beating or the location of individuals inside the concert hall. This example illustrates Spontaneous Synchronization, one of the most captivating cooperative phenomena in nature. Spontaneous Synchronization is observed in biological, chemical, physical, and social systems such as a stock market and it has attracted the interest of scientists for centuries.
Another paradigmatic example of Spontaneous Synchronization could be found in some South Asia forests. At night, a myriad of fireflies lay over the trees. Suddenly, several fireflies start emitting flashes of light. Initially they flash incoherently, but after a short period of time the whole swarm is flashing in unison creating one of the most striking visual effects ever seen.
The relevance of synchronization in the Stock Market has been stressed frequently although so far it has not been fully understood. In the case of the fireflies, synchronous flashing may facilitate the courtship between males and females where in the Stock Market Spontaneous Synchronization is responsible for the dramatic price fall or rise that no rational models could possibly explain.
Spontaneous Synchronization observed in complex systems can suddenly change the systemâs behavior from a disordered state to an ordered one. These sudden changes are known as phase transitions and occur in a whole range of systems â think, for example, of a group of chaotically moving birds suddenly coming together to form a "V" shape, or locusts simultaneously alighting on a field of valuable crops. Fish spontaneously assemble large schools and small birds form swarms to protect themselves from predators. The behavior of these kinds of systems is remarkably similar to the behavior of the stock market participants.
Spontaneous synchronization is a special case of complicated dynamic phenomena. Understanding the mathematics of how, and under what circumstances, entities can come into synchronization with one another provides a starting point for exploring the vast world of nonlinear dynamics in the Stock Market.
It could be assumed that Spontaneous Synchronization only occurs in the living organisms or their collectives. Could the same phenomenon exist in pure mechanical world? The most successful attempt to answer this question was first proposed by Yoshiki Kuramoto, who analyzed a model of phase oscillators running at arbitrary intrinsic frequencies, and coupled through the sine of their phase differences. The Kuramoto model is simple enough to be mathematically tractable, yet sufficiently complex to be non-trivial. The model is rich enough to display a large variety of synchronization patterns and sufficiently flexible to be adapted to many different contexts including the Stock Market prices behavior.
To demonstrate this phenomenon a primitive set up of five metronomes on a strip of balsa wood that was positioned on top of two aluminum cans on their sides was constructed and tested. This setup allows to metronomes to "feel" each other through motions of the board. The obvious Synchronization of otherwise non-synchronized metronomes in this demonstration leaves no doubt of the Spontaneous Synchronization persistence. It converts the individual behavior of mechanisms into their collective behavior thus creating a new unique system where each of its individual components plays an important role in maintaining the stability of this new system. Interesting enough that new system that is based on the synchronized behavior of its individual components exhibits by far grater stability to the changes in the environment in which those individual components exist. It makes the system much âstrongerâ and a lot more stable.
What makes five individual metronomes more stable as a group is the introduction of a common feedback (a strip of balsa wood) that provides an extra complexity to the group. This phenomenon of enhancing the systemâs stability by increasing its complexity was first introduced by an English psychiatrist and a pioneer in cybernetics William Ross Ashby in his groundbreaking book âIntroduction to Cyberneticsâ. Ashbyâs Law of Requisite Variety is the most influential concept in modern Cybernetics and Artificial Intelligence. It probably could use better name as it refers to the level of minimal or âNecessary Complexityâ for a system to withstand fluctuations of its environment, in other words, to be stable. The stock market is a wonderful example of a stable, self-synchronized system with necessary complexity introduced via its multiple information feedback loops that exist between its participants.
Time plays a key role for all living beings. Their activity is governed by cycles of different duration which determine their individual and social behavior. Some of these cycles are crucial for their survival. There are biological processes and specific actions which require a precise timing. Some of these actions demand a level of expertise that only can be acquired after a long period of training but others take place spontaneously. How do these actions occur? Could that possibly be through synchronization of individual actions in a population? Here is an example. Suppose we attend a concert. Each member of the orchestra plays a sequence of notes that, properly combined according to a musical composition, elicit a deep feeling in our senses. The effect can be astonishing or a fiasco (apart from other technical details) simply depending on the exact moment when the sound was emitted. In the meantime, our heart is beating rhythmically because thousands of cells synchronize their activity. The emotional character of the music can accelerate or decelerate our heartbeat. We are not aware of the process, but the cells themselves manage to change coherently, almost in unison. How? We see the conductor moving harmoniously his arms. Musicians know perfectly how to interpret these movements and respond with the appropriate action. Thousands of neurons in the visual cortex, sensitive to specific space orientations, synchronize their activity almost immediately when the baton describes a trajectory in space. This information is transmitted and processed through some outstandingly fast mechanisms.
Just a few seconds after the last bar, the crowds filling completely the auditorium start to applaud. At the beginning the rhythm may be incoherent, but the wish to get an encore can transform incoherent applause in a perfectly synchronized one, despite the different strength in beating or the location of individuals inside the concert hall. This example illustrates Spontaneous Synchronization, one of the most captivating cooperative phenomena in nature. Spontaneous Synchronization is observed in biological, chemical, physical, and social systems such as a stock market and it has attracted the interest of scientists for centuries.
Another paradigmatic example of Spontaneous Synchronization could be found in some South Asia forests. At night, a myriad of fireflies lay over the trees. Suddenly, several fireflies start emitting flashes of light. Initially they flash incoherently, but after a short period of time the whole swarm is flashing in unison creating one of the most striking visual effects ever seen.
The relevance of synchronization in the Stock Market has been stressed frequently although so far it has not been fully understood. In the case of the fireflies, synchronous flashing may facilitate the courtship between males and females where in the Stock Market Spontaneous Synchronization is responsible for the dramatic price fall or rise that no rational models could possibly explain.
Spontaneous Synchronization observed in complex systems can suddenly change the systemâs behavior from a disordered state to an ordered one. These sudden changes are known as phase transitions and occur in a whole range of systems â think, for example, of a group of chaotically moving birds suddenly coming together to form a "V" shape, or locusts simultaneously alighting on a field of valuable crops. Fish spontaneously assemble large schools and small birds form swarms to protect themselves from predators. The behavior of these kinds of systems is remarkably similar to the behavior of the stock market participants.
Spontaneous synchronization is a special case of complicated dynamic phenomena. Understanding the mathematics of how, and under what circumstances, entities can come into synchronization with one another provides a starting point for exploring the vast world of nonlinear dynamics in the Stock Market.
It could be assumed that Spontaneous Synchronization only occurs in the living organisms or their collectives. Could the same phenomenon exist in pure mechanical world? The most successful attempt to answer this question was first proposed by Yoshiki Kuramoto, who analyzed a model of phase oscillators running at arbitrary intrinsic frequencies, and coupled through the sine of their phase differences. The Kuramoto model is simple enough to be mathematically tractable, yet sufficiently complex to be non-trivial. The model is rich enough to display a large variety of synchronization patterns and sufficiently flexible to be adapted to many different contexts including the Stock Market prices behavior.
To demonstrate this phenomenon a primitive set up of five metronomes on a strip of balsa wood that was positioned on top of two aluminum cans on their sides was constructed and tested. This setup allows to metronomes to "feel" each other through motions of the board. The obvious Synchronization of otherwise non-synchronized metronomes in this demonstration leaves no doubt of the Spontaneous Synchronization persistence. It converts the individual behavior of mechanisms into their collective behavior thus creating a new unique system where each of its individual components plays an important role in maintaining the stability of this new system. Interesting enough that new system that is based on the synchronized behavior of its individual components exhibits by far grater stability to the changes in the environment in which those individual components exist. It makes the system much âstrongerâ and a lot more stable.
What makes five individual metronomes more stable as a group is the introduction of a common feedback (a strip of balsa wood) that provides an extra complexity to the group. This phenomenon of enhancing the systemâs stability by increasing its complexity was first introduced by an English psychiatrist and a pioneer in cybernetics William Ross Ashby in his groundbreaking book âIntroduction to Cyberneticsâ. Ashbyâs Law of Requisite Variety is the most influential concept in modern Cybernetics and Artificial Intelligence. It probably could use better name as it refers to the level of minimal or âNecessary Complexityâ for a system to withstand fluctuations of its environment, in other words, to be stable. The stock market is a wonderful example of a stable, self-synchronized system with necessary complexity introduced via its multiple information feedback loops that exist between its participants.
although winning % is not that important
The question is, does that program yield directional signals, such that with good money management, one can get a reliable profit factor significantly above 1.0??
For example, if it is 70% acurate, by telling you on small moves or once a large move is already partly underway, but one cannot lock in enough profit and prevent enough loss, then it is still a random process, from a trader's POV.
Your work is in line with much published work. But as TZ said, you need to be more concerned with net return than hit rate (most, not all, academic papers miss this).
The reason you see this can be better understood if you are familiar with PCA theory. By including only correlated (and co-linear) components, you are unnecessarily focusing on redundancy, while filtering important information.
You actually want to include dispersion information in the orthogonal components (non-correlated: which is what I think you are observing, inadvertently), to capture the overall variation in the index.
There are feature reduction techniques you can play with (such as PCA), to minimize the variables you are using as regressors.
------------------------------------------------
Maestro is giving away some big keys (although in a metaphorical sense); thankfully, not literal.
Based on the feedback so far, I don't think anyone truly understands how to utilize some of the information embedded in his observations.In the time domain people can look at:
- regressions of all sorts
- ARMA
- GARCH
- PCA/ICA
- cointegration
Otherwise, frequency domain and all the entertaining Fourier stuff.
- regressions of all sorts
- ARMA
- GARCH
- PCA/ICA
- cointegration
Otherwise, frequency domain and all the entertaining Fourier stuff.