Intuition Amplifiers 2

[Darklingg]

Quote from MAESTRO:

This is an excellent question! Any market analysis system can only process the information that is currently available and rely on what is considered more or less stable patterns established in the past to anticipate what might happen in the future. The IA systems are not different. We process the information extracted from the book and classify it using previously observed behavioral patterns of the traders that we consider to be the most stable patterns one can use for assessing the potential profit opportunities. So, the accuracy of the anticipated price moves depends whether or not the new and significant behavioral pattern has been observed since the time when a trader has already committed to a trade. Obviously, the longer the projected time a trader chooses to use the greater the probability that some new and potentially significant event happens and changes the previously generated outlook. In our experience the accuracy of the IA based systems is the best in the short term. The average trade duration that most of our systems generate is between 5 – 10 minutes. However, we do sometimes get into longer trades if the conditions do not change or if the newly emerged patterns prompting us to continue with the existing positions. So, the answer is: The sweet spot of the IA based systems is the Intraday Trading based on multiple relatively short duration trades.

Your explanation of new behavioral pattern is very interesting. I need to think more about that.

Of course, with increasingly shorter holding times you need increasingly better trades (since spread/commissions are eating a relatively bigger chunk of profit).

Can you estimate what % of your profit are eaten by commissions and spread on average?

 

[MAESTRO]

Quote from Darklingg:

Your explanation of new behavioral pattern is very interesting. I need to think more about that.

Of course, with increasingly shorter holding times you need increasingly better trades (since spread/commissions are eating a relatively bigger chunk of profit).

Can you estimate what % of your profit are eaten by commissions and spread on average?

According to our 5 year statistics trading with IB and using their standard commission rates our average profit/commission ratio is %88.2 for Futures, %73.4 for Stocks, %69.1 for ETFs and %77.3 for FOREX (that is b/a spread costs for pure FOREX or commissions through Ideal Pro)
Overall, we have paid 21% in commissions on our accounts over 5 year period.

 

[Darklingg]

Quote from MAESTRO:

According to our 5 year statistics trading with IB and using their standard commission rates our average profit/commission ratio is %88.2 for Futures, %73.4 for Stocks, %69.1 for ETFs and %77.3 for FOREX (that is b/a spread costs for pure FOREX or commissions through Ideal Pro)
Overall, we have paid 21% in commissions on our accounts over 5 year period.

That is quite impressive. I can't reach those kind of numbers at that timeframe.
I find it very interesting that the intuition-based trading also gives good results in the Forex markets.

I see your IA system as speedometers in cars. With your IA you're giving us a way to assess what's happening in the markets. Similarly speedometers allow to assess and adjust one's driving.
The interesting thing is that while everyone can be taught easily how to drive, it still takes a champion (both in terms of skills and work/dedication) to drive like M.Schumacher with the same tools.
Do you see the same kind of deviation in the results of your traders? Can the average trader be successful with IA or does it take a very skillful one? Can you teach a random person to become a successful trader?

I guess, stated differently, the question would be to what extend can intuition-based systems be automated to remove the human factor? I understand you operate as a grey box, but do you think it could be possible to turn your IA into a blackbox and still retain positive expectancy (obviously less positive than with a human trader )?

 

[fatality666]

I am still trying to connect the dots. No doubt, orderbook activity will occur before price change, but there is a lag in detecting it (ie. latency), especially after any kind of filtering. This is really only useful over very short timeframes and due to filtering one can literally miss the bus or get wasted by it. Why (mentioned many times here) natural cubic splines would make it better, especially as they constantly recalculate values? What is their use in filtering as shown on Maestro’s IA graph?

According to Maestro, splines are important due to their advantages over Fourier transform. But they share same huge disadvantage: filters created using either FFT or cubic spline do repaint. Some people inspired by Ehlers (who as I heard abandoned the idea) try to catch cycles in raw market data which is kind of funny…

So here is my best shot: put multiple splines with different bending (knot spacing) on non-periodical but bounded-scale pre-price representation (ie. bid-ask ratio, put-call ratio) at short TF (or even multiple closely related TFs). I can search for signal-to-noise ratio/minimum RMSE to separate strongest frequencies (but why? – I can’t recognize trading groups nor activity of those groups), OR try to find synchronization by phase and other parameters (hard, due to non-periodic character of signal and trading frequency of different groups of traders), OR try to measure dynamics of the signal (useless, splines repaint like hell, extrapolated or not). I have good FIR filters which provide normal distribution around price and other data without repainting.

I know this is the question of technical matter, but it's related and it concerned me for last 3 years after reading great topic about RTM: the proper use of spline and type of value/parameter one has to calculate with it. The more I analyze those IA graphs, the less I understand what I am really looking at

 

[MAESTRO]

Quote from Darklingg:

That is quite impressive. I can't reach those kind of numbers at that timeframe.
I find it very interesting that the intuition-based trading also gives good results in the Forex markets.

I see your IA system as speedometers in cars. With your IA you're giving us a way to assess what's happening in the markets. Similarly speedometers allow to assess and adjust one's driving.
The interesting thing is that while everyone can be taught easily how to drive, it still takes a champion (both in terms of skills and work/dedication) to drive like M.Schumacher with the same tools.
Do you see the same kind of deviation in the results of your traders? Can the average trader be successful with IA or does it take a very skillful one? Can you teach a random person to become a successful trader?

I guess, stated differently, the question would be to what extend can intuition-based systems be automated to remove the human factor? I understand you operate as a grey box, but do you think it could be possible to turn your IA into a blackbox and still retain positive expectancy (obviously less positive than with a human trader )?

The automation question was hunting me for years. On one hand, there are IA based formal strategies that could be written as logic statements, however, their abilities to adapt are limited to requisite variety (complexity, or sheer number of predicates) of the model that they represent. On other hand, using associative pattern recognition abilities of humans makes the implementation of IA based tools extremely adaptive and very reliable, but it, of course, requires extensive training and natural abilities (such as of Michael Schumacher) to create successful trading track record. After much experimenting and testing I have convinced myself that using automation to reduce the number of mechanical tasks performed by the operator (thus reducing his/her strain) while allowing the operator to make creative decisions using his/her trained visual cortex is the way to go. Obviously, this approach drastically reduces the number of individuals that are suitable for the IA based systems; however, the ones who have the abilities to “drive” this “formula 1” car can outperform traders who use any other method of trading quite dramatically. Unfortunately the number of such “gifted” individuals is quite low. On average, only 10% of all the traders trained in IA based system become “stars” and never look at any other methods of trading ever again. 30% of traders do “OK” and generally use the IA systems as the “second opinion” tool and enhance their average performance as much as by 30%. Unfortunately, as I said earlier, 60% of traders do not have abilities to use those types of tools in their trading at all. I still have not discovered what makes one trader extremely successful with IA based tools and other to suck at it, but for now I have no choice but test as many traders as I can for their abilities to perform with IA. I am not sure where it is going to take me, but as an extremely radical and unusual journey it makes me excited to investigate. At the end, if I end up with a handful of “M. Schumachers” I will be happy.

Cheers,
MAESTRO

 

[MAESTRO]

Quote from fatality666:

I am still trying to connect the dots. No doubt, orderbook activity will occur before price change, but there is a lag in detecting it (ie. latency), especially after any kind of filtering. This is really only useful over very short timeframes and due to filtering one can literally miss the bus or get wasted by it. Why (mentioned many times here) natural cubic splines would make it better, especially as they constantly recalculate values? What is their use in filtering as shown on Maestro’s IA graph?

According to Maestro, splines are important due to their advantages over Fourier transform. But they share same huge disadvantage: filters created using either FFT or cubic spline do repaint. Some people inspired by Ehlers (who as I heard abandoned the idea) try to catch cycles in raw market data which is kind of funny…

So here is my best shot: put multiple splines with different bending (knot spacing) on non-periodical but bounded-scale pre-price representation (ie. bid-ask ratio, put-call ratio) at short TF (or even multiple closely related TFs). I can search for signal-to-noise ratio/minimum RMSE to separate strongest frequencies (but why? – I can’t recognize trading groups nor activity of those groups), OR try to find synchronization by phase and other parameters (hard, due to non-periodic character of signal and trading frequency of different groups of traders), OR try to measure dynamics of the signal (useless, splines repaint like hell, extrapolated or not). I have good FIR filters which provide normal distribution around price and other data without repainting.

I know this is the question of technical matter, but it's related and it concerned me for last 3 years after reading great topic about RTM: the proper use of spline and type of value/parameter one has to calculate with it. The more I analyze those IA graphs, the less I understand what I am really looking at

I think there is a confusion between the spline based price regression analysis tools and the spline based order flow visualization tools used to teach, trigger and develop stable associations towards the behavioral patterns observed in the order book. The former has its benefits compared to FIR filters and other types of interpolation techniques but also has obvious disadvantages. Use of splines to create mean regression techniques could be fruitful, but not revolutionary. However, using splines to produce certain visual associations is essential. There are reasons why movies like Avatar, Shrek etc. are all based on splines. They have a unique ability to create the sense of “fair” movement or sense of “center of gravity” that is crucial for our visual analysis techniques. The use of splines in IA is related to just that. At the same time my previous discussions (a few years back) were related to the MRT tools and that, I suppose, led to some confusion.

 

[fatality666]

Quote from MAESTRO:

I think there is a confusion between the spline based price regression analysis tools and the spline based order flow visualization tools used to teach, trigger and develop stable associations towards the behavioral patterns observed in the order book. The former has its benefits compared to FIR filters and other types of interpolation techniques but also has obvious disadvantages. Use of splines to create mean regression techniques could be fruitful, but not revolutionary. However, using splines to produce certain visual associations is essential. There are reasons why movies like Avatar, Shrek etc. are all based on splines. They have a unique ability to create the sense of “fair” movement or sense of “center of gravity” that is crucial for our visual analysis techniques. The use of splines in IA is related to just that. At the same time my previous discussions (a few years back) were related to the MRT tools and that, I suppose, led to some confusion.

So it appears that I am overthinking the problem. I had some experience with computer animations few years back and now I clearly understand what you mean. Years before motion-capture techniques animations for games and movies (movement of body parts and clothes) were based on hand made frame after frame blocks. So splines help to make smooth and natural transitions between those frames and as you said - it's just it.

Here is an example: http://www.youtube.com/watch?v=qdVG7EFIijI

Many thanks for clearing it up for me, Maestro.

 

[Darklingg]

Here is AAPL today with IA-inspired indicators.

The first one measures order cancel rate on ask vs bid side. The second one measures ask vs bid ratio with a stiff spline applied (it represents the sentiment of higher frequency traders). And the third one is an average of the first two. The third one generates the signals.

Let me know what you think

 

[Darklingg]

Quote from MAESTRO:

The automation question was hunting me for years. On one hand, there are IA based formal strategies that could be written as logic statements, however, their abilities to adapt are limited to requisite variety (complexity, or sheer number of predicates) of the model that they represent. On other hand, using associative pattern recognition abilities of humans makes the implementation of IA based tools extremely adaptive and very reliable, but it, of course, requires extensive training and natural abilities (such as of Michael Schumacher) to create successful trading track record. After much experimenting and testing I have convinced myself that using automation to reduce the number of mechanical tasks performed by the operator (thus reducing his/her strain) while allowing the operator to make creative decisions using his/her trained visual cortex is the way to go. Obviously, this approach drastically reduces the number of individuals that are suitable for the IA based systems; however, the ones who have the abilities to “drive” this “formula 1” car can outperform traders who use any other method of trading quite dramatically. Unfortunately the number of such “gifted” individuals is quite low. On average, only 10% of all the traders trained in IA based system become “stars” and never look at any other methods of trading ever again. 30% of traders do “OK” and generally use the IA systems as the “second opinion” tool and enhance their average performance as much as by 30%. Unfortunately, as I said earlier, 60% of traders do not have abilities to use those types of tools in their trading at all. I still have not discovered what makes one trader extremely successful with IA based tools and other to suck at it, but for now I have no choice but test as many traders as I can for their abilities to perform with IA. I am not sure where it is going to take me, but as an extremely radical and unusual journey it makes me excited to investigate. At the end, if I end up with a handful of “M. Schumachers” I will be happy.

Cheers,
MAESTRO

I think you're on a good way to finding your Schumachers.

I believe there are unexploited edges in your research. I've tried reproducing some of your indicators (in a more simple way, certainly missing many of the subtleties) and they definitely exhibit correlation with future price moves.

Personally I have a lot of troubles trusting a system that won't exhibit significant positive expectancy with a couple of simple automated buy/sell rules. It's my bad habit of trusting data more than traders.

Ps. I wonder if you happen to be in the 10% with "star" potential?

 

[MAESTRO]

Quote from Darklingg:

Here is AAPL today with IA-inspired indicators.

The first one measures order cancel rate on ask vs bid side. The second one measures ask vs bid ratio with a stiff spline applied (it represents the sentiment of higher frequency traders). And the third one is an average of the first two. The third one generates the signals.

Let me know what you think

That is a very good start! IA is a very rewarding approach, take my word for it! The more you use it the more insightful it becomes! You will LOVE IT!