Well you have to ask this to the people who constantly emphasize the "cleansing, filtering, and feature selecting" as being more important than the actual network. And a NN does not "kick in and it does not do its magic". It does exactly what it is prescribed and designed to do.
Until anyone shares any NN type of systems for trading, there's no real basis for debate at all, and is why the signal to noise ratio is so low here.
Do you have any handy references emphasizing data cleaning and preprocessing for NN?
What NN does it replaces hundreds to millions of conditional statements with a grid of activation functions that converge to a single result which depends on having enough distance in statistical terms between groups. There is nothing complicated about it except for the complexity of the design.
Hi,
very interesting discussion.
My understanding from the AlphaGo story is that the beauty of it was precisely that the computer had been fed next to nothing in terms of Go rules or strategies, but instead managed to learn from raw data (from real games as well as simulated games against itself).
There is a consistent literature feeding known predictors (say RSI or MA) into ML algos and getting some results. In many ways you could just say this is merely normalising data / removing noise. But I wonder - is it just a constraint because of lack of data, particularly given changes in market regime? If we had a million years of EURUSD data tick by tick and the market behaviour was stable, couldn't we just feed raw data and get good results?
The algorithm is relatively irrelevant. Much more important are the data with which you feed it. Normally a bunch of indicators or other derivatives of the price curve.
When you selected the features, then you can find the best algorithm for them. All of your list are available in R. You can relatively quickly try them out.
I found deep learning networks the most versatile algo for financial prediction so far. There are several R implementations for deep learning.
ML is naturally a part of TA, when done as analysis of "technical" price-volume data. Any more, and you start to include the fundamental realm of markets. This is simplified understanding and everything can be mixed together also, probably even favourably so. I think the consensus here is reasonable, although people tend to argue about differences in semantics rather than finding common grounds.
Anyhow: I believe everyone agrees that given raw market data input, nobody here expects to plug in a generic ML/datamining algorithm or two, and then wait for a consistent trading plan and execution from the computer. You have to have some basic trading idea, data mine the possibilities (manually or semi-automatic) and then make a first attempt at a prototype. This is true for all technical trading, and you don't even need an ML algo. ML like everything else is just a tool. It's ass-backwards design to include it, if you don't really have a use for it yet. Instead, you include ML in your trading engine when you clearly see it can fit a specific need. Doing it the other way may provide much learning experience, but may waste alot of time towards a working implementation as well.
What ML may provide is more dynamic non-linear analysis and rules based on deeper statistics. In that way, the same algo could provide many different trading setups, mined from the data itself. So you have more automation in the analysis part of your trading engine. Or it could just be used for one specific little need or anything a little more involved than if-then-else.
This may sound great, but everyone who's had anything to do with software development, know from experience how hard it is to make something work in all circumstances. Such a general method as ML is even harder to create, monitor and maintain. Additionally, possibilities for analysis is endless, and ML doesn't really solve that problem of narrowing down the problem/questions in the grand scheme of things.
If the trading engine learns from new data continuously, you'd need to quality check that process continuously as well, as all kinds of artifacts and biases may develop over time. A static ruleset is easier to maintain, but will of course lack the adaptivity you may get from ML.
This is all very general, and may not be applicable to every implementation, but nobody is discussing specific implementations here yet.
There's no reason you can't make money in the markets, with or without ML, other than lack of faith - which is the basis for most successes in life. Ie. There's always up-front cost and risk that needs to be paid. Of course, one must create a trading plan as one has to mitigate being wrong most of the time, which is natural in most trading except scalping. So to anyone planning to "trade with ML", learn trading first, as ML won't make anything "easier" (although it may perhaps simplify the trade engine parts it's replacing!).
It is highly suspicious whether ML can recognize any statistically significant model of the market. So, it seems to be simple minded to apply ML directly to generate buying and selling signals.
R has a much larger user base than Matlab, and therefore also much more libraries for neural nets and other algos. For deep learning, look into DARCH, DEEPNET, and DNN.Can you be more specific? I am trying to find some algos in neural network to adjust my indicators.
Thanks !
As I became enthusiast to neural network quite recently, I am wondering whether R is a ideal software to implement neural net works? As far as I know, R is used by many professional statisticians and could run the Monte Carlo too. If the answer is yes, I don't have to waste my time on other platforms like Matlabs, right?
Absolutely not. You are way off the mark with comparing AlphaGo to trading and completely wrong that you can use raw data. Board games or video game play is an unsupervised machine learning problem, trading is a supervised machine learning problem. A win in a board game is the result of a series of moves, trading profits are the result of average profit factor for every trade. Completely different problems. In addition, historically any potential entry can be tagged as a trade loss or gain, making this a supervised problem. Treating it as unsupervised is simply the wrong approach. In addition, trading has considerable uncertainty, where board games have only one outcome.
A more similar problems are computer vision, eg identifying the object in a picture, or search engines, or natural language processing. The best example is the Viola-Jones algorithm for face recognition used on smart-phones. These algorithims are completely dependent on the "features", or mathematical conversion of the raw data into something the computer can understand and then convert into a probability. The breakthroughs in these fields come from the development of better features, and what algorithm (neural network, support vector machine etc) does not make much difference.
Also, computers can outplay a person in chess, but a two year old can describe images better than a computer. A person could also provide better answers than a search engine. What good answers the search engine does give such as local results based on knowing your location or knowing a number is a fed-ex tracking are a hard wired features. And look at the recent crashes from car autopilots. Similarly, good trading algos can be built but it is by no means as trivial as and requires considerable feature creation.
No, you can't just feed raw price data into a ML algorithm. That is absurd.