Neural Networks Revisited

[userque]

Just to be clear, since your reply to my post referred to a 'mechanical algorithm,' and I'm not sure exactly how you define that. My system is based upon a homemade, custom kNN algorithm. An ensemble of over 32 of them (currently). Their outputs are feed into a homemade simple NN. The NN is trained with the help of an evolutionary algorithm--Excel Solver. My Excel VBA code ties it all together. It's simple. But it's also very complex.

I've had several breakthroughs. I overcame the curse of dimensionality problem--for example. Etc. etc.
https://en.wikipedia.org/wiki/Curse_of_dimensionality

The system has many different aspects to it. It grew over time by me solving one problem at a time. I doubt I could have sat down in the beginning and dreamt up what I ultimately created. In a way, by using logic and common sense; it kind of was telling me how to build it--as I built it.

I can get even greater performance by running the algo on a more powerful computer (by increasing the size of the ensemble). I'm currently running it on a 2014 cheap Best Buy laptop. It takes 2-3 hours to create a forecast. The forecast applies to the following day. It forecasts one day at a time, per day.

I've read papers too. Mainly to see if anyone had discovered what I've discovered about ML. I'll also concluded it is real easy for the results of papers to be 'massaged.' (I recall a recent news story about papers being 'made up.') A few did probably inspire some of the concepts I use. I'm not certain.

My concepts can be used generally, and not just for financial time series.

Imo, it's less about what you're using; and more about how you do it. The first problem to overcome is: how to present the financial time series problem to an algorithm--the inputs.

Before building my own, I tried:
GMDH Shell
Eureqa
Weka
and some others.

I'll eventually get around to coding my system in Python, but if I use any of the ML libraries, it'll be in a very unique way, and not as they are intended to be used. My intent would be to take advantage of Cython eventually...for the speed. More speed: more inputs. More inputs (for my system) the better.

Whenever you get your deep learning system up and running, my challenge will still be on the table.

I am no expert here; just sharing some thoughts/issues that I have come across when I looked into this area; some of what I write here is the advice that I got from some deep learning experts who happen to be my colleagues at the company I work for.

One particular thing that I have noticed when I scratched this area is that open source software such as Theano/keras or tensorflow or torch have good CNN models, but lack/not upto-date in many RNN/RBM models that are currently being churned out at universities. RNN/RBMs have LSTM (Long-Short Term Memory) that support time series models; CNNs don’t for that matter. At one point, my colleagues advised me to simply ignore the open source packages for latest RNNs and start reading papers and use CUDA programming; Nvidia’s latest cuDNN supports RNNs. The other problem with LSTM models is that they are not as parallel as CNNs and so processing not so parallel neural networks can get time consuming. There is so much research going on in deep learning that one has to just keep reading all the papers coming out of the universities full-time (forget trading). Training RNNs is VERY hard; don't underestimate this; most of today's research focus is around ways to simplify training. As you see, once we get into the details, deep learning can get complicated and time consuming; unless you absolutely know what you are after, you can quickly get bogged down in a wild goose chase.

To be fair, there are a number of simple mechanical strategies that can be developed by hand; deep learning is always there for more complicated scenarios. Every approach can be designed by either traditional (mechanical algo) approach or a deep learning... it’s a matter of how complicated the mechanical algo gets for a required accuracy; guess that’s the deciding factor. Deep learning is always superior, but is it worth the effort for a good enough accuracy? At the end of the day, we are after ROI/PNL, so its just a matter of ones perception of where the alpha is given the resources/time/priorities we have at hand. I struggling to stop myself from jumping into this field, I have some low hanging fruits first

 

[userque]

Very nicely done. This gives me further encouragement for an ML project that I have currently going. I know an nth of that which the rest of you all seem to know, which is why I have a team working on mine for me.

Mine has a very specific focus but the results you've posted are also in line with what they're claiming they have produced for me given my desired inputs. I think we can get it better than what they claim it is right now, but time shall tell.

Good luck, and keep up posted!

 

[Simples]

Just to be clear, since your reply to my post referred to a 'mechanical algorithm,' and I'm not sure exactly how you define that. My system is based upon a homemade, custom kNN algorithm. An ensemble of over 32 of them (currently). Their outputs are feed into a homemade simple NN. The NN is trained with the help of an evolutionary algorithm--Excel Solver. My Excel VBA code ties it all together. It's simple. But it's also very complex.

I've had several breakthroughs. I overcame the curse of dimensionality problem--for example. Etc. etc.
https://en.wikipedia.org/wiki/Curse_of_dimensionality

The system has many different aspects to it. It grew over time by me solving one problem at a time. I doubt I could have sat down in the beginning and dreamt up what I ultimately created. In a way, by using logic and common sense; it kind of was telling me how to build it--as I built it.

I can get even greater performance by running the algo on a more powerful computer (by increasing the size of the ensemble). I'm currently running it on a 2014 cheap Best Buy laptop. It takes 2-3 hours to create a forecast. The forecast applies to the following day. It forecasts one day at a time, per day.

I've read papers too. Mainly to see if anyone had discovered what I've discovered about ML. I'll also concluded it is real easy for the results of papers to be 'massaged.' (I recall a recent news story about papers being 'made up.') A few did probably inspire some of the concepts I use. I'm not certain.

My concepts can be used generally, and not just for financial time series.

Imo, it's less about what you're using; and more about how you do it. The first problem to overcome is: how to present the financial time series problem to an algorithm--the inputs.

Before building my own, I tried:
GMDH Shell
Eureqa
Weka
and some others.

I'll eventually get around to coding my system in Python, but if I use any of the ML libraries, it'll be in a very unique way, and not as they are intended to be used. My intent would be to take advantage of Cython eventually...for the speed. More speed: more inputs. More inputs (for my system) the better.

Whenever you get your deep learning system up and running, my challenge will still be on the table.

@userque : Using ML, NN, AI or whatnot is very far from my current investigative approaches. Do you mind some questions about your system, as to get some rudementary understanding?

Regarding practical DIY approaches, this is a very long and solitary road, but often necessary for big/revolutionary advances. Am currently leaning on long-term analysis for stock selection, timing and management of trades on the daily TF. Can identify with the simple/complex dichotomy. Am constantly striving for simplicity, yet, some of the parts used are sometimes rather complex. Often though, I find complexity / non-linearity not warranted, needing simplification.

 

[userque]

@userque : Using ML, NN, AI or whatnot is very far from my current investigative approaches. Do you mind some questions about your system, as to get some rudementary understanding?

Regarding practical DIY approaches, this is a very long and solitary road, but often necessary for big/revolutionary advances. Am currently leaning on long-term analysis for stock selection, timing and management of trades on the daily TF. Can identify the simple/complex dichotomy. Am constantly striving for simplicity, yet, some of the parts used are sometimes rather complex. Often though, the complexity is not warranted and needs to be simplified.

The system grew by the addition, over time, of simple parts. (However, the simple parts are 'connected up' in such a way that makes them--the system--complex and extremely unique; as far as I can tell.)

Whereby the next 'addition' to the whole could have only been known or discovered by a thorough analysis of, and experience with, the system prior to the 'addition.'

I could not have started building the current system from scratch. What I did start building, evolved to what it currently is.

Likewise, electricity wasn't invented with the goal of creating an iPhone. Going back in time and showing a caveman the iPhone may cause the caveman to try to replicate it--but he'll be going about it all wrong. The iPhone is the result of the accumulation of simpler tasks/discoveries/inventions that necessarily (generally) had to be performed in a certain order to achieve the complicated iPhone result.

The simple components of the system also began with simple concepts. Academia, a lot of times, over-complicates things. It produces graduates that over-complicate things. That sends them in the wrong direction from the very start. It takes someone with the ability to ignore the status quo; start from scratch; and conclude that the Earth is round.

 

[Simples]

The system grew by the addition, over time, of simple parts. (However, the simple parts are 'connected up' in such a way that makes them--the system--complex and extremely unique; as far as I can tell.)

Whereby the next 'addition' to the whole could have only been known or discovered by a thorough analysis of, and experience with, the system prior to the 'addition.'

I could not have started building the current system from scratch. What I did start building, evolved to what it currently is.

Likewise, electricity wasn't invented with the goal of creating an iPhone. Going back in time and showing a caveman the iPhone may cause the caveman to try to replicate it--but he'll be going about it all wrong. The iPhone is the result of the accumulation of simpler tasks/discoveries/inventions that necessarily (generally) had to be performed in a certain order to achieve the complicated iPhone result.

The simple components of the system also began with simple concepts. Academia, a lot of times, over-complicates things. It produces graduates that over-complicate things. That sends them in the wrong direction from the very start. It takes someone with the ability to ignore the status quo; start from scratch; and conclude that the Earth is round.

Sounds very much like agile development. However, when implemented in methodologies such as scrum, organizations tend to try to follow scrum to the letter (mandating daily stand-ups and kanban-boards), rather than organizing the people and work to be more practical and smart. Fortunately for us going solo, agile development is not a group effort and thus much simpler, but also laborious and bewildering due to the many possibilities.

This is the same kind of issue with a caveman trying to replicate an iPhone. Since the caveman has no idea of the original requirements, domain knowledge and technologies, not even why or how it's supposed to work and interact, it's a futile attempt to try to clone it. Closest would probably be a stone tablet with some chisel marks in it. Anything deeper than the superficial would be considered "magical". However, people generally are encouraged to adhere to such "magical thinking", instead of creating solutions themselves.

Am wondering about some basic stuff:
1. Is my basic assumption correct in that kNN can be used for classification and boundary-detection, but do not by itself provide much predictive elements for the future? It sounds like some sort of advanced averaging tool.

2. How stable are your predictions? Ie. considering simplest to full versions of your system, are these rapidly converging towards a "better" solution, each adding some "element", or do solutions wary wildly with incorporations of various factors?

3. Have you tried to quantify each factor's level of contribution to success or does that not make any sense when using a type of NN?

4. Is your system based on some fixed ideas about your model(s) for trading, or can the system just "figure it out by itself"?

5. Are you incorporating risk management as part of your ML strategy?

Mostly just curious. Interesting topic!

 

[userque]

Sounds very much like agile development. However, when implemented in methodologies such as scrum, organizations tend to try to follow scrum to the letter (mandating daily stand-ups and kanban-boards), rather than organizing the people and work to be more practical and smart. Fortunately for us going solo, agile development is not a group effort and thus much simpler, but also laborious and bewildering due to the many possibilities.

This is the same kind of issue with a caveman trying to replicate an iPhone. Since the caveman has no idea of the original requirements, domain knowledge and technologies, not even why or how it's supposed to work and interact, it's a futile attempt to try to clone it. Closest would probably be a stone tablet with some chisel marks in it. Anything deeper than the superficial would be considered "magical". However, people generally are encouraged to adhere to such "magical thinking", instead of creating solutions themselves.

I agree.

Am wondering about some basic stuff:
1. Is my basic assumption correct in that kNN can be used for classification and boundary-detection, but do not by itself provide much predictive elements for the future? It sounds like some sort of advanced averaging tool.

The kNN can also be used for prediction by itself. I only added the NN as of May 24.
https://en.wikipedia.org/wiki/K-nearest_neighbors_algorithm

2. How stable are your predictions? Ie. considering simplest to full versions of your system, are these rapidly converging towards a "better" solution, each adding some "element", or do solutions wary wildly with incorporations of various factors?

In the beginning (2014), the system would work for several days; then for several days, the system would seem to be wrong 100% of the time. The ensemble/voting approach greatly improved those results. Then, the NN greatly improved upon those.

3. Have you tried to quantify each factor's level of contribution to success or does that not make any sense when using a type of NN?

No. Never considered doing that. Never imagined I'd have a need for it. My research and experiments were never rigorous to the extent that I could write a scientific paper on my findings. I only researched a 'question' well enough to where I was satisfied with my conclusions.

4. Is your system based on some fixed ideas about your model(s) for trading, or can the system just "figure it out by itself"?

After trying all sorts of NN setups, I rethought the problem and formed a premise about the markets that would explain why NN's generally fail at forecasting them. That lead to the use of the kNN.

I also had to devise a way to define the 'pattern building blocks' that the system looks for. Then, I had to devise a custom search algo to search through, and internally backtest on the fly, the pattern possibilities.

5. Are you incorporating risk management as part of your ML strategy?

Currently no. The system 'rights' itself quickly. However I do have plans to test the use of stop-losses/profit-targets.

I've toyed with algos to determine position sizing early on, but the system starting working well enough to where the best position size was always: 'all in.'

 

[dtrader98]

The system grew by the addition, over time, of simple parts. (However, the simple parts are 'connected up' in such a way that makes them--the system--complex and extremely unique; as far as I can tell.)

Whereby the next 'addition' to the whole could have only been known or discovered by a thorough analysis of, and experience with, the system prior to the 'addition.'

I could not have started building the current system from scratch. What I did start building, evolved to what it currently is.

Likewise, electricity wasn't invented with the goal of creating an iPhone. Going back in time and showing a caveman the iPhone may cause the caveman to try to replicate it--but he'll be going about it all wrong. The iPhone is the result of the accumulation of simpler tasks/discoveries/inventions that necessarily (generally) had to be performed in a certain order to achieve the complicated iPhone result.

The simple components of the system also began with simple concepts. Academia, a lot of times, over-complicates things. It produces graduates that over-complicate things. That sends them in the wrong direction from the very start. It takes someone with the ability to ignore the status quo; start from scratch; and conclude that the Earth is round.

You had a lot of really good observations and commentary. I too found myself on a similar path which required really understanding what ML algorithms are trying to accomplish, how they go about it, and more importantly what are their limitations. One big limitation is that common objective/loss functions are very limited in flexibility, requiring either trying to rework the built in ones, using different architectures entirely (e.g. evolutionary learning), or throwing them out and rebuilding from scratch. Sometimes you can get around this by combining methods as you did with NN and GA/EA. I designed a lot of novel approaches to solving these issues and haven't seen the approaches published anywhere.

That's one reason why I say few Deep Learning ventures will succeed with financial data, as they aren't really addressing the inherent limitations with common ML I/O and framework as applied to markets. People were blown away by DeepMind playing games, but, IMO, video game applications, cats, and text are far more repeatable and consistent than market data. Imagine what could be accomplished, if there wasn't such a (unfortunately necessary) veil of secrecy around development.

 

[userque]

You had a lot of really good observations and commentary. I too found myself on a similar path which required really understanding what ML algorithms are trying to accomplish, how they go about it, and more importantly what are their limitations.

Exactly!

One big limitation is that common objective/loss functions are very limited in flexibility, requiring either trying to rework the built in ones, using different architectures entirely (e.g. evolutionary learning), or throwing them out and rebuilding from scratch.

Exactly! I couldn't believe the lack of flexibility software packages offered. The cost/production functions can be of key importance, depending upon how you frame the 'question.' Sometimes you can split the 'question' into parts, and/or use multiple algos.

Sometimes you can get around this by combining methods as you did with NN and GA/EA. I designed a lot of novel approaches to solving these issues and haven't seen the approaches published anywhere.

Right! No one. No University. No student. Is going to reveal the 'good' stuff, should they happen to discover any, to the public, imo.

That's one reason why I say few Deep Learning ventures will succeed with financial data, as they aren't really addressing the inherent limitations with common ML I/O and framework as applied to markets.

Financial (market) data are simply too chaotic for NN's imo.

People were blown away by DeepMind playing games, but, IMO, video game applications, cats, and text are far more repeatable and consistent than market data. Imagine what could be accomplished, if there wasn't such a (unfortunately necessary) veil of secrecy around development.

Right again, imo. Those challenges aren't chaotic in nature.

Yeah, that would be interesting to see what would happen to the markets if all ML secrets held by any and all parties were suddenly made public and all traders were all using the best possible technologies to trade with. Such a scenario would probably tear a hole in space-time. But would the markets then become more, or less orderly.

 

[Jerry030]

"Yeah, that would be interesting to see what would happen to the markets if all ML secrets held by any and all parties were suddenly made public and all traders were all using the best possible technologies to trade with. Such a scenario would probably tear a hole in space-time. But would the markets then become more, or less orderly."

To me the answer is obvious: the ML systems would stop working. This has happened in the past..perhaps 30 years ago when PCs were to the point where everyone could use one to trade
with. This was the heyday for TA's. There was a fellow called J. Wells Wilder who would invent new TA's and proceed to hold seminars on how to make a mint trading them. People would pay $3,000 for the privilege. He invented one called RSI and added a few simple entry and exit rules and you had a money maker? Yep the back testing on a decade or two of data showed it consistently made money. Not long after it became the rage with everyone paying attention if not using it outright, it stopped working in any noteworthy fashion. Why?

To many cooks using the same method spoil the broth by altering the market itself through mutual collective anticipation.

 

[userque]

"Yeah, that would be interesting to see what would happen to the markets if all ML secrets held by any and all parties were suddenly made public and all traders were all using the best possible technologies to trade with. Such a scenario would probably tear a hole in space-time. But would the markets then become more, or less orderly."

To me the answer is obvious: the ML systems would stop working. This has happened in the past..perhaps 30 years ago when PCs were to the point where everyone could use one to trade
with. This was the heyday for TA's. There was a fellow called J. Wells Wilder who would invent new TA's and proceed to hold seminars on how to make a mint trading them. People would pay $3,000 for the privilege. He invented one called RSI and added a few simple entry and exit rules and you had a money maker? Yep the back testing on a decade or two of data showed it consistently made money. Not long after it became the rage with everyone paying attention if not using it outright, it stopped working in any noteworthy fashion. Why?

To many cooks using the same method spoil the broth by altering the market itself through mutual collective anticipation.

Conventional wisdom says it was because too many people were using it. I remain skeptical. There are other possibilities.

But unlike RSI, we're talking algos that can adapt.

When two chess programs that are the same play against each other at the same strength, do they draw every time. Not at all...I least not when I did this experiment as a teen.

There are to many unknowns. Algos controlling more bankroll many adopt different strategies than Mom and Pop's algo. Etc.

Keep in mine we are talking about the most advanced--hidden and known--algos out there.

And don't forget about the concept of the self-fulfilling prophesy. If everyone has the same algo. And all of them buy at the same time. Wouldn't they cause the market to rise, and therefore be correct in being long.

Chaos.