Out-of-sample trading strategy with neural networks

[userque]

Hello everyone. So i tried to predict the close price of S&P 500 using neural networks (there are a lot of papers on the web) with NeuroshellTrader. The results in-sample was amazing, i was already ready to live on the beach and left my job.
When i tried to predict out-of-sample (the result that matters!) the dissapointment was huge.

Do you think it´s impossible to have at least good results?

With ML, it can be done, but you have to really know what you're doing if you want to solve trading problems to a profitable degree. I'm not talking about Neuroshell as I don't know enough about it; I'm talking about actual, full featured ML software.

Having the correct software/platform is one thing; knowing how to properly use it is quite another.

See here for many examples:

https://www.kaggle.com/search?q=stock+market

 

[PoundTheRock]

First, most of the machine learning models for markets are garbage. I even saw one example with bidirectional LSTMs, which as a previous poster mentioned, has possibly millions of trainable parameters.

Second, your target variable is not numeric; it's a simple buy-or-sell signal with probabilities. So, let's say you take the whole universe of technical analysis variables and train your model with cross-validation. Again, your model will fare poorly because of the changing nature of the market (you've basically trained the model for one snapshot in time).

Finally, you're better off predicting conditions such as "wide range" or "narrow range". This way, you can apply a trend or counter-trend system based on a high probability within a calibrated model, filtering out the low-signal results.

You can take a look at some of my work here:

https://github.com/ScottfreeLLC/AlphaPy

 

[guowei58]

Hello everyone. So i tried to predict the close price of S&P 500 using neural networks (there are a lot of papers on the web) with NeuroshellTrader. The results in-sample was amazing, i was already ready to live on the beach and left my job.ou
When i tried to predict out-of-sample (the result that matters!) the dissapointment was huge.

Do you think it´s impossible to have at least good results?

ML is a tool and like all tools, used in the right way, it will give you good results. it's no different than using technical analysis, fundamental analysis, price action or your own intuition...

For ML particularly, I would try to limit the number of variables, degrees of freedom or increase the sample size. typically, you need 2K of trades in order to have statistical significance. as you increase the number of variables, you should increase the sample size too. the bigger the sample size, the less overfit the model is. Linear regression works great because you have 1 degree of freedom, and are guarateed to find that small advantage without overfitting the data. Those small advantages really add up...ML should be able to do better given enough data.

 

[userque]

ML is a tool and like all tools, used in the right way, it will give you good results. it's no different than using technical analysis, fundamental analysis, price action or your own intuition...

For ML particularly, I would try to limit the number of variables, degrees of freedom or increase the sample size. typically, you need 2K of trades in order to have statistical significance. as you increase the number of variables, you should increase the sample size too. the bigger the sample size, the less overfit the model is. Linear regression works great because you have 1 degree of freedom, and are guarateed to find that small advantage without overfitting the data. Those small advantages really add up...ML should be able to do better given enough data.

Good post.

But the problem isn't necessarily the number of variables. The real problem is having variables that don't correlate, and not doing out of sample validation (or turning OOS data, into in sample by repeated testing).

 

[ssp729]

thanks for all the answers. I decided, after pros and cons, giving up trying to predict the close price, percent change, open, high or anything related to financial time series.