Out-of-sample trading strategy with neural networks

[easymon1]

and left my job

...was it a good job?

 

[ssp729]

Edit2: Maybe your post is mostly ironic and you're already fully aware of the issue, but I leave my post for any other readers that do not.

You can configure basically any kind of algorithm to output a nice looking return in-sample. Literally, you optimize the seed of a pseudo random number generator or a hashing algorithm to do that. It does not imply that it works out-of-sample. Not working in-sample does suggest not working out of sample (but is not a binary implication since it depends on what you have in-sample).

An interesting case I observed was on a site where people submitted Python codes/bots to play Rock-Paper-Scissors. One of the best performing bots was a few characters long as just calling a particular MD5 hash of the sequence of moves observed thus far. It turned out the guy that wrote it had downloaded all bots and checked which particular MD5 hash output (when transformed into an output move) would win against all deterministic bots. Against the best bots incorporating randomness it still loses almost every time though (and has 50% win rate against lesser ones).

Absolutely not ironic... I am really asking for help and help other in this tough journey of trading.

 

[ssp729]

...was it a good job?

not at all

 

[shatteredx]

Can you show some results? Now i think its´impossible to predict financial series

Well, I'm up 8% using linear regression the last two months compared to the market only returning 5%. So, linear regression can beat the market, but in my recent experience, only very modestly. It's making correct predictions 55% percent of the time, not 95%.

Is it even worth the bother compared to buy and hold? Perhaps not unless you enjoy the calculations (it's fun for me).

I have not tried the Ernie Chan method of incorporating ML but it's on my to-do list.

 

[guru]

Currently it is basic student homework to predict stock prices using various AI/NN techniques, likely to demonstrate how pointless this is.
There are now over 10,000 related open source projects and the list is growing every day:
https://github.com/search?q=stock+price&type=

 

[Russell Shuffle]

Anyone who is genuinely interested in getting some experience applying ML to financial data should look into the data-science competition at Numerai. It will quickly fix any questions you have about, does it work and what works.

It's not something you can do half way though, it takes a lot of work in the beginning to grind through the processes required to really put together a meaningful pipeline. I was one of the original participants in the comp, and I just pulled out of the main competition this fall after nearly five years of participation.

My six month performance average has been in the top twenty many times over the years and I've had dozens of models land in the +90% performance percentile out of nearly 2000 models. Now I'm focused 100% on their new "Signals" comp, which is significantly harder. Only a couple hundred models in that comp right now as we're all just getting our pipelines sorted.

I will say this. If you do join the comp and the community, leave any bias you have at the door. Opinions don't matter, results do.

 

[MarkBrown]

i flew out and hung around with si

I will say this. If you do join the comp and the community, leave any bias you have at the door. Opinions don't matter, results do.

so if results matter and nothing else would i be able to not use ai to win? is there somewhere i can see the results of actual trading is someone independently auditing the returns?

btw i am really put off from the touting of Renaissance Technologies cause i know they are bs.

 

[Russell Shuffle]

I've probably said all I will about it. If someone wants to get their hands dirty with some deep ML modeling, it is, imo, a very good place to do it. I'll leave it at that...

 

[globalarbtrader]

Hello everyone. So i tried to predict the close price of S&P 500 using neural networks (there are a lot of papers on the web) with NeuroshellTrader. The results in-sample was amazing, i was already ready to live on the beach and left my job.
When i tried to predict out-of-sample (the result that matters!) the dissapointment was huge.

Do you think it´s impossible to have at least good results?

It might be possible, but it's very difficult. And it's unlikely that someone who has just read a few papers and used an off the shelf package will be able to find the holy grail. As you've discovered, the most likely outcome is that you will end up massively overfittedd. A more benign outcome is that you'll reinvent the wheel of something much simpler (I remember someone doing a similar exercise, and coming up with a highly complex model that essentially did the same as a pair of moving average crossovers).

I'd advise you to spend time learning and understanding more old fashioned techniques first (like someone said nothing wrong with linear regression, on which almost the entire equity market quant fund industry is run), then you'll have a better appreciation of statistical significance and an intuition about fitting that a black box method like NN won't give you. And then read Marcus' book.

GAT

(I'm using old fashioned statistical backtesting to trade. Not living on a beach, but left my job 7 years ago)

 

[fan27]

FWIW, I wrote a custom ML implementation and initially had great success trading the models. Once they started the fall apart, I pulled the plug and reassessed to discover there was a flaw in the process. Fortunately, I was still quite ahead from where I started. I have abandoned the ML approach and instead now look for ways to best exploit known tendencies of particular markets.