Careful with Historical Analysis v. Real Performance

El OchoCinco

El OchoCinco

As someone who has been programming and running automated strategies for testing past year or so, I wanted to share an experience for any newbies.

I created an automated strategy on Ninjatrader and ran a 6 month historical analysis:

upload_2020-3-9_9-19-27.png


Yeah.....ready to run this for real and buy an island in a year! haha ......NO.

Well as a good smart individual I ran it in SIM for a few days and it lost a few thousand dollars most days. I let it run live with 1 contract on NQ which it was tested on the display above and it just kept generating red.

BE CAREFUL about unrealistic historical analysis. I have no issues with NT actual trader platform and running strats but their analysis of strategies is really unrealistic so be careful before you think you found the next strat that will generate $191,143.84 in a month on 1 contract haha.

Always sim test going forward and live test before letting a bot run.

I like comparing historical SIM versus actual going forward and the only way to test your automated strat is to let it run on a live SIM account with realistic fills and commissions or with any confidence run it with 1 contract for a few days.

Just a word of advice for the newbies or those who come on here posting their amazing backtest results. As the backtest results I posted here, anyone can make it look amazing but reality is far from true.
 
Of course today the SIM is up $8,500 so far but today is a huge swing day and any system would make money today with 30 point swings haha
 
Would be a lot of effort to learn another platform. I am trying to see why the historical is so overstated but I would have to study almost each trade to see if the entries were not realistic or whatever. Even if real live lead to 25% of the performance predicted I would take it but the live SIM and actual live have been poor for weeks (treating real losses as invested capital into the system haha).

Of course I am ignoring today's sim result as today is the most abnormal market possible and not too hard even for the most basic system to go long and short on these swings.

System should produce small losses on choppy trendless rambling random days but seems to produce red live on swing days as well so just curious if anyone else uses NinjaTrader and sees huge difference between historical analysis on strategies and actual?
 
I have tried NT and it was harder than expected to translate my strategies to code (I can code, but defining the things I use in each trade is a lot of work), but I was struggling to get much useful information out of the historical data. I know it is also lacking (for NQ futures) several years back, so I imagine other platforms could be more accurate.

I had varying levels of success, getting my strategy to recognize certain moves the way I wanted, but then having it pick up on random movements as well, leading me to try more constraints and ultimately leaving it for a future day when I have more time. I never tried it live yet.
 
Last edited:
Ninja is one of the most robust platforms out there. You have full power of C# at your disposal. And NT8 is multi threaded, so performance is amazing as well.

Backtest vs real have nothing to do with the platform.
 
El OchoCinco said:
As someone who has been programming and running automated strategies for testing past year or so, I wanted to share an experience for any newbies.

I created an automated strategy on Ninjatrader and ran a 6 month historical analysis:

View attachment 221519

Yeah.....ready to run this for real and buy an island in a year! haha ......NO.

Well as a good smart individual I ran it in SIM for a few days and it lost a few thousand dollars most days. I let it run live with 1 contract on NQ which it was tested on the display above and it just kept generating red.

BE CAREFUL about unrealistic historical analysis. I have no issues with NT actual trader platform and running strats but their analysis of strategies is really unrealistic so be careful before you think you found the next strat that will generate $191,143.84 in a month on 1 contract haha.

Always sim test going forward and live test before letting a bot run.

I like comparing historical SIM versus actual going forward and the only way to test your automated strat is to let it run on a live SIM account with realistic fills and commissions or with any confidence run it with 1 contract for a few days.

Just a word of advice for the newbies or those who come on here posting their amazing backtest results. As the backtest results I posted here, anyone can make it look amazing but reality is far from true.
More...
So the island is still up for grabs?

upload_2020-3-9_13-47-24.jpeg
 
Are you using limit orders or market orders? I imagine that would/should make a big difference.

Regardless, these are not normal market conditions and past results does not guarantee future performance.

Even as a discretionary day trader, I've noticed significant differences between market days going from day to day lately. Two weeks back, I did well on the first 4 days of the week, only to get chopped up on wild counter swings on Friday.

This market easily moves 10-20 ES points against you in 1 minute - only to resume the intended/original direction.

I can only imagine that NQ is even worse, both with regards to speed, size of moves and fill price.
 
So I had a slow afternoon and poured over hundreds of trades randomly chosen over past 6 months.

Noticed unusual fills consistently and adjusted the slippage factor in the strategy and re ran the analysis...

SURPRISE...losing strategy.

Historical performance often can have unrealistic fills. NY lets you adjust the slippage factor to bring them back to reality which is what I did now and the Analysis shows RED final account. I am still tweaking to solve the problems but if I can resolve the issue, will be a nice bot to run from the morning to afternoon and not stare at these @#$%ing screens so much.
 
You must log in or register to reply here.
Back
Top