backtesting vs real trades

[DTorSwing]

I did some backtesting of my strategy of few stocks for 18 months and also tried each 6months. backtesting was done in Ninja trader and Multicharts at for $1000/ stock trade
my strategy is ATR based strategy
1st 6 months $3351
2nd 6 months -$4250
3rd 6months $1452
and I am testing this in sim now and have NO IDEA what would be for the next 6 months. it could be even -PnL

so how do you guys use backtest results? without knowing it can reproduce a +PnL in the future.
I dont expect a exact replica of backtest in real trades, so should be closer?

 

[d08]

I did some backtesting of my strategy of few stocks for 18 months and also tried each 6months. backtesting was done in Ninja trader and Multicharts at for $1000/ stock trade
my strategy is ATR based strategy
1st 6 months $3351
2nd 6 months -$4250
3rd 6months $1452
and I am testing this in sim now and have NO IDEA what would be for the next 6 months. it could be even -PnL

so how do you guys use backtest results? without knowing it can reproduce a +PnL in the future.
I dont expect a exact replica of backtest in real trades, so should be closer?

First off you shouldn't preselect stocks as that introduces bias. If your strategy only works on 20, 50 stocks then most likely it's just a curve fit. 18 months for me is nowhere near a reasonable period, we've had a massive bull market for longer than that.

 

[DTorSwing]

First off you shouldn't preselect stocks as that introduces bias. If your strategy only works on 20, 50 stocks then most likely it's just a curve fit. 18 months for me is nowhere near a reasonable period, we've had a massive bull market for longer than that.

ok, but if I don't pre select stocks, how would I even run backtest on what instruments?

so 18months less, what period would you backtest, like 3 yrs or 5yrs? i doubt i have intraday data for that long

 

[IAS_LLC]

According to simple sampling theory (emphasis on simple... maybe too simple), for a strategy to be statistically significant with a confidence level of 99% and a margin of error of 5%... You would need to conduct at least 666 trades, and 363 of them would have to be profitable to reject the null hypothesis.

If you wanted to reduce the margin or error to 1%, than you would need 16641 trades, with 8470 profitable to reject the null hypothesis.

 

[DTorSwing]

According to simple sampling theory (emphasis on simple... maybe too simple), for a strategy to be statistically significant with a confidence level of 99% and a margin of error of 5%... You would need to conduct at least 666 trades, and 363 of them would have to be profitable to reject the null hypothesis.

good point. I never took period to test based on # of trades, always did 1yr to max 3 yrs
but problem to get 666 trades is, I may not have so much historical datafeed
example, I test a symbol called ACHN with 30min or 60 min chart
approximate, it takes a trade or two /month
so i get only 24 x 5 = 95 trades in 5 years (I have Barchart that gives this data)
666 is way too far from datafeed I can get

 

[IAS_LLC]

good point. I never took period to test based on # of trades, always did 1yr to max 3 yrs
but problem to get 666 trades is, I may not have so much historical datafeed
example, I test a symbol called ACHN with 30min or 60 min chart
approximate, it takes a trade or two /month
so i get only 24 x 5 = 95 trades in 5 years (I have Barchart that gives this data)
666 is way too far from datafeed I can get

Well consider this:

Lets say you can test 100 trades, and want to do 99% confidence. This would mean your margin of error is about 13%. To reject the null hypothesis, you would need to have 61/100 profitable trades. But there is a stochastic element to the market, and the backtest is only accurate to 61 +/- 13 trades, 1 Sigma. A one standard deviation error is not an unlikely event... It is reasonably likely that your 61% win rate is actually a 48% win rate.

With only 100 trades, you really can't reject the null hypothesis reliably. Even with 666 trades, it is difficult.

If you want to trade systematically, I would recommend shelling out a bit of cash for decent historical database. Otherwise, your backtests will not be statistically significant, when margin of error is factored in.

 

[syswizard]

According to simple sampling theory (emphasis on simple... maybe too simple), for a strategy to be statistically significant with a confidence level of 99% and a margin of error of 5%... You would need to conduct at least 666 trades, and 363 of them would have to be profitable to reject the null hypothesis.

If you wanted to reduce the margin or error to 1%, than you would need 16641 trades, with 8470 profitable to reject the null hypothesis.

And that would mean if his average time in trade was 1 day and his system was in a position half the time, he would need to test for over 10 trading years: 666/0.5/250
In the second case, he could be dead before the testing is complete.

 

[IAS_LLC]

And that would mean if his average time in trade was 1 day and his system was in a position half the time, he would need to test for over 10 trading years: 666/0.5/250
In the second case, he could be dead before the testing is complete.

Depends on the nature of the strategy. If he is trading the same strategy, with the same parameters, on multiple assets, it could be significantly less.

You could also invoke "Population Size" and say only one trade per day is possible... but that is cheating in my opinion. For trading, population size shoudl be infinite... but, if you only had 250 trading days of data.. you could say 182 trades is a large enough sample size.

 

[syswizard]

Depends on the nature of the strategy. If he is trading the same strategy, with the same parameters, on multiple assets, it could be significantly less.

Good idea. IMHO He would need to test with 10 different symbols to get the testing time down to one year.

 

[DTorSwing]

Depends on the nature of the strategy. If he is trading the same strategy, with the same parameters, on multiple assets, it could be significantly less.

You could also invoke "Population Size" and say only one trade per day is possible... but that is cheating in my opinion. For trading, population size shoudl be infinite... but, if you only had 250 trading days of data.. you could say 182 trades is a large enough sample size.

I do use same strategy, with the same parameters, on multiple assets
except 1 parameter, which is data series of the chart
I might do 15min on few symbols, 30min on others and 60 on others. I have 3 data series so far