how much data is needed for backtesting?

Quote from radolym:

Is this an intraday system or EOD? Also, can you test on basket of stocks in Tradestation or these results based on 1 stock?

It's an intraday system. What is EOD? I don't have much experience with trading stocks. I only focus on future, and this system is designely solerly for YM. I have backtested it on ES, and it was profitable, but it doesn't apply too well. If I ever use it to trade other products, I will have to reoptimize the parameters again.


Quote from nononsense:

You're almost there. Start working now on predicting the transitions from one "zone" to another.
If everything goes well, this might require you only 5 to 10 years.

PS: don't get sidetracked by "Trend" guru's & addicts.

By "zone", do you mean different phases of the market? like trendy or sideway?
 
Quote from FulltimeTrainee:


By "zone", do you mean different phases of the market? like trendy or sideway?
I meant "zone" as priorly defined by you in:
Quote from FulltimeTrainee:

There's another thing I worry about is the optimisation. The performance of the system is 5 times better with the optimisation. There are 4 time zones for trading in this system, and each use the same indicators for entry and exit, but the parameters are different. Those parameters were found with optimisation. I'm affraid that the optimisation is overly done which boost up the misleadingly good results.

I tried to convey that you only scratched the surface of the problem. The remaining part is a very tough nut to crack. Don't believe old-wive tales coming from self-styled guru's. I tried to give you some highly competent advice.
 
How to optimize a trading system without curve-fitting it

There’s a fine line between “optimizing” and “curve-fitting”.
You have to make sure that your don’t over-optimize.
In the following example I’ll show you how to determined the “best” parameter without curve-fitting a
system.

The underlying system is a simple breakout system. The system contains a parameter called <strong>TF_Param</strong>.
In the following I will explain how we optimized the parameter and why we selected 0.7 as the current value for the parameter.
First I run the optimization and look at the <strong>net profit</strong>, since that&#8217;s one of the most important figures.

optimization_01_net_profit.gif


As you can see, any parameter between 0.35 and 0.8 produces robust results.
Next I am looking at the max drawdown.

optimization_02_max_drawdow.gif


Any parameter above 0.4 produces a rather low drawdown, so using the combined information I would pick a TF_Param
between 0.4 and 0.8.
Now I am looking at the average profit per trade:

optimization_03_avg_trade.gif


The parameter range between 0.45 and1.2 looks promising, so I am limiting the TF_param to 0.45-0.8.
Now I am looking at the winning percentage. The higher, the better:

optimization_04_win_percent.gif


Anything above 0.5 produces a quite robust winning percentage above 50%.
As it seems, anything above 0.6 is quite good.
Combining all the previous results we should look for a TF_Param between 0.6 and 0.8
As a last test we look at the number of trades. Again: The higher, the better:

optimization_05_trades.gif


No surprises here: The lower the parameter, the more trades we get.
By combining all the above findings we see that a TF_Param between 0.6 and 0.8 produces best results.
That&#8217;s why 0.7 is definitely a good choice. Even if the market changes slightly, the system would still produce excellent
results.
 
Quote from BenMillane:


That&#8217;s why 0.7 is definitely a good choice. Even if the market changes slightly, the system would still produce excellent
results.

Now explain using 3d surface charts for 2paramteres....god, I hate reading those surface charts ^_^
 
Quote from Markus:

There are a few factors that you need to consider when determining the period for backtesting your trading system:

Trade frequency
How many trades per day does your trading system generate? It's not important how long you backtest a trading system; it's important that you receive enough trades to make statistically valid assumptions*: If your trading system generates three trades per day, i.e. 600 trades per year, then a year of testing gives you enough data to make reliable assumptions*. But if your trading system generates only three trades per month, i.e. 36 trades per year, then you should backtest a couple of years to receive reliable data.

Underlying contract
You must consider the characteristics of the underlying contract. The chart below shows the average daily volume of the e-mini S&P:

eminiSP_monthly_volume.gif


It doesn't make sense to backtest a trading system for the e-mini S&P before 1997, because the contract simply didn't exist! In my opnion it doesn't make sense to backtest an e-mini trading system before 2002 because at that time the market was completely different; less liquidity and different market participants. I believe that a reliable testing period for the e-mini S&P are the years 2002 - 2004.

Margin of Error
The more trades you use in your backtesting (without curve-fitting), the higher the probabilities that your trading system will succeed in the future. Look at the following table:

Number of Trades 50 100 200 300 500
<>Margin of Error 14% 10% 7% 6% 4%

The more trades you have in your backtesting, the smaller the margin of error, and the higher the probability of producing profits in the future.

My 2 cents.... Hope it helps :-)

Markus

============
That helped.

FullT Trainee;

As you can see from above 500 is better than 50;
& include bull & bear years, which will get your sideways,slop chop trends. Wisdom is profitable to direct.


And some have posted on YM summer slop sidways trends;
a thorough [all derivatives section ] read of elitetrader.com will give you some specific numbers , which may help you avoid sideways slop-chop,which could be a real killer
:cool:
 
Wow, thanks Ben, that's a helluva study.

Hope this is working for you Fulltime, it's great info, which if you can just get the concepts can you help you go a long way towards creating a profitable system.

I did a search on google and came up with www.Price-Data.com as a potential source for backtesting data.

I am sure there are plenty more out there.

Best,

Jimmy
 
Thank you guys! I got a lot of useful thoughts from reading your posts. Now I will try to get more data to include different phases of the market, then I will have to redo the parameters part. My system has probably done a lot of curve fitting since the net profit was the only thing I looked at. Also I would probably replace the indicators by something else that has less parameters in order to make the optimisation simpler.

Actually, my system have 11 parameters for each time zone, so together I have 44 parameters, and it takes me 2 to 3 days to optimize it (fortunatly, not all parameters are not related, some could be optimized individually which saves some time). Anyway, I will replace the entry and exit indicators by something that have similar effect but with less parameters. Hopefully, it's going to work.

Thank you very much!
 
Back
Top