Enough sample data?

M

mvalley

I've been trading a particular system recently that has proven to be profitable, but I have a question about some of the measures.

I run the backtest in AMI Broker and it reports:

Profit Factor 9.29
Sharpe Ratio of trades 5.24
All trades 285
Avg. Profit/Loss % 11.88 %

I backtested on stocks since the year 2000.


So, it generated 285 trades in around 6 years.


Is 285 trades enough sample data to assume that the 9.29 profit factor is actually correct?

Is there a way for me to calculate a statistically "valid" amount of time I should backtest against?

I'm concerned that the ratios reported aren't reflecting reality even though I am testing over 6 years of data.
 
Hi there.

I´m probably not the most qualified person to talk about this subject, but here are some things you may want to consider when backtesting:

1. Most successful system builders stress that more is always better. That means a larger quantity of trades, different types of markets and different market situations, preferably along decades of data. In a recent thread, someone presented a link to an article by William Eckhardt, where he talked about this subject. According to him, subtle statistical techniques employed to extract large amounts of meaning from small samples have no place in trading. You may want to run a google search to find this article, as I don´t have the link.

2. You can calculate the standard error from a sample with the following formula: 1 / square root of sample size. In your case this gives an error of about 6%. Since you want to be conservative, you should subtract 6% from all your metrics.

3. Are you backtesting on an index or are you also including delisted stocks? If you intend to trade stocks then you should also include delisted ones, since stocks are very prone to survivorship biases (index survivorship).

4. Have you included transaction costs and slippage in your simulation? Are your assumptions realistic? Be very wary of these two, since they are system killers.

5. Be very skeptical of your results. Many ass-kicking systems don´t produce even half of the profit factor and sharpe you mentioned.

If your system passes through this and still kicks butt then congratulations, you are probably very talented and I would be very glad to hear more about it :D.

Just my 2 cents. I´m certain that the more experienced members will have more to offer.

Regards and good luck.
 
(1) What is the trend of the profitability? Were the best years early in the data sample or more recently? (2) If the best years were recent, trade the system now because it appears to be better-optimized to the present instead of the past. (3) Ultimately, you have to decide when you'll feel confident enough to trade the system with real money. Don't get too bogged down with backtesting.
 
Does your profit factor 9.25 mean 9.25:1 profit/loss ratio?
If yes, that's a very high number, and 285 trades should be statistically sufficient.

One thing you want to look at is the average profit per trade. If that's too low (e.g. <0.5%) you must carefully examine your trade calculations, making sure all costs (spread, fees, slippage, etc.) are included. This could easily reverse your statistics.

Otherwise, you may have hit a jackpot.
 
Thanks for the replies, everyone.

Its a good suggestion to find out if the profits are centered on certain years. I don't think AMI has this capability, but I can do it via excel.

I finally downloaded more data into AMI from 1990 - 2006 and reran the test.

The results were:
# trades: 441
Winners 345 (78.23 %)
Avg. Profit/Loss % 10.19 %
Sharpe Ratio of trades 4.72
Profit Factor 7.67
Net Profit % 3922.56 %


The measures dropped some, but still it's profitable, overall.

I've just started to trade this particular variant of my filter, so we'll see how it does in real life.
 
Quote from OverKill:


3. Are you backtesting on an index or are you also including delisted stocks? If you intend to trade stocks then you should also include delisted ones, since stocks are very prone to survivorship biases (index survivorship).

More...


I just started reading about this topic last week and I'm having a problem locating delisted stocks for back testing.

Is there some place that records stocks that aren't listed anymore?


-Mike
 
Mike,

Is there some place that records stocks that aren't listed anymore?

CSI is the only supplier that I know of for delisted stocks. Are you looking for stock lists for a certain indices? If so what indice?

nt
 
Under ideal conditions (meaning the back-testing time period is an ideal representative of what you can expect from your system in the future) an estimation of the validity of above metrics can be made using the formula&#8230;.

Actual number is between:

backtesting result &#8211; 1.96 * standard deviation /sqrt(sample size)

backtesting result + 1.96 * standard deviation /sqrt(sample size)

with 95% confidence.

For example if average profit per trade is 500$ and standard deviation is 2000$ then with 250 samples the actual true average must be between 250 and 750 with 95% confidence. Of course your profitability is not stable and so in real life the error in the estimation could be significantly higher.

regards
 
Your numbers are real good, but numerous trades lacks a little bit.

Did you choose your stocks in accordance to the system?

Did your AmiBroker report list equal quantity of trades on each symbol?

I suggest you may run your test on a wider sample of stocks so you can have more trades( 1000 + ).
 
Quote from mvalley:

I just started reading about this topic last week and I'm having a problem locating delisted stocks for back testing.

Is there some place that records stocks that aren't listed anymore?


-Mike
More...

As far as I know, like ntfs posted above, CSI is the only provider that sells delisted stock data. I have also heard about Worden Brothers (www.worden.com) as good suppliers of this type of data, but haven´t looked much further. You may want to check them out.

Another option you may want to try out is to download historical data from the yahoo.com database. It´s a pretty good place to find centralized stock data from all over the world, since they have the quotes from most stock exchanges in the world. You could get data from Argentina, France, Germany, Singapore, throw it in your portfolio and see how your system fares. There are a bunch of vendors in the internet selling utilities that interface with yahoo and streamline the download process. The problem with their data is that it is filled with errors, unadjusted splits, spikes, etc... But then again, it´s free. You may try a stab at it while you research for a more reliable vendor.

Finally, in my first post I forgot to ask: are you checking for postdictive erros? This is the worst mistake one can make when coding a system. Look very carefully in your code for them.

Regards.
 
You must log in or register to reply here.
Back
Top