Backtesting Sufficiency

Quote from Lefty62151:

... So what I am trying to say, is that the professionals have moved on. At this point in time, they are approaching the problem from another direction. I don't want to bore you with the details. What I willl say is that there have been a few posts relating to how fit a program to a market and vice versa. The posts were largely ignored. .... Lefty
More...

Hi Lefty,

Personally I would not be bored with the details.

Would you mind pointing to me where these posts are? Being new to this board and somewhat green in trading I probably would have missed the point even if I had read them.

Would love to take a look (again?) and explore further.

Thanks & cheers!
 
Thanks for the replies. My timeframe per trade is around 2 days to a couple weeks, and usually on the shorter end of that scale.

Also I don't really understand what Lefty means when he says "in sample/out of sample" testing is ineffective. Are you suggesting that backtesting itself is ineffective, or that adaptive optimization is ineffective?

If you use a learning algorithm to either tune parameters or select from a fixed number of strategies, you run the risk of "overfitting" or "overtraining." The algorithm may have been able to exploit specific memorized patterns that only occur in the "in sample" data. By using the out of sample data, you are simply verifying that your solution does not have this problem. You should expect similar (although usually worse) performance on the out of sample data, but if it is significantly worse, this points to a definite problem.

This is not a methodology that has been developed solely by those who analyze the market. This is widely accepted practice in almost any kind of academic adaptive optimization research. In fact, I wager that no work on the subject could be published anywhere respectable without a demonstration of performance on out-of-sample data.

The other opinion, that there is no such thing as true out-of-sample data, might make sense with other ways to approach the problem.
 
More on my strategy: I want to focus on two things: gapping stocks, and ETFs. I want to determine if I can swing trade the ETFs (like QQQ, SPY, etc.).

I'm not sure about the ETF's yet, but I have a strategy for gapping stocks that looks good when backtesting on historical data from Yahoo. However, I am currently working on getting intraday data to make sure.

I think strategies that focus on individual stocks are limited as account size grows, because proportionately larger trades have more chance of disturbing the market price. I therefore limit any trade my backtesting program makes to be 50K or less in terms of money, and I adjust this if the average volume wouldn't support even that. This is why I'd like to also focus on getting good at the ETF's, since trades are more liquid there. Having a combination of strategies would help as well I think.
 
:
I have a few minutes here after the open. Choppystride, one option or alternative that has already been posted involves techniques to characterize the target market. Use search. Also there are several books on the subject. For folks without the math background it is going to be difficult to pick it up, but "Mathematics of Technical Analysis" by Clifford Sherry is probably the simplest read. As far as questions about in sample/out of sample testing, yes it is very useful in specific circumstances. For instance In sample/out of sample methods are often used to evaluate research for medical/biological agents and other types of studies where one can anticipate that the distribution is normal or lognormal. However where the data itself is dynamic rather than static and stationary, this technique has its limits. By the way, how is it working for you in the markets? :D Lefty.
 
Quote from gorgonzola:

...I don't really understand what Lefty means when he says "in sample/out of sample" testing is ineffective. Are you suggesting that backtesting itself is ineffective, or that adaptive optimization is ineffective?
More...

i won't speak for Lefty. when i say it i mean walk forward optimization ends up being no different at all from whole-sample optimization when applied to market endevours. the search button will lead you to long, detailed and polite conversations on the subject.

good luck!
 
Thanks for the pointers, Lefty. You may have given me enough information to get headed in the right direction on this.

I have a Ph.D. in applied science and work on intelligent analysis of signals for a living, just got a wild hair up my ass to try applying some of the techniques to stock trading once I found a couple logical strategies that have potential to be made consistent. So what I'm trying to say is, if you have a couple favorite references on the subject regardless of technical difficulty, I'd be interested in seeing if I could hunt them down.

Even if I don't require that level of sophistication to be successful, one of the reasons I'm doing this is because it's interesting to me. I think analyzing the stock market should be a great hobby, even if I'm only marginally profitable. I certainly don't want to fail and have it be an expensive hobby, if you know what I mean :).

Anyway, you've definitely got my curiosity up.
 
Quote from gorgonzola:

I have a Ph.D. in applied science...
More...

iron ring? you're not alone, there are lots of current and former engineering types 'round these parts.
 
By the way, about the question "how is it going in the markets?"

I'm still sitting on the sidelines for now. I'm not going to jump in until I've had more opportunity to study the problem, but then again I've only been applying significant time to this problem for the last 2-3 weeks (although I have a lot of good [and bad] trading experience in the past, perhaps more luck than anything). In the meantime I've been paper trading the "gap" strategy, and every day it does just fine... sometimes outstanding (like today for instance... check out STEM, CRS and IST today). I probably won't swing trade ETF's until I've spent significantly more time on the problem, since that should prove to be a lot more difficult.
 
Quote from gorgonzola:

I am trying to develop a strategy for trading, and I want to get some opinions on what some of you think about what constitutes "sufficient" backtesting. I will supply my own ideas and perhaps that will help elicit some feedback.

How many years back to test? I think roughly 5-7 years is good, since that is "recent history" in terms of the equity market trading patterns, given new technologies etc. Any difference of opinion on that?
More...
Yes.

It isn't time, it's trades. The number of trades in your backtest minus the number of indicator parameters you're optimizing has to be 30 or greater for you to be able to say anything statistically valid about your system performance.

That's all. :cool:
 
The issue about how long of a period to use for a backtest is not simple.

1) Is the system based on a sound premise, for example a seasonal corn system which shorts in July , does not need as many trades as a price pattern based system because the premise is sound and based on the end of drought risk. Another example would be a bond system based on intermarket analysis, like using gold,silver,chemical stocks as the intermarket. A system based on a sound premise is the most important issue in deciding how long to backtest for.

2) You want to cover at least one bear and bull market in your backtest period. Sometimes when markets are in a long term trend like Nasdaq was before the 2000 bear market started the best you could have done is a major correction in 1998.


3) Developing a system on a basket of market using the same rules and parameters gives you more trades.

4) Advance Optimization methods like walk forward testing can give you a more valid ,longer period for out of sample testing.

If anyone wants to discuss these issues in more detail , let me know.
 
You must log in or register to reply here.
Back
Top