In-Sample? Out-of-Sample? "Simple" Sample?

A

abattia

I have sufficient data to backtest a simple intraday strategy to end up with approx 1500+ trades in each test case.

With this data, I can (METHOD A) optimize for the three strategy variables by running all possible variable combinations over the whole data set (i.e. 1500+ trades), and look then for stable value ranges for each variable that give me the best results (in terms of PF, or Sharpe, or cumulative profit, or win/loss ratio, or average trade, etc, or whatever my criteria are).

Alternatively, I could (METHOD B) divide the data set into “training” (“in-sample”) and “testing” (“out-of-sample”) data sets, then optimize over the first group, and then test my results in the second.

My questions are:
1) What benefits will METHOD B bring me that I don’t get with (the much simpler) METHOD A? After all, I can run METHOD A over 1500+ theoretical historical trades …. seems like a lot to me, and enough to give confidence (or at least as much I would get from any method) that the strategy may have an edge ….

2) Do the benefits of METHOD B outweigh the extra time and complication of going through all the steps of METHOD B (and the additional hazards of selection bias in deciding which data to include in-sample, and which out-of-sample, etc.). A forward test will need to be the next step anyway in both cases, after all …

Unless the benefits of METHOD B are material, keep-it-simple-stupid still seems the best mantra to follow.

I await the flaming …

Please feel free simply to point me in the direction of other ET threads, if you feel this matter has already been hashed thoroughly and effectively elsewhere.

As always, thanks in advance.
 
The purpose of testing on historical data is to give yourself confidence that a strategy will perform acceptably-to-you, in the unseen future. This rests upon the often unstated assumption that market price behavior in the unseen future will strongly resemble market price behavior in the past. It may, but then again, it may not. Keynes said "Markets can remain irrational longer than you can remain solvent", and to a strategy trader, this becomes "Future prices can behave differently than past prices, long enough to put your strategy into a deep drawdown that makes you cry uncle."

Developing in-sample and testing out-of-sample is an approach that gives some people extra confidence. Whether it gives YOU extra confidence, is a question that only you can answer. If you test out-of-sample and observe that the strategy performs acceptably-to-you across a slice of unseen historical data, it may give your confidence a boost. It may help you feel enthusiastic that the strategy will perform acceptably-to-you on another slice of unseen data, namely, next week + next month + next year.

However, nobody can guarantee that future prices will behave in a way that strongly resembles past prices (with the happy result that strategy performance in the future strongly resembles strategy performance in the past). The best you can hope for is to perform computer tests and mental gyrations, that make you more and more confident. No matter how confident you are, things can still go wrong.
 
out_of_sample testing is important especially when a system involves variables, like for example a simple SMA crossover. It allows you to get an indication how the system performs on data not used in its analysis and design. But tis is not enough to grarantee future performance but only historical.
 
You didn't specify the time duration of the sample period, which to me is important.

Not really answering your question, but two things. First, three variables is IMO two too many, unless two of them are stop loss and profit target. Second, I have never liked two-stage testing. I test over a rolling period of six months once every week. After I get about three months of stable results, I trade it.

A great question BTW, thoughtfully phrased! You must be new here.
 
Thanks.
Quote from Arthur Deco:
You didn't specify the time duration of the sample period, which to me is important.
More...
The sample period covers approx last 3.5 years of 1-minute data.
 
Quote from abattia:

Thanks.

The sample period covers approx last 3.5 years of 1-minute data.
More...

The sample is significant but 1-mimute data have a lot of noise in them. In the past I burned a lot of midnight oil testing on 1 - 5 mimute data and got me nowhere. Maybe that is only me. I found more success with 30 minute data but that got me into the position trading territory.
 
Quote from abattia:

...optimize over the first group, and then test my results in the second.
More...

If you tested against out-of-sample once, it's not out-of-sample the second time.
 
FWIW I test on one minute data. But 3.5 years of it? IMO that seriously risks blurring the efficacy of a strategy by averaging in the migration of it over that time. IMO strategies "breathe."
 
You must log in or register to reply here.
Back
Top