The p-value you get from bootstrapping returns will be a function of how many observations you have and what your realised Sharpe Ratio is. So for example if your Sharpe is really 0.5 then on average with 20 years of data you'll have a p-value of 2.5%. Bootstrapping deals well with non Guassian returns as you get a parametric p-value. You get higher p-values for a positive skew strategy like trend following.
So this is just a fancy way of saying that trend following SR was lower in the past.
“There’s a creative moment when you think of a hypothesis, maybe it’s that interest rate data drives currency rates. So we think about that first before mining the data. We don’t mine the data to come up with ideas.”
I call this ideas first testing. Starting with the data, and getting the model, I call 'data first' testing.
Both methods have their advantages and drawbacks.
The following is an excerpt from my forthcoming book:
Systematic trading assumes that the future will be like the past. Hence we should create rules that would have worked historically, and hope that they will continue to work in the future.
But there are at least two different ways to find rules. One common method, which I call
data first, is to analyse some data, find some profitable patterns and create some trading rules to exploit them. This is sometimes called
data mining. The alternative,
ideas first, is to come up with an idea, then create a rule, which is then tested on data to see if it works.
(There is a third method which is to use an idea which you cannot or will not test on historical data. This falls outside the scope of this book.)
Designing an ideas first system is like saying:
“I want to design a system that captures this kind of market behavior or source of return. I hope this behavior or source of return is still around in the future”.
Whereas for a data first system:
“Here is a system that was profitable in the past given the patterns in the market (which I won't try and explain or understand). I hope these market patterns persist in the future”.
Advantages of ideas first
If an idea works no further dangerous fitting is compulsory.
Any fitting will probably be done in a small subset of alternatives.
We tend to get simpler and more intuitive trading rules with ideas first.
We can construct rules that make intuitive sense, with a story behind them.
It's easier to classify the trading rule, and work out where its profits are coming from.
Advantages of data first
There will be a bias with ideas first to testing things that we know will work, either because of 'market lore' or academic studies. This is a form of hidden over-fitting. This is essentially the problem highlighted in the blog.
With ideas first it is tempting to try a large number of ideas to find the ones that work. This is also a form of over-fitting.
All the fitting that is done is explicit, so the degree of over-fitting. can be controlled.
A compelling theory or story does not guarantee that the source of returns is repeatable, and could give a false sense of security.
Clever data analysis might unearth novel strategies that were previously unknown.
Given my preference for things I can trust and understand, I favor the ideas first method. This usually results in intuitive, simpler and more transparent rules. As long as a small number of ideas are tested
over-fitting is less likely.
But in some situations the data first process is better; for example in high frequency trading where there is plenty of data, rules can be refitted regularly and novel ideas are more likely to be found as market structure evolves.
The important thing is to be aware of the strengths and weaknesses of each method, and use them appropriately.
(Disclosure - I know a few of the BT guys so might not be an unbiased observer)
