I also like the metrics you mentioned. Good suggestions. The important thing is that you provided a reason for looking at those ratios. Most people throw suggestions without justification. For example a profit factor of 5 may be a terrible thing if DD is excessive. This is in reference to the other poster who recommended the profit factor.
You are using an out of date browser. It may not display this or other websites correctly.
You should upgrade or use an alternative browser.
You should upgrade or use an alternative browser.
Backtesting Metrics
- Thread starter DustyFoot
- Start date
I also like the metrics you mentioned. Good suggestions. The important thing is that you provided a reason for looking at those ratios. Most people throw suggestions without justification. For example a profit factor of 5 may be a terrible thing if DD is excessive. This is in reference to the other poster who recommended the profit factor.
Quote from HowardCohodas:
My trade filled, so I'm back.
Cornered. Oh, I can't wait to hear an explanation for that. Do tell!
I have no idea why saying I have friends in the military would scare anyone away from anything. My friends in the military primarily come from two of my areas of interest. I am a private pilot and make many acquaintances who are military. Also among air traffic controllers, commercial pilots and other private pilots. Cleveland Center is a wonderful place to visit. Not so scary yet.
I am also a Concealed Carry Weapons instructor in the State of Ohio. One needs a permit from the state in order to carry concealed and must take a class in order to obtain one. I've met many in the military in that venue as they want to carry when out of uniform or after military service. That may be scary to some who do not believe in self reliance and self defense.
Do you have to say anything about Pardo's book or you will keep trolling forever?
Quote from ronblack:
Why are the rest noise and profit factor isn't? All metrics are related in a way or another.
Why 10+ years for intraday?
Why 25+ years for daily?
I think you should offer some justification for these bold statements of yours. You may be right but I have seen more people being wrong than right in this business. Bold statements without justification mean absolutely nothing to serious traders.
Sure,
The rolling pf gives you the distribution of your profit curve. From this distribution and the trade frequency you can compute profit and drawdown (and anything else you like) using monte carlo.
It is the only statistically correct approach as measuring profit and DD directly from the equity curve gives you only the estimate from this one run that was realized in the past. The future values might vary widely from your estimate if you're "unlucky".
You need 15+ years simply because most mean-reversion strategies have a huge regime shift around 1997-1998.
Ninna
Quote from ronblack:
I also like the metrics you mentioned. Good suggestions. The important thing is that you provided a reason for looking at those ratios. Most people throw suggestions without justification. For example a profit factor of 5 may be a terrible thing if DD is excessive. This is in reference to the other poster who recommended the profit factor.
Then you clearly don't understand what the profit factor is...
From the pf and trade frequency you can compute the maximum DD (within your chosen confidence interval).
In other words, if your pf (or trade frequency) is good enough, the DD can't be excessive. The DD depends only on the pf and the trade frequency.
Ninna
Being wrong and being rude. An interesting combination. I didn't know answering dubious questions was now considered trolling. I could just ignore you, but then some newbies might be misled.Quote from ronblack:
Do you have to say anything about Pardo's book or you will keep trolling forever?
I loaned my copy to a friend working for an investment firm on some strategies of their own.
However, the most important thing I recall learning (I believe from this text) was how to avoid "curve fitting" by dividing your historical data into two non-contiguous sets. One set to develop your strategy and the other to confirm your results on data it had never seen before.
Also of value was learning to understand the value and proper use of some of the metrics.
I don't recall if this book was the inspiration of my four step strategy qualification process.
- Back Testing
- Paper Trading
- Small Money Trading
- Serious Money Trading
Sometimes I abandon the change rather than go back, but I never move forward unless the results are confirmed as an improvement.
Quote from nLepwa:
Sure,
The rolling pf gives you the distribution of your profit curve. From this distribution and the trade frequency you can compute profit and drawdown (and anything else you like) using monte carlo.
It is the only statistically correct approach as measuring profit and DD directly from the equity curve gives you only the estimate from this one run that was realized in the past. The future values might vary widely from your estimate if you're "unlucky".
You need 15+ years simply because most mean-reversion strategies have a huge regime shift around 1997-1998.
Ninna
Thank you for the insight.
Quote from HowardCohodas:
Being wrong and being rude. An interesting combination. I didn't know answering dubious questions was now considered trolling. I could just ignore you, but then some newbies might be misled.
I loaned my copy to a friend working for an investment firm on some strategies of their own.
However, the most important thing I recall learning (I believe from this text) was how to avoid "curve fitting" by dividing your historical data into two non-contiguous sets. One set to develop your strategy and the other to confirm your results on data it had never seen before.
Also of value was learning to understand the value and proper use of some of the metrics.
I don't recall if this book was the inspiration of my four step strategy qualification process.Every time I make a modification to my strategy, I go back to at least to step 3. If the extent of the modification is significant or if small money trading does not show good results I will go back to steps 2 at least. Sometimes all the way back to step 1.
- Back Testing
- Paper Trading
- Small Money Trading
- Serious Money Trading
Sometimes I abandon the change rather than go back, but I never move forward unless the results are confirmed as an improvement.
I have the Pardo book. Interesting stuff and I would think it the definitive guide on proper backtesting procedure for any automated strategy. For myself, however, I don't create fully automated stuff so it was a useful read, but that's about all.
As for some of the metrics. It's the usual stuff.
# of trades
P/L
Annualized P/L
% winning trades
largest win
lenght of largest win
average win
length of average win
same as above for losses
standard deviations of averages
max drawdown
lenght of max drawdown
longest winning run
longest losing run
blah blah bla
What I do, to keep it simple, is just track number of wins, losses, averave win, average loss, # of trades, # of trades per day, pf...
I then put these into a kind of do it yourself monte carlo excel sheet that I created that allows me to run several iterations of a thousand trades and see what the worst case scenario might be.
As for needing 15+ years of data to test an intraday strategy, I call bullshit.
You'd first need to know just how often the strategy actually trades. Let's say that you have a strat that generates 30 trades per day. So, what? You need 100,000+ trades to know whether it works or not? That's just nonsense and belies a lack of understanding of statistics.
Quote from iagainsti:
You'd first need to know just how often the strategy actually trades. Let's say that you have a strat that generates 30 trades per day. So, what? You need 100,000+ trades to know whether it works or not? That's just nonsense and belies a lack of understanding of statistics.
The trade frequency is irrelevant if you suffer from regime shift.
I have mean-reversion systems with thousands of trades per month that perform extremely well from 1990 to 1998 and then crash.
1998 really seems to be a turning point for a lot of mean-reversion strategies. Since you don't know when the next turning point will come, backtesting back to 1997-1998 gives you a nice overview of how things could turn out...
If your strategy exploits some kind of mean-reversion even remotely (and most intra-day strategies do) I would suggest to backtest atleast 15 years.
Personally I backtest 25 years. And I expect my strategies to work for (atleast) another 25 years. Trading isn't a get-rich-quick scheme.
Ninna
Quote from iagainsti:
I have the Pardo book. Interesting stuff and I would think it the definitive guide on proper backtesting procedure for any automated strategy. For myself, however, I don't create fully automated stuff so it was a useful read, but that's about all.
As for some of the metrics. It's the usual stuff.
What I do, to keep it simple, is just track number of wins, losses, averave win, average loss, # of trades, # of trades per day, pf...
IIRC he not only lists the metrics, but discusses their benefits and limitations. This permits the strategy designer to make informed decisions on which metrics to use.
I'm not one for a completely automated system. I'm a retired software engineer and I know my limitations. I always want my brain and fingers between the trade recommendations and their execution.
Totally automated or not, IMO, Pardo is a good reference in learning how to evaluate your strategy.
Quote from nLepwa:
The trade frequency is irrelevant if you suffer from regime shift.
I have mean-reversion systems with thousands of trades per month that perform extremely well from 1990 to 1998 and then crash.
1998 really seems to be a turning point for a lot of mean-reversion strategies. Since you don't know when the next turning point will come, backtesting back to 1997-1998 gives you a nice overview of how things could turn out...
If your strategy exploits some kind of mean-reversion even remotely (and most intra-day strategies do) I would suggest to backtest atleast 15 years.
Personally I backtest 25 years. And I expect my strategies to work for (atleast) another 25 years. Trading isn't a get-rich-quick scheme.
Ninna
Ninna,
With an intended lack of respect to those you're addressing; you're talking to a few clowns in this thread. The range of belief systems held on ET can be quite diverse, unfortunetely, when it comes to these topics, most choose to betlittle information if it goes counter to their limited experience base.
Point in hand, no one should trade a strategy that doesn't survive in all regimes, no matter the frequency. It's about confidence in the model. If one is putting on large positions, it comes down to confidence in the process one took and the method that resulted.
The posters you're addressing don't even know what a regime shift is...
Mike