Quote from bluedemon77:
Can you guys tell me what your experience has been in designing systems that are less dependent on precise parameters, i.e. avoiding curve fitting so that the in-sample performance predicts out-of-sample results? I have been optimizing based on Ulcer Performance Index or Net Profit, but I'm not entirely satisfied with the results of either of those. I also tried Sharpe Ratio and K Ratio, but those seem to be worse than UPI. Any thoughts?
Quote from Mike805:
I think your design goals might be suspect... what type of system are you working with?
If you're dealing with a directional, multi-day system, then optimizing for net profit might be a mistake. The reason being is that the optimization will focus on the larger winners (i.e. the tails of trade distribution). I would avoid system parameters that emphasize catching those tail trades.
If you don't mind, can you post a distribution of trades for non-optimized and optimized results? That will likely tell you if you screwed something up... you'll likely see a less normal looking distribution with over-optimization.
By distribution I mean histogram with individual trade PnL on the x-axis and frequency on the y-axis.
Mike
Quote from bluedemon77:
Thanks for your response, Mike. No I don't think I ever used maximum drawdown to optimize, but I tried Recovery Factor. The problem I had with that is it tends to narrow the possibility of trades to a very small number, where the maximum drawdown was zero or one tick, something like that. I don't use exhaustive testing because I use a tick database and if you have more than one or two parameters it takes forever to optimize. I use CMA-ES to optimize.
I used to optimize based on net profit and then pick the best combination of parameters by comparing net profit, average profit per trade, profit factor, maximum drawdown and recovery factor. Then in the course of my research I found out that a lot of traders liked to use Sharpe or K-ratio, but those don't seem to work as well as UPI, which is what I'm using to optimize now. The UPI criterion makes sense to me because it seems to be a good measure of consistency, better than Sharpe or K-ratio.
Quote from Mike805:
FYI, Amibroker has CAR/MDD as one of its optimization selections.
When I develop a system (i.e. single concept), I look at the parameters you mention as a whole; they're all useful.
The next step is to incorporate that system into a portfolio of systems and that's where the CAR/MDD ratio becomes very useful. For example, two systems combined may produce a lower Profit factor (since one is meant to hedge the other), but, the overall CAR/MDD will be vastly improved... meaning one can scale up with *some* comfort.
Mind you, the systems have to be decent on their own right, but, this approach (Sys A + Sys B, where conceptually the systems oppose each other) allows for more consistent returns.
I guess what I'm getting at is that I don't optimize that much on single systems, I optimize for an individual systems' effect in a portfolio of systems.
Quote from ronblack:
+
I have to admit this post summarizes well what one will learn after reading 10 books on the subject.
Do you also optimize for optimum performance across multiple markets?
To me optimization is not the important issue. I'm worried when my combined systems fail to perfom well when applied to similar markets. For example, when a portfolio of systems with excellent forward test results in S&P 500 Index fails to perform well in Russell 2000 Index.