You are using an out of date browser. It may not display this or other websites correctly.
You should upgrade or use an alternative browser.
You should upgrade or use an alternative browser.
Activate/Deactivate System?
- Thread starter EricP
- Start date
Quote from EricP:
First, before I forget, let me address one of the biggest mistakes people make when developing a system: optimization.
Typical traders will take a program like TradeStation or WealthLab or the equivalent, and create a spectacular trading system with clever and creative ideas, which has a near-linear P&L curve, and with very few minor drawdowns... They are excited and can't wait to start making the 'easy money' with live trading.
Will they actually make money with their new methods? Actually, we don't have a clue, as they've never tested whether their new method will make money in the future on not. All they have actually done is prove that their new system does any excellent job of being optimized to fit the past data, which in and of itself, is worthless.
How do we improve upon this? Test your system in the future! Hmmm... the future, you ask, how would we do that?
Let's assume that you are doing your backtesting on 5 years of data, and the system generates 150 trades per year, for a total of 750 trades. In order to tell how well the system performs in the future, you must withhold all of your data from the optimization tests. Instead, for example, only optimize on the first two years of data (300 trades, pretty decent number). Then, use your optimized system to now test on the 'future' => i.e. test year 3 of data with the system optimized on years 1 and 2.
Next, reoptimize your system on the data from years 2 and 3, and give it a future test with the data from year 4. Similarly, optimize your system with the data from year 3 and 4, giving it a 'real-time' test against the data from year 5. In this way, you can see how your system will perform in the 'future' based upon whatever optimization methods you are using.
This sort of method should give you increased confidence when you go to live activation that it should work well in the 'future'.
-Eric
But then the question becomes: what to do when the out-of-sample results look bad?
Many people will feel tempted to redesign the trading system, and repeat the procedure, until they discover a system that worked well in both the in-sample and out-of-sample period. Needless to say, such a system is over-optimized as well and will likely disappoint when it goes live.
Quote from NoWorries:
I use mean-variance optimization to determine the weights of the systems in my portfolio. Because I reoptimize regularly, systems whose performance starts to deteriorate are "gradually" stopped out, i.e. their weight will diminish over time, and may reach zero eventually in case they haven't offered anything in terms of reducing the variance of the returns or improving the profitability of the portfolio.
No Worries,
You are using a standard Markowtiz approach (Efficient Frontier)? And what frequency or events trigger a re-opt?
Thanks,
bolter
Quote from EricP:
As I probably mentioned earlier in the thread, I prefer to have at least 30 trades before using the formula for statistical significance. However, to answer your question => Yes, based upon your data, I would certainly be happy to activate the system you describe. Obviously, going forward, the results might improve or deteriorate, but my activation decision must be made with the information we currently have at hand.
Just to be clear what the formula is saying. Let's assume that a certain trading strategy makes 5000 trades over the course of two weeks (we'll call this the entire 'population' of trades that we have). Now, let's write the results of each of the 5000 trades on a piece of paper and put them in a bucket. Next, we will draw out 18 random pieces of paper from the bucket and do the calculation above, based on the results of those 18 trades. The formula tells you that there is a ~98% chance that the entire population of 5000 trades has an average profit above zero (i.e. net profitable). That's all it tells you. But, I find this to be extremely useful in providing an unbiased foundation for making activation/deactivation decisions.
Best of luck,
-Eric
P.S. I've now 'subscribed' to this thread, so that I will be able to see when future posts are made and will hopefully do a better job of replying to questions in the future. (this is the first thread I've subscribed to, so hopefully it will work). Also, if I fail to address anyone's that posts to me on this thread in the future, please feel free to send me a PM and point out the post. Thanks.
Eric,
How many systems in how many markets are you now running?
Michael
Quote from bolter:
No Worries,
You are using a standard Markowtiz approach (Efficient Frontier)? And what frequency or events trigger a re-opt?
Thanks,
bolter
Yes, I use standard Markowitz (quadratic programming) and reoptimize monthly. This is quite arbitrary and I wouldn't really know how to estimate the "optimal reoptimization interval", as it will obviously suffer from overoptimization if I would base it on a historical simulation. Because the number of systems is usually reasonably small (4-6), I sometimes also run a brute-force optimization with some other output parameters of interest (e.g. mean-VaR) just to check.
Quote from robinxing:
thank you Eric
another question
in your formula
x = abs(Avg-Profit)*(Number of Trades)^0.5 / (Std Dev of Profits)
my question is what is the best number of trades for an intraday
trading system whose frequency is about 100 trades in one month
Unfortunately, there is no 'best' number. You have to pick a number by making a balance judgment. You are balancing the increased accuracy of a larger number, with the increased 'response time' of a smaller number (i.e. when you use a relatively smaller number, your confidence value will more quickly change with respect to recent trading performance).
If it were me, and I was testing a known successful system on a new market (i.e. you have already traded the system, and know it works for you, but now trying it in corn instead of T-bonds, for example), then I might use a starting value of 30-40 trades to draw initial conclusions, and increase the number of trades used until you get to ~100 or 120. Once at ~120 or more trades in your log, then I would just use the most recent 120 for future calculations.
Again, there is no right answer, and you have to make this sort of judgment based on your own preferences.
Quote from NoWorries:
But then the question becomes: what to do when the out-of-sample results look bad?
Many people will feel tempted to redesign the trading system, and repeat the procedure, until they discover a system that worked well in both the in-sample and out-of-sample period. Needless to say, such a system is over-optimized as well and will likely disappoint when it goes live.
When the out of sample results look bad, you have now had the benefit of discovering this important fact with paper trading, and not with real money, avoiding potentially substantial losses as a result. Granted, you go back to the drawing board in trying to develop a good system, but at least you've done it for 'free'.
Hopefully, if you come across some good ideas, then one of your early efforts will show promise in the out of sample tests. However, you are right in that a trader that merely takes a shotgun approach and tests dozens of systems in out of sample testing is likely to find one (worthless system) that gets 'lucky' and tests reasonably well in out of sample testing. In that case, the trader might be inclined to activate the system using real money, and will lose money during the live test. However, the benefit remains that the trader may have only lost real money on one system, and not the dozen systems that he may have thrown money at.