Do not post please, It is Only For Me
- Thread starter mcgene4xpro
- Start date
In all the books I have read about trading, I don't think I have ever read a truer statement than this from MB:
"Maybe markets appear to have a directional bias. But really they do not, all markets are in a transition to surprising the person who thinks he has the direction figured out."
D.C.
"Maybe markets appear to have a directional bias. But really they do not, all markets are in a transition to surprising the person who thinks he has the direction figured out."
D.C.
No truer words were spoken. This is exactly why a system trader must be completely comfortable with a system before trading. The only way I know how to do this is to get all the numbers and historical data before trading. I believe all to often traders don't do this and that is why they bail at the wrong time.
Regards,
Michael
-----Original Message-----
From: Neal Hughes [mailto:tradero@xxxxxxxxxxxxx]
Sent: Thursday, December 12, 2002 5:25 PM
To: omega-list@xxxxxxxxxx
Subject: Re: Limited life span of mechanical systems?
Hi Traders,
It is normal for any mechanical system to "stop working" at times. Actually it has not stopped working. It is working as designed.
A mechanical system may experience flat, losing, or winning streaks at any time, if it was designed that way.
If the system is back-tested on market data which is similar to our recent history, it would most likely provide flat results. The system needs different market conditions to perform better.
The challenge with any system is to stick with it. Even during periods of draw-down, if the system works you must stick with it. This is a common problem with traders. It's not so much the system that stops working, it's the trader that stops working with the system. Too often, traders quit using a system at exactly the worst time. Similar to gaming in Las Vegas, quit when you are winning, rather than losing.. But this is counter to human nature. Trading is counter to human nature in so many ways.
System trading can be very emotional, just ask all those traders who have abandoned systems, or skipped some of the signals during losing streaks. Or those who decided they need to modify/re-assess a system which seemed perfectly good before.
Best wishes,
-Neal T. Weintraub
Regards,
Michael
-----Original Message-----
From: Neal Hughes [mailto:tradero@xxxxxxxxxxxxx]
Sent: Thursday, December 12, 2002 5:25 PM
To: omega-list@xxxxxxxxxx
Subject: Re: Limited life span of mechanical systems?
Hi Traders,
It is normal for any mechanical system to "stop working" at times. Actually it has not stopped working. It is working as designed.
A mechanical system may experience flat, losing, or winning streaks at any time, if it was designed that way.
If the system is back-tested on market data which is similar to our recent history, it would most likely provide flat results. The system needs different market conditions to perform better.
The challenge with any system is to stick with it. Even during periods of draw-down, if the system works you must stick with it. This is a common problem with traders. It's not so much the system that stops working, it's the trader that stops working with the system. Too often, traders quit using a system at exactly the worst time. Similar to gaming in Las Vegas, quit when you are winning, rather than losing.. But this is counter to human nature. Trading is counter to human nature in so many ways.
System trading can be very emotional, just ask all those traders who have abandoned systems, or skipped some of the signals during losing streaks. Or those who decided they need to modify/re-assess a system which seemed perfectly good before.
Best wishes,
-Neal T. Weintraub
You have three devils against you. We all talk about the first two,
but permit me to really harp on the third:
1) A random series that pretends to be non-random. You can fight this
by only accepting systems that are robust across many different
markets, as well as performing sizable forward tests.
Equity curve-fitting is a problem because a random series can
masquerade as a sexy equity curve. Then, for a number of months of
forward testing, it could even perform a random walk that stays within
the standard error bands, giving the impression that the system is
true.
It's important to forward-test systems to provide confidence that the
strategy is in fact reflecting something non-random about the markets.
The problem is that the forward-test itself is statistically
insignificant.
So you decide to trade it anyway, telling yourself that you will stop
or lighten up if the equity curve crashes through the standard error.
You watch it and see what it does next, and compare this movement with
the movement in the past.
2) You have a system that works, passes the test above, and has worked
for some time. Then the market changes because times are changing.
Your trading has to change too. The good news is that when the market
begins to play different tunes, it combines it with the old ones, to
give you a chance to change before you are wiped out.
3) You have a system that works, passes the test above, and has worked
for some time. Then the market changes because it is responding to
YOUR system. This is why I stay clear away from any strategy that is
talked about. I don't understand why people think that you can just
grab a publicized system, start trading it, and expect an easy road to
riches.
Added to this, don't fall for the fallacy that your trades are
anonymous. I highly doubt it. It's all too tempting for the
exchanges/brokers to process data that contains not only what and when
markets are traded, but WHO is trading what and when. It doesn't take
much to discover who is making a lot of money. If they can't figure
out what you are doing directly, they can at least run computer
algorithms that reverse-engineer your trading patterns. This opens the
door for front-running, intraday price running, and fading
opportunities.
Indeed, there are other successful people, too. Their systems may be
different, but perhaps they are the same because we all hear the same
stuff. Either way, the people who know WHO is doing what and how
successful he is, have an enormous advantage.
Jessie Livermore knew this very well, and had to go to great lengths
to deal with this problem. He had several different accounts under
several different aliases executing only parts of his grand plan. Yes,
the market was a small town back then, but cheap powerful computers
and well-paid programmers have changed the market community, and have
made it small town once again.
This concern may be overkill for the trader who makes a decent living,
verses someone who discovered something more. But be aware that your
systems may be very similar to the ones several other traders use. The
sum of 100 successful traders doing the same thing = a crowd that
makes a lot of noise. So the system becomes the Big Shot that insiders
notice.
This is a tough game. Mechanical systems must be adaptive and clever.
If what you have is hot as hell, cloak yourself and your system.
From The Omega List
http://www.purebytes.com/archives/omega/2002/thrd7.html#09640
but permit me to really harp on the third:
1) A random series that pretends to be non-random. You can fight this
by only accepting systems that are robust across many different
markets, as well as performing sizable forward tests.
Equity curve-fitting is a problem because a random series can
masquerade as a sexy equity curve. Then, for a number of months of
forward testing, it could even perform a random walk that stays within
the standard error bands, giving the impression that the system is
true.
It's important to forward-test systems to provide confidence that the
strategy is in fact reflecting something non-random about the markets.
The problem is that the forward-test itself is statistically
insignificant.
So you decide to trade it anyway, telling yourself that you will stop
or lighten up if the equity curve crashes through the standard error.
You watch it and see what it does next, and compare this movement with
the movement in the past.
2) You have a system that works, passes the test above, and has worked
for some time. Then the market changes because times are changing.
Your trading has to change too. The good news is that when the market
begins to play different tunes, it combines it with the old ones, to
give you a chance to change before you are wiped out.
3) You have a system that works, passes the test above, and has worked
for some time. Then the market changes because it is responding to
YOUR system. This is why I stay clear away from any strategy that is
talked about. I don't understand why people think that you can just
grab a publicized system, start trading it, and expect an easy road to
riches.
Added to this, don't fall for the fallacy that your trades are
anonymous. I highly doubt it. It's all too tempting for the
exchanges/brokers to process data that contains not only what and when
markets are traded, but WHO is trading what and when. It doesn't take
much to discover who is making a lot of money. If they can't figure
out what you are doing directly, they can at least run computer
algorithms that reverse-engineer your trading patterns. This opens the
door for front-running, intraday price running, and fading
opportunities.
Indeed, there are other successful people, too. Their systems may be
different, but perhaps they are the same because we all hear the same
stuff. Either way, the people who know WHO is doing what and how
successful he is, have an enormous advantage.
Jessie Livermore knew this very well, and had to go to great lengths
to deal with this problem. He had several different accounts under
several different aliases executing only parts of his grand plan. Yes,
the market was a small town back then, but cheap powerful computers
and well-paid programmers have changed the market community, and have
made it small town once again.
This concern may be overkill for the trader who makes a decent living,
verses someone who discovered something more. But be aware that your
systems may be very similar to the ones several other traders use. The
sum of 100 successful traders doing the same thing = a crowd that
makes a lot of noise. So the system becomes the Big Shot that insiders
notice.
This is a tough game. Mechanical systems must be adaptive and clever.
If what you have is hot as hell, cloak yourself and your system.
From The Omega List
http://www.purebytes.com/archives/omega/2002/thrd7.html#09640
vita
Registered: Jul 2008
Posts: 103
03-24-09 05:16 PM
Quote from jacksmith:
One common story I had heard about trading system is that,
Successfully backtest for 3 year's data, yet the system blows up in 3 days,
what causes this ?
Thanks.
IMO, the most robust way to backtest a systematic strategy is to do the following. I will give a simple example later to clarify.
1) use a historical data set that is long enough to cover multiple periods of market regimes including bullish, bearish and ranging.
2) select "Training Period (TP)" which is a subset of the data where the market exhibited regime changes. This training period is a rolling subset of data that is used to optimize you system parameters.
3) select an "Out-of-Sample Period (OOSP)" which is also a rolling subset of data. This period is used to backtest using the optimized parameters from step 2.
4) to find parameter, you need to start with the first TP and optimize your parameters. The objective function to maximize can be the cumulative return, the Sharpe Ratio, etc. I personally use the product of the two.
5) to back test, use the optimized parameters from step 4 to trade in OOSP which should start right after the last data used in TP. Log the trading results in OOSP.
6) to complete backtest, you need to repeat steps 4 & 5 after rolling TP subset forward by OOSP.
Here is a simple example:
Lets assume you have ~10 years of daily historical data starting from Jan 2000. Lets also assume that the Training Period (TP) and the Out-of-Sample Period (OOSP) are 3M and 1M respectively. First you start with the first 3M data i.e. Jan-Mar 2000 and optimize/calibrate the parameters to maximize the performance of your model. Then utilize these parameters to trade the month of April (without any further calibration!!!). Log your trading performance for April. Next you need to roll forward your TP by OOSP, i.e., use Feb-April as the new TP to optimize/calibrate parameters and use the new parameters to trade and log the month of May. This procedure requires you to optimize/calibrate 12 times for every year of data and trade only out-of-sample without calibration. The final backtest is now reflected in the log of all your out-of-sample trades. By following this, you avoid curve fitting.
One final check is to repeat the entire procedure but using a different start date. For example choosing your first TP from Jan 15 to April 15 instead. This is essentially like rolling your historical data set forward or backward to make sure your trading strategy is robust irrespective of when your start trading.
I hope this helps.
Registered: Jul 2008
Posts: 103
03-24-09 05:16 PM
Quote from jacksmith:
One common story I had heard about trading system is that,
Successfully backtest for 3 year's data, yet the system blows up in 3 days,
what causes this ?
Thanks.
IMO, the most robust way to backtest a systematic strategy is to do the following. I will give a simple example later to clarify.
1) use a historical data set that is long enough to cover multiple periods of market regimes including bullish, bearish and ranging.
2) select "Training Period (TP)" which is a subset of the data where the market exhibited regime changes. This training period is a rolling subset of data that is used to optimize you system parameters.
3) select an "Out-of-Sample Period (OOSP)" which is also a rolling subset of data. This period is used to backtest using the optimized parameters from step 2.
4) to find parameter, you need to start with the first TP and optimize your parameters. The objective function to maximize can be the cumulative return, the Sharpe Ratio, etc. I personally use the product of the two.
5) to back test, use the optimized parameters from step 4 to trade in OOSP which should start right after the last data used in TP. Log the trading results in OOSP.
6) to complete backtest, you need to repeat steps 4 & 5 after rolling TP subset forward by OOSP.
Here is a simple example:
Lets assume you have ~10 years of daily historical data starting from Jan 2000. Lets also assume that the Training Period (TP) and the Out-of-Sample Period (OOSP) are 3M and 1M respectively. First you start with the first 3M data i.e. Jan-Mar 2000 and optimize/calibrate the parameters to maximize the performance of your model. Then utilize these parameters to trade the month of April (without any further calibration!!!). Log your trading performance for April. Next you need to roll forward your TP by OOSP, i.e., use Feb-April as the new TP to optimize/calibrate parameters and use the new parameters to trade and log the month of May. This procedure requires you to optimize/calibrate 12 times for every year of data and trade only out-of-sample without calibration. The final backtest is now reflected in the log of all your out-of-sample trades. By following this, you avoid curve fitting.
One final check is to repeat the entire procedure but using a different start date. For example choosing your first TP from Jan 15 to April 15 instead. This is essentially like rolling your historical data set forward or backward to make sure your trading strategy is robust irrespective of when your start trading.
I hope this helps.