if backtesting works, what could go wrong in real trade?

The primary problem with back-testing is that you may see something that works, but it may only "seem" to work.

Unless you are able to answer the WHY as per your hypothesis, then it is just a coincidence.

Don't look at the data to TRY and find correlation - that will only end in ruin.


You have to have a hypothesis prior to making any test.

Simple example:
1. I believe that X meets X objective when X happens.
2. I believe this happens because of X force.
3. I believe that this event has a probability of x%


Now test for it.

The majority of the back-testers I run across are blindly searching for a connection. They see patterns in everything. That only works if you are able to answer the WHY? Otherwise you are putting the cart in front of the horse.

Also -

If you are not able to clearly answer your expectations of your strategy (risk/reward) than any back-testing is pretty useless. Back-testing should verify a hypothesis and check for statistical results. It should not be used to justify the risk/reward expectation of the strategy. Many forget this simple yet important difference.


I also believe that forward (live) testing is as important. Using a sim-route for execution and keep forward logs will check for execution risk, liquidity, etc.
 
Quote from lionfish42:

The primary problem with back-testing is that you may see something that works, but it may only "seem" to work.

Unless you are able to answer the WHY as per your hypothesis, then it is just a coincidence.

Don't look at the data to TRY and find correlation - that will only end in ruin.


You have to have a hypothesis prior to making any test.

Simple example:
1. I believe that X meets X objective when X happens.
2. I believe this happens because of X force.
3. I believe that this event has a probability of x%


Now test for it.

The majority of the back-testers I run across are blindly searching for a connection. They see patterns in everything. That only works if you are able to answer the WHY? Otherwise you are putting the cart in front of the horse.

Also -

If you are not able to clearly answer your expectations of your strategy (risk/reward) than any back-testing is pretty useless. Back-testing should verify a hypothesis and check for statistical results. It should not be used to justify the risk/reward expectation of the strategy. Many forget this simple yet important difference.


I also believe that forward (live) testing is as important. Using a sim-route for execution and keep forward logs will check for execution risk, liquidity, etc.
More...
Wisdom indeed,
congrats, great post
 
Quote from Gcapman:

Cool-calm-collected demeanor gets tossed out the window when this month's/year's mortgage/car payment is at risk...
More...

While that is certainly good comedy, you should not be trading the rent in the first place.
 
1. Real world slippage costs
2. Backtest finds a strategy that is profitable in market condition A. You then trade it under market conditions not-A
3. Competition - a former edge gets discovered by other market players, and competed away.
4. Curve-fitting - there was no causation at all, it was just chance correlation.
 
Quote from lionfish42:


Also -

If you are not able to clearly answer your expectations of your strategy (risk/reward) than any back-testing is pretty useless. Back-testing should verify a hypothesis and check for statistical results. It should not be used to justify the risk/reward expectation of the strategy. Many forget this simple yet important difference.


I also believe that forward (live) testing is as important. Using a sim-route for execution and keep forward logs will check for execution risk, liquidity, etc.
More...

thank you. this makes sense to me. How can you tell what's your your expectations of your strategy (risk/reward) w/o testing it? I might have some idea that the strategy might have over 50% winning odds, but that is far from enough because you have to overcome trading cost.
 
Quote from traderzhangSan:

thank you. this makes sense to me. How can you tell what's your your expectations of your strategy (risk/reward) w/o testing it? I might have some idea that the strategy might have over 50% winning odds, but that is far from enough because you have to overcome trading cost.
More...


A strategy (regardless of probability) should have a risk/reward ratio.

Example:

Covered Call -

1. You can calculate static yield
2. You can calculate maximum gain
3. You can calculate 1SD RISK/REWARD values
4. You can calculate expected return

You know have a set of data with expectations.
You can draw a general conclusion
You draft your hypothesis

Now you back test.

Back testing - should show several things.

1. Statistical win/loss %
2. Volatility vs. expected return.
3. Max draw down expectations and are they with in your hypthesis
4. Max gains hit.
5. Opportunity costs
6. Real Price vs. Volatility change

and a host of other data.

Then you forward test (or Live test).
This will test for

1. Execution Risk
2. Liquidity Risk
3. Net Fee impact

etc...



Whether it is a stock, futures, or options strategy - you should know what your expectations are prior to back-testing.
 
Quote from traderzhangSan:

thank you. this makes sense to me. How can you tell what's your your expectations of your strategy (risk/reward) w/o testing it? I might have some idea that the strategy might have over 50% winning odds, but that is far from enough because you have to overcome trading cost.
More...

This article explains well I think. You may need much higher winning odds in reality than what your system achieved during backtesting due to increased volatility in the case of swing and intraday trading or due to smaller size trends in the case of trend following.

This is pretty basic stuff.
 
BTW:

You can have two methods of winning.


You could be right only 40% of the time , but have a 3:1 payout.

Or

You could be right 60%, but only have a 1:1 payout.



You need to know what your strategy expectation is, prior to figuring out probability.


As they say in poker it is all about the pot odds.
 
One of the things that could go wrong in the real trade is overtrading. Of course, if backtesting works, it is purely in theory and done entirely in your own computer.

If you do real trading, you need to pay commissions to the broker, if you did not factor in the commission costs during back testing, then the commission alone can substantially eat your profits in the long run provided you over trade.

One of the things that could go wrong is your discipline, of course it looks good in paper/backtesting but when you trade real money, things can be very different because emotions is attached to trade. And if you are the kind of trader who is emotional and not so discipline then you can violate your own trading rules because you are afraid of losing money despite losses being normal.

There is no emotions attached during backtesting but there will be in real trading.
 
You must log in or register to reply here.
Back
Top