Help me be a pessimist about my automated system

Quote from segv:

If it looks to good to be true, it IS too good to be true. Have you followed a scientific process? I suspect that you have not, given that you are posting on Elitetrader in an attempt to understand and validate your results.

1. Observe, Classify, and Define. Can you describe theoretically how your system is supposed to generate profits within the constraints of the marketplace (fair pricing, fair value, efficient market)? Is it arbitrage? Is it liquidity provision? Does it depend on the microstructure of the market in question? Can you classify the phenomena on which the system depends and track them discretely? Does the system use a weakness of some other theoretical framework?

2. Form a Hypothesis. Use your observations, theoretical reasoning, and phenomena to develop logic that can be tested empirically. Define the relationships. Does it predict or correlate? Does A cause B? If A changes does B change?

3. Test the Hypothesis Emperically. Define the test criteria, and identify the data that is to be collected. Think out of the box and create "stress tests" for your hypothesis. Intentionally invalidate your assumptions and observe the result. Is your data valid? Are there any outlier trades that are creating unusual results? Test the system with out-of-sample data, then with random data. Does it behave as you would have predicted? Why or why not?
...
More...
Excellent.
 
Quote from segv:


1. Insufficient Data. Three months of intraday data is simply not enough samples. You have already "curve fit" by having such a small sample.

2. Improper Testing. In your post, you said that you tested the same 3 month interval with different periodicity. This is not sufficient, you need to perform "walk forward" testing. That means taking a random sample of 3 months from 3 or more years ago, then testing each 3 month interval thereafter. You have confirmed only that your rules "work" during the period in question.

3. Invalid Assumptions. Your slippage is far too small. You have to account for computer, software, network, exchange, and human error somewhere. Using 5 or more ticks per trade will give you a better "stress case".

4. Data Validation. Is there an abnormal pattern of returns somewhere in your sample? Huge price swings or trades? Are all of your trades the right size?

5. Stress Testing. Permute the variables that are inputs to your testing. Does what you thought would happen actually happen?
More...


Good points.

One more. Bear in mind, you're testing on a basket of *highly* correlated pairs.

The effect? Compounded returns when the market is churning $$$. But drawdowns will be just as spectacular when the market stalls.

Test the system on two correlated pairs -- max. Anything more and you're courting disaster.

For example, I only trade the pound and euro (correlated), or pound and yen (uncorrelated).

Best of luck.
 
When dealing with timeframes of less than even 300 minute bars (you are at 90 minutes), I have found that simply subtracting a few pips for 'slippage' is not sufficient.

My shorter term strategies all use limit and stop orders. These get filled with at times only a few pips to spare... if the market moved only a couple of pips less, I would not get a fill. Or, if my TradeStation data indicates a fill - many I bet would not have been filled in a real market environment.

Playing with real money answers many of these questions.

Good luck,
Granville
 
Quote from segv:

If it looks to good to be true, it IS too good to be true. Have you followed a scientific process? I suspect that you have not, given that you are posting on Elitetrader in an attempt to understand and validate your results.

1. Observe, Classify, and Define. Can you describe theoretically how your system is supposed to generate profits within the constraints of the marketplace (fair pricing, fair value, efficient market)? Is it arbitrage? Is it liquidity provision? Does it depend on the microstructure of the market in question? Can you classify the phenomena on which the system depends and track them discretely? Does the system use a weakness of some other theoretical framework?

2. Form a Hypothesis. Use your observations, theoretical reasoning, and phenomena to develop logic that can be tested empirically. Define the relationships. Does it predict or correlate? Does A cause B? If A changes does B change?

3. Test the Hypothesis Emperically. Define the test criteria, and identify the data that is to be collected. Think out of the box and create "stress tests" for your hypothesis. Intentionally invalidate your assumptions and observe the result. Is your data valid? Are there any outlier trades that are creating unusual results? Test the system with out-of-sample data, then with random data. Does it behave as you would have predicted? Why or why not?

In your post, you spoke only of the testing process, or Step 3 above. There are several problems with your testing thus far:

1. Insufficient Data. Three months of intraday data is simply not enough samples. You have already "curve fit" by having such a small sample.

2. Improper Testing. In your post, you said that you tested the same 3 month interval with different periodicity. This is not sufficient, you need to perform "walk forward" testing. That means taking a random sample of 3 months from 3 or more years ago, then testing each 3 month interval thereafter. You have confirmed only that your rules "work" during the period in question.

3. Invalid Assumptions. Your slippage is far too small. You have to account for computer, software, network, exchange, and human error somewhere. Using 5 or more ticks per trade will give you a better "stress case".

4. Data Validation. Is there an abnormal pattern of returns somewhere in your sample? Huge price swings or trades? Are all of your trades the right size?

5. Stress Testing. Permute the variables that are inputs to your testing. Does what you thought would happen actually happen?

I will say it again, just to make sure you heard me. If it looks to good to be true, it IS too good to be true. Just ask an option trader.


-segv
More...

ditto...

I have built a few winning systems using neoticker as well...

the biggest thing for me was...the amount of data...like was mentioned 3 months is nothing...especially on 90 min bars...you will probably be disappointed when you run it on 3 yrs...they say you should have a system that can last through 30 trades of backtesting...I think that is a joke...get a system that runs clean for about 300 trades (on a long time frame) and you'll have something...but still...your system is built on past data...and all indicator built systems tend to enter markets late...in other words...90 min bars is a intra-day timeframe and more than likely your system does not do value comparisons from day to day
 
My systems (they are end-of-week) are backtested to the 1950s. You need to have hundreds of trades and many market cycles included in your data to have any confidence.
 
You may have configured your system to find the same profitable trades, over and over and over again.

Perhaps these conditions will never occur again.

I've just recently begain to study price action (equities), I built a database and continually fine tuned my queries to maximize profits. Man, I envisioned my jet parked on the runaway, after I optimized to the point of "astronimical returns".

Then it hit me, ...
 
Achilles 28
You will need to forward test your system
as your next step.

Providing it confirms your backtesting results then you can open an account and began trading with YOUR money.

When your account balance reaches zero you should have some idea of the factors that are applied to Fx trading that your system failed to take into account and you will be both poorer and hopefully wiser for the experience.
Goodluck
 
Quote from hardyards:

Achilles 28
You will need to forward test your system
as your next step.

Providing it confirms your backtesting results then you can open an account and began trading with YOUR money.

When your account balance reaches zero you should have some idea of the factors that are applied to Fx trading that your system failed to take into account and you will be both poorer and hopefully wiser for the experience.
Goodluck
More...


uh.... what makes you think you know anything about me, my system or testing?

I think you've got me confused with the original poster.

take care.
 
You must log in or register to reply here.
Back
Top