Confidence based on a couple O' points....

[amigasearch]

Ok. Question for systems guys:
(bear with my trouble in communicating the idea):

Lets say you develop a system, that relies on, a certain theory you have (dont we all).

When the system recognizes a signal, to buy or sell, at that point, nothing is done. Instead, you rely on a constant value AWAY from the generated signal point to enter the market.

Example: 1146 is the signal generated. BUT, nothing is done until price reaches, lets say, 2 points up or down (1148, buy, or sell, AND 1144 buy, or sell).

Fine. The system backtests very well, without any fitting - you just happen to like how 2 points works (the results are nice). BUT, adjust that 2 points - play with that constant value, and the system falls apart.

What would you make of this, and how to determine if this is / is not just a random number which happens to backtest well over 10 years data (nothing spectacular, just well?) Finally, would you trust it to trade it?
Thanks.

 

[ig0r]

Quote from amigasearch:

Ok. Question for systems guys:
(bear with my trouble in communicating the idea):

Lets say you develop a system, that relies on, a certain theory you have (dont we all).

When the system recognizes a signal, to buy or sell, at that point, nothing is done. Instead, you rely on a constant value AWAY from the generated signal point to enter the market.

Example: 1146 is the signal generated. BUT, nothing is done until price reaches, lets say, 2 points up or down (1148, buy, or sell, AND 1144 buy, or sell).

Fine. The system backtests very well, without any fitting - you just happen to like how 2 points works (the results are nice). BUT, adjust that 2 points - play with that constant value, and the system falls apart.

What would you make of this, and how to determine if this is / is not just a random number which happens to backtest well over 10 years data (nothing spectacular, just well?) Finally, would you trust it to trade it?
Thanks.

I no longer trade systems but I have had experience with them in the past, I would advise you not to trade it, even if you did not mean to chances are that it is fit to historical data. Look for values that are not outliers, values above and below the selected should not change the outcome of the system by much. Most successful systems traders will agree.

 

[Mathemagician]

Agreed. The best advice is to watch it for a few months to see what happens. That way you have nothing to lose. The market and the system will both still be there in six months. By deploying only systems you are sure of you can ensure that your account will be there, too.

jj

 

[kserra]

Have you considered adjusting the value of 2 points accordingly with the volatility of the average daily range ?

If so, how does this affect the system ?

 

[harrytrader]

Quote from amigasearch:



Lets say you develop a system, that relies on, a certain theory you have (dont we all).

Not all theories are equivalent, if your theory is to use stochastic indicators you will have the same problem than with classical time series so that backtest will reflect the past and not the future so nothing astonishing that backtest is not reliable then.

http://www.elitetrader.com/vb/showthread.php?s=&threadid=28808

 

[amigasearch]

Kserra:
This is a good idea (Range). I will try it, and see if it has a positve effect.

 

[WDGann]

Quote from amigasearch:

Ok. Question for systems guys:
(bear with my trouble in communicating the idea):

Lets say you develop a system, that relies on, a certain theory you have (dont we all).

When the system recognizes a signal, to buy or sell, at that point, nothing is done. Instead, you rely on a constant value AWAY from the generated signal point to enter the market.

Example: 1146 is the signal generated. BUT, nothing is done until price reaches, lets say, 2 points up or down (1148, buy, or sell, AND 1144 buy, or sell).

Fine. The system backtests very well, without any fitting - you just happen to like how 2 points works (the results are nice). BUT, adjust that 2 points - play with that constant value, and the system falls apart.

What would you make of this, and how to determine if this is / is not just a random number which happens to backtest well over 10 years data (nothing spectacular, just well?) Finally, would you trust it to trade it?
Thanks.

It all depends on the system itself.

Timeframe, style(swing, scalp, etc.), markets traded (single / multiple), money management rules, linear/non-linear, etc. etc.

One thing I can add is if it's a non-linear / robust (non-curve fitting edge system) some linear stuff like using volatility is fine... if it's not then I wouldn't add another linear criteria.

 

[Mathemagician]

Looks like you've attracted a bunch of very impressive "I don't knows" from the academic crowd and one good suggestion (from kserra)!

Bottom line is that your best bet is to sit on the system for a few months to see how it goes. Usually when they break down, it's not subtle. It's obvious. Conversely, when they work, they just keep on working. That's readily apparent, too. Better to find out on paper than in your account.

jj

 

[amigasearch]

Yes - sounds more like everyone is leaning toward hesitance, due to the reliance on a number. Maybe the range function will make it more agreeable. I will let you know.

 

[amigasearch]

John - as far as watching it - it tests well over 10 years. The backtest results instill alot of confidence.