Trading Systems Robustness & Market Structure

Bill,

There is an even simpler way to resolve the opinions on Optimization. It's called OOS or Out of Sample testing.

To do this:

1) take a set of market data and create the best system you can, with 2 rules or 20 rules. Make it simple, make it complex, whatever.

2) Freeze the rules and logic.

3) Apply you new system to a new data set that you have not seen in step 1 and 2.

4) If you had a winner in step 1 but now you have a system not worth trading or even a looser, then you system is not valid. It is over optimized or curve fitted.

However if your system still works, then take another OSS period of even longer length and test. IF you system continues to work then you have created something that is very likely to work in real trading regardless of the simplicity or complexity of the system.

The process above is the basis of neural networks and genetic algorithms and these tools have created some very profitable trading systems.

Jerry


--------------------------------------------------------------------------------
Quote from Donchian:

Curtis does a thorough explanation of this paradox in his book Way of the Turtle, chapters 11-13. You may read an extract at: http://www.tradingblox.com/tradingb...ion_paradox.htm
--------------------------------------------------------------------------------



The paradox is only in his mind. There is no paradox. Optimized systems are fitted for vest past performance. Past performance is no guarantee of futures results I heard. Simple.

Paradox is an apparent contradiction that is resolved when one of the false premises is removed and replaced by a true one. In this case, we have a true conclusion, no paradox.

Bill
 
Quote from OddTrader:

I think you've already answered your own question. :p
More...

Actually I'm still puzzled because there were around 9000 nonoptimized trades on in sample data and my out of sample trades were not significantly different than the in sample. The strategy itself makes sense and is very simple.

The timeframes that I'm concerned about were in the past and maybe due to changes in the interworkings in the market. The thing that bothers me is that I can't understand why and the more I try to, the more likely it'll be biased.

I was reading a post by Acrary mentioning random testing for the same data period for the same duration of trade length and then using chi square for determining significant difference. It seemed to me he was relating it to preventing against overfitting, but this appears to be more useful for significane testing with regard to the market character (sorry I lost his post). If anybody has any feedback, I'd appreciate it.

Jeff
 
Quote from CPTrader:

Trading Systems Robustness & Market Structure

It is commonly agreed that the measures of a robust trading system are:
More...

A system would be fairly common and generate some common results when merely based on commonly agreed/ found stuff.
 
Quote from CPTrader:

Trading Systems Robustness & Market Structure


It is commonly agreed that the measures of a robust trading system are:

1. Common sense and valid underlying logic.

2. Identical parameters, rules for all markets

3. Consistent profitable performance across multiple markets in multiple market complexes

4. Simple rules



However, I wonder in addition to the above, what role “Market Structure” as defined by:

a. volatility,
b. liquidity of specific markets,
c. format of trading (pit vs electronic),
d. continuity of trading (24 hr markets vs daytime only markets),
e. type of market participants,
f. number of market participants,
g. market volume

can affect a trading system and how this impacts on system robustness.

So for example can one I have a system that is seemingly “robust” and profitable in one time zone and less profitable in another time zone due to market structure issues.

So for example if a trading system were to show good results say between 2000 – date, but a mix of fair, good and poor results between say 1987 – 1999, should one deduce from this that :

a. the system is fully robust and the less consistent performance in the period 1987 – 1999 (note I am not saying the performance was bad, just not as good) is due to a different market structure in the 1987 -1999 window. The 1987 – 1999 window had less markets available for trading, less liquid markets, fewer market participants, more discontinuous markets i.e. no 24 hour trading) or


b. the system showed less inconsistent performance in the 1987 – 1999 window due to inherent system weaknesses; in which case one should expect that the period going forward could also show less consistent performance.


Is there any way to even scientifically confirm which hypothesis from the two above is valid.

I hope my thoughts will provide the basic for a hearty and intelligent dialogue on this important issue.

Many thanks.
More...
 

Attachments

I keep on repeating the same ol' thing but:

Understand your "edge". Depending on the edge you are using, how you test (optimize and etc.) and how you set the risk management criteria changes.

To make things easier, let's say I found the specific edge:

Let's say you exposed a specific tendency in the equity market like... "All stocks that hit the 'Daily Bar' 20-day moving average on the 3rd Friday of the month, bounces off the MA." After some research you, understand that funds liquidate their position on the 3rd Friday and they use 20-day MA.

(Note that this is just an example)

I test the system with a code like below under intraday data:
If (DayofWeek == "Friday" && WeekofMonth == '3' and Price == Average(DailyDataClose, 20)
{
Trade();
// Trade checks if it's bouncing below or above and deciding whether to buy/sell
}

ExitAllPositionAtEndofDay();


Running the above on all symbol universe under:
- 30 years of data.
- Set contract size.
- All statistic values are calculated within the symbol. (eg. AvgTrade or RiskRewardR. will not be cross referenced with other symbols)

Total Profitability = 70%.
Average Risk/Reward Ratio: 3.4
Average Sharpe: 1.9

Now... what would you optimize? and based on what fitness?

Would you do it the same way as an trend-following model? Scalping/Market Maker? Would Out-sample... Walk-Forward be significant (relatively high Sharpe).
 
You must log in or register to reply here.
Back
Top