Why Optimization Won't Kill My System and Will It Work In The Future?

Andy,

I wanted to elaborate a bit on the system that you are looking at. As you probably know, moving average systems such as this are trend following. When you catch a trend, the profits just keep rolling in.

These can be viable systems, but it's good to understand them before you put you money on the line.

--------(For anyone reading this posting: I neither endorse nor disclaim the following system, all results are based on simulated backtesting of actual contract data. Everything below is stated for informational and demonstration purposes only.)---------

So as an example, I've written a system to backtest in Metastock on the NQ (NASDAQ Mini's). I think you mentioned 1 minute charts, but I don't remember for certain. For this example, I'll use it anyway. I've chosen a 5 minute simple moving average compared with a 30 minute simple moving average. No overnight positions and I'm trading it only between 10 am and 4pm. All positions are entered at the open of the bar following the signal.

The only stops I'm using are at the end of the trading day. All other exits are based on a reversal of the moving averages, so you remain in the market all day, either long or short.

For an out of sample backtest, I'll use the NAZ Mini, Dec00 contract (NQ00Z in QCharts). From Sept to Dec, the system generated 248 trades (103 winners and 145 losers). Winning long trades were 51 of 124 total long trades and winning short trades were 52 of 124 total short trades.

The net Points profit for those three months was 808 points with no allowance for slippage or commission. So, if you were trading 1 contract of the NQ, you would have generated (808*20)=$16,160 in profit per contract.

Sounds great so far, $16,160 in profit and you put up only $11K or so in margin. Nice return in three months. I ran the same test on an out of sample set of data: the most recent NQ for Dec to March (NQ01H in QCharts), these results were similar, with about half the profits per contract at 431 points. Using the same analysis of the t-statistic that I posted a while back, it shows slightly better than an 81% probability of continuing to perform in the future.

Now for the bad stuff, and this is where someone really has to understand a system in order to trade it. The maximum open trade drawdown was 31.5 points (over $600 per contract). The maximum open system drawdown was 57 points ($1140 per contract) which means that at some point during your trading, your net account balance was negative by $1140 per contract traded.

The low win to loss ratio produces another trader psychology question: the greatest consectuive wins is 4 and the greatest consecutive losses was 6. Can you trade this through 6 losses in a row and still take the next trade? Oftentimes, in trend following systems that next trade could be the biggie that makes 3 times what you've just lost.

I'd start to worry that this system is broken if I were to get 10 losses in a row, although there's a lot written that says this could happen statistically. These are personal decisions that have to be considered by each trader contemplating this sort of a system--before any money gets traded in the system.

Also consider that I've set up the backtest with the assumption that a trader would enter on the open of the following bar. That's pretty quick, considering you got the signal just one second ago. Consider some slippage and staring at your computer screen pretty intently when the moving averages start to come together. For comparisons sake, if you were to enter on the CLOSE of the following bar, the net system profit is just 401 points in sample, approximately half of your profits are made in the first minute after the crossover. (That in itself may be worth looking at in another system...a very fast system).

So, Andy, those are some hard questions for your consideration. It also gives you an idea of just some of the things you can do with some of this software.
 
Hi Dave,


thank you so much for your time in answering and also in running the test.

I'll go over the system I'd been trying to test "live" I guess, just in case you see something that might add or take away from what you ran.

As far as when I would enter a trade, I'd wait for the next candle to close after the signal before entering the trade, I figured playing worsecase should make it more realistic ?


1 min'charts, ndx-mini, using the 9, 18, 50 EMA's. When the 9 first crosses the 50 that's my entry (long or short) then when it comes back to cross the 18 that's my exit, sometimes it's a quick whipsaw and it'll cross the 50 first, that's still my out. I don't enter another trade until the 9ema crosses the 50ema again. Of course on a whipsaw (9 comes back to the 50 before the 18) it would become a trade in that direction.

What I have noticed is that the NDX seem to have a 10pt wiggle, so using a 12pt trailing stoploss could keep you in a trade. I also had lots of 20-30 pt gains before the price suddenly fell/rose leaving me with just a few points or a slight loss using the worse case fills, after the MA's told me to exit.

Not to curve fit the system, which I may end up doing :-( This next week as well as the above system I'll also try using a system where when a 4ema crosses the 40ema that's my entry until I exit on a 12pt loss or every positive 12pts I'll move my stop up. Start risk -12pts, when +12pt's I'm at breakeven etc. At +24 stop is at +12. Realisticly I'm not sure if the IB order entry will work on such a quick system ? (Though allowing for 12pts wiggle room will give me time to get orders ready to send etc)

PFG has a work station where it looks like you can just click on a row of prices which might work faster, of course commissions will add up alot more.

I did try working alongside the EMA's with other indicators, stochastic's, RIS etc, but really didn't see anything.

Anyway thanks again, Andy

p.s. hope I'm not over complicating this, I'm just trying to find something that'll work in one market, hey, but then I guess everyone is :-)
 
Hi Andy,

It'll probably be a couple of days before I get to check out the parameters that you discussed. It should be interesting to see though!

What I've seen of the PFG system is that the quotes are not streaming. I think it's a snapshot that you have to keep refreshing. For very short term trading as you describe, I think this would prove very difficult.

Unless the NAZ is moving really fast, I think you will find IB to be as fast as any system even with your stops. The NAZ can trade really fast in a five or ten point range, so even the fastest system will see slippage. It might be interesting to include these amounts to see what the "worst case" might be.
 
Hi Dave,

thanks, no rush. It's actually given me some time to run them as best I can, live. The 9,18,50 seems to leave some big moves on the table, by getting out on the 18 instead of waiting for the 50 doesn't seem to save too much especailly considering how it missis big moves.

The 4 and 40ema's, I guess all in all close to the 5 and 30ema's that you ran seems to work better. Which if what I remember on what I've read about systems that's good !! They say you should be able to change the numbers around in the parameters which are close to what you started with and over all it shouldn't make too much difference, that way it shows you not curve fiiting a system.

A 12pt stoploss doesn't seemed to help much right at the begining of a signal, especially with quick whipsaws, probably it's better to just be as quick as possible on the signal given and not to wait around for a confirming candle. Though once a trend has started and your in the money 15-20pt's the 12pt stoploss can save big. Today for instance waiting for the crossover signal to exit that were in the money good would have left as much as 25-30pt's on the table. I'd like to say, "I could try using common sense on keeping some profits." But then that gives way to emotions on where to place the stop. Looking for support/resistance could work, but again it gets away from a clear signaling system. Maybe a set "12pt" is the best way ?

There seems enough time to get a order ready and waiting and when you see the crossover send it (or alsowatching new highs/lows for the 12pts). I was thinking of using a limit order set a couple of points higher/lower in the direction of the signal, that way you could guess/anticpate what you think the signal price will be and with it being more (long) or less (short) than needed it would act as a market order for a point or two ? Saving a couple of seconds of imputting a price, that might not get filled. Hope that makes sense ?

I'm trying to set up with IB now for futures, just to see if they'll give me an account :-) I have a stock account already with then, not that I think it'll make any difference, guess I'll wait and see.

Back to the orders. If I start the day with 1 long contract, when the signal shows a crossover, I believe I can then go short 2 contracts, canceling out the first ? Do I need enough margin for the 2 or is enough for just 1 alright (of course they'll be extra funds included for drawdowns etc)

I never did notice how do you roughly trade the mini's ?

I guess I'm trying to see if you use a mechanical system or something like support/resistance, not the exact system.

Thanks again, Andy
 
Hi Dave,

just to give you an update. The funny thing is either using the 5,30 or the 4 and 40ema, the signal comes out about the same. I've pretty much given up on using 3 different EMA's. The two seem just as good and a lot simplier.

Also as far as using two contracts to close the first while putting you in the opposite direction at the same time really helps. I'm still playing with using a 12pt stoploss as the trade moves in my favour, at the moment it's a kind of give and take. I've got to admit for myself so far the 12pt stoploss is easier to handle than leaving 20-30points on the table waiting for the MA's to cross. Of course some bigger runs were missed, but not enough for me to get upset over, yet :-)

How did your week go.? I've been busy with other stuff, I've still to get my IB paperwork done. I think I did get Q-charts figured out, just change to different servers if I notice problems, duhh !!

Oh well, later, Andy
 
Andy,

I wasn't sure about putting an order up to short two contracts so that you can exit one and end up net short by one. That's great to know. Some brokers will allow it, and others require separate transactions. Thanks for the info!

My trading is currently very short term in the two to five minute range. I am able to trade only the first hour of the regular trading session, so I'm focused on very short trades. I have done longer trades intraday as well as three to five day swing trades. It just depends on my schedule.

I do have mechanical systems for each. I would really like to say that I follow these exclusively, but sometimes betweeen signals, my discipline breaks down, and I'll take a trade or two.

I posted a response to your myTrack question on another thread.
 
The whole process of system building must be based on optimization. In order to create a profitable system on some instrument we must fit it's trading rules to curve...

Of course, the better results are, the smaller probability of getting better ( or higher probability of getting poorer ) results in the future is.

If we build a non-optimized system with net profit =0, there's about 50% chance that system's profit will be higher in the future
:)

DT-waw
 
One of the best indications that a system has a good chance to work in the future, and that it is not overly curve fitted or optimized (in addition to the test mentioned here), is to see if the system also performs well across many markets with no changes (or minimal changes to account for things like tick size, etc.).

In other words, after you've gone through what I call the "anti-optimization" exercise to make sure that your system is not over-optimized, test it across many different markets or stocks in different sectors and if it is still profitable, you might be on to something.
 
DT-waw
I have been following your posts for some time and I really must say it: you have no clue.
Prediction: You are going to be searching for the edge for few years. I guarantee you won't find anything beyond 60% probability of working (which is practically as good as 50%). If by the chance you find something it's going to disappear fast. Realize the simple fact that you are not the only one with computer!!!! Then if you save enough money you will find out you can't trade at all. After that you might actually realize what really matters in trading.

P.S. I wrote this because of your "curve fitting" note. It's really naive. Read William Eckhardt interview in "The New Market Wizards" at least 100 times!!!!
 
dottom, et al -

I'm curious why you expect that a system must work across multiple markets to be valid?

Private trading firms I've done work for have all used computerized systems specific to the individual market - whether index futures, metals, grains, stocks, etc.

Their quants develop technical systems (often quite complex) geared to each individual market/security and then further refine the technical parameters and then run with it. If for some reason one of the systems starts dropping below their profitability expectations, they rework it or replace it. The philosophy being of course that they're looking for a profitable edge, not the single, end all, beat all holy grail.

There is always a chance of overoptimizing certain parameters, rules, or conditional qualifiers, but there are also methods to help avoid doing so. An "optimized" system tailored to a specific instrument/market is not by default "over-optimized".

Expecting a single set of rules and technical parameters to work on all instruments/markets would be great but ignores that differences exist between markets and even between individual stocks.
 
You must log in or register to reply here.
Back
Top