did i apply curve fitting to my system

N

newguy05

i finally finished my system development and started backtesting.

I am backtesting using 10 years of ES intraday tick data, and analyzed all the losers (it's an intraday system that only makes a few trades a week, so really isnt that many cases to eyeball over). Anyway, i noticed of all the losers majority took big loss due to my stop loss been set way too high.

So i modified the system logic for stop loss to be a lot narrower, for example instead of having a 1:2 reward/risk (take profit once reach 4pt, hit stop loss once reach 8pt). I changed it to 1:1 (4pt profit, 4pt stop loss). This in term increased the p&l of my backtesting significantly.

Now the question i have is this curve fitting? I dont want to tweak backtesting so it looks good while failing miserably in production. Thanks
 
Yes, it is. You went back, say what didn't work and tweaked it until you got the results you wanted.

That is like me taking my models back to 1987, seeing that they do poorly on the day of October 19th, and decide to tell them not to trade on that day.

Okay, not as bad, but you get the point.

Instead, split your historical data up. Three or four fifths for developing your model on, one to two fifths for forward testing your model on.

Basically, tape this above your computer:
DO NOT DEVELOP YOUR MODEL USING THE SAME DATA YOU WILL TEST IT ON.

The best thing you can do is make your risk reward metric adaptive ... but if it has to be static, find what works on the first 3/5ths of your data and see if it works on the last 2/5ths. If it doesn't, then you know you curve fit.

Unfortunately, at this point, it will be hard for you to take that step backwards because you already used all your data in development. See if you can get some data from a random time period ... say the 90s ... and see if your system still works.
 
Newguy,

If you only tried changing it from 1:2 to 1:1 then it's statistically very unlikely that you'll be affected significantly by curve fitting. You've only added one degree of freedom to the model, which is insignificant because of how much data you're using: intraday data over 10 years. You're model is still well over-determined by the data.

If you tested other risk/reward ratios, say 1:1.1, 1:1.2, 1:1.3 etc then you might have more of a problem.

Of course Corey is 100% correct:
"DO NOT DEVELOP YOUR MODEL USING THE SAME DATA YOU WILL TEST IT ON."

Curve fitting happens during initial development, not just in optimization. I just wanted to clarify that the change you are asking about is not likely to add a large amount of curve fitting beyond what you may already have done.

Good luck.

Regards,
Max
 
I designed in such a way that it reads es intraday tick data, yet the trade signals are based on price action of at least 3-6 pt movement. That way it makes sure the backtesting is relatively accurate to actual production.

i ran the backtesting for 5 years after "curve-fitted" the risk:reward from 2:1 to 1:1. This is based of 3 contracts, it can be scaled up pretty well up to 30 contracts in production as es is very liquid.

Next step i am going to hook it up to IB's java api to test on the demo account.

Not sure what to make of the result, is it acceptable to have a FULL YEAR with a loss?

YEARLY
tradeyear , sum(profitloss_dollar)
2002 112.5
2003 10487.5
2004 -4000
2005 5162.5
2006 -1562.5
2007 19225
2008 6625


MONTHLY
tradeyear , trademonth, sum(profitloss_dollar)
2002 12 112.5
2003 1 2675
2003 2 1850
2003 3 1800
2003 4 1150
2003 5 -112.5
2003 6 2550
2003 7 512.5
2003 8 -1812.5
2003 9 -1200
2003 10 2075
2003 11 -350
2003 12 1350
2004 1 -575
2004 2 -137.5
2004 3 225
2004 4 -2075
2004 5 -200
2004 6 1537.5
2004 7 -2700
2004 8 450
2004 9 -975
2004 10 2037.5
2004 11 -1262.5
2004 12 -325
2005 1 325
2005 2 287.5
2005 3 887.5
2005 4 2162.5
2005 5 -1137.5
2005 6 637.5
2005 7 -275
2005 8 -862.5
2005 9 1575
2005 10 1500
2005 11 250
2005 12 -187.5
2006 1 -850
2006 2 -212.5
2006 3 1575
2006 4 -2012.5
2006 5 1137.5
2006 6 100
2006 7 -3287.5
2006 8 1375
2006 9 475
2006 10 100
2006 11 50
2006 12 -12.5
2007 1 -500
2007 2 2000
2007 3 2475
2007 4 375
2007 5 987.5
2007 6 1912.5
2007 7 6075
2007 8 5637.5
2007 9 -4687.5
2007 10 3462.5
2007 11 4962.5
2007 12 -3475
2008 1 4475
2008 2 4212.5
2008 3 -1250
2008 4 1050
2008 5 -950
2008 6 -912.5
 
Quote from newguy05:

I designed in such a way that it reads es intraday tick data, yet the trade signals are based on price action of at least 3-6 pt movement. That way it makes sure the backtesting is relatively accurate to actual production.

i ran the backtesting for 5 years after "curve-fitted" the risk:reward from 2:1 to 1:1. This is based of 3 contracts, it can be scaled up pretty well up to 30 contracts in production as es is very liquid.

Next step i am going to hook it up to IB's java api to test on the demo account.

Not sure what to make of the result, is it acceptable to have a FULL YEAR with a loss?

More...


I don't think you need tick data to do what you're doing. Minute bars should suffice.

Adjusting exits is not curve fitting in the destructive sense, it is optmization of you model, which is fine. As long as the model concept is sound then the parameters that dictate profit/loss stops are relatively insignificant. In fact, the best thing you can do is walk forward test your targets/stops and see what happens.

Here are the important questions:

1. How many trades (what is your sample size?). For an intraday model over 10 years I'd like to see 500 trades or more.

2. Average profit per trade. Minimum $25 per trade for the ES AFTER COMMISSION AND SLIPPAGE. Use $2.5 for commission and $6.25 slippage per trade.

3. Did you use limit orders? Do not use limit orders unless you have software that can look forward one bar and simulate a fill only when the future price crosses one full tick under or over your limit price.

Generally, year end lossess are not acceptable, especially for an intraday model.

Without giving away the method, what is the market movement you are trying to capture?

Mike
 
Quote from newguy05:

So i modified the system logic for stop loss to be a lot narrower...Now the question i have is this curve fitting?
More...

Seems you already know the answer to this is an emphatic "Yes!"
 
Quote from TSGannGalt:

What system is not curve-fitted?
More...

There are folks who generate (and pick from) random systems. Those are the only ones I know of who do not curve fit.
 
Quote from TSGannGalt:

What system is not curve-fitted?
More...

very good question.
the one not curve-fitted at all is the system which generates no profit over the long run.
the more profit or the more consistent returns it produces - the more it is fitted to the curve.

yet 99.9% of traders are unable to grasp this simple concept!
 
Quote from newguy05:

So i modified the system logic for stop loss to be a lot narrower, for example instead of having a 1:2 reward/risk (take profit once reach 4pt, hit stop loss once reach 8pt). I changed it to 1:1 (4pt profit, 4pt stop loss). This in term increased the p&l of my backtesting significantly.

Now the question i have is this curve fitting? I dont want to tweak backtesting so it looks good while failing miserably in production. Thanks
More...
You didn't curve fit your system.

You changed the reward:risk parameters from negative (double the amount for loss as there is for the potential gain) to 1 to 1 (same amount for loss as there is for potential gain).

Now, provided your system trades with positive expectation, you should do well.
 
You must log in or register to reply here.
Back
Top