did i apply curve fitting to my system

[newguy05]

looks like there are some conflicting opinion on curve fitting, but in the end i think the way i modified it seems ok. Whether it is actually called curve fitting or not, doesnt really matter.

Mike, i dont mind sharing details. The system and its signals are generated from point figure charting technicals (support/resistance etc..). I like point-figure because i can take tick data(the most granular datafeed) yet the actual signal generation is based of a much higher level data. This way i can mimic production as closely as possible in term of fast volatile moves / slippage.

The bad part is there are no existing pf backtesting system available, so the whole thing was built from scratch using java - real time pf engine, order/position management module, datafeed module, blackbox, etc..

1. How many trades (what is your sample size?). For an intraday model over 10 years I'd like to see 500 trades or more.

each entry involves 3*X contract at market, 3*X stop loss, and 3*X limit. As either stoploss or limit is hit, the other side will get canceled. I think IB actually has a order entry for this (dont remember the name) to actually make it even simpler in production.

About 15 days a month have trading going on that averages 1-2 entries. So that's about 30 trades a month?

2. Average profit per trade. Minimum $25 per trade for the ES AFTER COMMISSION AND SLIPPAGE. Use $2.5 for commission and $6.25 slippage per trade.

The profit/loss are above $150 per entry. It's not one of those systems that make tons trade scalping pennies.

3. Did you use limit orders? Do not use limit orders unless you have software that can look forward one bar and simulate a fill only when the future price crosses one full tick under or over your limit price.

Well there is no software, everything is custom coded by myself. The backtesting is just a while loop, each iteration reads 1 es tick data from the database. So interation1 i submit limit order, interation2 i check the limit order vs the new tick price, if it's equal or over, the limit order is filled.

I thought about this problem, ES at 1000, i create:
1) long market order
2) limit order 1004
3) stoploss order 996

but market is very volatile, so by the time i submit ES already moved to 1005. So what that means is now my long order got filled at market 1005, and instantly my limit order got filled too at 1005. So a winning trade now turns into a loser due to slippage.

Really cant think of a way to deal with this other than rely on the fact that my system reaction is faster than the es movement, my limit/stoploss is at least 3-6 pts from last tick, and eat up the rare(hopefully) slippage due to this.

Generally, year end lossess are not acceptable, especially for an intraday model.

Yeah i figured as much, after analyzing the winners i realized if didnt strictly use limit orders to take profit, instead leaving 1 out of 3 contracts as a trailing order, similar to letting the winner run, profit will be even higher. But then that means curve fitting the system even more.

Guess not much left to do but to hook it up to IB's api. Another fun weekend ahead

Quote from Mike805:

I don't think you need tick data to do what you're doing. Minute bars should suffice.

Adjusting exits is not curve fitting in the destructive sense, it is optmization of you model, which is fine. As long as the model concept is sound then the parameters that dictate profit/loss stops are relatively insignificant. In fact, the best thing you can do is walk forward test your targets/stops and see what happens.

Here are the important questions:

1. How many trades (what is your sample size?). For an intraday model over 10 years I'd like to see 500 trades or more.

2. Average profit per trade. Minimum $25 per trade for the ES AFTER COMMISSION AND SLIPPAGE. Use $2.5 for commission and $6.25 slippage per trade.

3. Did you use limit orders? Do not use limit orders unless you have software that can look forward one bar and simulate a fill only when the future price crosses one full tick under or over your limit price.

Generally, year end lossess are not acceptable, especially for an intraday model.

Without giving away the method, what is the market movement you are trying to capture?

Mike

 

[intradaybill]

This is an important topic. I think as long as you develop a system by analyzing its performance in relation to a price series than some type of curve-fitting is involved inevitably. The issue is not whether their is curve-fitting but whether it is major or minor in the sense of this rule:

"curve-fitting adversely affects the actual performance of a trading system if it involves adjustment of parameters that have the greatest impact on it."

The author of this rule presents three different cases of curve-fitting. This is the link to the analysis.

A must read IMO.

 

[maninjapan]

I have a question in regards to the comments about using seperate data for developing and testing. How do you choose what data to use for which stage? I mean if you use a period that is strong bullish then test it on a period that is bearish your results should probably be terrible. But does that invalidate your system? Im not a believer that a single strategy works well in all markets. Id be very suspicious of a system that works well in all scenarios. (not saying its impossible, just that Id be very suspicious....)

 

[TSGannGalt]

Ignore Corey's comment.

Your question as a statement makes it obvious that it's a bad thing to do.

 

[Corey]

Ignore my comment at your own risk.

I agree with you -- not every system works in every market -- especially if you develop niche systems.

My point was this: if your system's parameters are non-adaptive, and you choose the parameters to maximize returns over a given data set, you should test on another data-set to ensure that you haven't over-fit your parameters. If you have, you may find that even if you enter another market phase that is conducive to your strategy, your system may still fail.

You should be choosing the data in a non-conscious manner. In machine learning, they will often split a data-set into five parts and rotate the learning/testing parts. So you might learn on the first 4/5ths then test on the last 1/5th. Then they will train on the last 4/5ths and test on the first 1/5th. If the parameters don't vary greatly and neither do the results, they know that their model is robust.

It is a little harder to do with time-series, but you can still choose random market segments to learn/test on.

If you design your system for a specific market type in mind, find a couple examples of that market type. Train your system on one and test on the others.

 

[John Paterson]

Quote from maninjapan:

How do you choose what data to use for which stage? I mean if you use a period that is strong bullish then test it on a period that is bearish your results should probably be terrible. But does that invalidate your system?

During the bulish period your system should generate many more long trades as compared to short. The other way around during bearish periods. If your system generates many long trades during a bearish period it is not a good system.

 

[John Paterson]

Quote from intradaybill:

The author of this rule presents three different cases of curve-fitting. This is the link to the analysis.

A must read IMO.

Good work by Michael Harris. I think his classification of the different types of curve fitting based on entries, exists and timeframes makes a lot of sense.

 

[TSGannGalt]

Quote from John Paterson:

During the bulish period your system should generate many more long trades as compared to short. The other way around during bearish periods. If your system generates many long trades during a bearish period it is not a good system.

I can just change your statement and it can be equally true:

During the bulish period your system should generate a higher Risk/Reward Ratio as compared to short. The other way around during bearish periods. If your system generates higher risk/reward on long trades during a bearish period it is not a good system.

 

[TSGannGalt]

Quote from intradaybill:

This is an important topic. I think as long as you develop a system by analyzing its performance in relation to a price series than some type of curve-fitting is involved inevitably. The issue is not whether their is curve-fitting but whether it is major or minor in the sense of this rule:

"curve-fitting adversely affects the actual performance of a trading system if it involves adjustment of parameters that have the greatest impact on it."

The author of this rule presents three different cases of curve-fitting. This is the link to the analysis.

A must read IMO.

I skimmed through the link. It's a newbie developer site. There's plenty more ways of "curve-fitting" than the categories.

One example would be a misuse of statistics like....

Let's say, I have a .001% chance that the system will make 4 losses in a row. Should I go "All-In" on the next trade?

Plenty of others that don't fit the 3 categories mentioned. As a matter of fact, any component of the trading system you implement is an extension of "curve-fitting" in some way and extent.

 

[nephos]

Quote from intradaybill:

The author of this rule presents three different cases of curve-fitting. This is the link to the analysis.

A must read IMO.

Useless and possibly dangerous drivel. The author implies you can only curve-fit with parameters but not with hard coded rules. Quotes from the website:

"The term "curve fitting" applies when parameters are adjusted (or optimized) so that a specific function "fits" a time series in a certain "best" sense, (...)"

"It is not possible to change the timing of the entry signals of a specific price pattern simply because there are no parameters to adjust for doing it."

However, their commercial product APS may test thousands of patterns like
O[1] > C[1] AND O[2] > H[3] AND C[3] > L[4] AND...

Of course you can curve-fit and over-optimize that way like crazy. Chance the identifiers OHLC, change how many bars ago, change the combination of the (un)equations.

I am not saying that data mining like that inevitably leads to over-optimization, but it certainly is a possibility.