Backtesting and Curve fitting

K

KarthikK

I am trying to backtest my strategies and have been having difficulty design it in the right way. I am using 1 minute OHLC bar data (forex) for dates from 2009-05-01 to 2015-08-31 (It comes to 1666 days) and my system will attempt to make at best one trade a day. So I have some input indicators along with few parameters that needs to be optimized. If I run my backtest over all the training days to optimize my parameters, I have already peeked into the data. I could do 80%-20% split, even so I fear I am curve fitting. How do you optimize your trading parameters ?

Another idea was to actually curve fit and hopefully land in the right side of market. For example, use sliding window of last 6-months days and optimize your daily trading parameters. Has anyone done this ? This would curve fit, but intentionally. Appreciate pointers.
 
It really depends on what your 'indicators' are, if you're just optimizing a bunch moving averages and oscillators then no optimization scheme is going to produce agreeable results. If you have something more solid then I would optimize over all my data then paper-trade going forward, you'll have to decide on what criteria you 'go live', some sort of 'not random' test is a good idea, there are many around (Monte-Carlo, T-test etc.).
 
I don't have lot of indicators. It's a very simple system based on EMA20 and SMA20 and the current price. If I use ALL the data, wouldn't data snoop bias cause issues ?
 
Curve fitting may make your backtest show better results but it tells you very little about what the future might bring and you may end up sacrificing a more logically and technically sound strategy for one that merely 'fits' better.
 
@Victory5: You are talking about the perils of curve fitting and I get it. But backtesting over a window and optimizing is anyways curve fitting. May be if the window is long enough over many years, we expect it will show similar pattern. But the other point I was making was, has anyone done an intentional curve fitting ? For example, every month, you will find an optimal strategy that has worked in last 6 months or so and keep moving your sliding window every month. You will curve fit - but would this actually land you in a strategy that is more aligned with market trend ? Or market is random enough, you will get whipsawed.
 
KarthikK said:
@Victory5: You are talking about the perils of curve fitting and I get it. But backtesting over a window and optimizing is anyways curve fitting. May be if the window is long enough over many years, we expect it will show similar pattern. But the other point I was making was, has anyone done an intentional curve fitting ? For example, every month, you will find an optimal strategy that has worked in last 6 months or so and keep moving your sliding window every month. You will curve fit - but would this actually land you in a strategy that is more aligned with market trend ? Or market is random enough, you will get whipsawed.
More...

If your EA is based on the 'common' indicators such as MAs and you curve-fit hoping to find the market 'behaviour', you are doing it wrong.

I have done it before and it doesn't work, at least not for long. The market is more random than you think.
 
KarthikK said:
@Victory5: You are talking about the perils of curve fitting and I get it. But backtesting over a window and optimizing is anyways curve fitting. May be if the window is long enough over many years, we expect it will show similar pattern. But the other point I was making was, has anyone done an intentional curve fitting ? For example, every month, you will find an optimal strategy that has worked in last 6 months or so and keep moving your sliding window every month. You will curve fit - but would this actually land you in a strategy that is more aligned with market trend ? Or market is random enough, you will get whipsawed.
More...


I view it as part of the scientific method: You start with an underlying ideology or understanding of the market and your strategy or system is a manifestation of that ideology. You can use a backtest as one way to test the validity of your system (ie test your hypothesis) but changing the core tenants for optimization purposes will put you in big trouble later on.

I have no experience in intentionally curve fitting, nor am I sure it would work well over any time frame.
 
carrer said:
If your EA is based on the 'common' indicators such as MAs and you curve-fit hoping to find the market 'behaviour', you are doing it wrong.

I have done it before and it doesn't work, at least not for long. The market is more random than you think.
More...

I agree with you. That's another reason I am giving up on any kind of averaging and trying to understand pure price action. May be some MA could be used as a guide - but am thinking about purely price action based strategies. If you have any thoughts, I'd love to hear.
 
6 months or even days isn't relevant, 2 days back tops to curve fit to, maybe even down to 30mins, what's working in the last 30mins, odds are it'll work for maybe an hour or 2 more, maybe.
 
KarthikK said:
I agree with you. That's another reason I am giving up on any kind of averaging and trying to understand pure price action. May be some MA could be used as a guide - but am thinking about purely price action based strategies. If you have any thoughts, I'd love to hear.
More...

You have to find patterns/somethings which have statistical edges.

You have to be careful with hindsight analysis, cognitive biases, etc. They can be very misleading and deceitful.

It takes time.
 
You must log in or register to reply here.
Back
Top