Insample Good / Out of Sample good!...?

P

ProgrammerGuy

I'm kinda bummed here. I've come up with a backet of stocks like 15 or so, and created a strategy...tested from Jan 2002 to May 2005. I detrended the data and showed a good profit. It is a moving average model using 3 parameters and found points where the parameters closed by worked. I tested on out of sample data (Detrended Data) from May 2005 to May 1, 2007 and I still got a good profit. I've ran the strategy from May 1, 2007 to current date June 13, 2007, and wow haven't done good @ all. I've been trading 50 shares for each stock, so I'm not down much.

How is my approach? Should I still run the strategy? Comments?
 
I just recently gave up backtesting. What a frigging headache, just getting the tests to not lie is hard. I kept finding more ways to inadvertently use data that was not available at decision time and getting fantastic results. Then I would look down in the bowels of the code and find a subroutine that used the Close of the bar when I was getting in earlier in the bar or something. I even found ways that the backtester I was using could bias things similarly. It did a bouncing tick approximation that would evaluate the bar low or high first depending on some statistical truth, not a bad idea but I found that could prejudice my tests a lot if I misapplied the tester.. and there were no published guidelines about misapplication of the tester. I have a pretty high IQ especially regarding math and tech stuff and I had to really work to understand these difficulties, I was putting in lots of 18 hour days for awhile.

I made more progress by working straight off the screen and trading realtime, watching the charts, in the last week than I did in months of backtesting. From now on out, I'll backtest something only after I have traded it awhile. The backtest won't give me a working strategy it will just optimize it or maybe tell me what kinds of markets it is good or not good for.
 
Quote from ProgrammerGuy:

I'm kinda bummed here. I've come up with a backet of stocks like 15 or so, and created a strategy...tested from Jan 2002 to May 2005. I detrended the data and showed a good profit. It is a moving average model using 3 parameters and found points where the parameters closed by worked. I tested on out of sample data (Detrended Data) from May 2005 to May 1, 2007 and I still got a good profit. I've ran the strategy from May 1, 2007 to current date June 13, 2007, and wow haven't done good @ all. I've been trading 50 shares for each stock, so I'm not down much.

How is my approach? Should I still run the strategy? Comments?
More...

I have a few questions for you.

1) What stocks are you trading ?.
2) Did you optimize and find one set of parameters that worked best on the basket ?

3) Are you using daily or intra-day data ?.

4) Where you using a dual moving average or a three moving average system ?.

5) Did you look at the parameter space in 3D ?

You need to realize that smaller stocks don't move the same as large caps. Also stocks in general are not good trend following. This is not totally true , but this topic could be the subject of another post.

You need an understanding of how the markets you are trading move to develop good systems. If system development was just optimizing moving averages everyone would be rich.

Key in system development is that you need to ask yourself ; What is your premise ?. A good premises leads to systems that hold up in the future.
 
i can relate too. simple backtested rulesets that used to work quite well in real trading seem to fail faster and more often on me. i made it from 2003 to early 2006 with just paramater tweaks on one simple er2 system, and since then nothing has ever really held up as well here. i was able to adapt it to metals from early to mid 2006, and been struggling for traction in a variety of mkts and systems since then.

a few months ago i had one strategy with a great backtest based on 2 overlapping time cycles. the shorter term cycle would only trade in the direction of the longer term cycle bias. the longer term cycle still performs, but the shorter term one failed right from the getgo. something with only one time-of-day param that had been scarily consistent for 3 years fell apart immediately and ever since with real contracts even though the underlying trend in that market appears consistent.

the list goes on, some were definitely ill-conceived over-fitted ideas, 4 params where a coefficient is applied to ATR to create a sort of vol band with seperate long and short params, etc. but in general i notice much more of a tendency for simple futs backtested strats to just fail outright for me and lock into endless sequences of losers ... i've wondered if it's a liquidity effect where i'm simply too conspicuous even with small lots due to something like clearing code tracking - and big parties can simply jam the illiquid stuff i'm trading when need be... or if the brokers are picking me off... or what

it seems like some competition is closer to the root of the market, instead of some technical ruleset, ppl can trade size against whatever 'flow' shows up in their superior liquidity information and bam... or as if the competition is more consolidated nowadays

not sure. it's been so profound at points though, at times it feels like i'm the only one being traded against. i know that's obviously not the case, but it can feel that way

there was a time here when copious positive excursion was much easier to achieve.
 
Quote from Murray Ruggiero:

I have a few questions for you.

1) What stocks are you trading ?.
2) Did you optimize and find one set of parameters that worked best on the basket ?

3) Are you using daily or intra-day data ?.

4) Where you using a dual moving average or a three moving average system ?.

5) Did you look at the parameter space in 3D ?

You need to realize that smaller stocks don't move the same as large caps. Also stocks in general are not good trend following. This is not totally true , but this topic could be the subject of another post.

You need an understanding of how the markets you are trading move to develop good systems. If system development was just optimizing moving averages everyone would be rich.

Key in system development is that you need to ask yourself ; What is your premise ?. A good premises leads to systems that hold up in the future.
More...

1) all the stocks in the transportation average
2) yes
3)60 min
4) it's a systems that uses 2 moving averages and I also draw a moving average above and a moving average below based on ATR
5) Yes, and no. I have a function that I wrote in C# that measures the fitness of the parameters. I'm pretty confident on this function to find optimal points. I've gone through a number of trails and the program always picks the points that I would have picked had I put in on a surface plot.
 
Quote from maxpi:


I made more progress by working straight off the screen and trading realtime, watching the charts, in the last week than I did in months of backtesting. From now on out, I'll backtest something only after I have traded it awhile. The backtest won't give me a working strategy it will just optimize it or maybe tell me what kinds of markets it is good or not good for.
More...

do you just watch the screen and get a feel for how the market is moving?
 
Quote from ProgrammerGuy:

do you just watch the screen and get a feel for how the market is moving?
More...

No way, I can't really do that.

I have a set of indicators and drawing tools. I'm shying away from backtesting but I am working towards automation so it has to be rules 100% of the time. I had a couple of situations that called for discretion in the last 2 days and looking back I can see that what I did could be rule based so as I trade I add things in. I find this method of system development to be way more satisfying intellectually and emotionally than backtesting.
 
If your model discovers a statistical pattern, it is natural to expect that the stability of this pattern deteriorates as you move further away from the training dataset. Especially if you go 2 years out.

I assume you weren't planning on training once and using that model indefinitely. A better approach would be to-retrain sequentialy as new data arrives.

I detrended the data and showed a good profit.
More...

For my edification purposes, may I ask what you meant by detrending. Is that some sort of a procedure where you subtract some moving average from the price data?
 
You must log in or register to reply here.
Back
Top