General Statistics Question

H

HotTip

I have a question for the stats studs out there:

If I developed a system that when backtested creates a smooth upward daily equity curve (i.e., >99% correlation) incorporating >3611 trades over 427 days, is it even necessary to do an in-sample/out-of-sample test on that? I can understand doing that when the in-sample results have a high beta and generates an erratic equity curve, but if my equity curve has a nice upward trajectory regardless of the time period I'm looking at, doesn't that already imply that the system is consistently profitable, or is an out-of-sample test still necessary regardless of the equity curve my in-sample generates?

Thanks in advance for any insight!
 
Quote from HotTip:

I have a question for the stats studs out there:

If I developed a system that when backtested creates a smooth upward daily equity curve (i.e., >99% correlation) incorporating >3611 trades over 427 days, is it even necessary to do an in-sample/out-of-sample test on that? I can understand doing that when the in-sample results have a high beta and generates an erratic equity curve, but if my equity curve has a nice upward trajectory regardless of the time period I'm looking at, doesn't that already imply that the system is consistently profitable, or is an out-of-sample test still necessary regardless of the equity curve my in-sample generates?

Thanks in advance for any insight!
More...

On what instruments? On what timeframes / volume / tick ?

Not to rain on your parade, and I have no idea what you are doing, but 427 days is meaningless IMO.

Others may refute that.
 
Quote from HotTip:

I have a question for the stats studs out there:

If I developed a system that when backtested creates a smooth upward daily equity curve (i.e., >99% correlation) incorporating >3611 trades over 427 days, is it even necessary to do an in-sample/out-of-sample test on that? I can understand doing that when the in-sample results have a high beta and generates an erratic equity curve, but if my equity curve has a nice upward trajectory regardless of the time period I'm looking at, doesn't that already imply that the system is consistently profitable, or is an out-of-sample test still necessary regardless of the equity curve my in-sample generates?

Thanks in advance for any insight!
More...
It's not a matter of whether it's necessary or unnecessary, it's a matter of how confident you are that the feature you are exploiting exists in any data other than what you are currently testing against. For example, if I have a development process that starts with a bunch of data and finds a system that trades it perfectly, then I'm not going to have much confidence in it. I will probably want to see how it performs on different data. On the other hand, I could also imagine a situation where I had high enough confidence, based on the rigor of my development process, to take a system live after seeing only a few test results.
 
All my backtests use some form of walk forward analysis.
Do not trust in sample backtests.
Whenever I see a very smooth equity curve,
the most likely explanation is some kind of mistake.
Most common mistakes are look ahead and suvivorship bias,
or unrealistic execution costs.
 
as a pseudo qualified statistician....there is no question there...



analogous to randomly asking somebody "is this important"
 
You must log in or register to reply here.
Back
Top