Statistical Analysis of Intraday Data - Any worthwhile resources?
- Thread starter Howard
- Start date
Well, like you said, the audio was really bad. I skimmed parts of it, but it didn't seem to be similar to what I'm doing or trying to do. But perhaps I'll give it another go some other day when I have better time.
It could perhaps seem like I'm a complete newbie starting from scratch, but that's not the situation here. I've spent a huge amount of hours the last year developing my system after a break from the markets. I have extensive amounts of experience with the markets prior to that as well.
The funny thing is that my system or model seems really simple now that it's (almost) done, but there is a huge amount of work and thinking behind it's logic and operations.
One of the things I've been curious about is for example how much data to look at and how to incorporate data that deviates from what's 'normal'. Like for instance the financial crisis period.
I'm also mostly looking at patterns based on the last trading day and the day before, but I'm also curious about whether I should add more days, such that I'm looking at for instance 3 days in sequence.
Anyway, I've realized that these are questions I can answer myself by doing backtesting and research. In fact, I figured out a way to include financial crisis type of data while 'discriminating' it, so to speak.
I'm not sure if anyone else has experienced this, but I tend to ask questions about where I'm stuck and often the answers just pop up by themselves. Like my brain is working subconsciously while I'm doing other stuff most of the time. Pretty cool.
One of the challenges for us, trading system developers, is to assess the quality of our work, and especially the likelihood that forward performance will match or come close to backtest performance.
Most anyone can datamine a good system in backtest, and see it fail to make any money in forward testing. My belief is that the traditional credo re. keeping 1/3 of the data for forward performance assessment is fundamentally flawed (as well as any walkforward optimizations), since any system not performing good enough through these tests is rejected, talk about selection bias here.
You cannot evaluate your system quality based on its p&l, not even its sharpe ratio, because these are too easily engineered through datamining (aka over-optimization or overfit). The "solution" is called null-hypothesis testing (which is about measuring how far your trade series is from random selection), but you have to find a proper way to do that (which I very much described in that webinar) ... try as hard as you can to understand that aspect because it is way more important than a p/f of 1.5 vs 2+ in your backtest results.
If you don't mind me posting some backtest results here, I will show you the best pattern I have ever identified - not best according to p&l & other classic metrics, but best as the further away from a random selection. That pattern isn't highly profitable per trade, but it the least susceptible of dramatic counter-performance in forward execution.
(I'll wait for your consent to post that backtest)
Most anyone can datamine a good system in backtest, and see it fail to make any money in forward testing. My belief is that the traditional credo re. keeping 1/3 of the data for forward performance assessment is fundamentally flawed (as well as any walkforward optimizations), since any system not performing good enough through these tests is rejected, talk about selection bias here.
You cannot evaluate your system quality based on its p&l, not even its sharpe ratio, because these are too easily engineered through datamining (aka over-optimization or overfit). The "solution" is called null-hypothesis testing (which is about measuring how far your trade series is from random selection), but you have to find a proper way to do that (which I very much described in that webinar) ... try as hard as you can to understand that aspect because it is way more important than a p/f of 1.5 vs 2+ in your backtest results.
If you don't mind me posting some backtest results here, I will show you the best pattern I have ever identified - not best according to p&l & other classic metrics, but best as the further away from a random selection. That pattern isn't highly profitable per trade, but it the least susceptible of dramatic counter-performance in forward execution.
(I'll wait for your consent to post that backtest)
The Machine by CG3 is all u need to know
Knock yourself out. I don't own this thread, so everyone can post whatever they want. : )
What's that?
A statistical computational engine built on the world's largest database of short term trading info. https://clients.themachineus.com/Login.aspx
So here are 2 patterns, both trading pullbacks in the direction of the last complete trend.
Both are "naked" here, meaning only the core-pattern is used, no additional filters.
Core-pattern vs. Filter :
Pattern X :
Pattern Y :
Both are "naked" here, meaning only the core-pattern is used, no additional filters.
Core-pattern vs. Filter :
- Core-pattern is always an AND of single-metric criterias, each criteria can only be to accept or reject a continuous range of values for that metric.
- Filters can use more than 1 metric (AND) to reject setups - this amounts to an OR condition to accept setups, which is prone to overfit (hence I always measure Filters quality on the entire market-model space, not on the setup space).
Pattern X :
Pattern Y :
and the winner is ...
... pattern Y, which is 6 stdev from the market model on its sample size of actually 2250 (even though only 2150 show-up in this backtest), despite its lower win%, lower avg.net / trade, and higher historical max DD, vs pattern X.
I am going to digress a little and rant about out-of-sample testing (also applicable to walk forward) ... say you have a system with 500 trades in-sample, and you test it out-of-sample which yields 200 trades, and find out the results out of sample are quite different than in-sample (to make things simple, let's say that your primary concern is that the win-rate goes from 62% in-sample to 58% out-of-sample). What does that tell you?
My answer is, if you just look at those results, it tells you nothing else than the win% is different. There are 2 key aspects that you would need to find answers for, to make any useful conclusion:
... pattern Y, which is 6 stdev from the market model on its sample size of actually 2250 (even though only 2150 show-up in this backtest), despite its lower win%, lower avg.net / trade, and higher historical max DD, vs pattern X.
I am going to digress a little and rant about out-of-sample testing (also applicable to walk forward) ... say you have a system with 500 trades in-sample, and you test it out-of-sample which yields 200 trades, and find out the results out of sample are quite different than in-sample (to make things simple, let's say that your primary concern is that the win-rate goes from 62% in-sample to 58% out-of-sample). What does that tell you?
My answer is, if you just look at those results, it tells you nothing else than the win% is different. There are 2 key aspects that you would need to find answers for, to make any useful conclusion:
- How does the performance of random-entries compares out-of-sample to in-sample ?
- Is the win% difference between in-sample & out-of sample statistically significant, or is it just noise? (if the random entries performance between the 2 periods are "quite" different, you need to account for that difference as well when working on that question)