Damn you, out of sample trades!

[sle]

Quote from logic_man:

Hope that makes sense. At least, it makes sense so that you understand what I'm trying to do, even if you disagree with me trying to do it.

It does, thank you. I can't disagree or agree, since I really have no opinion on the underlying assumption, I was just puzzled by the idea of having large gaps as good (I don't buy the Oraclewizard explanation at all).

I am sure you test for all sort of N-th worst/best losing streak, N-th worst/best trade set etc. Dumb question - the recent trades, are these outliers vs the back tests or simply bad coincidence?

 

[logic_man]

Quote from sle:

It does, thank you. I can't disagree or agree, since I really have no opinion on the underlying assumption, I was just puzzled by the idea of having large gaps as good (I don't buy the Oraclewizard explanation at all).

I am sure you test for all sort of N-th worst/best losing streak, N-th worst/best trade set etc. Dumb question - the recent trades, are these outliers vs the back tests or simply bad coincidence?

OK, good. The thing with the large gaps is that there is a logic to how the initial stop gets set and it is very difficult to optimize beyond that logic. If I were to, say, change the algorithm such that the initial stop was only 50% as far, yes, my loss to win ratio would become more favorable, but the winning percentage would go way down. Of course, I know you know this, so I'm just kind of stating the obvious, which is that there are tradeoffs to each decision. If I had to estimate from historical incidents where I've seen price go very close to my initial stop and then turn back in my direction, I would say that maybe the sweet spot would be to set the initial stop at 85-90% of the distance it gets set to now. That would decrease the win rate, but decrease the average loss size and may lead to a better overall profit factor. One way I can eventually figure this out is to capture maximum adverse excursion data for each trade, but I don't have that right now.

One other way to decrease the size of the average loss would be to only take trades where the maximum potential loss was below a given threshold. Now, I did look at that and the data showed that when the initial stop was further away, the average trade result was $~230 profit per contract per trade, but when the initial stop was closer than average, the average trade result was ~$110 profit per contract per trade. Clearly, just a couple of big losses with far away stops could flip that result, but, if we are just looking at the data to drive decisions, a "risk-neutral" observer would say to definitely take the trades even with the further away stops because even when you eventually do take that huge loss, the numerous smaller gains will still leave the overall ledger positive. At least, that's the theory.

After a suggestion to look at the worst drawdown, I did see a larger drawdown in the historical results, so these live trades appear to be, at worst, marginally worse than any similar size subset of the historical trades. Which, if you buy into the idea that "your worst loss is always ahead of you", makes sense.

What I do to test at least the winning percentage against historical data is plug the sample I'm looking at winning percentage into the binomial distribution function in Excel to see what the odds of this particular sample winning percentage occurring. When I did that for the Crude trades, I got that there was a ~1.5% chance of those outcomes even if the overall winning percentage was the much higher rate that had been seen historically. It is just a matter of fact that even with a winning percentage above 90%, you are going to have scenarios in which 2 of 3 trades are losers.

I have just noticed that, ironically, those scenarios seem to occur right when I go live with an idea. It's almost as if the market says, "OK, now you see that idea is a good one, but I am going to screw with you a little bit here". Now, I'm not one of those people who think the market is out to screw anyone or help anyone, it just is what it is, but even I have to wonder at times.

 

[sle]

Quote from logic_man:
If I were to, say, change the algorithm such that the initial stop was only 50% as far, yes, my loss to win ratio would become more favorable, but the winning percentage would go way down.

Yeah, I hear you. It just might be my personal bias against negative skewness, since many strategies in my space show a very high percentage of winners to losers, but when you lose, you lose big (and liquidity disappears so stops might not help). I guess in a linear product space it's not as big of a deal.

I usually do some partial re-sampling of the historical trade set to see what sort of distributions of win/lose, best/worst etc I can get. Kind-of removes the element of surprise and also (more importantly), if the out of sample trade set is on the fringes of one of the re-sampling studies, it makes me think harder about possibly missing something important.

 

[logic_man]

Quote from sle:

Yeah, I hear you. It just might be my personal bias against negative skewness, since many strategies in my space show a very high percentage of winners to losers, but when you lose, you lose big (and liquidity disappears so stops might not help). I guess in a linear product space it's not as big of a deal.

I usually do some partial re-sampling of the historical trade set to see what sort of distributions of win/lose, best/worst etc I can get. Kind-of removes the element of surprise and also (more importantly), if the out of sample trade set is on the fringes of one of the re-sampling studies, it makes me think harder about possibly missing something important.

I don't like the negative skew, either. It smacks of those forex bots with 90% win rates and negative expectancy. They tout the win rates and put the part about negative expectancy in the fine print.

I look at the un-optimized base results and if they are at or near profit factor of 1, I think that's a sound base from which to start optimizing. In the case of this negative skew strategy, the base profit factor is close to 1.5. From there, with the optimizations, it goes higher. That 1.5 is actually higher than the base profit factor of the unoptimized "intraday swing" strategy, which is about 1.15. From these results, it appears that my real "edge" is in identifying situations where price will move from my entry price to my initial stop movement price, which can be anywhere from 1 tick to over 100 ticks. Given the difficulty of making any directional strategy work, I consider this a very good outcome.

From there, the "fat tails" I occasionally catch with the "intraday swing" strategy are where all the profits come from for that component of the trade.

What prompted me to split the two parts of the trade was that in Crude the trades which triggered had an ever so slightly higher probability of triggering and not reaching the initial stop movement price. Because I have been unable to figure out why, I thought that it might be best to simply enter Crude trades at the initial stop movement price, thus eliminating the negative skew at the beginning of the trade. The trades that get eliminated from the strategy had such large losses that it made sense even to get in at the more adverse price. Put simply, Crude is just marginally less predictable in the space between my ideal entry price and my stop movement price, so the cautious thing to do is make those prices one and the same. The nice thing about this is that it eliminates losing trades like the ~$5 loser I would have had on a short in mid-July, but still lets me catch the tail trades like the ~$3.5 winner I had in early July. Yes, it would have been a ~$3.8 under the old entry model, but it would have taken me ~17 more winning trades with that extra $0.30 on the front end to make up for that 1 ~$5 loser. The math made sense and, psychologically, no one wants to take that big a hit on a trade, so it made sense there, too.

If I can figure out what makes one trade take the maximum loss and not another, I will be even better off, but that is a really tough nut to crack. I have some ideas on what it might be (basically, volume and or volatility needing to confirm what price is doing at that moment of entry) but right now I lack the tools to really test those ideas.

 

[logic_man]

A post, subsequently deleted, stated that there are no non-random moves of 3 points in the ES, so it is impossible that such moves could ever be captured via a true "edge".

Well, I will concede that my data is less than exhaustive, so I can't speak to every 3 point move that has ever happened, and I will also concede that let's say 99.99999% of all 3-point moves appear random to me as well.

Where I won't concede is that there are 0.00001% of 3-point moves that are not random because the data seem to say they aren't. Obviously, short of telling someone exactly which 3-point moves those are and letting them validate it for themselves, any debate is futile. Just as obviously, I'm not going to tell anyone which 3-point moves those are because it took me a long time to figure it out and telling someone else does nothing for me.