Why wouldn't those types of opportunities get picked off by market makers and large highly automated trading firms?
This question may be similar to why IV smile exists when it isn’t predicted by BSM, and actually it’s the source of everyone’s problems with trying to calculate option prices & vol surface that constantly shifts, while creating jumps/skews, and other phenomena. The main job of quants nowadays seems to be figuring out vol surface and skew dynamics, and there are hundreds of papers written on this, each with different findings, opinions, and conclusions. Some people like Cem Karsan specialize in just this domain and run funds profiting from related inefficiencies. Many more funds try but also can lose if they don’t get this right. While the profit/alpha may be created by institutional investors who simply want/need to or are required to hedge. Cost of hedging is expensive, adds large slippage, and is the necessary evil for many large investors and funds, so it should produce substantial alpha for someone else. Not really different from insurance companies collecting insurance from people who need it even if they won’t use it. Then the question is whether all of this alpha is harvested by MMs and automated trading, and whether there is no way to chip at it. Logically there is no monopoly here so chipping at it should be possible. While the smartest person I’ve ever chatted with who consults to MMs, simply told me that MMs don’t do anything creative, just the basic market making and delta+vega hedging. This would mean even more alpha available for the taking. And later I found that MMs also buy otm options for vega hedging, which adds an additional player that may be in need of buying insurance.
Although I suspect that MMs can manipulate option prices, for example buying cheap hedges even at the last minute during an unexpected event, and then cranking up the IV and/or reshaping the IV surface, which can instantly make their hedges more valuable and basically would make it impossible for them to lose. At least I think I see sign of that happening in how the options are being repriced, and can understand certain aspects of convexity from studying how options get repriced during and post-event. This is what let’s me make free or nearly free bets on unexpected events before they may happen, or as they happen.
Btw, during backtests I was also able to find/confirm that simply selling puts can generate alpha, again coming from institutions and anyone else needing to hedge. That alpha is fairly small and can easily be mishandled, but it seems measurable, and can be explained by the necessity of hedging and vol smile. I guess it could be an arb between standard BSM and vol smile where you’d be selling overpriced vol. You’d still lose on the market crashing, but you’d save a few bucks on those sold puts, so you’d have a bit of alpha besides beta. While I’m trying to focus on just the alpha while being somewhat neutral to beta.
Anyway, that’s an argument for the existence of alpha, at least in one aspect.
I’m still working on couple other forms of alpha that can be proven logically, but also being more dependent on beta. Basically you can beat any underlying’s returns, profit off theta while being hedged for crashes, but most of the time having to ride with the underlying when the market doesn’t crash, so still having substantial volatility. So that’s a different approach than I’m showing here, but I believe I could prove the existence of couple sources of alpha if I needed to.
Lastly, couple years ago I’ve seen someone spraying exchanges with thousands of highly complex/mathematical options combos/orders that seemed to be the work of math quants, maybe even worthy of Rentech. And I found them because I came up with similar method but every time I tried to place an order, there was already another such order in the spread book, so I could see someone else’s limit price beyond my own. The option/combo structure had highly stable greeks, similar to buying a box and locking the profit, but it just wasn’t the box. Such combos can be used for market making, just as boxes could when they still worked. So that’s yet another potential source of alpha: market making using only options, without a need to hedge with shares, though small amount of shares can be used to hedge certain option/combo structures that have very stable greeks but have pockets of risk. Basically boxes will no longer work, but there may be other fairly stable option structures that might be used for market making or alpha harvesting, though having pockets of risks. Basically riskless opportunities are gone, but others can show up when taking some risk.
Yet another method of harvesting alpha may be from the market’s efficiency itself! Market efficiency results in the IV following the historical or realized volatility, so some people specialize in trading just the volatility - buying it when it’s low while selling when it’s high. The markets, being efficient, allow some people to profit while other people lose…
I’ll try to address your other questions later.