Thanks for the generosity with your time and thoughts as always. To add some context to my motivations, I run an automated high/mid frequency equities business focused on the entire US listed stock/etf universe. It's over 10k transactions/day, so I'm wary of adding any additional data management headaches/chores. As you'd expect, capacity is a problem so I'm hoping to incorporate the AFTS trend following ideas as part of a longer term portfolio with stocks/bonds/RE. I only share that to frame some of my views on ETFs vs futures data handling. To some of your points:
"Well in fact to trade ETFs properly you would have about as much pain as you have to keep track of dividends unless you stuck to accumulated only, but if you are only going to use indicies then I suppose that's not a problem, although the logistics of shorting and borrowing are MUCH worse."
There's no doubt that I'm biased based on already having systems in place for ETFs. I actually didn't mind the stitching together of the futures contract series and back-adjusting as it's essentially a one time programming project, but the amount of data quality issues I ran into for mature contracts using a mature data provider (IQFeed) surprised me. Also, the lack of symbol mapping between broker/data provider/exchange seemed medieval. Lastly, and this is strictly personal, I had no confidence that my lack of experience with some of the contracts in a large universe wouldn't result in my doing something immensely stupid and potentially catastrophic (miss a roll, take delivery, bungle negative prices, or some other unknown thing).
--Incidentally, I do have historical borrow availability and rates (at IBKR) back to mid-2013 for US stocks and ETFs...I'd be more than happy to share the files if the need ever arises while you're writing book #5 or conducting other research.
"not sure I follow here, ETFs have management costs, and except for a few cases the risk adjusted trading cost is usually higher than in the futures. You would also probably pay more to get leverage than in futures. I do a full comparison in book #3 LT. Lower costs is a poor reason to"
I agree about the expense of portfolio margin (although there are ways to mitigate it), but my findings were a bit different on the other costs. Free commissions and executing in the closing auction make execution costs near-zero (still minor slippage possible in the auction if size is enough to tip the imbalance). Also, I would think management costs are largely a wash in long/short strategies. The exception would be in assets where you're primarily long (e.g. equities), but those are typically where expense ratios are de minimis. Anyhow, I'm not trying to make a case or be argumentative; just hoping to add to the discussion. I have two of your books but LT isn't one of them - I'll get a copy and see what you've written on the topic.
"EDIT: I just saw your later post; OK if you want to replicate some index that can be replicated with a perfect regression on some ETFs, then do that regression! Don't do it implicitally by plunging into an optimisation, that's batshit crazy and your eignevalues will be bonkers."
Ha! We're getting to the point where I should be typing less and thinking more...but I wanted to clarify something if you don't mind. So I start with my fantasy universe of 1000 independent indices that has a trend following SR >5. Each night, I calculate the optimal exposures for the indices the next day. Now I want to replicate those exposures using some subset of ETFs. Are you suggesting that I regress the ETFs against that night's optimal set of exposures in the indices? Or that I have a replicating basket for each of the indices in my fantasy universe (based on regressions) and then run them through the optimization process? I am aiming not to drop the optimization process because I ultimately need to make trades in the real world and it neatly deals with various constraints like expenses, borrow availability, etc..
Again, I appreciate the willingness to indulge me on this. I get that it's likely garbage in = garbage out, but I have a nagging feeling that I'll learn something useful.
)