I recall reading acrary's posts around mid 2000s. He made a great impression on me back then.
Lately I was looking back at it and I can't help but think that not everything is as perfect as it seemed.
For example the edge test which is supposed to prevent overfitting. It's a p-value against random results. It's very easy to overfit - by definition a certain fraction of tested systems will pass the randomness test simply by random chance. It's a basic multiple comparisons problem.
Next there was the famous thread "System Development with acrary". It could be summarised as optimisation of portfolio weights for maximum Sharpe. His method was an exhaustive iteration through all weight combinations. However completely unnecessary as there are analytical solutions that don't require exhaustive search, namely Markovitz optimisation - stuff that's been known since the 1950s. It's yet another exercise of naive overfitting, since Sharpe ratio estimations typically have high uncertainty. In many cases an equal weights portfolio outperforms naive optimisations.
Lastly, the alleged performance. I checked the thread, there's an example of 5 trading systems portfolio (4 for ES and 1 for NQ) that shows Sharpe ratio of 1.61... monthly! That's approximately Sharpe ratio of over 5 annualised. Are we supposed to believe that it is achievable with a few day trading systems on a couple of index futures markets? It is simply unrealistic. There are real world cases having annual Sharpe ratios of 2-3 (excluding HFTs and option selling blow up artists). I can think of a couple, one is the famous Renaissance Technologies certainly trading more than a couple of index futures markets and the other Ed Thorp's Princeton Newport Partners running statistical arbitrage on thousands of stocks.
Let's assume the performance was true after all. With this kind of ability one would be at least a billionaire in short order. We would very likely know about this, yet there's nothing out there.
A secret billionaire or a fake guru?
Lately I was looking back at it and I can't help but think that not everything is as perfect as it seemed.
For example the edge test which is supposed to prevent overfitting. It's a p-value against random results. It's very easy to overfit - by definition a certain fraction of tested systems will pass the randomness test simply by random chance. It's a basic multiple comparisons problem.
Next there was the famous thread "System Development with acrary". It could be summarised as optimisation of portfolio weights for maximum Sharpe. His method was an exhaustive iteration through all weight combinations. However completely unnecessary as there are analytical solutions that don't require exhaustive search, namely Markovitz optimisation - stuff that's been known since the 1950s. It's yet another exercise of naive overfitting, since Sharpe ratio estimations typically have high uncertainty. In many cases an equal weights portfolio outperforms naive optimisations.
Lastly, the alleged performance. I checked the thread, there's an example of 5 trading systems portfolio (4 for ES and 1 for NQ) that shows Sharpe ratio of 1.61... monthly! That's approximately Sharpe ratio of over 5 annualised. Are we supposed to believe that it is achievable with a few day trading systems on a couple of index futures markets? It is simply unrealistic. There are real world cases having annual Sharpe ratios of 2-3 (excluding HFTs and option selling blow up artists). I can think of a couple, one is the famous Renaissance Technologies certainly trading more than a couple of index futures markets and the other Ed Thorp's Princeton Newport Partners running statistical arbitrage on thousands of stocks.
Let's assume the performance was true after all. With this kind of ability one would be at least a billionaire in short order. We would very likely know about this, yet there's nothing out there.
A secret billionaire or a fake guru?
I member’