Quote from hftvol:
As NetTecture correctly pointed out there are retail platforms out there that are so badly coded up that they indeed do result in different backtest results over the same data sets, even without any errors on the user's end. Come on, you cannot deny the work that someone like him has put into for a year or so. Its an affront. I have spoken to him and he comes across as honest and knows what he is talking about. And I believe I also know what I am talking about.
I am in regular contact with NetTecture for about 9 months now, and while I respect him for what he does, I also know that he is not the one programming his strategies, he has a couple of guys doing that for him, and who knows how good they really are? Anyway, I am the one who pointed out to him the biggest flaw in his backtests back in July of last year, not his programmers.
But his issue with "random" results is not what it seems. He was using several PCs for backtesting, and was having different results on different PCs. There are many good reasons for this to happen:
- different historical data on different machines
- different rollover dates and/or offsets on different machines
- using even slightly different start/end dates on different machines
Anyone using any type of recursive indicator (for example an EMA, ATR, or something using an EMA) should expect different results if the entire dataseries (start date/time and back-adjustement) is not *exactly* the same.
I am doing thousands of backtests & optimizations with Ninja, and I always get the same results if I do twice in a row the same backtest. But it appears there are many ways Ninja can mess-up the rollover database, and I suspect a bug or two when Ninja reconstruct its data cache, which can lead to different results before/after reconstructing the cache. Yes, this is annoying as a difference in results may not be related to the strategy, on the other hand I know it, and every time it happens the 1st thing I do is backtest a reference strategy, and see if its results are identical or changed. In the later case, it is a data cache issue.
On a more general standpoint, I have been in high-tech R&D (senior) management for over 20 years, and the NIH syndrom (Not Invented Here) remains a major waste of time & money, despite all the software components available (including retail platforms). To each his own, but I guarantee spending time & money to re-develop what's already out there is a recipe for being late to market (as well as be cost-ineffective).
