Quote from Sparohok:
Hi again Prophet,
Another day of this? I guess so. 
Hi Martin, Yeah it is getting old, and taking too much time. Maybe we should call a truce? Agree to disagree?
If by optimization you mean designing good, efficient algorithms, then by all means we are in agreement. But, that is not the generally accepted definition in the computer science community.
Perhaps I am using the term âoptimizationâ a little broadly. The choice and design of an algorithm is usually considered optimization of the overall program. Some of the most successful low-level optimizations are actually algorithmic changes such as loop-order rearrangement.
The design process should be focused on getting the lowest computational complexity. Optimization, on the other hand, is the process of improving performance without changing computational complexity. In other words, good design will change an O(n^2) problem to an O(n log n) problem. Optimization will take an O(n log n) design and make it two, three, four times as fast... but not any more scalable. I seldom waste time with optimization until I run into an unacceptable performance problem. Good design, on the other hand, needs to be built in from day one.
Why split hairs here? Optimization usually means reducing running time. You are saying that simple weakly-algorithmic optimizations like caching of computational intermediates are a waste of time because they donât fall in the category of algorithimic complexity. That goes against common sense.
A profiler will never help you design a good algorithm. It is exclusively an optimization tool. Although I guess it may help you find out when you are using a very bad algorithm, and as such it can be a useful learning tool.
I never said a profiler would produce a good algorithm. Why donât you try replying to the things I actually said?
As for flat files versus databases, consider the problem of filtering tick data for bad ticks. With a CSV file, you read the whole thing into memory, run your filter on it, and write it out again. What happens if the power goes out while you are writing the file? You are screwed, your data is gone.
Donât be silly. Never heard of redundant power? Never heard of computational checkpointing or caching of computational intermediates to disk?
What happens if you want to change just one tick? You still have to rewrite the whole file, or write some pretty hairy code to patch one value in an ASCII file. What happens if the file doesn't fit in physical memory? What if you want to access a single tick? What if you want to delete a tick? An O(1) problem just became an O(n) problem.
Conserning my systems... There is one file per day, per market⦠minimal memory requirements. Data is corrected for gaps when itâs downloaded from the server, each night, ONCE per file, and only if there was a data outage which is rare (5 times in the last year). When the data is demanded by the backtester, the CSV is loaded, filtered for obvious last/bid/ask out-of-line ticks, then written out as binary, fixed record length file, ONCE. Fortunately, my Globex/Interactive Brokers data is very clean. Subsequent requests access the binary file. There is very little overhead in reading a single tick, not that I do that. Just seek to the record in the appropriate file. Normally I load one day, one market at a time to process. Very few algorithms do random access to single ticks! Most look at blocks at a time. You know that right?
What if you want to access the file from more than one program at once? More than one computer? Yes, all these issues can be addressed by writing more code, but someone has already written all that code and more, it is called a database. More important, they've debugged it for you. Why reinvent the wheel?
Just share the files out among multiple computers. I only have one computer writing to a given set of data, partitioned by market. This isnât credit card transaction processing, and does not need the complexity and management issues of a 100 GB transactional database. Itâs fast and efficient market data analysis.
If your methodology is statistically invalid and you are overfitting the data, more market data and more analysis will appear to increase your returns when all it is really increasing is the implementation shortfall. The sheer quantity of data hides the logical flaws in the design. It is much easier to see why something is working or not working when applied to one stock than 1000. Given that the vast majority of backtested strategies do not work in practice, I think this is a real problem.
Back to the competence versus amount of computation debate. Two separate issues, which can interact together in many different ways. Why do you keep confusing these issues, and the discussion?
Iâm no market God. Sure my systems have flaws and imperfections. All systems have flaws. Sure there are incredibly broken methodologies out there, but we're not talking about that incompetent extreme. We're talking about real systems that struggle to find statistically valid patterns in the noise. All things being equal, most statistical measures are more valid over larger data sets. Why should trading systems be different?
Iâm sure not going to reduce the amount of data I process in the hopes that Iâm too incompetent to handle it, or that the extra data is tricking me into thinking Iâm making more money than I am! I now live off my trading system income. My systems trade 6E, NQ and ER2 full time, fully mechanically, 1 or 2 contracts per market, 23 hours/day 5 days/week. I currently operate 51 real time trading systems organized in voting groups, each group managing a fixed position size per market. I am working to bring ES and YM systems online, as well as some European index futures and other FX futures markets too, maybe stocks or options someday too. The point is I see and feel the effect of my actions and choices every single trading day. I can say with great confidence that computaional power and lots of data are tremendous advantages.
