Holy Grail Trading Software
- Thread starter BullFighter
- Start date
I'm processing every trade for every issue in the TAQ database. I start with 36 months of data (49 GB), which contains 14958 unique CUSIP numbers. I filter this down to about 120 statistical channels per-day per-issue (2.4 GB cached data). Most of these statistics are derived from T&S. That operation takes about an hour. That data gets loaded into Matlab, as needed to design indicators, which are cached between simulations, along with data from any redundant calculations. These additional caches occupy a few more GB. My simulations only examine the top 3000 to 6000 stocks by liquidity. But, to avoid survivorship issues, I rebalance that set and smaller sets rapidly.
The TAL data interface is very similar. In any case, I like to use my own database and file formats, structured as simple 2D or 3D arrays, ordered according to how the data is demanded. If demand patterns are different, I'll just transpose, select or sort rows or columns and cache it again. Matlab can do this in one line of code.
It's not easy! But, you can always learn something as long as you truly understand issues of statistical significance, hindsight bias (overtraining), survivorship bias, and the problems relating to singular systems (inputs that cannot explain outputs). Neural networks and high level AI methods can easily get confused trying to predict aspects of price changes that are inherently unpredictable. System complexity is not the answer. If you cannot generate a reliable forecast for a future variable A, it may help to find some future variables B and C that can be forecasted accurately, then find a route from B and C to reach A. The point is to maximize the statistical significance of all forecasted quantities, and to allow the designer to understand what is going on. A lot of AI methods have convergence issues, and thus can never solidly explain what they are learning.
The most important thing I have learned from designing quantitative systems is: If you don't know how stable your statistics are over time, and over different issues, you are inviting the market to evolve and break your systems. I always start with an over-determined linear system and understand the statistics before I add non-linearities. Otherwise, it's a black box system and you learn nothing except by brute force trial and error.
I'll use whatever 3rd party tool that gives me the most design dexterity, elegance and power. Matlab seems to be best in this regard. As for database issues, that has taken about 2% of my time because the TAQ files are rather simple in structure.
I don't mean to sound argumentative or arrogant. This stuff really fascinates me. I also want to invite criticism and refine my ideas.
Jay
that much quant work is a waste of time unless you are a large organization, if your interested get a job with one of them
complicated is not better, a friend of mine runs a few different, simple mechanical systems and is always ridiculing a few choice competitors.
better is in asking the right questions and interpreting the answers correctly.
complicated is not better, a friend of mine runs a few different, simple mechanical systems and is always ridiculing a few choice competitors.
better is in asking the right questions and interpreting the answers correctly.
Why is it a waste of time? If I can develop and trade a profitable system on my own, why share it? I'm not after really big money. I'm already financially secure. I don't want devote all my life to this either. I don't want to worry about my employer stealing my ideas and systems. I want to work at home, or anywhere I choose, travel while I work, live near my friends and family, and have lots of unrelated hobbies and interests. I would work with other quants if there was more trust in this business.
Who ever said this was complicated? The math behind my systems is utterly simple. You could describe everything in a couple of matrix equations and a few simple algorithms. What good would it be if I can't understand the dynamics of my own systems? Of course, those equations don't include getting the data infrastructure correct, normalizing everything properly, and designing the testing methods. It may seem complicated. But, it's all grounded in common sense statistical and linear algebra theory that you can pick up in two or three quality college courses.
Very true.
How much computation are you doing in real-time? Are you looking at all issues and markets with equal attention? Are your systems heavily dependent on execution-speed? Mine are not. The shortest term market rules seem to evolve the fastest over the years. Do you agree or disagree?
Jay
metoo,
I believe he means that the more compressed the time frame that one has developed rules for, the less likely that (the validity of) those rules are "working" the more time goes by.
I actually believe that as a generalization, what he is saying probably true due to the APT (tho you seem to have proved that wrong in the specific
Still, you are probably reduced to picking up nickels and dimes, and even that may "go away") However, I don't believe _ANYTHING_ works in the "long" term, save Arb a la LTCM, and buy and hold forever...nitro
Originally posted by nitro
metoo,
I believe he means that the more compressed the time frame that one has developed rules for, the less likely that (the validity of) those rules are "working" the more time goes by.
I actually believe that as a generalization, what he is saying probably true due to the APT (tho you seem to have proved that wrong in the specific
nitro
Nothing works forever; things work, stop and start again.
Picking up nickels and dimes in front of a steam roller is what we do; to plagiarize someone who said it first.