Sharing Automated Trading System Design or Code?

[prt_systems]

Quote from stephencrowley:

Which libraries specifically? I've learned a lot writing my own matrix package.. I think it is robust enough to release publically but don't want to go thru the trouble right now..
specifically most matrix libs/languages are fairly ineffecient

for instance, a matrix transpose actually creates a fully copy of the matrix, whereas my package makes the new transposed matrix share the same underlying native buffer and only swaps the num rows/cols and the col/row increments.. many of my native fortran libs can deal with matrices such as these and it is not necessary to do a full matrix copy before doing some operation. If a native func actually needs a column-major matrix when I just do 'matrix.trans().copy()' before passing it along in the case where a full copy really is necessary.

Same concept goes for matrix slicing, reversing.. swapping, etc.

I've reimplemented the unscented kalman filter with matrix package, translated directly from the matlab version.. it is about 15% faster and infinitely more readable and fully object oriented.

Things like this make it more apparant about the memory access and runtime of your algorithms and makes it more readily apparent where optimizations can happen.

I am surprised that you would tackle such an enormous task - unless I am missing something writing general purpose solvers is no simple task . I have written a couple for commercial products from scratch - outside of the BLAS.

I really have not had any issues using Atlas nor interfacing with it.

We dont use Matlab - for some of the reasons you mention but there are many other options, especially with respect to filtering.

Of course I dont know all the details of your system but it sure sounds like you are spending time writing things that are already available to be integrated and which offer performance that would only be an issue in the largest, most demanding of applications - far beyond those of even a very ambitious ATS.

 

[stephencrowley]

I've used R and gnuplot.. and octave extensively before switching to matlab for serious analysis.. I'll definately check python out again. I've heard of some python->java bytecode compilers which might integrate well with my existing code. I'll put it on my todo list.

And I didn't write any complicated solvers myself.. just spent time interfacing them into java by writing native wrappers around them and making sure the OO design matches the reality of the lowlevel underyling stuff (fortran, atlas, C, whatever)

Quote from nononsense:

Hi Stephen,

To get a quick idea, start looking at:
http://www.scipy.org/
http://www.r-project.org/
http://rpy.sourceforge.net/
http://www.gnuplot.info/
http://gnuplot-py.sourceforge.net/
also MLab.py and matplotlib.py modules.

These above libraries contain an unbelievable wealth of thoroughly tested and optimized code going back 20 years or more. In fact some falls back on hard to beat Fortran. Most of this would be impossible to duplicate during an individual's lifetime. Before you code - first look. To acquire these for other platforms cost a fortune (or used to?). In fact MLab is still very pricey!

Your experience of transcribing MatLab code into any serious language is revealing! For those who don't know, doing this, quickly teaches the difference! (You ought to try some in Python.)

 

[stephencrowley]

Quote from prt_systems:

I am surprised that you would tackle such an enormous task - unless I am missing something writing general purpose solvers is no simple task . I have written a couple for commercial products from scratch - outside of the BLAS.

Well, you are right.. it took me about 2-3 weeks to re-write the square-root version of the unscented kalman filter in java.. but the bulk of that was teaching myself how the algorithm works.. the *only* working implementation comes in the form of matlab code from the authors. I read all the papers and understood how the algorithm works but to really understand it in and out I basically had to rewrite it. It's basically a "black-box" algo that requires no analytic derivatives.. highly effecient.. thus handling online optimization of functions that could be time consuming and error prone to differentiate.

I really have not had any issues using Atlas nor interfacing with it.

We dont use Matlab - for some of the reasons you mention but there are many other options, especially with respect to filtering.

I didn't have any problems interfacing with atlas, but building a powerful and intuitive interface on top of it (or any fortran native code) took a bit of design.. but now that it's done I can interface with nearly any native fortran code in about 5 minutes and very cleanly.

As far as filtering, there were no alternatives for the type of filter I needed here.. only working implementation was a non-optimized reference implementaion done in matlab.

Of course I dont know all the details of your system but it sure sounds like you are spending time writing things that are already available to be integrated and which offer performance that would only be an issue in the largest, most demanding of applications - far beyond those of even a very ambitious ATS.

I've spent a lot of time making all my of stuff pluggable, well designed and generic. I learn by taking things apart and putting them back together.. I've read all the papers but to get a true feel of how all of these unrelated algorithms fit together I really have to build it myself.

The good thing is 99% of my framework is done and I get to spend all of my time now on the fun stuff..

 

[prt_systems]

Quote from stephencrowley:

... the *only* working implementation comes in the form of matlab code from the authors. ...

..

Well .... Yes ...The normal flavors of Kalman filtering techniques can veer offcourse in highly nonlinear situations ... however, in many cases there is no need for the unscented version ... although as you know the algorithm is in many ways just better with respect to implementation and is a good almost general purpose replacement - there are still cases where you want to compute jacobians.

The *only* working implementation ... ? No not really but probably the only working implementation you can download off the net ....

 

[stephencrowley]

When would you need jacobians? And yes.. freely availably implementation was implied..

Sigma-point particle filters are really interesting as well.

Quote from prt_systems:

Well .... Yes ...The normal flavors of Kalman filtering techniques can veer offcourse in highly nonlinear situations ... however, in many cases there is no need for the unscented version ... although as you know the algorithm is in many ways just better with respect to implementation and is a good almost general purpose replacement - there are still cases where you want to compute jacobians.

The *only* working implementation ... ? No not really but probably the only working implementation you can download off the net ....

 

[ptunic]

Quote from nononsense:

Your experience of transcribing MatLab code into any serious language is revealing! For those who don't know, doing this, quickly teaches the difference! (You ought to try some in Python.) [/B]

Btw nononsense, I took your advice and and looked into Python a month ago finally. I totally agree with you now, it is an incredible language and I am now using it as my language of choice.

For speed reasons, I can see how some applications would require compiled languages like C++ potentially, but for my applications so far it has not been a limitation.

Also this is more web-related, but I don't like the mod-python Publisher module (since it only dynamically recompiles the .py file in the URL without any dependancies in custom modules that are called) so I wrote a quick little 20-line replacement that dynamically checks the datestamp of the specified library directory and loads the modules into memory if they have been refreshed. This makes web development with mod-python much easier, since you don't need to restart the web server. For production you would turn this off obviously. Let me know if you want me to email it to you.

Good stuff

-Taric

 

[nononsense]

Quote from ptunic:

Btw nononsense, I took your advice and and looked into Python a month ago finally. I totally agree with you now, it is an incredible language and I am now using it as my language of choice.

For speed reasons, I can see how some applications would require compiled languages like C++ potentially, but for my applications so far it has not been a limitation.

Also this is more web-related, but I don't like the mod-python Publisher module (since it only dynamically recompiles the .py file in the URL without any dependancies in custom modules that are called) so I wrote a quick little 20-line replacement that dynamically checks the datestamp of the specified library directory and loads the modules into memory if they have been refreshed. This makes web development with mod-python much easier, since you don't need to restart the web server. For production you would turn this off obviously. Let me know if you want me to email it to you.

Good stuff

-Taric

Taric,
Happy you like it.
As to web applications, I'm not really an expert on these. Following the newsgroup regularly, I often see references and discussions about this. I would say that many avenues seem to be available. Your above procedure looks interesting. Maybe you should raise this question while showing your approach. You will be surprised about the possibly many replies.

In fact, Ruby also appears to be a quite convincing programming tool. However, almost every day I realize more and more that Ruby is no match for Python when you look at the wealth of available libraries and tools.

As to speed. Since I switched 100% to Python, I am still amazed about the few times that I still had to write in C/C++. In fact, the only times I seem to touch it is to wrap existing C/C++ code to make it callable from Python. Of course C and C++ remain great tools as Python is built on it.
Python "speed" is another interesting topic of discussion - raw speed is absolutely meaningless. Very interesting developments are in gestation, e.g. PyPy.

Be good,
nononsense