modify an exising automated trading package or build a new one ?

[ecoscien]

Thank you so much for your helpful answers, nbates!

Quote from nbates:


Q. How do you deal with the "hard real-time" computing issues in your trading algo?

A. Multi-threading with an optimal balance between processing logic, shared memory and networking. You should hand-tune Algo's and use real-time 'cached, state aware' techniques.

Is there some book that I can read on the 'cached, state aware' techniques?

I finally found this link, they hide it pretty well:
http://www.progress.com/realtime/products/event_engine/index.ssp

see what you think of their 'distributed in-memory caching', is that really a new software architecture? Is that something that I can build with the way I structure my database and query? Is that their kind of architecture can save you all the time in networking and therefore have a big time margin so that you don't necessarily need to hand-tune your algo?

I heard that their platform is written in Java, is there any reason that Java could be better for this kind of tasks, or just that it is simpler for people to use?

Q. If you can't really predict the possibility of fill and have to change your orders on the fly.

A. Execution is an art in itself. Things like "sweet-spot" discovery for the Bid and Offer, complex-conditional-secondary execution triggers (e.g. on canceled, on filled, on quote change, on time interval w/o completion, etc) and the ability to modify order behavior dynamically and change routing on the fly are a must (imho).

So can you make your algo fast enough to automatically respond to all these situations, with computations?

Q. What about writing algo to track positions and risk management?

A. Waste of time, just use trailing stop orders with modify on the fly 'stop trigger' behaviors.

So that takes care of the risk management part. So what you are saying is that if the risk management part is simplified then the algo for positions tracking is easy.

Q. Order management and risk management software for free, would it be feasible to integrate?

A. Avoid 3rd party 'bits' as much as possible and only integrate when there's an absolute requirement that you can't meet and the benefits are crystal clear.

Got that, I finally got that advice that everyone is telling me here.

Q. Is Java just as good, or better language, for trading algo, since I am starting fresh without any legacy concerns?

A. Most prefer C++, but don't make it too complicated (think 'C'), some like Python and .Net is good for GUI only.

-jmho

That is interesting. I never quite understand why the preference for C++, because people like its class hierarchies or some other reason? So maybe I can write my GUI part in .Net and have my trading algo to call on that module, or call it through Octave? Would you use something like Octave in your trading algo?

It seems that everybody uses Solaris or Linux, because Windows insecure or unstable? Anybody care to comment on operating systems for algo trading?

Thanks a lot.

 

[ecoscien]

Here I finally figured out how to use quotes.

Quote from cosine:

That's pretty much it, yes. Depending on how much data you use.
First you get octave to wait on a dl-function listening on a fifo (or whatever equivalent under windows). If your trading strategy is high-frequency and does not depend on data beyond a short time frame, what you can have is your C++ algo pushing your data (wherever you got it from, tons of providers available) into octave matrices, then sending a fifo msg to octave to execute whatever computation you want from the data, and give the results back through the fifo. Given the results, you can have your algo manage your positions and fills, and send/cancel orders accordingly. No need for a database for that, you can keep that in a mmap (or whatever equivalent under windows). Also make sure you keep logs of all the orders, completed or canceled, just in case you lose your positions.

So fifo is some kind of swtich you use for networking and you use two processors, one run octave all the time, and another the trading algo? and mmap is some kind of temporary database you use? See other posts on questions on networking and memory time.

If you have more data, you may need a SQL database. In that case you may want a stream feeder that stores all the order flow you're monitoring in a db. Then you have to write another dl-function for octave to read the content. Fortunately, you know what kind of data structure you deal with, so this shouldn't be much of a problem. In this case your algo will most likely iterate at some frequency and query octave to see if it finds anything new from the data, using the same method as previously described.

See the other link on progress software and their event streaming processing. So here you are doing the equivalent of what they call 'event streaming processing' (esp) there, what is the relative advantages and distadvantages of this way of doing esp versus their way of doing esp? Comments from everybody welcomed.

There is no need to compile anything with octave. You have to compile with QuickFIX librairies, however. Octave and your trading algo both run standalone, communicating through a fifo, or whatever method you want (if you have a large infrastructure, though a socket, on different machines)

So this octave runs on a separate processor, it doesn't need to be complied even though it is a high level language? on the other hand I do need to compile the trading algo with QuickFIX, of course.

Keeping track of your positions and fills is complicated enough. If your strategy is execution-dependant, all that becomes quite complex. You also have to consider risk allocation, and depending on the expected profitability of the strategy given the type of execution, put on new trades, or cancel previous ones, etc. Using both markets and GTCs, depending on how the trade should be optimally executed.

So what you are saying is that you want to anticipate the outcome of your execution and ask octave to do computations before the fact instead of responding to the outcome of execution after the fact, and do computations on the fly. Is this motivated by a constraint of the speed of the algo?

Ideal is when you can clearly separate the strategy from the trading (strategy does not depend on execution). Then you can manage the strategy and the execution as two separate processes, and you estimate the viability of the strategy given an expected vwap for the trade.

The question here is whether your strategy uses historical data, such as vwap for decision-making, or uses current market condition. The criterior for that I guess is either the trading model or computing speed of algo. I would take that execution is an indication of market condition. But then nbates' post would indicate that it is an art itself, to find the 'Sweet spot'.

 

[mrtwo]

Quote from ecoscien:

A couple of questions (for everyone): how do you deal with the "hard real-time" computing issues in your trading algo, in that you need the intercommunication and computing all be done within short time frame so that you can respond to the market in a timely manner. Of course this is only important if one's strategy is very high frequency or the market very volatile. Also if you can't really predict the possibility of fill and have to change your orders on the fly. ('Hard real-time' as in aviation as opposed to the 'soft real-time' as in video games)

Hey ecoscien,

Only realtime operating systems can do real 'hard-realtime' processing. I am pretty sure there are not many hard-realtime trading systems out there. No chain is stronger than its weakest link, right? Well, in our case the weakest link is the network. TCP/IP isnt 'hard-realtime' so as long there is a TCP/IP network between you and your broker, all you do will be 'soft real time'.

Just as a curiosity, the average current generation game crunches more floating point data per second than a trading system would crunch in a week

Now, about keeping your system response/reaction times low:

-> Make good, educated use of threads
-> Optimize for L2 cache usage: treat RAM the same way you treat your harddisk - not as infinitely fast memory!
-> Do your own memory management, avoid having your application being paged to the disk as much as you can
-> Keep your codebase simple and small, do not overdesign and do not sacrifice performance to favor reuse.

I hope that helps

 

[mrtwo]

Quote from ecoscien:

Is Java just as good, or better language, for trading algo, since I am starting fresh without any legacy concerns? Do you run into memory leak problems with C++, does Java have an advantage in handling real time issues? (My experience is more in Fortran, both C++ and Java are new beginnings for me)

The DMA brokerage seems mostly support C++ and Java, and everybody seems only use Unix or Linex, so I am not sure there is any reason to consider .Net or Python.

If you are concerned about performance, your only choice is C++.

Really.

Java is a great language, but all it is 'great' features comes at the cost of execution perfomance. Same thing for all .Net languages. I personally love C# but it isnt a hammer for all nails, that is for sure.

Python is interpreted and should only be used in situations where performance isnt a critical issue. It is a great, well designed language though

Now, about memory leaks on C++, remember: THEY ARE MAN-MADE. Yeah, I said what no one wants to hear, lol, it is ok, things are the way they are. There are no excuses for memory leaks on C++. really! You allocate, you deallocate, it is all about discipline

But most important thing is: IT IS ALL ABOUT INTENT

What do you want to do? Crazy AI running at high frequency monitoring 200 symbols? That is a machine-language compiled language's job, and by that I mean C/C++

Deterministic systems running on a couple dozen of symbols? That is a Python job for sure!

Anything in between? Java and .Net will do

I would recomend that you sit down for a couple of hours in a quiet room with a piece of paper and write down exactly what are you trying to accomplish, and then make your decision.

 

[nbates]

Adaptive Weight Estimation for Interference Nulling on a DSP
http://www.astron.nl/tl/thea/publications/awe2002_alliot_CTIT_submitted_twente_2002.pdf

The case for real-time visibility
http://www.embedded.com/showArticle.jhtml?articleID=174300427
[great site!]

Real-time Systems with Windows NT
http://www.theragens.com/misc/MR - Windows NT Real-Time.htm

 

[ecoscien]

Thanks, Mrtwo,

Quote from mrtwo:

Hey ecoscien,

Only realtime operating systems can do real 'hard-realtime' processing. I am pretty sure there are not many hard-realtime trading systems out there. No chain is stronger than its weakest link, right? Well, in our case the weakest link is the network. TCP/IP isnt 'hard-realtime' so as long there is a TCP/IP network between you and your broker, all you do will be 'soft real time'.

Good Point.

Now, about keeping your system response/reaction times low:

-> Make good, educated use of threads
-> Optimize for L2 cache usage: treat RAM the same way you treat your harddisk - not as infinitely fast memory!
-> Do your own memory management, avoid having your application being paged to the disk as much as you can
-> Keep your codebase simple and small, do not overdesign and do not sacrifice performance to favor reuse.

I hope that helps

what does 'good, educated use of threads' mean?
How do I manage my own memory?
Thanks.

 

[ecoscien]

Thanks a lot, mrtwo, for the ranking of languages.

I wonder whether any one here can give a comparision of operating systems--- Unix, Linux, or Windows---their algo trading pros and cons. Thanks

Quote from mrtwo:

If you are concerned about performance, your only choice is C++.

Really.

Java is a great language, but all it is 'great' features comes at the cost of execution perfomance. Same thing for all .Net languages. I personally love C# but it isnt a hammer for all nails, that is for sure.

Python is interpreted and should only be used in situations where performance isnt a critical issue. It is a great, well designed language though

Now, about memory leaks on C++, remember: THEY ARE MAN-MADE. Yeah, I said what no one wants to hear, lol, it is ok, things are the way they are. There are no excuses for memory leaks on C++. really! You allocate, you deallocate, it is all about discipline

But most important thing is: IT IS ALL ABOUT INTENT

What do you want to do? Crazy AI running at high frequency monitoring 200 symbols? That is a machine-language compiled language's job, and by that I mean C/C++

Deterministic systems running on a couple dozen of symbols? That is a Python job for sure!

Anything in between? Java and .Net will do

I would recomend that you sit down for a couple of hours in a quiet room with a piece of paper and write down exactly what are you trying to accomplish, and then make your decision.

 

[mrtwo]

Quote from ecoscien:
Thanks, Mrtwo,

Good Point.

what does 'good, educated use of threads' mean?
How do I manage my own memory?
Thanks.

It means, pick your language, study how threading is implemented on it, study threading best pratices, parallel computing theory, scheduling, etc. Study it until you know how to manage threads as well as you can drive your car

About memory, it is quite simple actually, use C++, write your own memory poll and overload the new and delete operators to use it and your good to go!

Parallel, high performance systems are made 80% of education and 20% of code so be prepared to become a master C++ programmer if you want to follow this path

Peace,
MrTwo

 

[mrtwo]

Quote from ecoscien:

Thanks a lot, mrtwo, for the ranking of languages.

I wonder whether any one here can give a comparision of operating systems--- Unix, Linux, or Windows---their algo trading pros and cons. Thanks

I sense this will be the beginning of a flame war lol

DISCLAIMER: I LOVE ALL OPERATING SYSTEMS!

Windows 2003 Server: Use it if you actually want to get anything done. The Visual Studio IDE and its optimizing compiler are the ONLY serious development platform out there right now.

Windows 2000 / XP: NO NO NO NO. (NO)

Linux: Awesome operating system but their development tools are primitive to say the least. Optimizing compilers are still more to 'promised land' than reality at this moment. Even the best Linux based IDE pales in comparison to Visual Studio 2005.

Unix: I gave up Solaris and HP-UX about 5 years go. Simply put, Unix stopped in time

Peace,
MrTwo

 

[Sam123]

Quote from ecoscien:

Does anyone know whether it is easy to write your own indicators that does some more complicated statistical analysis ( is it doable in Easylanguage?) and then put that in an automated trading software package(commercially available ones)? Is there any good auto-trading package out there that allows that?

Unless you tell us the details of your system ideas, you have to find this out for yourself. I would try some commercial trading programs like Tradestation and discover for yourself their limitations with respect to your system design, before you realize it’s worth the extra time, blood, sweat and tears to write your own programs from scratch. Because, alongside the time you need to work your statistical algorithms, you will also need to learn about asynchronous function calls, threading, hashtables, class libraries, and so on.

Quote from ecoscien:

Or it is better (and not too much more work) to just write a new automated trading algorithm (in C++?). How complicated will the program be? Are there any source codes that one can start with and modify?

Thanks a lot for any inputs.

Yes, you can do “anything” if you could write your own automated trading platform, but it will take a lot more of your time. If you are not an experienced programmer it will require a lot of time (a lot of traders in here were programmers-turned-traders and they'll act as if it's easy.) Programming is a lot more accessible and “user friendly” today, but there is no way around the extra-time issue.