Building an ATS - Logbook

[hft_boy]

Quote from CT10Gov:

Updating an SMA calculation with each incoming quote is literally one subtraction, one addition, and one division; Granted, these are floating number ops, but the amount of instruction needed for multiple threads will be FAR larger!

(std is a few more instructions)

Know your algorithms!

Yeah. If you know where your bottlenecks are, the catch-all solution is not to just start spawning threads willy nilly, but to attack the bottleneck. In your case, if you're just calculating moving averages, just update one quote at a time. You should be worried about numerical instability -- so just recalculate the whole damn thing once every 100 ticks or something. Bam. If you were spending 95% in that area of the code, now your code is like 20 (amortized) times faster.

If you really start trading hundreds of instruments, and you really need shared memory for something (the best case I can think of for multithreading is in using shared memory for reads and writes), then I can see the case for multithreading. Premature optimization is the root of all evil!

 

[vincegata]

Quote from hft_boy:

Yeah. If you know where your bottlenecks are, the catch-all solution is not to just start spawning threads willy nilly, but to attack the bottleneck. In your case, if you're just calculating moving averages, just update one quote at a time. You should be worried about numerical instability -- so just recalculate the whole damn thing once every 100 ticks or something. Bam. If you were spending 95% in that area of the code, now your code is like 20 (amortized) times faster.

If you really start trading hundreds of instruments, and you really need shared memory for something (the best case I can think of for multithreading is in using shared memory for reads and writes), then I can see the case for multithreading. Premature optimization is the root of all evil!

My trading is a one-man show, it's easier for me to start a new thread than to spend hours on optimizing an algo that might turn to be a flop.

If I think I might be using multithreading in the future I might as well start developing for it now to catch all the bugs while I am trading paper/small.

Non-multithreading is a viable option that I might try. I've actually asked questions on this forum and elsewhere how people design ATS. Thx for input.

 

[2rosy]

Quote from vincegata:

OP is using Python, I develop using C++/STL/POSIX on Ubuntu, I am almost done with infrastructure and soon to start with strategies whenever I get time for this again. At least I need to calculate SMA and st. deviation on, say 1000 quotes, I haven't done profiling to estimate how long that would take. I'd love to get rid of multithreading however I am afraid I am going to limit myself to 1) the number of instruments, if I want to grow to hundreds of instruments; 2) the strategies, if I find a strategy that would take 1sec to do calculations that would be prohibitively long to run on multiple instruments.

I think it's okay to have some delay for entry, but it should be as fast as possible for exits. Disclaimer: I've traded only with paper.

I am unaware of anyone who does a thread per instrument. So you have a connection to some datafeed and all the messages are going down that feed so why would you then thread out a consumer per instrument? As I mentioned before the reactor pattern is what you need and some (not many) threads.

As a side note, I just installed cuda and it is sweet. Noticeably faster from just playing around

 

[vincegata]

Quote from 2rosy:

I am unaware of anyone who does a thread per instrument. So you have a connection to some datafeed and all the messages are going down that feed so why would you then thread out a consumer per instrument?

So to process quotes from each instrument in parallel not sequentially.

As I mentioned before the reactor pattern is what you need and some (not many) threads.

As a side note, I just installed cuda and it is sweet. Noticeably faster from just playing around [/B]

I have yet to look at the reactor pattern - I did not get to it. I remember your post from before. But if it is not multithreaded then it's a single threaded. Let's say not one thread per instrument but one thread per few instruments. I can still change that part though to non-multithreading. Frankly I just do not know how long my algos are going to take to run.

I am waiting for the Xeon Phi to be available to the general public, it should to be easier to program.

 

[hft_boy]

Quote from 2rosy:

As a side note, I just installed cuda and it is sweet. Noticeably faster from just playing around

Cool. What are you using it for?

 

[2rosy]

Quote from hft_boy:

Cool. What are you using it for?

Right now nothing. But I would like to use it for pricing on the fly. Currently, my firm does a lot of precalculating and caching.

 

[Craig66]

Quote from 2rosy:
As I mentioned before the reactor pattern is what you need and some (not many) threads.

+1 for this, boost::asio::io_service is the modern version of this, jeepers, it's been a while since I've seen someone mention ACE.

 

[vincegata]

Just to make sure what ya all touting is to implement a reactor pattern instead of using multithreading which processes quotes for each instrument in a synchronous, sequential way? So if I trade twenty instruments (that's my near plan) then the algorithm will be processing the instruments sequentially.

I still think this is much depends on algo, if algo takes for example more than a half a second (all optimized and dilly) then I do not think it's possible to get away w/o multithreading.

I am going to branch my code to implement reactor pattern version of my strategies execution system and see what happens. Thanks for input, guys.

 

[jack hershey]

Quote from vincegata:

Because processing each instrument takes time, say it takes 10ms, so if I trade 10 instruments it takes 100ms to process them all. That means I am 100ms late on the quotes, so when I start processing quotes in the next loop some of the the quotes will be 100ms old, which is essential during fast moving market. I am not talking low latency, only mid latency. It also depends on the algo, of course. Such delay is okay for a longer term strategies, but for day trading strategies imo such delay should be avoided. You do not agree?

P.S. you're right about I/O - it's a pain.

Change your data sampling to a different scheme. What you suggest is discouraging for readers to continue to read your posts.

 

[vincegata]

Quote from jack hershey:

Change your data sampling to a different scheme. What you suggest is dicouraging for readers to continue to read your posts.

what do you mean?