Building an ATS - Logbook

Quote from hft_boy:

Why thread? It's expensive, and more importantly, bug prone (it's tricky to get the locking and syncing right). Locking and unlocking can take between 30-100 ns, in C. Assuming your code doesn't actually do much (and there's not much you can do in 600 lines), you're not actually saving that much, and you can save a lot more just by writing more efficient code and data structures. Most threaded code I've seen is written that way because the developers are lazy and start a new thread for every random task they have; they can't be bothered to write it correctly.
More...

Trading a single instrument is ok not to multithread, but I want to have multithreading to handle processing of multiple instruments.
 
Quote from vincegata:

Trading a single instrument is ok not to multithread, but I want to have multithreading to handle processing of multiple instruments.
More...

Okay -- I don't mean to be rude or anything, but why do you need multiple threads for multiple instruments? Just loop through them. Otherwise you have to sync the I/O which is often the most expensive part anyways.
 
Quote from hft_boy:

Okay -- I don't mean to be rude or anything, but why do you need multiple threads for multiple instruments? Just loop through them. Otherwise you have to sync the I/O which is often the most expensive part anyways.
More...

Because processing each instrument takes time, say it takes 10ms, so if I trade 10 instruments it takes 100ms to process them all. That means I am 100ms late on the quotes, so when I start processing quotes in the next loop some of the the quotes will be 100ms old, which is essential during fast moving market. I am not talking low latency, only mid latency. It also depends on the algo, of course. Such delay is okay for a longer term strategies, but for day trading strategies imo such delay should be avoided. You do not agree?

P.S. you're right about I/O - it's a pain.
 
Quote from vincegata:

Looks like an interesting read from a practitioner, looking forward for your review.
More...

A self-published book from someone who appears not to have verifiable experience in the field (at least nothing googling turned up)... not to mention billed as "Part career-travelogue, part self-help book".....

... this doesn't appear to be a serious book.
 
If your strategy can't survive 100ms delay, you are not going to trade the strategy in a retail setting anyway.

(yes yes, threading is not that hard, nor is spawning processes; it's just not that necessary)

Quote from vincegata:

Such delay is okay for a longer term strategies, but for day trading strategies imo such delay should be avoided. You do not agree?

P.S. you're right about I/O - it's a pain.
More...
 
Quote from vincegata:

Because processing each instrument takes time, say it takes 10ms, so if I trade 10 instruments it takes 100ms to process them all. That means I am 100ms late on the quotes, so when I start processing quotes in the next loop some of the the quotes will be 100ms old, which is essential during fast moving market. I am not talking low latency, only mid latency. It also depends on the algo, of course. Such delay is okay for a longer term strategies, but for day trading strategies imo such delay should be avoided. You do not agree?
More...

That's fine reasoning. I agree that if it takes you 10ms to process each quote, you could look into threading. But do you know that it takes 10ms per quote? That is awfully slow, especially considering that 1 ms on a modern computer is 2-3 million clock cycles. Think of it as about 10 million instructions.

There are usually better ways to optimize than multithreading. I don't write much in interpreted languages, so I don't know how slow they can get. But presumably somewhere in there you're looping through the threads trying to figure out which one to give the work to. I don't know exactly what you're doing. But I'm guessing that the overhead of finding the thread, assigning work to it, locking it and the whole shebang probably takes more time than any of your quote processing does. The only way to really know is to profile it, of course.

Also, quick Google indicates that Python threads don't actually work like they are supposed to. Apparently there is a 'Global Interpreter Lock', which means that (to prevent nasty lock and sync on the part of the end user), the interpreter only ever uses one CPU at a time anyways. Ouch. http://kurt.seifried.org/2010/05/31/python-performance-part-2/
 
Quote from hft_boy:

That's fine reasoning. I agree that if it takes you 10ms to process each quote, you could look into threading. But do you know that it takes 10ms per quote? That is awfully slow, especially considering that 1 ms on a modern computer is 2-3 million clock cycles. Think of it as about 10 million instructions.

There are usually better ways to optimize than multithreading. I don't write much in interpreted languages, so I don't know how slow they can get. But presumably somewhere in there you're looping through the threads trying to figure out which one to give the work to. I don't know exactly what you're doing. But I'm guessing that the overhead of finding the thread, assigning work to it, locking it and the whole shebang probably takes more time than any of your quote processing does. The only way to really know is to profile it, of course.

Also, quick Google indicates that Python threads don't actually work like they are supposed to. Apparently there is a 'Global Interpreter Lock', which means that (to prevent nasty lock and sync on the part of the end user), the interpreter only ever uses one CPU at a time anyways. Ouch. http://kurt.seifried.org/2010/05/31/python-performance-part-2/
More...

OP is using Python, I develop using C++/STL/POSIX on Ubuntu, I am almost done with infrastructure and soon to start with strategies whenever I get time for this again. At least I need to calculate SMA and st. deviation on, say 1000 quotes, I haven't done profiling to estimate how long that would take. I'd love to get rid of multithreading however I am afraid I am going to limit myself to 1) the number of instruments, if I want to grow to hundreds of instruments; 2) the strategies, if I find a strategy that would take 1sec to do calculations that would be prohibitively long to run on multiple instruments.

I think it's okay to have some delay for entry, but it should be as fast as possible for exits. Disclaimer: I've traded only with paper.
 
Quote from vincegata:

At least I need to calculate SMA and st. deviation on, say 1000 quotes,
More...

Updating an SMA calculation with each incoming quote is literally one subtraction, one addition, and one division; Granted, these are floating number ops, but the amount of instruction needed for multiple threads will be FAR larger!

(std is a few more instructions)

Know your algorithms!
 
You must log in or register to reply here.
Back
Top