The apparent reality here is that EVERY person who trades an automated system is experiencing these issues. Some are making money at a certain level and do not question, remaining content. Others struggle on a daily basis. 95% of the streets automated traders have no clue and simply "trust" their brokers. If they do question their broker, they usually ask the wrong questions not knowing the correct ones. The reality is that they are missing out big time and don't even know it.
Why is it that ALL brokers do not understand true latency?
- Thread starter sigsegvboogman
- Start date
How much capital would it take to develop your Tech Approach Idea divorced from the whole cost of setting up a brokerage in total...
Cost $
cj...
_________________
HAVE STOP - WILL TRADE
If You Have The Vision We Have The Code
Interesting question. Well, if the broker was existing, I imagine the process would be as follows:
1. Examine the current set up (From api software, feed software, all hardware and all network components).
2. Take detailed latency measurements of all existing equipment as is when we walked into the place. This would mostly be used by in house tools we built.
3. Based on all of information gathered, make recommendations from : Hardware and OS as a first level. Also, how to tune that OS for stability and performance. There is a way that 99% overlook when it comes to tuning Linux (If this was what they used). As a second and deeper level: analyze the source code of all software (with an NDA of course) and make recommendations as to how to write or re-write the architecture. (FYI, I'm laughing as I type this. How many firms would actually admit they needed help and allow this to happen. I would really have respect for a firm if they actually admitted this). We have extensive experience in writing very low level software (IMDB's, feed gateways, execution software).
4. When all was said and done and improvements were in place: Run a new analysis of latencies within the brokers systems. Compare and show them the difference.
Cost would be a tough question. A smaller approach would be to examine all hardware and operating systems. Just changing that on existing legacy API/feed software would be a huge improvement (once the OS was tuned properly, not stock or simply compiling the kernel as best as you can). A deeper and more costly approach would entail as stated above. My guess is that it would be based on the scope of the project. It wouldn't be something a broker couldn't afford. In fact, I easily believe it would pay itself off very fast. And after we were done, nobody could ever complain about latencies within the broker for a very long time. My guess 5 years. Any latencies would be Data center related, or client related. The broker would be free.
I would love to have us do it on a consulting basis, but there would be a huge barrier to over come. Just getting the in house developers, network engineers and anyone else involved to allow someone from the outside to come in and tell them how to do their job would be impossible without friction. Our expertise is very narrowly scoped, but very important and overlooked 99% of the time. I am not the best network engineer in the world, but I know exactly where to look.
1. Examine the current set up (From api software, feed software, all hardware and all network components).
2. Take detailed latency measurements of all existing equipment as is when we walked into the place. This would mostly be used by in house tools we built.
3. Based on all of information gathered, make recommendations from : Hardware and OS as a first level. Also, how to tune that OS for stability and performance. There is a way that 99% overlook when it comes to tuning Linux (If this was what they used). As a second and deeper level: analyze the source code of all software (with an NDA of course) and make recommendations as to how to write or re-write the architecture. (FYI, I'm laughing as I type this. How many firms would actually admit they needed help and allow this to happen. I would really have respect for a firm if they actually admitted this). We have extensive experience in writing very low level software (IMDB's, feed gateways, execution software).
4. When all was said and done and improvements were in place: Run a new analysis of latencies within the brokers systems. Compare and show them the difference.
Cost would be a tough question. A smaller approach would be to examine all hardware and operating systems. Just changing that on existing legacy API/feed software would be a huge improvement (once the OS was tuned properly, not stock or simply compiling the kernel as best as you can). A deeper and more costly approach would entail as stated above. My guess is that it would be based on the scope of the project. It wouldn't be something a broker couldn't afford. In fact, I easily believe it would pay itself off very fast. And after we were done, nobody could ever complain about latencies within the broker for a very long time. My guess 5 years. Any latencies would be Data center related, or client related. The broker would be free.
I would love to have us do it on a consulting basis, but there would be a huge barrier to over come. Just getting the in house developers, network engineers and anyone else involved to allow someone from the outside to come in and tell them how to do their job would be impossible without friction. Our expertise is very narrowly scoped, but very important and overlooked 99% of the time. I am not the best network engineer in the world, but I know exactly where to look.
I agree. What specifically are the bottlenecks you've found in concrete terms. Are they using ridculous hashing algorithms or haveingconcurrency issues? tcp stack too slow? Ineffecient queing/locking algorithms?
Most of what is being discussed here is sillines anyway. There are always going to be Internet latency when you are using a public network unless you get a PTP connection to your broker.
Secondly, the latency that should be being judged most is not internet latency because there is no control in it (except of choice of ISP) by either the client or the broker without a PTP, but the latency in the order once it gets to the broker (the order goes through middleware of all sorts, like business logic and transaction processing.) The best brokers have written the critical areas of their software by hand to overcome these issues and have optimized them in clever ways.
Anyone that has ever used IB or Redsky Securities can tell you that when you send an order "by hand" to an ECN using one of these brokers, before the mouse button comes up the order is already at the ECN. If you send in the order automatically by computer the roundtrip times are measured on the order of 15-20 milliseconds over a public network and faster still over a PTP. These brokers have PTP 100 MBit connections to the ENCs.
Finally, I have seen brokers that implement their datafeed collection/dissemination correctly. This can be huge and what may seem like lag to you in other parts of the transaction, could very easily be that you are judging the rest of your experience with them on the one thing that can make everything else worthless even in inplemented correcly - the efficiency of collecting and disseminating the data feeds from exchanges or aggregators. If you are (mis)judging a broker's execution (particularly limit orders) or freshness of data based on the data you get from a third party data vendor like say eSignal, you are never going to know where the lag truly resides. Garbage in Garbage out.
nitro
Secondly, the latency that should be being judged most is not internet latency because there is no control in it (except of choice of ISP) by either the client or the broker without a PTP, but the latency in the order once it gets to the broker (the order goes through middleware of all sorts, like business logic and transaction processing.) The best brokers have written the critical areas of their software by hand to overcome these issues and have optimized them in clever ways.
Anyone that has ever used IB or Redsky Securities can tell you that when you send an order "by hand" to an ECN using one of these brokers, before the mouse button comes up the order is already at the ECN. If you send in the order automatically by computer the roundtrip times are measured on the order of 15-20 milliseconds over a public network and faster still over a PTP. These brokers have PTP 100 MBit connections to the ENCs.
Finally, I have seen brokers that implement their datafeed collection/dissemination correctly. This can be huge and what may seem like lag to you in other parts of the transaction, could very easily be that you are judging the rest of your experience with them on the one thing that can make everything else worthless even in inplemented correcly - the efficiency of collecting and disseminating the data feeds from exchanges or aggregators. If you are (mis)judging a broker's execution (particularly limit orders) or freshness of data based on the data you get from a third party data vendor like say eSignal, you are never going to know where the lag truly resides. Garbage in Garbage out.
nitro
I cannot do that. I am currently trading with the broker I used as an example (and alot of you may be as well) and it wouldn't be fair seeing how, in my opinion and based on my experience; every other one out there is in the same boat with regards to where they are with latency and where they easily could be if they just listened. As I have said: They just need to step up , admit it or at least be open to the idea that they are not as fast and stable as they could and should be.
With regards to latency: I have spoken to them all. I have requested stats from most of them. They all come back the same. Some stronger in one area, but weaker in others. Some are co-located better than others. Others give you execution reports on the latency through their API. They are proud at "1000" executions per second, which isn't a slow number, but I ask: What is the latency at that load? What is the CPU level at that load? How does it compare to the the latency of one order? You know, sometimes 1 order is slower than 1000. They give me really "low" numbers as far as latency and in EVERY case, my results and systems are around 900-1000 faster. Why? They never look that deep and think their "time" is acceptable.
The things that matter with regards to execution API latency:
1. Stability
2. Consistency - meaning no matter the large load or small load of traffic, latencies do not change, or I should say: Nothing within your broker system should change them. Yes folks, it is possible if you know what you are doing.
I really hate it when in pre market, executions to a destination are 2 milliseconds average and when regular market came in executions jumped to an average 3 milliseconds. You just slowed down 33%. Now, that doesn't even factor in the "Red Hat 7.3 feed latencies" on the feed side. Everything mentioned was solely execution. Sometimes it's not the API that caused the slow down because I know that ECN traffic heats up and changes loads, but I guarantee that feed server is not having fun. But that's transparent to you if you never thought about it. And if you did think about it, I guarantee you don't config it to 100% to it's ability.
The things that matter with regards to execution API latency:
1. Stability
2. Consistency - meaning no matter the large load or small load of traffic, latencies do not change, or I should say: Nothing within your broker system should change them. Yes folks, it is possible if you know what you are doing.
I really hate it when in pre market, executions to a destination are 2 milliseconds average and when regular market came in executions jumped to an average 3 milliseconds. You just slowed down 33%. Now, that doesn't even factor in the "Red Hat 7.3 feed latencies" on the feed side. Everything mentioned was solely execution. Sometimes it's not the API that caused the slow down because I know that ECN traffic heats up and changes loads, but I guarantee that feed server is not having fun. But that's transparent to you if you never thought about it. And if you did think about it, I guarantee you don't config it to 100% to it's ability.
.... or you could just accept the fact that this is part of playing the game and figure out an advantageous way (for you) to deal with it. That's what entreprenurialism is all about. Maybe even start your own broker and then let the market decide if it agrees with you.
Thanks for the post Nitro,
First off, none of this discussion has anything to do with Internet latency. Everything I discussed is internal to the broker.
Yes, alot of brokers write their API correctly, but I 100% guarantee that the way they are running that very fast API through say a "Linux" OS is much slower than it could be. And i'm not saying hardware needs to be "overclocked" or throw the "nitro" patch in the kernel (no pun intended). They just don't optimize the OS so that the software can perform even better and you are just accepting to the current results, not realizing there is a bigger world out there. My systems live in that world.
First off, none of this discussion has anything to do with Internet latency. Everything I discussed is internal to the broker.
Yes, alot of brokers write their API correctly, but I 100% guarantee that the way they are running that very fast API through say a "Linux" OS is much slower than it could be. And i'm not saying hardware needs to be "overclocked" or throw the "nitro" patch in the kernel (no pun intended). They just don't optimize the OS so that the software can perform even better and you are just accepting to the current results, not realizing there is a bigger world out there. My systems live in that world.