Tibco

[byteme]

Quote from nitro:

I just claim that mixing high frequency trading and this stuff can be hazardous to your pocket book.

Of course is has value in general.

OK fair enough. If the "high" in "high frequency" is high enough to warrant spending significant effort performance tuning and optimizing your trading code (in assembly language of course), then throwing a non-optimized third-party message middleware into the mix doesn't make sense.

It depends on the specific non-functional requirements of your trading system e.g. latency, throughput, reliability. You can have "good enough" today with an off-the-shelf messaging middleware or "much faster" tomorrow with a roll your own.

Opportunity cost aside, your trading system simply might not make money with a "good enough" solution because it's edge is dependent on not having the latency introduced by the middleware. I suspect this may be your situation.

However, there are several use cases where the message middleware is still useful even in the highest high-frequency trading systems such as your recent use case of distributed logging where the logging doesn't necessarily have to happen NOW.

Even when you have to throw out MQs that have some inherent latency due to the fundamental architecture where messages from producer to consumer have to travel via an intermediary server/broker (i.e. all JMS providers that I know of) there are still other options that I'd personally look at before "rolling my own".

Specifically, I might look at OMG's Data Distribution Service (DDS) for real time systems initiative:
http://www.omg.org/docs/formal/05-12-04

Some open source implementations:
OpenDDS (http://www.opendds.org)
OpenSplice (http://www.opensplice.org)

A releveant webcast from the commercial makers of opensplice:

http://www.opensplice.com/section-item.asp?id=816

I personally, haven't reached a point where I need to look at this technology in anything more than a cursory way. Your situation may be different.

 

[nitro]

Quote from byteme:
OK fair enough. If the "high" in "high frequency" is high enough to warrant spending significant effort performance tuning and optimizing your trading code (in assembly language of course), then throwing a non-optimized third-party message middleware into the mix doesn't make sense.

That's right. And it would be programed in C++, not assembler.

It depends on the specific non-functional requirements of your trading system e.g. latency, throughput, reliability. You can have "good enough" today with an off-the-shelf messaging middleware or "much faster" tomorrow with a roll your own.

In the trading world, there is no such thing as an unreliable live system that auto trades. It is never a compromise you are willing to give up. People adapt their style of trading to fit poor systems. But no one says consciously, ok I have a terrible latency system, let's make markets in options. You will get obliterated.

If your systems are unreliable, you have a team poorly matched to the job at hand.

Opportunity cost aside, your trading system simply might not make money with a "good enough" solution because it's edge is dependent on not having the latency introduced by the middleware. I suspect this may be your situation.

You argue to argue. The original post in this thread was about high frequency trading and using off the shelf middleware. I have already put forth that these things are useful in some situations, just not this one. I am astonished when I see the two together by firms that you would think know better.

However, there are several use cases where the message middleware is still useful even in the highest high-frequency trading systems such as your recent use case of distributed logging where the logging doesn't necessarily have to happen NOW.

Yes, that would not be part of the core systems. Just because a module is linked in to a program doesn't mean it is part of the high frequency core. Often front ends to high frequency programs are written in C#. That doesn't mean you will ever find C# on the server side on high frequency, low latency automated systems.

Even when you have to throw out MQs that have some inherent latency due to the fundamental architecture where messages from producer to consumer have to travel via an intermediary server/broker (i.e. all JMS providers that I know of) there are still other options that I'd personally look at before "rolling my own".

Specifically, I might look at OMG's Data Distribution Service (DDS) for real time systems initiative:
http://www.omg.org/docs/formal/05-12-04

Some open source implementations:
OpenDDS (http://www.opendds.org)
OpenSplice (http://www.opensplice.org)

A releveant webcast from the commercial makers of opensplice:

http://www.opensplice.com/section-item.asp?id=816

I personally, haven't reached a point where I need to look at this technology in anything more than a cursory way. Your situation may be different.

 

[rufus_4000]

I guess I will throw my $0.02 in. If the question is general "off the shelf middleware" vs "rolling your own specializing messaging", then I think the answer would be "it depends" on your perspective.

In one of my previous incarnations, as divisional CTO at an ibank, I would probably kill any projects that proposes to "roll their own messaging", as versus "off the shelf solutions. The business risk there is just too great, the big technology push ibanks is to "consolidate" computing platforms, not making them more disjointed. The managability of "home grown messaging", usually does not have the capability (at minimum SNMP), fault tolerance, etc, that a global firm requires.

Note I made a distinction between "messaging" and "middleware". To me, messaging is only the base line capability of a "middleware stack", it provides basic, prototol level recovery, and protocol compliance, not much else. Middleware to me, has someway to expose message definition, fully integratible with network monitoring and system monitoring capabilities, guaranteed 24x7 up time, and full extensibility. So hence, to me, FIXFAST is a messaging specification, not a middleware specification.

Now, in my current position, as owner and head of a trading firm. We did *exactly* that, aka, rolling our own messaging infrastructure. We started off using Spread (a very good, robust, open source messaging toolkit), and we changed Spread so much, that we ended up using a custom subset of it, that has very little resemblance to th original source package. But again, since we have a full open source oriented computing infrastructure, it is not hard to plug *all* of our platform services into it.

So, it depends on which perspective you have. That's my take anyways

On to TIBCO a little bit, like some1 else mentioned, back 6-8 yrs ago, TIB-RV was really the only game in town. And don't forget, TIBCO Hawk (th SNMP based network monitor), was one of the best Enterprise cross segment network monitor too. So that's why a lot of large firms chose TIBCO (not just RV). And 29West, is basically part of the original Talarian team, after their non-compete expired, and the product looks awfully like what Talarian was work on (next gen), before they got bought by TIBCO. And frankly, Talarian was never good.

 

[nitro]

I have nothing more to add.

Quote from rufus_4000:

I guess I will throw my $0.02 in. If the question is general "off the shelf middleware" vs "rolling your own specializing messaging", then I think the answer would be "it depends" on your perspective.

In one of my previous incarnations, as divisional CTO at an ibank, I would probably kill any projects that proposes to "roll their own messaging", as versus "off the shelf solutions. The business risk there is just too great, the big technology push ibanks is to "consolidate" computing platforms, not making them more disjointed. The managability of "home grown messaging", usually does not have the capability (at minimum SNMP), fault tolerance, etc, that a global firm requires.

Note I made a distinction between "messaging" and "middleware". To me, messaging is only the base line capability of a "middleware stack", it provides basic, prototol level recovery, and protocol compliance, not much else. Middleware to me, has someway to expose message definition, fully integratible with network monitoring and system monitoring capabilities, guaranteed 24x7 up time, and full extensibility. So hence, to me, FIXFAST is a messaging specification, not a middleware specification.

Now, in my current position, as owner and head of a trading firm. We did *exactly* that, aka, rolling our own messaging infrastructure. We started off using Spread (a very good, robust, open source messaging toolkit), and we changed Spread so much, that we ended up using a custom subset of it, that has very little resemblance to th original source package. But again, since we have a full open source oriented computing infrastructure, it is not hard to plug *all* of our platform services into it.

So, it depends on which perspective you have. That's my take anyways

On to TIBCO a little bit, like some1 else mentioned, back 6-8 yrs ago, TIB-RV was really the only game in town. And don't forget, TIBCO Hawk (th SNMP based network monitor), was one of the best Enterprise cross segment network monitor too. So that's why a lot of large firms chose TIBCO (not just RV). And 29West, is basically part of the original Talarian team, after their non-compete expired, and the product looks awfully like what Talarian was work on (next gen), before they got bought by TIBCO. And frankly, Talarian was never good.

 

[jjw]

perspective is the key. high frequency and low latency messaging and middleware is an arms race. my firm wites this stuff, licenses it and gets paid for it, but we always run the risk of a competitor launching a new and lower latency product (or even a claim of one), and worse, a private firm launching its own lower latency product for its own use.

Before jumping into the "lowest latency" game, a trading firm should answer the following questions:
1. does the trading algorithm absolutely, positively need the lowest latency to be consistently profitable ?
2. can the algorithm tolerate the fact that the latency of market data from an exchange is variable and is at least 1 magnitude greater than the latency that can be acheived by off the shelf products and by home grown products ?
3. does the trading firm have the resources to constantly focus on low latency issues and is it ok with putting those resources to work on computing issues instead of on trading issues ?
4. can greater low latency be achieved by changing other aspects of the trading system rather than by developing and maintaing one's own messaging and middleware software ?

 

[nitro]

Quote from jjw:

...

Before jumping into the "lowest latency" game, a trading firm should answer the following questions:
1. does the trading algorithm absolutely, positively need the lowest latency to be consistently profitable ?

I disagree with the way this is phrased. In my case, if your latency is worse than another firm, you get picked off (picked off is a term meaning you got arb'ed). Period. There is no second place.

2. can the algorithm tolerate the fact that the latency of market data from an exchange is variable and is at least 1 magnitude greater than the latency that can be acheived by off the shelf products and by home grown products ?

Again, this is focusing on the wrong issue. If this is true for me, it will be true on average for firms that I am in competition against. Hence, we can normalize this away and assume it is zero. You can not extract energy from random events.

3. does the trading firm have the resources to constantly focus on low latency issues and is it ok with putting those resources to work on computing issues instead of on trading issues ?

This line of reasoning is again missing the point. If you want to make markets in options, you don't have a choice. Either get out of the kitchen, or if you want to be in the kitchen, you have no choice but to worry about it. People don't understand that option market makers have their autohedge on 95% of the time. That means if you fill an option and the autoquoting software hedges the deltas, you may have locked in a loss if your systems are lagging (unless you get lucky, not a good business plan.) There is no second place!

4. can greater low latency be achieved by changing other aspects of the trading system rather than by developing and maintaing one's own messaging and middleware software

This is the one thing I agree with wholeheartedly, but it is no excuse for let down your guard and focusing on all the places where you can minimize disadvantages. Hygiene doesn't mean wash your hair, and then you can ignore brushing your teeth.

 

[jjw]

i should have been a bit more explicit about this. When considering things from a latency point of view, focusing on the messaging and the middleware is not enough. if you have a system that takes 0 time to react to data, especially market data, you can still get beat because someone else might have received the data before you and reacted as fast as you. since it generally takes a few milliseconds to get market data from an exchange, there is enough room timewise for someone to take action that you do not see before you take your action. for options market makers the problem is compounded by the fact that an exchange can be quite slow in absorbing quotes when many, many, many are submitted at the same time.

the thrust of this thread was on build .vs buy of middleware/messaging. my point is that even if you have software, built or bought, and it takes 0 time, there are other factors that have a more siginificant latency affect on an algorithms's performance. the conclusion i draw, and this is of course why i am in this business, and i may be somewhat biased about this as i am in this business, is that traders are likely to do better focusing on the whole problem and leaving the midleware/messaging issues to those who do it for a living.

 

[nitro]

Quote from jjw:

i should have been a bit more explicit about this. When considering things from a latency point of view, focusing on the messaging and the middleware is not enough. if you have a system that takes 0 time to react to data, especially market data, you can still get beat because someone else might have received the data before you and reacted as fast as you.

Huh If someone else received the data before you, you have latency. All other things being equal, you don't want these middleware in your high frequency software, unless it is expertly done. Of course if you have hardware latency or datafeed latency, focusing on middleware is a waste of time. All other things being equal.

You keep trying to sweep under the rug that most messaging middleware, because they must deal with the general case, almost always sacrifice performance to something that is a custom fit. It is like buying a suit. You may be Macys trying to sell me a suit or you may be a tailor in London selling me a suit made exactly to fit me. You don't stand a chance in end product if you are Macys in this situation.

since it generally takes a few milliseconds to get market data from an exchange, there is enough room timewise for someone to take action that you do not see before you take your action. for options market makers the problem is compounded by the fact that an exchange can be quite slow in absorbing quotes when many, many, many are submitted at the same time.

What? I am totally lost, or you don't understand the problem. Look, Let's assume we have identical feeds. We are both on sight at the ECN matching engines. Identical switches and cables, identical hardware. We have identical software expect that yours is linked in with generic messaging software, maybe even your own, and I use custom messaging software optimized for my application and my style of trading. You lose. What is so hard to understand about this? I think you are assuming that I am running some shit feed or I am some retail trader. I am saying I am Goldman Sachs, or Getco. Now, what middleware do you suggest for me?

the thrust of this thread was on build .vs buy of middleware/messaging. my point is that even if you have software, built or bought, and it takes 0 time, there are other factors that have a more siginificant latency affect on an algorithms's performance.

Yeah, that is true for a trader in the mddle of kansas, that sends in a limit order to his broker on a retail account. That is not true for a firm that makes markets that have direct connections to the quoting engines, and has spent hundreds of thousands of dollars on infrastructure so they can compete in the high frequency space.

the conclusion i draw, and this is of course why i am in this business, and i may be somewhat biased about this as i am in this business, is that traders are likely to do better focusing on the whole problem and leaving the midleware/messaging issues to those who do it for a living.

I think we must be talking apples and oranges. Every firm that I know of in Chicago has a dedicated IT staff. We all do it for a living.

 

[jjw]

Quote from nitro:

Huh If someone else received the data before you, you have latency. All other things being equal, you don't want these middleware in your high frequency software, unless it is expertly done. Of course if you have hardware latency or datafeed latency, focusing on middleware is a waste of time. All other things being equal.

You keep trying to sweep under the rug that most messaging middleware, because they must deal with the general case, almost always sacrifice performance to something that is a custom fit. It is like buying a suit. You may be Macys trying to sell me a suit or you may be a tailor in London selling me a suit made exactly to fit me. You don't stand a chance in end product if you are Macys in this situation.


What? I am totally lost, or you don't understand the problem. Look, Let's assume we have identical feeds. We are both on sight at the ECN matching engines. Identical switches and cables, identical hardware. We have identical software expect that yours is linked in with generic messaging software, maybe even your own, and I use custom messaging software optimized for my application and my style of trading. You lose. What is so hard to understand about this? I think you are assuming that I am running some shit feed or I am some retail trader. I am saying I am Goldman Sachs, or Getco. Now, what middleware do you suggest for me?


Yeah, that is true for a trader in the mddle of kansas, that sends in a limit order to his broker on a retail account. That is not true for a firm that makes markets that have direct connections to the quoting engines, and has spent hundreds of thousands of dollars on infrastructure so they can compete in the high frequency space.


I think we must be talking apples and oranges. Every firm that I know of in Chicago has a dedicated IT staff. We all do it for a living.

The Macy's analogy is good but i think that it is not quite the point beacuse it is based upon the folloing premises:
1. all things being equal; and
2. that the tailor on saville row is always as good or better as any tailor in the world.

i address premise 2 first. the middleware/messaging produced by my firm has been used by firms like GS and Getco after having been tested against their own home grown messaging/middleware. the bake-off was not a middelware/messaging comparison but a comparison of a trade particular to the trading firm - our rimplementaiton .vs theirs. we won, they chose us over their own. this is not to say that we are better (we were in that instance) but just because GS is GS or Getco is Getco, does not mean that they always have the best. they certainly have money to get the best but it could be that the best resides outside their respective firms.

with respect to premise 1 - all things being equal - all things are never equal, it is just a matter of the latency tolerance of a trading algorithm to keep it in the game or not(assuming it is otherwise a profitable algorithm). consider that some feeds (equity options in particular) distribute their market data via tcp. in such a case someone gets the data first - all others are not first. depending upon the number of recipients of the market data, latency realized by the last recipient can be several milliseconds. unfortuantely such an exchange will not disclose to its recipients where they are in the market data distribution list. now consider that most market data these days is distributed via udp multicast. a common myth is that every one on the same switch gets the data at the same time. each recipient may get the data much closer in time, say 10 microseonds difference, but again, someone is first and some one is last. in the world of ultra low latency trading, 10 microseconds makes a difference.

as this kind of latency is beyond a recipient's control (and also may be different from day to day) a trader, even a GS or Getco, in my opinion, is still better off focusing on what they can control and on what can save them milliseconds and leave the microseconds to the microsecond maniacs.

 

[nitro]

Ok,

Do you have a trial of your messaging software?

Do you have any metrics showing comparisons against other solutions? Is this documented somewhere?

Quote from jjw:

The Macy's analogy is good but i think that it is not quite the point beacuse it is based upon the folloing premises:
1. all things being equal; and
2. that the tailor on saville row is always as good or better as any tailor in the world.

i address premise 2 first. the middleware/messaging produced by my firm has been used by firms like GS and Getco after having been tested against their own home grown messaging/middleware. the bake-off was not a middelware/messaging comparison but a comparison of a trade particular to the trading firm - our rimplementaiton .vs theirs. we won, they chose us over their own. this is not to say that we are better (we were in that instance) but just because GS is GS or Getco is Getco, does not mean that they always have the best. they certainly have money to get the best but it could be that the best resides outside their respective firms.

with respect to premise 1 - all things being equal - all things are never equal, it is just a matter of the latency tolerance of a trading algorithm to keep it in the game or not(assuming it is otherwise a profitable algorithm). consider that some feeds (equity options in particular) distribute their market data via tcp. in such a case someone gets the data first - all others are not first. depending upon the number of recipients of the market data, latency realized by the last recipient can be several milliseconds. unfortuantely such an exchange will not disclose to its recipients where they are in the market data distribution list. now consider that most market data these days is distributed via udp multicast. a common myth is that every one on the same switch gets the data at the same time. each recipient may get the data much closer in time, say 10 microseonds difference, but again, someone is first and some one is last. in the world of ultra low latency trading, 10 microseconds makes a difference.

as this kind of latency is beyond a recipient's control (and also may be different from day to day) a trader, even a GS or Getco, in my opinion, is still better off focusing on what they can control and on what can save them milliseconds and leave the microseconds to the microsecond maniacs.