Why IB API echoes same order status multiple times?

[6yaNYCjm5m]

***Capture of the messages received as a result of market order being submitted. Not all fields are present, only ones I deem relevant for myself.

Msg [EXECUTION_DATA]->[(2, 37955052, 'CL', 'FUT', '20110520', 'NYMEX', 'USD', 'CLM1', '0001f4e5.4dc4368a.01.01', '20110506 11:11:31', 'DU***', 'NYMEX', 'SLD', 1, '101.95', 1687424472, 55555, 0, 1, '101.95')]

Msg [OPEN_ORDER]->[(2, 37955052, 'CL', 'FUT', '20110520', 'NYMEX', 'USD', 'CLM1', 55555, 1687424472, 'SELL', 1, 'MKT', '101.95', '0.0', 'GTC', None, 'DU***', 'O', 0, None, '0', '1', 0, None, None, None, None, 'Filled', '1.7976931348623157E308', '1.7976931348623157E308', '1.7976931348623157E308', None, None, None, 'USD', None)]
Msg [ORDER_STATUS]->[(2, 'Filled', 1, 0, '101.95', 1687424472, 0, '101.95', 55555, None)]

Msg [OPEN_ORDER]->[(2, 37955052, 'CL', 'FUT', '20110520', 'NYMEX', 'USD', 'CLM1', 55555, 1687424472, 'SELL', 1, 'MKT', '101.95', '0.0', 'GTC', None, 'DU***', 'O', 0, None, '0', '1', 0, None, None, None, None, 'Filled', '1.7976931348623157E308', '1.7976931348623157E308', '1.7976931348623157E308', '2.32', None, None, 'USD', None)]
Msg [ORDER_STATUS]->[(2, 'Filled', 1, 0, '101.95', 1687424472, 0, '101.95', 55555, None)]

Msg [OPEN_ORDER]->[(2, 37955052, 'CL', 'FUT', '20110520', 'NYMEX', 'USD', 'CLM1', 55555, 1687424472, 'SELL', 1, 'MKT', '101.95', '0.0', 'GTC', None, 'DU***', 'O', 0, None, '0', '1', 0, None, None, None, None, 'Filled', '1.7976931348623157E308', '1.7976931348623157E308', '1.7976931348623157E308', '2.32', None, None, 'USD', None)]
Msg [ORDER_STATUS]->[(2, 'Filled', 1, 0, '101.95', 1687424472, 0, '101.95', 55555, None)]


***Capture of the messages received as a result of limit order being submitted.

Msg [OPEN_ORDER]->[(3, 37955052, 'CL', 'FUT', '20110520', 'NYMEX', 'USD', 'CLM1', 55555, 1687424473, 'SELL', 1, 'LMT', '100.0', '0.0', 'GTC', None, 'DU***', 'O', 0, None, '0', '1', 0, None, None, None, None, 'Submitted', '1.7976931348623157E308', '1.7976931348623157E308', '1.7976931348623157E308', None, None, None, 'USD', None)]
Msg [ORDER_STATUS]->[(3, 'Submitted', 0, 1, '0', 1687424473, 0, '0', 55555, None)]


***Capture of the messages received as a result of stop order being submitted.

Msg [OPEN_ORDER]->[(5, 37955052, 'CL', 'FUT', '20110520', 'NYMEX', 'USD', 'CLM1', 55555, 1687424474, 'BUY', 1, 'STP', '99.18', '100.0', 'GTC', None, 'DU***', 'O', 0, None, '1', '1', 0, None, None, None, None, 'PreSubmitted', '1.7976931348623157E308', '1.7976931348623157E308', '1.7976931348623157E308', None, None, None, 'USD', None)]
Msg [ORDER_STATUS]->[(5, 'PreSubmitted', 0, 1, '0', 1687424474, 0, '0', 55555, 'trigger')]


As I've mentioned earlier, it is mimicking the FIX behaviour. Not really interested digging too much into the IB backend, I don't know if they are running FIX server as a central place for trade management or something proprietary, I know there is a FIX gateway so they are either mapping TWS API into FIX or vice-verse.
Regardless of what's in place, system has to keep track of different order statuses, hence multiple OPEN_ORDER and ORDER_STATUS messages when market order submitted. For example, order life cycle goes through Submitted, Accepted, Pending, Working, Cancelled, Filled, etc. For some (stupid) reason IB have decided not to show the status change and have 'Filled' in all messages, but still transmit all of them instead of only the final one.
You can notice that in the first OPEN_ORDER msg, the fifth field from the end does not have the value, which is commission, but subsequent messages does. I am speculating
(unfortunately), this is the msg for Submitted/Accepted/Pending order status and the next two are for the Working and Filled order status.

Submitting the limit or stop order results in a single message where you can see the status Submitted/PreSubmitted.

Quote from dcraig:

Multiple status events reporting the same status should never be a problem. The operation is, in the jargon, idempotent. http://en.wikipedia.org/wiki/Idempotence#Computer_science_meaning

Never heard of that before, what a name... I personally do not like when stuff like that creeps into the software architecture, too abstract and largely unnecesary in software design, there are already plenty specific methodologies we can use to design the software and explain the software.
Having said that, I believe this is the wrong example for "idempotent operation" because the status is not the same. Those are multiple events that are changing the status. The fact that somebody in IB decided not to show it, is wrong. In such case they should be sending only one message. If, on the other hand, they don't care about the status order for market orders and process only Filled status (which I do not believe), then it's plain stupid to send multiple messages after single event.

Quote from dcraig:

Some of it is not the way I would have done it, but it is reliable and robust which does count for quite a bit. The historical data portion of the API is a bit horrible and unnecessarily complicated to use. It is, however, perfectly possibly to make decent charts by using it.

Well, it's working and that's about it. Hardly robust, there is no structure in protocol low-level, no validation, no recovery, and list goes on and on....
Historical data msgs are same as everything else, what makes it additional nightmare is actually extremely restrictive policies regarding the amount of data you can get and the fact that it is not separated from the real-time communication, which forces you to spin-off something (I do separate process, they do that horrible Java threading).
At the first opportunity, I will ditch the data portion and retain only execution. For data, I am looking into DDF BarChart, looks well designed and platform agnostic. Thankfully no Java involved. No secret I can't stand it, nor the whole OOP, gone bezirk, mental exercise. However, I perfectly do not mind anybody enjoying it...

 

[clearinghouse]

Quote from maxpi:

IB's API is known to be so krappy... I just use a chart package and script it, the chart package developers have to deal with IB's garbage, not me... and the refuse to implement certain things just because IB makes it so difficult for them that they just won't go there... I wonder if OpenEcry is better?

IB is the best retail broker, hands down. They aren't perfect, but they have the most customizable, usable interface. My gripe is that I wish their EWrapper and SocketClient classes were "broken up" a little more, so I could just inherit / reorganize the classes I need. It's not a huge deal, and I worked around it. The only cost was just a lot of time and testing.

Assuming they don't change their API substantially, I figure the investment to build a decent set of classes around their API is a worthwhile investment. Have they historically changed their API substantially, or has it been relatively stable over time?

 

[clearinghouse]

Quote from dcraig:

Some of it is not the way I would have done it, but it is reliable and robust which does count for quite a bit. The historical data portion of the API is a bit horrible and unnecessarily complicated to use. It is, however, perfectly possibly to make decent charts by using it.

Multiple status events reporting the same status should never be a problem. The operation is, in the jargon, idempotent. http://en.wikipedia.org/wiki/Idempotence#Computer_science_meaning

Is it possible to cancel-decrement orders in IB? i.e., I have 1000 shares, I want to cancel 500, but not lose my spot on the order queue on the other 500? Seems like I have to cancel all 1000 and put an order out for 500. Or maybe I am missing something here.

 

[asiaprop]

yes you are...queue priority has nothing to with IB, its about how the exchange to which you route will handle partial cancels. Even if IB would not support it you could easily do it yourself through the IB API. Generally, most exchanges support partial cancels without forcing a loss of queue priority on the remaining unfilled portion.

Quote from clearinghouse:

Is it possible to cancel-decrement orders in IB? i.e., I have 1000 shares, I want to cancel 500, but not lose my spot on the order queue on the other 500? Seems like I have to cancel all 1000 and put an order out for 500. Or maybe I am missing something here.

 

[clearinghouse]

Quote from asiaprop:

yes you are...queue priority has nothing to with IB, its about how the exchange to which you route will handle partial cancels. Even if IB would not support it you could easily do it yourself through the IB API. Generally, most exchanges support partial cancels without forcing a loss of queue priority on the remaining unfilled portion.

Sorry I wasn't clear, I was trying to figure out how to do it through the API. There are two functions, placeOrder and cancelOrder. There's no changeOrder. Is that done through placeOrder() with the same order ID again?

 

[uexkuell]

Quote from asiaprop:
queue priority has nothing to with IB, its about how the exchange to which you route will handle partial cancels.

That's wrong.
Obviously you didn't follow/don't remember the discussion on this board going on just recently where IB people admitted that under certain conditions you loose priority in the queues.

See http://www.elitetrader.com/vb/showthread.php?s=&threadid=215023

Quote from asiaprop:
Even if IB would not support it you could easily do it yourself through the IB API.

Wrong again.
If IB doesn't let you do things you can't do them no matter if the exchange supports it or not.

Quote from clearinghouse:
Is it possible to cancel-decrement orders in IB? i.e., I have 1000 shares, I want to cancel 500, but not lose my spot on the order queue on the other 500? Seems like I have to cancel all 1000 and put an order out for 500. Or maybe I am missing something here.

Sorry, I can't answer your question.
Obviously other people here also cannot.
You might get valid answers at the Yahoo IB API discussion board (free).

 

[clearinghouse]

Quote from asiaprop:

yes you are...queue priority has nothing to with IB, its about how the exchange to which you route will handle partial cancels. Even if IB would not support it you could easily do it yourself through the IB API. Generally, most exchanges support partial cancels without forcing a loss of queue priority on the remaining unfilled portion.

Just posting this so someone else can Google find this and avoid the hassle:

On TWS, IB's getting started manual says:

"Modifying an Order
Please note that if you want to modify an order via the API, you need to manually complete
the same steps that TWS would process automatically. This means that you need to cancel the
existing order, then create and resubmit a new order with the modified parameters."

So, there is no direct way to modify an order down. You have to cancel and resubmit. I don't know if their FIX connection has the same limitation or not, but for TWS, this is the final word.

 

[LeeD]

Quote from 6yaNYCjm5m:

As I've mentioned earlier, it is mimicking the FIX behaviour.

....

Having said that, I believe this is the wrong example for "idempotent operation" because the status is not the same. Those are multiple events that are changing the status. The fact that somebody in IB decided not to show it, is wrong. In such case they should be sending only one message. If, on the other hand, they don't care about the status order for market orders and process only Filled status (which I do not believe), then it's plain stupid to send multiple messages after single event.

The fundamental reason for multiple order status reports is exchnages try to deliver order status as fast as possible to their high-frequency clients. For example, the order matching algorithm on the exchange has to match orders before it can calculate the fill price. As soon as the order is matched it reports that the order has been filled (without the fill price). A few milliseconds later the fill price has been identified and it is reported (again as "oreder filled" message, but now with the price).

P. S. Note that not all "high-frequency" traders are "prop". High-frequency infrastructure may be used to achieve VWAP for a discretionary trader or a large fill for an investment manger.

 

[DAV]

Quote from clearinghouse:

On TWS, IB's getting started manual says:
"Modifying an Order
Please note that if you want to modify an order via the API, you need to manually complete
the same steps that TWS would process automatically. This means that you need to cancel the
existing order, then create and resubmit a new order with the modified parameters."

The geting started guide is incorrect and will be updated.
Generally whatever can be changed for an order submitted using TWS can be changed via API. No manual cancellations are needed.

 

[clearinghouse]

Quote from DAV:

The geting started guide is incorrect and will be updated.
Generally whatever can be changed for an order submitted using TWS can be changed via API. No manual cancellations are needed.

Well, can you tell me how to do that in the forum then? I may have missed the example code for how to do it.