Futures Order Book parsing methodology

[IAS_LLC]

Hello All,

I am currently writing some software that monitors the CME futures orderbook (I am using DTN NxCore as my data provider and using their C++ API). Orderbook data is what is the driver of my automated trading strategy.

As you may know, the NxCore orderbook is updated in chunks (I suspect the CME FIX feed is as well...). For Example, lets say at one instant the best ask price for a given contract is 1.3250 and the second best price is 1.3251. A market maker quote update is recieved, and the new best ask is 1.3249, and the second best price from the feed will still read 1.3251 until another update is received, at which point the 2nd best price will read 1.3250 and the third will also read 1.3250 (assuming no sales took place at 1.3250 and removed all of the liquidity at this level). The third level will then update and read 1.3251, etc.

How do the pro's deal with this type of thing? I know its not an overly complicated situation, but I would like to hear what elegant solutions the experts have developed. I can't imagine they would allow their algorithms to make decisions based on a partially update order book?

Thanks,
Brad

 

[uexkuell]

Have an array for bid and ask quotes separately.
One entry for every possible price value.
It holds the actual volume information at this price level.
When updates come in the program updates this volume information only. Everything else unchanged.

If the question is when to take action (buy/sell) then a condition regarding order book and/or price has to be defined very clearly.

If a single update in the order book or in price might lead to this condition then the reaction will follow immediately after this update.

Not sure if this answers the question . Perhaps you might want to specify the question further.

 

[2rosy]

if nxcore sends the order data then have 2 trees bids and asks. each level of the tree is a price and each node is a list of orders

 

[IAS_LLC]

Have an array for bid and ask quotes separately.
One entry for every possible price value.
It holds the actual volume information at this price level.
When updates come in the program updates this volume information only. Everything else unchanged.

If the question is when to take action (buy/sell) then a condition regarding order book and/or price has to be defined very clearly.

If a single update in the order book or in price might lead to this condition then the reaction will follow immediately after this update.

Not sure if this answers the question . Perhaps you might want to specify the question further.

Thanks for the reply. Your response does answer my question, although it does inspire another question regarding memory footprint. When you say "every possible" price value you surely cant mean that literally. Do you use a fixed array size or dynamic memory?

 

[uexkuell]

Thanks for the reply. Your response does answer my question, although it does inspire another question regarding memory footprint. When you say "every possible" price value you surely cant mean that literally. Do you use a fixed array size or dynamic memory?

You're welcome.

Recommendation: Two fixed arrays (bid and ask separately) for storage.
This is very fast, easy to implement and effective.

I meant "every possible price" literally.
There aren't so many possible values for price during one given day (see maximum expectable volatility). No problem with today's large main memory.
E.g.:
ES yesterday 1844-1872 -> 28*4 = 104 levels
CL yesterday 100.3 - 101.5 -> 120 levels

So you might just take the first price value in a stream that you see and allow for 1000 levels up and down from this value.
You won't see a volatility that exceeds this range for one day.
If looking over a longer time period increase the 1000 levels accordingly.

 

[nonlinear5]

I meant "every possible price" literally.
There aren't so many possible values for price during one given day (see maximum expectable volatility). No problem with today's large main memory.
E.g.:
ES yesterday 1844-1872 -> 28*4 = 104 levels
CL yesterday 100.3 - 101.5 -> 120 levels

So you might just take the first price value in a stream that you see and allow for 1000 levels up and down from this value.

CME only broadcasts the top 10 levels of the order book for both the ES and the CL. There is no visibility to the levels below, so I don't understand what is it that you propose to maintain on levels 11 and below.

 

[uexkuell]

CME only broadcasts the top 10 levels of the order book for both the ES and the CL.

Sure. True.

... what is it that you propose to maintain on levels 11 and below.

Didn't propose to maintain the information.
Sure you don't have to use the information outside the scope of 10 levels.

The approach (array) just simplifies code.
No need to use trees or whatever.
Only sacrifices a little memory for the sake of speed and simplicity.

 

[nonlinear5]

The approach (array) just simplifies code.
No need to use trees or whatever.
Only sacrifices a little memory for the sake of speed and simplicity.

Ok, I understand the intent now. I have not used the DTN NxCore API, but I have used both the IB API, and the FIX/FAST CME data stream, to build the order book. In both cases, you receive messages to "remove level N and to shift all the levels below level N up". This would be a very expensive operation if your implementation is a simple array, rather than a linked list.

 

[uexkuell]

Ok, I understand the intent now. I have not used the DTN NxCore API, but I have used both the IB API, and the FIX/FAST CME data stream, to build the order book. In both cases, you receive messages to "remove level N and to shift all the levels below level N up". This would be a very expensive operation if your implementation is a simple array, rather than a linked list.

Used the array approach with IB API and Zenfire API.
In both it works like a charm.

IB API updateMktDepth has indeed a field "operation" that can have values like "insert" or "delete". But there is no need to use this.
Just take the price value, convert into an index into the array and store (new) volume information.
No need to shift any values.
In contrast a linked list needs much more code (and time).

 

[nonlinear5]

Used the array approach with IB API and Zenfire API.
In both it works like a charm.

IB API updateMktDepth has indeed a field "operation" that can have values like "insert" or "delete". But there is no need to use this.
Just take the price value, convert into an index into the array and store (new) volume information.
No need to shift any values.
In contrast a linked list needs much more code (and time).

Ok, I understand what you are doing now. Instead of directly applying the "delete" and "insert" market book operations, you are doing effectively the same by tracking the level by the price. Yes, it sure can be done that way.

Just to verify something, you keep saying "volume", but it's really the size of the bid/ask at the particular price level that you are updating, right?