Emulating CME DataMine FIX Format

cjbuckley4

cjbuckley4

I currently have the ability to stream limit order book data from IQFeed and Rithmic. They each have their own protocols through which they disseminate data: IQFeed has a messaging format that must be parsed by methods and Rithmic passes a few different market data objects to abstract methods. What I am trying to do is take these two datafeeds and normalize the output to be similar to what CME DataMine provides (examples available, open them in vim or something because they're too big for notepad et. al) so that I can work with both datasets with the same code and use the FIX Observer Rebuilder API to rebuild the limit order book according to the CME book management rules.

Before I invest the effort into outputting the data in this format, can anyone with experience in this area confirm whether the CME Globex Legacy Market Data format is a format that will be compatible with FIX Observer with or without some tweaking? I'm completely new to FIX, so I don't know what is or isn't required in each message or if the Legacy Market Data format is the correct format of messages to be using. I tried comparing the CME DataMine Market Depth examples and they general match up, but some tags appear to be omitted and some that are present are not discussed (in the part of the documentation I'm view at least). Any help would be greatly appreciated. You can just post a link to the right area of the CME documentation linked above to use and that will be highly appreciated.

Thanks in advance.
 
I fail to understand what your grand objective here is. What are you trying to achieve? You have multiple data feeds and want to merge the streams and feed into one centralized order book? For what purpose? And why does it have to conform to some "fix observer" or "cme book management rule"?


cjbuckley4 said:
I currently have the ability to stream limit order book data from IQFeed and Rithmic. They each have their own protocols through which they disseminate data: IQFeed has a messaging format that must be parsed by methods and Rithmic passes a few different market data objects to abstract methods. What I am trying to do is take these two datafeeds and normalize the output to be similar to what CME DataMine provides (examples available, open them in vim or something because they're too big for notepad et. al) so that I can work with both datasets with the same code and use the FIX Observer Rebuilder API to rebuild the limit order book according to the CME book management rules.

Before I invest the effort into outputting the data in this format, can anyone with experience in this area confirm whether the CME Globex Legacy Market Data format is a format that will be compatible with FIX Observer with or without some tweaking? I'm completely new to FIX, so I don't know what is or isn't required in each message or if the Legacy Market Data format is the correct format of messages to be using. I tried comparing the CME DataMine Market Depth examples and they general match up, but some tags appear to be omitted and some that are present are not discussed (in the part of the documentation I'm view at least). Any help would be greatly appreciated. You can just post a link to the right area of the CME documentation linked above to use and that will be highly appreciated.

Thanks in advance.
More...
 
@volpunter First of all, let me just preface this response by saying that I've spent a few hours looking at the FIX Observer code and believe I've answered the question myself. My apologies, I may have been unclear. Let me clarify--I have multiple feeds, but my intention is not to merge them into one book, rather I want to establish one uniform standard for the two feeds and other feeds that may follow. The output of the two feeds will be stored separately but with the same format. I just thought that using FIX protocol as the storage format would be sensible because it's ubiquitous and I could avoid any possible oversights in my own design by using something that has served as an industry standard in this space.

The purpose I envisioned was to create a uniform standard for both data sets such that I could perform the same analysis on both sets of data with the same code as well as use available FIX oriented tools like the aforementioned FIX Observer, which rebuilds the CME limit order book.

To briefly digress from my answer and leave a note to anyone that follows my path: upon playing with FIX Observer, it appears a lot of FIX Protocol versions will work, but it is designed to use FIX 5.0 SP2 according to some comments in the Decoder.cs class.

Another reason I wanted to use FIX was because both Rithmic and Trading Technologies have FIX adapters you can use to send (not receive, or this problem would be trivial) messages, so using it now will hopefully allow me to move more easily between the two in the future as I diversify FCMs and platforms. A lot of the bigger FCMs that offer SPAN margins and have better infrastructure/commissions seem to use Trading Technologies (Advantage, RCG). I looked into starting with TT instead of Rithmic, but the cost/benefit wasn't there and Rithmic is an excellent product for my current needs.

With regard to the CME book management rules, I just want to make sure that I'm rebuilding the book correctly. I also wanted to start with what someone who seems to know what they're doing has done with CME as an example because I'll probably be tweaking his work for ICE and Eurex in the future.

I know you must have spent a long time using FIX because you had to connect to a number of FX trading venues. The CME documentation is really good, but if you have any resources that helped you, feel free to point me that way. I've been looking at the CME documentation page linked above and QuickFix/n. Let me know if I'm still being unclear.
 
@2rosy, I'm far from an expert. Everything I know about FIX has come from what I've learned in this process, so I can't say for sure. I haven't tried to use QuickFix to parse the files, and I don't think that's what QuickFix is intended to do. I was just reading some of their documentation as I went. As far as I know from what I've read in FIX Observer comments, the CME files from DataMine are in FIX 5.0 SP2. The format discussed in their documentation is called CME Globex Legacy Market Data, which may or may not be the same. I'm still figuring it all out, but until otherwise informed, I'm gonna roll with the FIX 5.0 SP2 discussed in the FIX Observer comments.
 
I am not sure why you make your life so complex. Most market data interfaces are based on UDP streams and expose an end user API. Most professional data streams are not based on FIX. FIX is generally perused for order handling purposes.



cjbuckley4 said:
@volpunter First of all, let me just preface this response by saying that I've spent a few hours looking at the FIX Observer code and believe I've answered the question myself. My apologies, I may have been unclear. Let me clarify--I have multiple feeds, but my intention is not to merge them into one book, rather I want to establish one uniform standard for the two feeds and other feeds that may follow. The output of the two feeds will be stored separately but with the same format. I just thought that using FIX protocol as the storage format would be sensible because it's ubiquitous and I could avoid any possible oversights in my own design by using something that has served as an industry standard in this space.

The purpose I envisioned was to create a uniform standard for both data sets such that I could perform the same analysis on both sets of data with the same code as well as use available FIX oriented tools like the aforementioned FIX Observer, which rebuilds the CME limit order book.

To briefly digress from my answer and leave a note to anyone that follows my path: upon playing with FIX Observer, it appears a lot of FIX Protocol versions will work, but it is designed to use FIX 5.0 SP2 according to some comments in the Decoder.cs class.

Another reason I wanted to use FIX was because both Rithmic and Trading Technologies have FIX adapters you can use to send (not receive, or this problem would be trivial) messages, so using it now will hopefully allow me to move more easily between the two in the future as I diversify FCMs and platforms. A lot of the bigger FCMs that offer SPAN margins and have better infrastructure/commissions seem to use Trading Technologies (Advantage, RCG). I looked into starting with TT instead of Rithmic, but the cost/benefit wasn't there and Rithmic is an excellent product for my current needs.

With regard to the CME book management rules, I just want to make sure that I'm rebuilding the book correctly. I also wanted to start with what someone who seems to know what they're doing has done with CME as an example because I'll probably be tweaking his work for ICE and Eurex in the future.

I know you must have spent a long time using FIX because you had to connect to a number of FX trading venues. The CME documentation is really good, but if you have any resources that helped you, feel free to point me that way. I've been looking at the CME documentation page linked above and QuickFix/n. Let me know if I'm still being unclear.
More...
 
@2rosy thanks for the parser. I probably will eventually write something like this in python myself.

@volpunter You might be right. I could be complicating my life. I initially just went down this route because I wanted to make sure that the format I serialize the data to was comprehensive, i.e. I didn't omit any necessary information when serializing my feed data to binary. I thought, to this end, just copying the CME format would be the best path. I played with Observer quite a bit with the CME DataMine example files, and it is really well done. That being said, after a lot of research, I've determined that a mix of designing my encoding of messages in a format fairly close to FIX for many messages and adapting Observer to fit my needs will be the best solution.
 
You must log in or register to reply here.
Back
Top