Best practices?

[ScoobyStoo]

I am currently building a fully automated futures algo trading system from the ground up and would appreciate the input of any others that have done the same thing.

I have 10+ years professional coding experience and have architected complex business applications before. However, at the moment I am working on the order management module and running into a few design headaches relating the asynchronous nature of interfacing with the market. I always knew this part of the project would be the hardest due to the formidable number of scenarios that must be handled arising from the decoupled nature of the beast. I won't go into examples here as any of you that code trading systems at this level will know exactly what I mean.

Are there any books/white papers in the public domain which address the best practices for handling these architectural decisions? I want to make sure I get the core design correct the first time round.

Many thanks as always...

 

[Trader922]

What about looking at some of the open source projects such as TradeLink or Marketcetera for some ideas/concepts?

Have you tried that? It would seem that may help you out a lot more.

Eric

 

[ScoobyStoo]

Quote from Trader922:

What about looking at some of the open source projects such as TradeLink or Marketcetera for some ideas/concepts?

Have you tried that? It would seem that may help you out a lot more.

Eric

Thanks Eric.

I spent a few days looking at both the TradeLink and Marketcetera codebases before Christmas.

To be honest, I really wasn't impressed with the quality of what I saw in the TradeLink codebase. The structure of some of the code is messy to say the least, the design approach seems to chop around a lot and the documentation/commenting is lamentable. The concept of handling rejections relating to order placement/modification/cancellation doesn't seem to be addressed at all. I could well be wrong but from my brief foray it seems to suffer from all the foibles typical of open source projects. A fairly amateur affair and not worth the time to investigate further...

Marketcetera is a far more professional setup (as you would expect) and looks to be well architected. Sadly though it doesn't really address my issues because it is in essence simply an execution platform and intentionally doesn't handle the underlying order management issues.

I'm really looking for guys that have coded systems for banks and hedgies (if any frequent this forum) to point me in the right direction. Might post on a few of the more heavy weight forums such as Wilmott as well to see if any of them bite...

 

[LeeD]

Are you looking to route order to a broker or directly to the exchange? If yopu route orders to teh broker you'd expect the broker has handled the most difficult issues at the expense of latency...

 

[ScoobyStoo]

Quote from LeeD:

Are you looking to route order to a broker or directly to the exchange? If yopu route orders to teh broker you'd expect the broker has handled teh most difficult issues at the expense of latency...

I'm routing direct to the exchange once the order has been through the risk management module of my FCM. Doesn't really matter anyway since no broker will handle any of the order management logic for you. For instance, if an order modification request gets rejected by the exchange then they will just pass that message onto you via their API and let you decide what to do next.

 

[LeeD]

Quote from ScoobyStoo:

I'm routing direct to the exchange once the order has been through the risk management module of my FCM. Doesn't really matter anyway since no broker will handle any of the order management logic for you. For instance, if an order modification request gets rejected by the exchange then they will just pass that message onto you via their API and let you decide what to do next.

For example, the broker will likely handle partial execution messages coming in the wrong order in such a way that you always get consistent information about the size that has already been executed and the average execution price.

On the topic, as I understand you are working on the module that handles the market connection and provides a higher-level inteface to the actual decision-making algorithms. If you don't get the exact answer you seek below in this thread (i.e. recommended books or web links) consider looking at OpenQuant documentation and examples. (Despite the name it's neither open-source nor free.) I am far from advertising the actual product but their event-driven interface (between trading strategies and broker connection) could be an example of sound design you seek.

 

[ScoobyStoo]

Quote from LeeD:

For example, the broker will likely handle partial execution messages coming in the wrong order in such a way that you always get consistent information about the size that has already been executed and the average execution price.

On the topic, as I understand you are working on the module that handles the market connection and provides a higher-level inteface to the actual decision-making algorithms. If you don't get the exact answer you seek below in this thread (i.e. recommended books or web links) consider looking at OpenQuant documentation and examples. (Despite the name it's neither open-source nor free.) I am far from advertising the actual product but their event-driven interface (between trading strategies and broker connection) could be an example of sound design you seek.

Thanks Lee.

Indeed, this is the module that interfaces with the FCM's trading platform and internally manages order state. It both sends and receives all messages relating to those orders.

Life becomes very complex when you try to handle those situations that can arise when your application is out of synch with the exchange. Consider the following very simple sequence of events:

1. Application: Sends entry (buy market) order (quantity: 10).
2. Exchange: Receives entry order.
3. Exchange: Sends entry order confirmation report.
4. Exchange: Executes entry order.
5. Exchange: Sends entry order execution report.
6. Application: Receives entry order confirmation report.
7. Application: Receives entry order execution report.

8. Application: Sends stop loss exit (sell stop market) order (quantity: 10).
9. Exchange: Stop loss exit order entered into the stop order book.
10. Exchange: Sends stop loss exit order confirmation report.
11. Application: Receives stop loss exit order confirmation report.

12. Application: Sends scale out exit (sell market) order (quantity: 1).
13. Exchange: Receives scale out exit order.
14. Exchange: Sends scale out exit order confirmation report.
15. Exchange: Executes scale out exit order.
16. Exchange: Sends scale out exit order execution report.
17. Application: Receives scale out exit order confirmation report.
18. Application: Receives scale out exit order execution report.

19. Application: Sends stop loss exit order modification (quantity: 9)
20. Exchange: Executes stop loss exit order in response to stop price being hit (quantity: 10).
21. Exchange: Receives stop loss exit order modification.
22. Exchange: Rejects stop loss exit order modification.
23. Application: Receives stop loss exit order modification rejection report.

As you can see, things get interesting around event 19 where the application tries to balance up the stop loss exit order. As a result of the pseudo race condition the strategy ends up short 1 contract instead. Obviously it's possible to detect this and handle it, but there are a million and one of these type of scenarios that can arise. I'm really looking for any best practice software development patterns for firstly avoiding them and secondly handling them if they do crop up.

I did indeed look at OpenQuant many months ago and from what I remember it doesn't handle these kinds of issues either. Like all platforms it tends to leave the cleaning up to the individual coder. Will take another ride by the docs and examples though to see if I can pick anything new up.

Does anyone know of any forums or online groups where coders working at this level fairly close to the metal congregate to discuss such things?

 

[LeeD]

Now I understand what was the question in the opening post about. It wasn't about aggregating the information that comes from the Exchange so that the most up-to-date state of orders and position is effectively "cashed" locally in an easy-to-interpret state.

If the question is about handling situations like you described, when the trader finds him/hersef in an unwanted position, then it's much more complex than what I thought originally.

The thing is software availabel for sale tends to be aimed at trading strategies with sufficiently large target and stop. As a result, the state like you described would be quite rare and can be handled by simply closing an unwanted position. Note some "retail" platforms like Neoticker allow delay in exchange accepting orders to be incorporated into backtesting. So, you can actually get a good flavour of how often such unwanted positions would be accuring and how their handling affects performance.

Trading systems, in which such "exceptional" states would be occuring routinely are likely to be "high-frequency" strategies and hardly anything is available to general public on this subject.

 

[Trader922]

Quote from ScoobyStoo:

Thanks Lee.

Indeed, this is the module that interfaces with the FCM's trading platform and internally manages order state. It both sends and receives all messages relating to those orders.

Life becomes very complex when you try to handle those situations that can arise when your application is out of synch with the exchange. Consider the following very simple sequence of events:

1. Application: Sends entry (buy market) order (quantity: 10).
2. Exchange: Receives entry order.
3. Exchange: Sends entry order confirmation report.
4. Exchange: Executes entry order.
5. Exchange: Sends entry order execution report.
6. Application: Receives entry order confirmation report.
7. Application: Receives entry order execution report.

8. Application: Sends stop loss exit (sell stop market) order (quantity: 10).
9. Exchange: Stop loss exit order entered into the stop order book.
10. Exchange: Sends stop loss exit order confirmation report.
11. Application: Receives stop loss exit order confirmation report.

12. Application: Sends scale out exit (sell market) order (quantity: 1).
13. Exchange: Receives scale out exit order.
14. Exchange: Sends scale out exit order confirmation report.
15. Exchange: Executes scale out exit order.
16. Exchange: Sends scale out exit order execution report.
17. Application: Receives scale out exit order confirmation report.
18. Application: Receives scale out exit order execution report.

19. Application: Sends stop loss exit order modification (quantity: 9)
20. Exchange: Executes stop loss exit order in response to stop price being hit (quantity: 10).
21. Exchange: Receives stop loss exit order modification.
22. Exchange: Rejects stop loss exit order modification.
23. Application: Receives stop loss exit order modification rejection report.

As you can see, things get interesting around event 19 where the application tries to balance up the stop loss exit order. As a result of the pseudo race condition the strategy ends up short 1 contract instead. Obviously it's possible to detect this and handle it, but there are a million and one of these type of scenarios that can arise. I'm really looking for any best practice software development patterns for firstly avoiding them and secondly handling them if they do crop up.

I did indeed look at OpenQuant many months ago and from what I remember it doesn't handle these kinds of issues either. Like all platforms it tends to leave the cleaning up to the individual coder. Will take another ride by the docs and examples though to see if I can pick anything new up.

Does anyone know of any forums or online groups where coders working at this level fairly close to the metal congregate to discuss such things?

Maybe I'm missing something but I'm not seeing how you could not do this using an architecture like OpenQuant (and others use). I think its worth looking at the API guide and re-thinking some of this.

PM me to discuss more if needed.

 

[ScoobyStoo]

Quote from LeeD:

The thing is software availabel for sale tends to be aimed at trading strategies with sufficiently large target and stop. As a result, the state like you described would be quite rare and can be handled by simply closing an unwanted position. Note some "retail" platforms like Neoticker allow delay in exchange accepting orders to be incorporated into backtesting. So, you can actually get a good flavour of how often such unwanted positions would be accuring and how their handling affects performance.

Trading systems, in which such "exceptional" states would be occuring routinely are likely to be "high-frequency" strategies and hardly anything is available to general public on this subject.

That's what I found when I evaluated the retail software packages and is one of the reasons I decided to build my own application from scratch. The default approach that seems to be taken by these packages is that if any order placement/modification/cancellation is rejected then an exception should be thrown and the strategy should shut down. I have a big problem with this as there are plenty of scenarios in which rejections can legitimately be generated during the normal operation of a strategy (e.g. sending an order cancellation request for an order which has simultaneously been fully filled at the exchange).

I'm not building an HFT strategy (my position holding times vary from seconds to hours rather than milliseconds) but I do have multiple orders in the market simultaneously so to me it would seem like gross negligence to not handle these situations in a robust manner. Sod's law dictates that if I don't then it will bite me hard in the arse at some point.