Handling overfills / order errors / order safety

abattia · Jul 5, 2012

For my first time, Iâm writing code (for NinjaTrader) to handle overfills / order errors / order safety.

While understanding that the specifics of such handling would depend on the specifics of the trading objectives/trading strategies involved, Iâm trying to lay my hands on past (and publicly available) examples of how these matters have been dealt with by others.

Iâm not fussed whether these examples are in C#, or whatever (including pseudo code!... or even just a detailed description in plain text!); I'm more interested in getting a feel for the scope of what is handled in each case.

Can anyone point me in the right direction?

2rosy · Jul 7, 2012

not sure about ninja but in general most APIs pass back errors/fills/etc. so you do something like

onfilll(messageobject){
print messageobject.quantity, messageobject.orderid
}

onerror(messageobject){
print messageobject.error
}

clearinghouse · Jul 7, 2012

Quote from abattia:

For my first time, Iâm writing code (for NinjaTrader) to handle overfills / order errors / order safety.

While understanding that the specifics of such handling would depend on the specifics of the trading objectives/trading strategies involved, Iâm trying to lay my hands on past (and publicly available) examples of how these matters have been dealt with by others.

Iâm not fussed whether these examples are in C#, or whatever (including pseudo code!... or even just a detailed description in plain text!); I'm more interested in getting a feel for the scope of what is handled in each case.

Can anyone point me in the right direction?

I don't think there's some document you can read to just do this. I think the best way to do this is to actually experiment and build a few systems and keep doing it until the kinks are worked out. If you're doing stocks, I advise using 200 shares for testing purposes so you can deal with odd and mixed lots and partial fills (which, honestly, are a real nuisance -- they should ban odd lots.)

It took me about 1-1.5 months of building and then mostly testing until my order management system was stable. It is something that does take time, because sometimes you don't anticipate corner cases and the market can surprise you with behavior you didn't quite plan out.

The coding part is easy; it's the testing part that sucks hardcore.

2rosy · Jul 7, 2012

Quote from clearinghouse:

The coding part is easy; it's the testing part that sucks hardcore.

If the coding part was so easy then the testing would be irrelevant. The coding is the whole thing :eek:

clearinghouse · Jul 7, 2012

Quote from 2rosy:

If the coding part was so easy then the testing would be irrelevant. The coding is the whole thing :eek:

"Writing the outline for a book is easier than writing the actual book" is the idea that is similar to what I was trying to get across.

dom993 · Jul 7, 2012

Quote from abattia:

For my first time, Iâm writing code (for NinjaTrader) to handle overfills / order errors / order safety.

While understanding that the specifics of such handling would depend on the specifics of the trading objectives/trading strategies involved, Iâm trying to lay my hands on past (and publicly available) examples of how these matters have been dealt with by others.

Iâm not fussed whether these examples are in C#, or whatever (including pseudo code!... or even just a detailed description in plain text!); I'm more interested in getting a feel for the scope of what is handled in each case.

Can anyone point me in the right direction?

I had to do a lot of this in the last couple of weeks, because of Ninja's bug with IB OCOs ... if you want to take-over the Ninja default error-handling, you need to set RealtimeErrorHandling=false; ... then you must keep track yourself of pretty-much everything that pertains to each order, and check the IOrder info (received through OnOrderUpdate()) against what this particular order was supposed to do.

Things I keep track of, per order:
private IOrder _TheOrderIOrder;
private OrderState _TheOrderState;
private OrderState _ThePriorOrderState;
private int _TheOrderRequestedQuantity;
private int _TheOrderFilledQuantity;
private string _TheOrderOCO;

The real pain in the butt is checking OCO order pairs (Target/Stop).

The nice thing about doing it myself, if that I get the exact behavior I want ... I also went ahead and addressed a number of other error cases, such as UPS Suspend, system Shudown, Internet loss of connection followed by reconnection.

abattia · Jul 8, 2012

Quote from 2rosy:

not sure about ninja but in general most APIs pass back errors/fills/etc. so you do something like

onfilll(messageobject){
print messageobject.quantity, messageobject.orderid
}

onerror(messageobject){
print messageobject.error
}

Thanks, 2rosy

How are different error messages responded to? I guess that's part of what I am trying to determine.

In the case of rejected orders, in some cases perhaps the response if to re-send the rejected/failed order again until N attempts have been made, and then fall back to some "next stage" process if the rejections continue?

And some errors can perhaps just be "caught" so that the strategy can continue? While shutting down the strategy and flattening might be the best response to other errors?

And overfills; it seems to me that overfills during an entry (when the strategy's intention was to take a position) aren't perhaps quite so "bad" as overfills during an exit (when the strategy was trying to get out)? And so different handling could be applied?

Etc ...

abattia · Jul 8, 2012

Quote from clearinghouse:

I don't think there's some document you can read to just do this. I think the best way to do this is to actually experiment and build a few systems and keep doing it until the kinks are worked out. If you're doing stocks, I advise using 200 shares for testing purposes so you can deal with odd and mixed lots and partial fills (which, honestly, are a real nuisance -- they should ban odd lots.)

It took me about 1-1.5 months of building and then mostly testing until my order management system was stable. It is something that does take time, because sometimes you don't anticipate corner cases and the market can surprise you with behavior you didn't quite plan out.

The coding part is easy; it's the testing part that sucks hardcore.

Thanks. Yes, fair enough.

I guess it follows a principle all too common in software design, in my experience at least. That the more unlikely to occur a particular situation is assumed to be, the more likely it is to occur the very first time you run the software "live" ...

abattia · Jul 8, 2012

Quote from dom993:

I had to do a lot of this in the last couple of weeks, because of Ninja's bug with IB OCOs ... if you want to take-over the Ninja default error-handling, you need to set RealtimeErrorHandling=false; ... then you must keep track yourself of pretty-much everything that pertains to each order, and check the IOrder info (received through OnOrderUpdate()) against what this particular order was supposed to do.

Things I keep track of, per order:
private IOrder _TheOrderIOrder;
private OrderState _TheOrderState;
private OrderState _ThePriorOrderState;
private int _TheOrderRequestedQuantity;
private int _TheOrderFilledQuantity;
private string _TheOrderOCO;

The real pain in the butt is checking OCO order pairs (Target/Stop).

The nice thing about doing it myself, if that I get the exact behavior I want ... I also went ahead and addressed a number of other error cases, such as UPS Suspend, system Shudown, Internet loss of connection followed by reconnection.

Thanks, dom993!

My laptop battery is about to run out, so can't give the detailed response now your post deserves. Will respond later ...!

dom993 · Jul 8, 2012

A couple more things I forgot in prior post:

If you are using the Managed approach, and rely of Ninja to manage OCOs (using SetProfitTarget() / SetStopLoss()) you will have to "catch" all Target & Stop IOrders in the OnOrderUpade() method, as they are not exposed anywhere else.

Be aware that by default, Ninja will issue an OCO pair for each Partial-Fill of the entry order, so when catching these Target / Stop order pairs in OnOrderUpdate() you'll need to expect any number of them (not just 1) (unless you restrict yourself to 1 contract per target).
This is where the Dictionary<K,V> class comes in handy (use the OCO string as Key)

I hope this helps.

Also ... You definitively need to connect to a broker-hosted simulated account to do your testing ... it will be much easier to 'force' error situations then.

Also ... You can look at Ninja trace files, with a little bit of effort you'll be able to see what's going-on at the order-feed level.