Writing Robust Trading Code

[dom993]

FWIW, the only thing I save to disk is the open position & the open orders. Upon strategy start, I execute the logic on past data without sending orders, just keeping track of what the current position & orders would be, which I compare to the position & orders as stored on disk, and I adjust those as required.

Regardless of the code robustness, running automated strategies is best done from a VPS or server hosted in a datacenter ... that in itself minimizes the number of occurrences of power & Internet issues, although these must still absolutely be addressed.

 

[vikana]

One things I've learned over the years, is that it's best to code defensively, when dealing with faults. A disconnect from a quote feed is typically easy to detect, whereas a slow feed is very difficult to detect. Both are dangerous. I know of no other way than to test for all the fault scenarios I can think of. This easily gets very complex and delicate (but necessary).

 

[Craig66]

In my experience, the best way to manage this stuff is by using the finite state machine pattern. FIX order cycles, TCP connection states, IQFeed login sequences can all be coded as FSMs. That way, if anything unexpected happens, you'll know real quick if you've coded defensively. Also +1 to the guy who said the backtest code should be the same as the live code, that cuts out another whole range of potential fuckups. As for what actually happens when the disconnect occurs, that depends on protocol, FIX can resend the messages, but if you're using the IB API (for example), your screwed, it's impossible to be sure of order states after a disconnect. For IQFeed disconnects I just keep on trucking for a couple of minutes, if it stays down I crap out...for the level of latency that one runs over the internet, some minutes of lost data shouldn't be a big deal.

 

[2rosy]

Quote from gmst:

Above kind of guys will typically experience following problems:
1) Temporary connection loss due to internet outages
2) Temporary Feed loss (and missing packets for extended duration say 5 seconds or more), even when internet is fine because their feed provider has a temporary outage.
3) Sometimes their application will crash and they will have to re-start the application during the trading day. (say you trade through Excel or MC or your custom written C# program and it crashes during the trading day).

For retail I wouldn't even worry about this. Just have the number of you broker so you can cancel over the phone if necessary. Focus on strategies , research

 

[NetTecture]

Quote from gmst:

This thread will be useful for Grey and Black Box Retail traders who:
1) are not co-located
2) have normal internet connections and
3) have average feed e.g. IB, IQ Feed, Nanex etc.

So, you talk about cheap idiots here?

Do not get me wrong, but technical co-location (i.e. putting things into a data center instead of home - the term is a standard term in IT and does not mean putting it into an exchange, check server colocation in google) is imho bottom line for anything automated.

That immediately kills a LOT of possible issues - internet (data centers typically have redundant links), power quality etc.

And whoever can pay nanex can also pay that

Exchange-Colocation is another thing, but that has low gains unless you go really HFT. My trading server is located in chicago and enjoys single digit latency to the CME - and I dont have to worry about a lot of things that would otherwise be quite - ah - challenging (mostly: my internet).

 

[CalVolibrator]

are FSM not slight overkill? When I code wrappers for APIs then by default I implement heartbeat mechanism and events that the user of such wrapper can hook up to to be called back when the heartbeat detects a disconnect or failed responses. Very simple, lightweight and almost trivial to code. FSM are usually attempting to solve more complex issues. What is very important for me is to always abide by the "Separation of concerns" principle. A FSM requires shared state by default and that is not a good thing in any programming paradigm.

Quote from Craig66:

In my experience, the best way to manage this stuff is by using the finite state machine pattern. FIX order cycles, TCP connection states, IQFeed login sequences can all be coded as FSMs. That way, if anything unexpected happens, you'll know real quick if you've coded defensively. Also +1 to the guy who said the backtest code should be the same as the live code, that cuts out another whole range of potential fuckups. As for what actually happens when the disconnect occurs, that depends on protocol, FIX can resend the messages, but if you're using the IB API (for example), your screwed, it's impossible to be sure of order states after a disconnect. For IQFeed disconnects I just keep on trucking for a couple of minutes, if it stays down I crap out...for the level of latency that one runs over the internet, some minutes of lost data shouldn't be a big deal.

 

[CalVolibrator]

That is not true. Colocation is a standard industry term and it almost always refers to exchange co-location, meaning you place your algorithms on racks that are physically located on exchange premises (we are talking within the realm of financial algorithmic trading not general IT). The slightly wider terminology is "proximity hosing" which refers to locations nearby exchanges or wherever you receive data from/execute orders, whereby the term nearby is relatively lose.

And if you refer to those then I disagree with you, too, because I would claim over 50% of non hft related trading volume is NOT originating from machines that are co or proximity located (just think of all the sell side investment banks, they all run their standard DMA algorithms on their own machines and submit orders through their exchange feed handlers).

Quote from NetTecture:

So, you talk about cheap idiots here?

Do not get me wrong, but technical co-location (i.e. putting things into a data center instead of home - the term is a standard term in IT and does not mean putting it into an exchange, check server colocation in google) is imho bottom line for anything automated.

That immediately kills a LOT of possible issues - internet (data centers typically have redundant links), power quality etc.

And whoever can pay nanex can also pay that

Exchange-Colocation is another thing, but that has low gains unless you go really HFT. My trading server is located in chicago and enjoys single digit latency to the CME - and I dont have to worry about a lot of things that would otherwise be quite - ah - challenging (mostly: my internet).

 

[NetTecture]

Quote from CalVolibrator:

That is not true. Colocation is a standard industry term and it almost always refers to exchange co-location, meaning you place your algorithms on racks that are physically located on exchange premises (we are talking within the realm of financial algorithmic trading not general IT).

Yes. Now we are down to a small fringe of people redefining widely used term. Gratulations. Maybe you also want oto be crowned king of your group of idiots?

Seriously.

Yes, you are right here - from the limited scope of a non-programmer, if you are a programmer you just made a mess of any spec you ever write.

But: for someone NOT investing into HFT (which is a money game) there is a LOT of value in Colocating in the TECHNICAL term. The one that shows up when you hit google with "server colocation" (hint: no near exchange link in the first pages). YOu can move close to the exchange (same city), which eliminates a lot of possible internet issues (less hops), in some cases even with a direct line (my data center in chicago has a direct connection to the CME) - and I have seen internet outages due to issues in the global infrastructure at times. You have decent power - which is hard to do otherwise at home, also because even if You have power, your internet hub locally may not and there you go. And you ahve fast repairs - if your machine fails you can normally have it repaired within hours, as the hoster has spare parts available. Not many small retail traders run a cluster of servers in their basement and would not caer about a machine dropping out because hey, they have enough spares to just restart the trading system on another one.

So, see, outside yourobviously very limited mental image of IT there is a lot of value in seeing colocation not as "hey, I have to pay HUUGH money because colocation means putting it at the exchange".

 

[CalVolibrator]

Listen, your whole bla can be boiled down to this: Stop accusing others if you are caught in your little IT world. Colocation in regards to any computer code running in regards to financial trading refers to co-location with exchanges, ECNs, ..., no matter what you think, your opinion is actually irrelevant. You look like a tool if you talk to anyone on the trading, bank IT, hedge fund IT, side and refer to co-location as simply server hosting. Period.

Quote from NetTecture:

Yes. Now we are down to a small fringe of people redefining widely used term. Gratulations. Maybe you also want oto be crowned king of your group of idiots?

Seriously.

Yes, you are right here - from the limited scope of a non-programmer, if you are a programmer you just made a mess of any spec you ever write.

But: for someone NOT investing into HFT (which is a money game) there is a LOT of value in Colocating in the TECHNICAL term. The one that shows up when you hit google with "server colocation" (hint: no near exchange link in the first pages). YOu can move close to the exchange (same city), which eliminates a lot of possible internet issues (less hops), in some cases even with a direct line (my data center in chicago has a direct connection to the CME) - and I have seen internet outages due to issues in the global infrastructure at times. You have decent power - which is hard to do otherwise at home, also because even if You have power, your internet hub locally may not and there you go. And you ahve fast repairs - if your machine fails you can normally have it repaired within hours, as the hoster has spare parts available. Not many small retail traders run a cluster of servers in their basement and would not caer about a machine dropping out because hey, they have enough spares to just restart the trading system on another one.

So, see, outside yourobviously very limited mental image of IT there is a lot of value in seeing colocation not as "hey, I have to pay HUUGH money because colocation means putting it at the exchange".

 

[vincegata]

Colocation is a tech term which means a practice of housing servers in a datacenter to take advantage of infrastructure, bandwidth, latency, security, etc. E.g. it's used for web-hosting, not just financial apps. In finance it's usually means hosting server in a datacenter close to exchange.