Best practices?

[nitro]

Storing Positions, Orders, etc, we might use couchDB:

http://books.couchdb.org/relax/

 

[autopilot]

Quote from ScoobyStoo:

I am currently building a fully automated futures algo trading system from the ground up and would appreciate the input of any others that have done the same thing.

I have 10+ years professional coding experience and have architected complex business applications before. However, at the moment I am working on the order management module and running into a few design headaches relating the asynchronous nature of interfacing with the market. I always knew this part of the project would be the hardest due to the formidable number of scenarios that must be handled arising from the decoupled nature of the beast. I won't go into examples here as any of you that code trading systems at this level will know exactly what I mean.

Are there any books/white papers in the public domain which address the best practices for handling these architectural decisions? I want to make sure I get the core design correct the first time round.

Many thanks as always...

I used Observer design pattern and Adaptor design pattern to deal with asynchronous problem. It works for me.
Maybe you can try it out.
home4trade.com has some infomation that I posted.

 

[byteme]

Quote from nitro:

Storing Positions, Orders, etc, we might use couchDB:

http://books.couchdb.org/relax/

What are the advantages of using a document oriented database for positions and order?

I can think of other use cases for document oriented databases but positions and orders seem well suited to the relational model.

 

[nitro]

Quote from byteme:

What are the advantages of using a document oriented database for positions and order?

I can think of other use cases for document oriented databases but positions and orders seem well suited to the relational model.

I currently tag all my orders with a reason for entry/exit. Soon, I will programming event based systems, and instead of a small "tag" the cause could be a long article or news story. I would like to store that along with the order for later analysis.

 

[rosy2]

Quote from nitro:

I currently tag all my orders with a reason for entry/exit.

is this a discretionary manual system then?

 

[nitro]

Quote from rosy2:

is this a discretionary manual system then?

The current one is Hybrid, but all entries and exits are 100% quantifiable. The only thing that is discretionary is the hedging part, and the size to do.

The event based system is likely 100% automated.

 

[rosy2]

Quote from nitro:

The current one is Hybrid, but all entries and exits are 100% quantifiable. The only thing that is discretionary is the hedging part, and the size to do.

The event based system is likely 100% automated.

then why would you need to tag all orders with a reason for entry/exit? isnt it easier to just say that this order is from this strategy.

 

[nitro]

Quote from rosy2:

then why would you need to tag all orders with a reason for entry/exit? isnt it easier to just say that this order is from this strategy.

Because the strategy can generate different order types. This is an option system where there could be several spreads, etc, that defines an "trade", which can be comprised of multiple orders. Also, some contracts get rolled, and they are part of the same trade, just different strikes. Options complicate things considerably over equities. For example, say I am trading a straddle. That is two orders if you enter both legs as separate orders, but as far as the strategy is concerned it is part of one strategy. Or what about a calendar, or things that have 8 legs, or ....

Also, as the system evolves, or I have systems that take trades on news stories for example, if I do things right from the beginning, I won't have to change a thing.

 

[ScoobyStoo]

Quote from nitro:

First, when talking about this stuff, it would be helpful if you differentiated between order state, position state, and marketdata (subscriptions etc) state.

Look, you have a class like this.

public class MarketData
{
    ThirdPartyAPI thirdPartyAPI;
    ....

    SubscriptionEntity SubscribeSymbol(Security s)
    {
         return thirdPartyAPI.SubscribeSymbol(...);
    }
}

as opposed to

public class MarketData
{
    ThirdPartyAPI thirdPartyAPI;
    Dictionary < Security, SubscriptionEntity > subscriptionEntities;
    ....

    SubscriptionEntity SubscribeSymbol(Security s)
    {
         SubscriptionEntity se;
         if(false == subscriptionEntities.TryGetValue(s, out se))
             return thirdPartyAPI.SubscribeSymbol(...);
    }
}

The point I am trying to convey is that you are keeping state in two places in the second case where the first case simply delegates state to the API (if it keeps it). This is poor design with no added benefit (this is a simplified case. Maybe you want to have multiple users. Then you have to keep a count of references to the subscription, so that you only unsubscribe from the thirdPary if there truly are no subscriptions. It is just a simplified example.)

In the case of OrderState, if the API is doing it for you already, same comment. It doesn't mean that you shouldn't duplicate some code in certain cases, it just means trying to delegate the state as much as possible to the owner of that state, to have lazy evaluation instead of eager evaluation as much as possible. The more the system needs to be fast, the more duplication that needs to occur. This is nothing more than saying you should cache state locally (although I have seen cases in functional programs where there is no state be faster than C++ threaded implementations that keep state. There is no one size fits all). But notice how perspective changes when you make explicit what you are doing. Now everything you learn about the perils of a cache immediately come to mind.

OK. I think I (and possibly some others such as propseeker) misunderstood your earlier comments.

I interpreted your previous posts as implying that you do not store order state locally at all and instead rely on making frequent data replay calls to the broker's API to request order state information from their servers whenever you needed it. What you are saying here is that you are maintaining local state (or rather the broker's API software is doing it for you). If that's the case then we can put the previous state persistance conversation to bed because we actually all agree that you have to keep state locally (whether it is done by the broker's software or your own is pretty much irrelevant) if you want to avoid making callback requests to the broker's servers every time you need order state information.

Let's draw a line under that debate.

Quote from nitro:

Your not understanding what I am saying. You are assuming that the API itself has to sync with the remote server. If the connection between the API and the provider goes down doesn't mean the API doesn't have the current state. Obviously someone is keeping state. either locally or remotely. I am saying if the API already keeps state locally, why would you duplicate that and offer another way that state can get out of sync? If the API has to query a remote server for that information, then of course you have to keep local state. I already said that. What I am saying is avoid duplication of local state.

Right. In the case where there is no local state maintained like FIX, which I mention above in another post, is another case where you have to do the local housekeeping yourself, you need to maintain local state yourself.

OK. As I said above. I think I and a few others interpreted your previous posts differently.

Quote from nitro:

Finally I am also saying that in exceptional cases like crashes, the last thing I want to do is rely on local state if at all possible. What I do is on login to the broker, I request by some mechanism the opening position coming into the day (that often comes in the morning in some manner that you have to parse and store in SQLServer or whatever), and then request all intra-day orders up to now. I then rebuild my position and act only on that. Some APIs will send you orders you missed so you only have to update those on your local state. They have a software order-ACK. But even in this case I might still prefer to relive the days orders and fills. It takes 1 minute instead of 1 second. The point is I ultimately trust what my broker says my position and my order state is in exceptional cases. If given the choice, I always rebuild state from a broker on major exceptions.

In the event of a restart following a crash I don't think there's any question that a strategy's state should be rebuilt from scratch using the position/order data persisted by the broker if possible. In fact, if the application misbehaves in any way I think the only safe approach is to consider everything held in memory as potentially tainted and therefore initiate a full restart and recovery.

What propseeker was saying earlier though is that if this data can't be obtained quickly and easily (i.e. through the broker's API) then no sane person is going to wait on the phone for it to be read back to them. They are going to use the locally stored state to make best efforts to put on any appropriate hedges. Far better to have hedges in place that reflect that last known state than be sat on the phone trying to get through to some broker who (at best) is going to be stressed and likely to make mistakes in reading data to you or (at worst) completely unobtainable because 100 other people are trying to do the same thing.

 

[NetTecture]

Quote from nitro:

Because the strategy can generate different order types. This is an option system where there could be several spreads, etc, that defines an "trade", which can be comprised of multiple orders. Also, some contracts get rolled, and they are part of the same trade, just different strikes. Options complicate things considerably over equities. For example, say I am trading a straddle. That is two orders if you enter both legs as separate orders, but as far as the strategy is concerned it is part of one strategy. Or what about a calendar, or things that have 8 legs, or ....

Also, as the system evolves, or I have systems that take trades on news stories for example, if I do things right from the beginning, I won't have to change a thing.

I like that. Do the same here, hopefully going live in a week or two.

The problem I ahve with most frameworks is that they have no idea what a (logical) trade is (taht may be composed of different legs). The result is that stuff like Ninja.... has totally fucked up statistics for complex scenarios.

Look at a spread. Large chance ONE of the sides will be a loss. Depending on how you enter - chance is nearly 100% one side will be at a loss.

So, if you loko at each individual item, you geta 50% profit rate. Yeah. Right. This is if you ahve a 100% profitable strategy - still 50% of the positions close out with a loss.

Why? No logical trade concept