Fully automated futures trading

[Kernfusion]

For negative prices I use a standard deviation filter that will pick up any large changes. Something you might want to consider.

I'm also somewhat worried if all the calculations will handle negative prices correctly.. Forecast calculations seem to swallow it well, at least no exceptions, but other places not sure.
Like for example position-sizing logic, if it gets a negative price, it will make volatilityScalar negative, not sure that's ok. I think I should use abs(price) when it goes into position sizing, because it's all about magnitude and not sign, e.g. instrumentCurrencyVolatility shouldn't be negative..

 

[Kernfusion]

I'm also somewhat worried if all the calculations will handle negative prices correctly.. Forecast calculations seem to swallow it well, at least no exceptions, but other places not sure.
Like for example position-sizing logic, if it gets a negative price, it will make volatilityScalar negative, not sure that's ok. I think I should use abs(price) when it goes into position sizing, because it's all about magnitude and not sign, e.g. instrumentCurrencyVolatility shouldn't be negative..

Also, when calculating portfolio risk with correlation matrix and weighted standard deviations, with negative prices the weighted standard deviations can become negative, I guess I should also apply Abs, although not sure here..

 

[Kernfusion]

I assert that the bid price is less than ask price

I think ticks (including Bid tick and Ask tick) are coming from IB independently, not both at once, I mean that's how their API works, they give you one thing at a time and then the next one., So when in reality Bid and Ask probably moved simultaneously (i.e. the whole spread shifted to a new level), in practice IB will first report change in bid then in ask (or vice versa) which will make the spread look wrong on the client until the other side is returned. I personally propagate prices to the logical level only if Ask >= Bid (I allow them to be equal but definitely not Ask<Bid)., and I also explicitly remove these exact price values 0.0, -1.0 and -100.0 - someone definitely hardcoded these in their software, and now it's probably next to impossible to take it out without collapsing the whole house of cards - the usual..

 

[HobbyTrading]

And a quick word about IB and data. I'm surprised almost every day how wonky getting prices is, especially when the market is closed (but you can't know that).

I noticed this a long time (a few years?) ago. Trying to receive historical data went awry when the instrument was not actively being traded. Since then I implemented a check to ensure that I only download historical data for an instrument if it is not night time. I check the instrument's local time and if it is between 10PM and 7AM I skip downloading historical data for this instrument. This does not hamper my system's performance as I only use end-of-day data (thus daily close prices).

 

[HobbyTrading]

I'm doing something similar right now, I currently collect data for 53 markets and trade only 20.

I used to do something similar. When the allocated capital was small I only allowed trading in the instruments which had the lowest value volatility per contract. As this volatility changes over time, I collected data for more instruments than only the ones I was allowing a position in.

 

[globalarbtrader]

Actually, now that I think of it - Rob, I assume your previous employer had some sort of a direct connection to the exchange (I know very little about microstructure but I assume something like that exists, and costs $$$), did you see issues like these there as well?
Also, I'd assume you'd purchase historical data from exchanges, is that data in general higher quality than the 3rd party?
Obviously, if you can talk about these things, feel free to ignore.

We got data via intermediaries like Bloomberg and Reuters; we weren't HFT so didn't need a direct exchange connection to shave off the slight latency that introduced. The BB product is called B-PIPE, can't remember what the Reuters product is called (I have a mate that used to work for them, will ask him at some point). Historical data for a new product we'd often get from Bloomberg as well; heck when you're paying many millions a year you make the most of it, although we also used people like Factset when appropriate.

Was this 'pro-level' data cleaner than IB? Yes, probably, in the sense that there are fewer 'glitches' and you are paying through the nose for things like accurate time-stamping: these peoples business is selling data to highly critical customers and they do it better than IB who at the end of the day are a brokerage for customers, 99% of whom are more concerned about price and a nice GUI than super clean data.

But when you think about it, unless there is some kind of manual or automated process in between you and the exchange, if the exchange produces a bad piece of data then it will propogate through the system to you. Using B-Pipe just means you'll get the data a bit quicker, and probably with no extra errors.

Historical data can be cleaned before use of course, which means that from someone like Bloomberg it's probably free of obvious errors (eg -1), which may not be true of IB.

My attitude has always been that all data is potentially bad. This was probably something I picked up when I was in IB, and our risk / p&l was produced through about 50 interconnected databases and excel sheets held together with Perl scripts, VBA, CTRL-C / CRTL-V and duck tape. Every morning I would come in and open the overnight marks with trepidation. I'd (a) check for anything massively wrong ("Oh look we lost a billion dollars again") and [if it wasn't obviously wrong] (b) trade in a way that if the marks were a bit off it wouldn't affect things too much (the greeks on an exotic book will change wildly during the day anyway, so if the closing snapshot is off a bit it will be irrelevant by 7:35am).

I still have that attitude now. So for historical data, I plot it and check it doesn't do anything obviously crazy; both before and after backadjustement (because I don't trust my backadjusted code eithier). For live data I flag large changes in price which look wrong and don't add them to my database until I've manually checked them. And then I design my system so that it is robust to bad data; moving averages, multiple trading rules, smoothing, buffering, multiple instruments.... these all mean that if one of my prices is off even by 50% it won't make much difference.

GAT

 

[globalarbtrader]

Also, when calculating portfolio risk with correlation matrix and weighted standard deviations, with negative prices the weighted standard deviations can become negative, I guess I should also apply Abs, although not sure here..

No they can't. A standard deviation is the square root of the sum of a series of squares of a return. If the return is the difference between two negative numbers, or one positive and one negative, well it will still be positive.

GAT

 

[globalarbtrader]

I'm also somewhat worried if all the calculations will handle negative prices correctly.. Forecast calculations seem to swallow it well, at least no exceptions, but other places not sure.
Like for example position-sizing logic, if it gets a negative price, it will make volatilityScalar negative, not sure that's ok. I think I should use abs(price) when it goes into position sizing, because it's all about magnitude and not sign, e.g. instrumentCurrencyVolatility shouldn't be negative..

Yes you should. It's worth double checking that your system can handle negative prices; testing this is easy. I just did some checks and whilst I'm confident it can handle negative adjusted prices in the past (which is common) I'm not 100% it deals okay with negative current prices: I've never had them but I'm still going to make sure.

GAT

 

[Kernfusion]

A standard deviation is the square root of the sum of a series of squares of a return

I meant that it's the "weighted" part that can produce negative values (intermediate values, not the final result, but that result will be different if some of the weights changed sign because of the negative price).
So the formula for portfolio variation is this: V(R) = w'Vw where V is a covariance matrix (I'm using correlation matrix instead and multiplying 'w' terms by the corresponding standard deviations, but the end result is the same).

So currently, even before allowing negative prices, 'w' terms can contain negative elements when I'm short. I'm also normalising the current weights such that their ABS values add up to 1. For example I have these positions:

A: pos:1, price:200$
B: pos:-1, price:500$
C: pos:1, price:300$

then my 'w' matrix will be [0.2, -0.5, 0.3].
Let's say I've got this covariance matrix, then my portfolio variance will be this:

if however the price of the asset B, which I'm short, is negative, say -500$, it's weight, I guess, should become positive?, which will change the result:

I.e. the question is if I'm short something with a negative price, should it's weight be considered positive when calculating portfolio risk?
Or maybe I should just always apply ABS to both positions and prices and be done with it ?

 

[globalarbtrader]

I meant that it's the "weighted" part that can produce negative values (intermediate values, not the final result, but that result will be different if some of the weights changed sign because of the negative price).
So the formula for portfolio variation is this: V(R) = w'Vw where V is a covariance matrix (I'm using correlation matrix instead and multiplying 'w' terms by the corresponding standard deviations, but the end result is the same).

So currently, even before allowing negative prices, 'w' terms can contain negative elements when I'm short. I'm also normalising the current weights such that their ABS values add up to 1. For example I have these positions:

A: pos:1, price:200$
B: pos:-1, price:500$
C: pos:1, price:300$

then my 'w' matrix will be [0.2, -0.5, 0.3].
Let's say I've got this covariance matrix, then my portfolio variance will be this:

View attachment 248437

if however the price of the asset B, which I'm short, is negative, say -500$, it's weight, I guess, should become positive?, which will change the result:

View attachment 248438

I.e. the question is if I'm short something with a negative price, should it's weight be considered positive when calculating portfolio risk?
Or maybe I should just always apply ABS to both positions and prices and be done with it ?

OK that's not how I do risk. I calculate standard deviation and positions as $ which I then convert into % of capital, so prices (negative or otherwise) never come into it. Applying ABS to positions is clearly a bad move, Also summing weights to 1 is also a bad move, since you're ignoring leverage effects.

GAT