Algorithmic Trading a large folio with small orders

[braincell]

Quote from fullautotrading:
Is the plan convenient with futures and fx too (i have not tried yet) ?

It is, when using limit orders. Use your paper trading account because it will also now be using cost-plus. You should be able to see the commissions there.

I'm glad I could help, maybe next month you can shave off $300 in comms.

Quote from fullautotrading:
In the next test i want (to try to) keep a rigorous risk allocation. Probably no more than 2% of capital for each instrument (please advise).

Having a constant % of capital for each instrument is good, but even better would be if you could include the volatility to scale that % also. Imagine a folio where you have fx, stocks and futures. Their volatilities will be different and the strategy may work differently. You probably want less capital on more volatile stocks.

Quote from fullautotrading:
One does not need to "go auto" immediately. Infact i also keep the instruments <b>under watch for some time</b> before starting it auto. Sometimes i wait days.

Well, i'm not sure it is always practical to wait for days. Maybe you want to add/remove or restart an instrument very quickly. Why wait for days when you can just "import" tick data and resume? I know maybe you don't have tick data but some of us who have easy access to it would find it easier.

Quote from fullautotrading:
I am also currently a little perplexed about the auto bounds for the new layers ("cloned" instruments). Currently i am not inheriting them. But i am thinking now that it would be safer to inherit them, and let the trader in case to reset them ("triangle" icon). I think i will modify this behavior in the next update.

I am not so sure i still see a major difference between that and setting manual CT bounds. Automatic CT bounds are just random, basically based on when the tick data starts to accumulate for that layer. Why not just use manual CT bounds always in that case?

 

[fullautotrading]

<b>>I'm glad I could help, maybe next month you can shave off $300 in comms. </b>

For sure. This was a <b>real great</b> piece of info! ;-))

<b>>Why not just use manual CT bounds always in that case? </b>

The "auto bounds" are a game feature. In case they can be disabled. I usually disable them only on a new "layer" (with hedging function). The "auto bounds" work great to avoid giving back all your profit, due to too unfavorable entry direction.

<b>>Having a constant % of capital for each instrument is good, but even better would be if you could include the volatility to scale that % also.</b>

Sorry i have been delayed by the tickdata resampling algorithm i have been refining these couple days. Now i am ready for this easier functionality.
Using volatility seems a <b>great idea</b>. (Although, we have to see in practice how useful it is to scale the packet size.)
(Also, for options, the scaling could be totally different, due to crazy volatility figures).

Here is a prototype of the functionality to set the packet size of folio instruments proportionally to a reference capital.

This is the feature i am designing now, and i should finish in a while: feel free to send suggestions so in can incorporate them immediately.

Following the suggestion from Braincell i have also added a "volatility factor" (which would multiply the volatility reduce the packet size,
a default might be 1/50 so that it would be 1 when volatility is 50%. In practice the box would contain the instrument volatility, which the bot computes realtime. Advise)

The scheme is simple.

There is a textbox with the "reference capital", which could possibly be retrieved from the account, in which case i think i could use the item called "NetLiquidation" (other items availablle are for instance "FullAvailableFunds" and "BuyingPower". From some discussion so far it seems that "NetLiquidation" is the preferred candidate. Advise.

Under this reference capital there the max percentage of capital to be allocated per each instrument. For instance 2%. And the max number of packets juggled with this instruments (for instance 6, if we use 2 layers each with "Bounded3" game)

Below the volatility factor, described before. Which can be retrieved as avg value computed by the bot.

Then a final line with min and max size allowed by the trader (for instance one might not want to trade less than 2 share, or
in any case would not want to exceed 50 shares ...).

While the trader fills these fields the packet size would appear interactively in the label below, and it can be applied by pushing the near button.

Finally there is a <b>"propagation"</b> feature which will clone the criterion to all similar instruments (for instance all STKs).
And doing so can also automatically compute the corresponding packet size and apply it. Also it can use the individual volatility
of each instrument when recomputing its packet size.

Let me know possible suggestions or ideas, while i am at it.

<img src="http://www.elitetrader.com/vb/attachment.php?s=&postid=3526215" />

 

[braincell]

Real-time measurement of volatility and using the "volatility factor" sounds like a practical and good idea. The only thing would be how you measure volatility?

Maybe you could measure volatility with the "speed" you talked about before, ie, how frequently the instrument hits the different 0.1% (grid) lines. That would probably be sufficient, simple, and practical. Meaning, each time it hits a new grid line (any direction), log it, and measure hits per time period (ie 1 day). However note that for example Forex is probably not going to hit those lines very often. You may want to use 0.001% instead for that reason, but then we have different problems. I have other suggestions if you're interested.

Reference Capital is probably better than automatically taking NetLiquidation. We can imagine sometimes a portion of the capital will need to be used for other means, or just to decrease exposure to G-Bot in total. I say let the user tell G-Bot what that is.

Also, when alocating capital, make sure to take into account the multipliers for futures, it's a small "gotcha" just in case you miss it.

 

[fullautotrading]

I have finally finished the new feature to "calibrate" the the folio
in order to <b>balance the exposition</b> according to <b>value and volatility</b>.

The picture below show the current interface.

<img src="http://www.elitetrader.com/vb/attachment.php?s=&postid=3531168" />

It's already different from the prototype we have seen before, as when actually implementing it, one finds out new things that are useful to have.

On top of the panel a button to show the current packet sizes of the whole folio.
For instance, one would get a list like (this is, as an example, a "constant" packet size of 20):

AGQ STK ARCA 20
BGU STK ARCA 20
BGZ STK ARCA 20
DDM STK ARCA 20
DGP STK ARCA 20
DIG STK ARCA 20
DUG STK ARCA 20
DXD STK ARCA 20
EEM STK ARCA 20
EFA STK ARCA 20
ERX STK ARCA 20
ERY STK ARCA 20
...

Then there is a section for input from the fund manager: the "reference capital" (which can be taken directly from the account balance, shown in the window on the bottom right or which can be manually inputed), the maximum % allocation for each instrument, and the max number of packets we are planning to use. On the top right a few options: <b>use volatility</b> - as suggested by braincell - and enforce min/max packet size.

On the bottom there is a "propagation" facility which can <b>apply automatically the "calibration criterion" to the whole folio</b> and apply the corresponding packet (it also can show a preview of all packets, before applying them).

For instance, running this with the shown parameters and clicking on the "preview" button, i get now (prices can be a little off, as mkt is closed at this time):

AGQ STK ARCA 4 [Price: 41.46 Avg Volat: 73.29%)
BGU STK ARCA 2 [Price: 69.05 Avg Volat: 62.00%)
BGZ STK ARCA 7 [Price: 24.53 Avg Volat: 64.90%)
DDM STK ARCA 1 [Price: 63.27 Avg Volat: 34.49%)
DGP STK ARCA 2 [Price: 47.96 Avg Volat: 43.30%)
DIG STK ARCA 4 [Price: 38.94 Avg Volat: 68.66%)
DUG STK ARCA 6 [Price: 27.95 Avg Volat: 70.44%)
DXD STK ARCA 2 [Price: 56.52 Avg Volat: 40.03%)
EEM STK ARCA 2 [Price: 37.75 Avg Volat: 26.47%)
EFA STK ARCA 1 [Price: 48.72 Avg Volat: 28.91%)
ERX STK ARCA 6 [Price: 37.6 Avg Volat: 97.37%)
ERY STK ARCA 18 [Price: 12.89 Avg Volat: 91.51%)
...

I think it's a quite useful and nice thing to have, as it makes matter of seconds to correctly balance the folio, while doing all this work manually, taking into account volatility too, would be tedious and error prone.

The implementation is essentially something like this (let me know if you see errors in the logic, or see improvements) :

NominalValueForOneUnit = Price * Multiplier;
MaxCapitalToBeAllocatedOnInstrument = ReferenceCapital * MaxPercentagePerInstrument * 0.01;
PacketSize_Dbl = MaxCapitalToBeAllocatedOnInstrument / (NominalValueForOneUnit * MaxNumberOfPacketsPerInstrument);

followed by min max check, optional shrinking proportionally to volatility:

PacketSize_Dbl *= Volatility / VolatilityFactor;

and finally rounding to integer.

I am planning to restart monday with a brand new folio "balanced" with this new toy ;-) and I am selecting a list of ETFs for the new test.

[ While i make the real $$$ test, you people interested can run a parallel session in paper trading, so you can also provide me with real time suggestions about the possible new <b>layers</b> or other discretionary aspects (force direction or discretionary bounds). Those interested send me a PM, i will send you the planned folio for the new test.

(I will be redistributing the app before the mkt reopens.) ]

 

[Occam]

Quote from fullautotrading:

Hi PocketChange,

However, to be fair, a few considerations are probably to be made. I started up the game with <b>really ridiculous small orders</b>, to see how the system responded, and i was not ashamed to issue orders of size <b>1 or 2 shares</b> only!

(Now i am using mostly 20-30 shares). Clearly, when one spends $1 commission to buy a couple shares, this is an expected consequence ;-))

Regular exchanges don't permit odd lot orders that initiate new positions. So my guess is most (or all?) of your orders are being internalized by IB's Timber Hill unit, hence IB is your counterparty. As the orders are not going to an exchange, my guess is that you may experience a disproportionate number of weird fills and/or other unexplained slippage/missed trades. Something to look out for, especially when using these very small orders.

 

[fullautotrading]

Quote from Occam:

Regular exchanges don't permit odd lot orders that initiate new positions. So my guess is most (or all?) of your orders are being internalized by IB's Timber Hill unit, hence IB is your counterparty. As the orders are not going to an exchange, my guess is that you may experience a disproportionate number of weird fills and/or other unexplained slippage/missed trades. Something to look out for, especially when using these very small orders.

Hi Occam,

yes i think you are right: i believe they are using some sort of internal "reservoir", as am able to short without problems even instruments which are not "shortable".

In practice i have seen no problems so far, as i only use LMT orders, so there is no room for "slippage". All orders executed fine and quickly (few millisecs) thus far.

It's good that they provide this service, or else juggling >100 shares at a time would not allow to play this game with small capital ;-)

 

[fullautotrading]

I am getting ready for the new real $$$ test, this time with <b>"balanced" sizing</b>.

Since yesterday i have put under observation a folio of 50 ETFs to collect data on realtime volatility and cointegrations.

Today when mkt open i will make a further screening to form a folio of about 40 and determine the "balanced" order sizes.

I will also enforce a <b>"correlation" rule</b> which will prevent an instrument to open if a strongly correlated is already open.

When mkt opens i will show the sizes suggested by the folio balancer.

<img src="http://www.elitetrader.com/vb/attachment.php?s=&postid=3533070" width="1000" />

[ PS
a line in the post about folio sizing should read:
PacketSize_Dbl *= Me.ReferenceVolatility / Volatility
as we are shrinking proportionally to volatility (while within the games, we enlarge the game proportionally to volatility). ]

 

[fullautotrading]

Today was busy. Talking to many friends trader and getting some new idea. Thanks to James (canada) who suggested to make a step further and automatize also the "packet size recalibration" to execute periodically. So i just added this new feature ;-)

Also discussing with Jens and Evgenios (germany) we come out with a nice improvement for the folio rules to "break" correlations.
I already had correlation rules, which were also discussed, at the time, with Danny (chicago) but i think i can make it a little
more powerful for the occasion. Currently the app computes realtime volatility and pairwise "cointegration" (scx) between instruments.
The rule said: "If an instrument is strongly cointegrated (say corr > 0.3 or corr < -0.3) with another open instrument, it will not be open."
Now, in hindsight, this rule is pretty drastic, as we can actually use cointegration for an hedging purpose and reducing risk,
instead of merely avoiding to take new positions.

So i have recoded it to the following version:

"Consider a instrument "Instr". Consider the set C(Instr) of all instruments which are correlated with it (say corr > 0.3 or corr < -0.3), that is having a correlation which exceeds a given threshold.
Instr will be allowed to take a position only <b>if the new position contributes to reduce the exposition.</b>"

Now this is a general formulation. I need to make it more precise.
First of all let's define a metric to measure "exposition" of a set of cointegrated instruments.
I define a "volatility-adjusted exposition" of one instrument as follow:

volatility-adjusted exposition = PositionWithSign * price * multiplier * volatility / ReferenceVolatility

that is: = NominalValueWithSign * volatility / ReferenceVolatility

This expresses (with sign) how much an instrument is "exposed".

<img src ="http://www.elitetrader.com/vb/attachment.php?s=&postid=3533791" width="1000" />

Now let's assume, for a moment, that the set C contains only instrument with *positive* correlations exceeding the
given threshold T (eg, T=0.3). If we make the algebrical sum of the expositions of all open instruments in the set C and Instr itself, we have the global exposition of this set and Instr. Let's call this Expos(CI). Let's distinguish the various cases happening when it's time to open or add:

if Expos(CI) > 0

BUY blocked
SELL permitted

if Expos(CI) < 0

BUY permitted
SELL blocked

This way we allow the instrument to take only positions which will contribute reducing the current exposition, considering its realtime correlations and instrument volatilities.

Finally, we need an ajustment for the general case where there are also negative correlated instruments. In such a case we will proceed exactly as before, with the only difference that the expositions will be added up with opposite sign to the rest of expositions.

In fact a negative exposition of negatively correlated instruments has the same relevance (for hedging purposes) as a positive exposition of positively correlated instruments. (A large "positive" "exposition" will result from either too many LONG with positive correlations or too many SHORT on instrument with with negative correlations)

(In practice, within the loop to accumulate expositions, we change the sign to those with negative correlations)

 

[fullautotrading]

ok, finally restarting the test with the new features.

I have started 45 ETFs on one instance and VM (alone) on another one:

<img src="http://www.elitetrader.com/vb/attachment.php?s=&postid=3534119" width="1000" />

The folio balancer, with 1.5% of a reference capital of 50K and max 6 packets for instrument gives the following sizes (which are automatically recalibrated by the app, according to volatility):

<code>
Current packet sizes for instruments of type STK

AGQ STK ARCA_USD 2
BGU STK ARCA 1
BGZ STK ARCA 4
DDM STK ARCA 3
DGP STK ARCA 3
DIG STK ARCA_USD 2
DUG STK ARCA 3
DXD STK ARCA 3
ERX STK ARCA 2
ERY STK ARCA 6
FAS STK ARCA_USD 1
FAZ STK ARCA 3
GASL STK ARCA_USD 2
GLD STK ARCA_USD 2
IDU STK ARCA 3
IJR STK ARCA 2
IVV STK ARCA 2
IWD STK ARCA 5
IWM STK ARCA 2
MDY STK ARCA 1
MWJ STK ARCA 3
QID STK ARCA 3
QLD STK ARCA 3
QQQ STK NASDAQ 4
SCO STK ARCA_USD 2
SDS STK ARCA_USD 8
SH STK ARCA 7
SKF STK ARCA 2
SLV STK ARCA_USD 5
SPY STK ARCA 2
SSO STK ARCA_USD 3
TBT STK ARCA 6
TLT STK ARCA 2
TNA STK ARCA 1
TVIX STK ARCA 3
TWM STK ARCA 2
TZA STK ARCA 3
UCO STK ARCA 3
UNG STK ARCA 5
UPRO STK ARCA 2
UVXY STK ARCA_USD 1
UWM STK ARCA 2
UYG STK ARCA 2
VIXY STK ARCA 1
VXX STK ARCA_USD 2
</code>

Ridiculously small sizes, which will cost a boatload in commissions (and will make very hard to scalp). Anyway, just testing ideas. These are both pointing to the same real $$$ account

Also the new "cointegration" mechanism, as described in the previous post, is working here.

 

[fullautotrading]

Finally i selected 42 instruments, allocating just 1.5% each, that is 0.5% per trade. VM is traded separately in another instance.

The application computes now automatically the packet sizes and recalibrates all automatically, while price and volatility
change (this was an idea of a smart follower!).

Current packets sizes (number of shares) are:
<code>
AGQ STK ARCA_USD 3 [exactly: 3.06 Real %:0.72 Price: 39.97 Avg Volat: 81.64% ]
BGZ STK ARCA 6 [exactly: 6.08 Real %:0.87 Price: 24.2 Avg Volat: 67.91% ]
DDM STK ARCA 4 [exactly: 4.37 Real %:1.50 Price: 62.43 Avg Volat: 36.65% ]
DGP STK ARCA 4 [exactly: 4.16 Real %:1.11 Price: 46.35 Avg Volat: 51.87% ]
DIG STK ARCA_USD 4 [exactly: 3.51 Real %:0.90 Price: 37.55 Avg Volat: 75.85% ]
DUG STK ARCA 5 [exactly: 4.77 Real %:0.86 Price: 28.6 Avg Volat: 73.29% ]
DXD STK ARCA 5 [exactly: 4.75 Real %:1.71 Price: 57.13 Avg Volat: 36.86% ]
ERX STK ARCA 3 [exactly: 2.77 Real %:0.65 Price: 35.95 Avg Volat: 100.46% ]
ERY STK ARCA 8 [exactly: 7.80 Real %:0.63 Price: 13.2 Avg Volat: 97.10% ]
FAS STK ARCA_USD 2 [exactly: 1.52 Real %:0.93 Price: 77.2 Avg Volat: 85.21% ]
FAZ STK ARCA 4 [exactly: 4.23 Real %:0.66 Price: 27.65 Avg Volat: 85.52% ]
GASL STK ARCA_USD 3 [exactly: 2.58 Real %:0.41 Price: 22.75 Avg Volat: 170.62% ]
GLD STK ARCA_USD 3 [exactly: 2.98 Real %:2.73 Price: 151.74 Avg Volat: 22.11% ]
IDU STK ARCA 5 [exactly: 5.39 Real %:2.62 Price: 87.36 Avg Volat: 21.24% ]
IJR STK ARCA 4 [exactly: 3.97 Real %:1.70 Price: 70.89 Avg Volat: 35.56% ]
IVV STK ARCA 3 [exactly: 3.47 Real %:2.37 Price: 131.88 Avg Volat: 21.85% ]
IWD STK ARCA 7 [exactly: 6.54 Real %:2.76 Price: 65.71 Avg Volat: 23.26% ]
IWM STK ARCA 4 [exactly: 4.37 Real %:1.83 Price: 76.07 Avg Volat: 30.05% ]
MDY STK ARCA 2 [exactly: 2.04 Real %:2.02 Price: 168.74 Avg Volat: 29.07% ]
MWJ STK ARCA 4 [exactly: 3.59 Real %:0.84 Price: 34.98 Avg Volat: 79.74% ]
QID STK ARCA 5 [exactly: 5.18 Real %:1.05 Price: 35.14 Avg Volat: 54.95% ]
QLD STK ARCA 4 [exactly: 3.74 Real %:1.20 Price: 50.06 Avg Volat: 53.42% ]
QQQ STK NASDAQ 6 [exactly: 6.31 Real %:2.24 Price: 62.18 Avg Volat: 25.49% ]
SCO STK ARCA_USD 4 [exactly: 3.75 Real %:1.17 Price: 48.71 Avg Volat: 54.75% ]
SDS STK ARCA_USD 13 [exactly: 13.38 Real %:1.34 Price: 17.13 Avg Volat: 43.63% ]
SH STK ARCA 11 [exactly: 11.48 Real %:2.52 Price: 38.14 Avg Volat: 22.85% ]
SKF STK ARCA 3 [exactly: 3.45 Real %:0.89 Price: 49.25 Avg Volat: 58.79% ]
SLV STK ARCA_USD 9 [exactly: 9.30 Real %:1.46 Price: 27.1 Avg Volat: 39.68% ]
SPY STK ARCA 4 [exactly: 3.67 Real %:3.15 Price: 131.42 Avg Volat: 20.71% ]
SSO STK ARCA_USD 5 [exactly: 4.54 Real %:1.52 Price: 50.61 Avg Volat: 43.51% ]
TBT STK ARCA 13 [exactly: 13.37 Real %:1.24 Price: 15.85 Avg Volat: 47.18% ]
TLT STK ARCA 4 [exactly: 4.12 Real %:3.03 Price: 126.25 Avg Volat: 19.22% ]
TNA STK ARCA 2 [exactly: 2.49 Real %:0.57 Price: 47.29 Avg Volat: 84.80% ]
TVIX STK ARCA 6 [exactly: 5.65 Real %:0.32 Price: 8.97 Avg Volat: 197.24% ]
TWM STK ARCA 4 [exactly: 4.28 Real %:0.83 Price: 34.38 Avg Volat: 67.90% ]
TZA STK ARCA 5 [exactly: 5.15 Real %:0.66 Price: 21.92 Avg Volat: 88.67% ]
UCO STK ARCA 6 [exactly: 5.72 Real %:1.08 Price: 30.01 Avg Volat: 58.24% ]
UNG STK ARCA 10 [exactly: 9.62 Real %:1.02 Price: 16.99 Avg Volat: 61.18% ]
UPRO STK ARCA 2 [exactly: 2.33 Real %:0.82 Price: 68.32 Avg Volat: 62.85% ]
UWM STK ARCA 4 [exactly: 4.07 Real %:0.88 Price: 36.66 Avg Volat: 66.97% ]
UYG STK ARCA 3 [exactly: 3.41 Real %:0.91 Price: 50.41 Avg Volat: 58.20% ]
VXX STK ARCA_USD 4 [exactly: 4.27 Real %:0.50 Price: 20.78 Avg Volat: 112.59% ]
</code>

In a while I will probably either increase the % allocated or add more instruments. Clearly with such small allocation and small capital the relative weight of commissions is enormous.

So far $ 16 in commissions (thanks also too "cost plus" suggested by braincell).

Net Liquidation gain: 127.35 EUR (elapsed: 9.21 days). Margins:

FullInitMarginReq: 1260.78 (Min: 1146.54 Max: 1309.80) EUR
FullMaintMarginReq: 1111.48 (Min: 1005.17 Max: 1174.24) EUR

So just 1K margin, using such large folio and VM.

Considering we are juggling minuscole packets with relatively high commission, one gets anyway the picture.

The <b>folio cointegration rules</b> work very well. Exposition is balanced by blocking all the entries which increase the exposition
of the cointegrated instruments, and allowing only those entries which decrease the exposition (hedging).

The picture below shows for each instrument, the exposition of correlated instruments. The threshold here is cointegration > 0.5
(in absolute value).

In practice, if an instrument has correlated instruments exposed in a given direction, it is forced to scalp only in a way that
reduces the global exposition. This is a powerful mechanism as we know well how important is the problem of cointegration.
In this way actually we transform correlations in a "positive" force, useful to reduce risk.

The bottom picture shows in highlighted cyan the pairs of instruments which have abs cointegration bigger than 0.5. This gives immediately the idea of how much "blocking" is going on in the folio, for a given chosen threshold. I am using 0.5 right now.

<img src="http://www.elitetrader.com/vb/attachment.php?s=&postid=3539637" width="1000" />

I am doing massive changes now to the application, also rewriting some part of the peaking mechanism, to make it vastly more powerful. Next update will be very rich ;-)

I will also prepare higher frequency games, as the correlation rules are blocking a lot of activity (the "dangerous" part of it), and probably we can allow higher frequency entries.