Automated folio trading - Role of Information and Hedging

[fullautotrading]

It's time to finally complete the management of the players at "logical" level, so that we have a complete set of tools which allows us to change the "logic representation" of a given set of entries waiting to be closed.

It is important to realize that a given set of "physical" orders can have, at the logical level of the players, infinite representations, and clearly sometimes it is useful to switch from a representation to another one.

Let's see this, starting with simple examples and gradually escalate our insight.

Imagine we have an ETF, TNA for instance and place 3 "automated entries":

BUY 100 @ 78.33
BUY 200 @ 78.41
SELL 301 @ 78.36

Now to each "real" order which has been executed, there is "naively" associated a player (here represented by a square), that is a logic unit which is keeping all the information about the entry and also follows independently the rules to automatically close and any other rules defined by the scalping/hedging game fixed by the fund manager.

At this, say "naive", level there is a one-to-one correspondence between entries and players. Now let's make the "conceptual leap" which will be useful to deal with more complex situations.

Look at the picture above: we have our 3 players, and if we look into the yellow rectangle, the monitor is telling us something. It says:

open players: Buy 300 @ 78.38, Sell 301 @ 78.36, Equiv: -1 @ 71.36

The meaning of this text is pretty simple. Since the buys are:
BUY 100 @ 78.33
BUY 200 @ 78.41

we have Buy 300 @ 78.38, where 78.38 represents the average buy price.

Similarly, 301 @ 78.36, represents the sell side (in this case it is coincident with the unique sell we have made: SELL 301 @ 78.36. If we had more, clearly we would have the cumulative position and the average price, just like we have for the buy side.)

Now, let's look at where it says: Equiv: -1 @ 71.36. What does it mean ?

This is saying that the 3 players, taken together, are globally "equivalent" (reasoning in terms of price and position alone) to have just 1 sell player of 1 share @ 71.36. In practice, the 2 buys have the effect to lower the price of the unique sell we have made, and this is completely intuitive: it's like we had sold just 1 share at a "less convenient" price (instead of 301 @ 78.36).

Now, what would change if we replaced the 3 players with a unique player Sell 1 @ 71.36 ?
Well, we have now the feature to do it and let's do it:

proceeding with the "grouping" we get a unique player:

Now what has changed ? If you look at the PNL figure, of course it has remain unchanged. This is obvious because it is completely independent of the players and based on the "physical" (real) orders actually executed.

If we look at the text on the screen, as before, we notice that the global player situation has also remained unchanged,and in fact we have:

open players: Buy 0 @ 0.00, Sell 1 @ 71.36, Equiv: -1 @ 71.36

so the equivalent position and price is always the same.

Now, it becomes obvious that we can play all the "alchemical" transformations we want on the players passing from an equivalent representation to another one. And this is useful to change the "logic view" of a given set of orders.

Let's now make a step ahead. I have discussed previously the "splitting" feature, where we could take 1 sell player and split into multiple sell players. Or we could take 1 buy player and then split it into multiple buy players.

As cursory remark, I noted that given 1 player, there are of course infinite equivalent representations. I also mentioned the possibility to represent it with a combination of buy/sell players. This seemed kind of mind blowing and heretic when we mentioned it, but now we are actually in a position to achieve this result easily, and actually it will turn out handy.

In fact, I have added now a "binary splitter" which allows representing 1 player through 2 players: 1 buy and 1 sell, of arbitrary sizes and prices (chosen by the manager). Since we already have the feature to further split a single buy or a single sell, this actually quite completes the set of tools we need for "logical manipulation" of the players.

Let's see a practical case, by continuing our TNA example above. (Later we will see an application on a complex real-world situation, where it actually makes sense and it is very useful to provide a more meaningful representation of our open players.)

So we apply our "binary splitter" to get a representation in terms of 1 sell and 1 buy (which can be later further split if we want to):

where the "previewer" text is saying:

Original player: CT_SELL 1 @ 71.36

Avg of new players: 71.36
Global signed position of new players: -1
Distance of new opposite player: 7.14 (10.00%)
Price range: [78.16, 78.50] 0.48%

CT_BUY 20 @ 78.50 (Reversing) Value: 1,569.92
CT_SELL 21 @ 78.16 (Reversing) Value: 1,641.28

and we get this new representation:

Note how after all these transformations nothing has essentially changed (certainly not the PNL, nor the average position of all players):

open players: Buy 20 @ 78.50, Sell 21 @ 78.16, Equiv: -1 @ 71.36

but, in general terms, we may get a representation which is far more useful to trade within a given price range.

An example of application might be case where our average has been thrown outside the typical trading range of the instrument: with this sort of conceptual alchemy we are able to somehow "recenter" the game in a more meaningful trading range. This is a pretty powerful conceptual tool which can come handy in several situations. In other words, nothing changes about the "physical orders" which have been executed: what changes is merely the "logical" representation of them, and therefore also the scheme which is eventually applied to recover the losing players.

In a next post, I will explain in some detail how the "binary splitter" works and how the 2 buy/sell players are obtained through this feature I have just added. We will see there are a couple of parameters (one about "sizing" and one about "spacing") which in practice give us infinite freedom in choosing the "representation" that we like and that is most useful for our purposes.

 

[fullautotrading]

Let's see now why and how this kind of "splitting" can be useful in general to "reorganize" the general logic of players.

Note that normally, when things are going smooth , you do not need usually much "reorganization": the opportunity arises, rather, when there is too much intervention of the fund manager, or large losses or, in general, we have somehow "lost track" of or got confused about the situation, and we need to globally reorganize the player logic.

Let's make a down-to-earth example. Let's consider our layer of ERX which for various reasons of poor management has accumulated a "monster" loss of almost 400K (I have loaded it temporarily on a "demo" acct, while the mkt is still closed):

We see here that the general situation and the monster losses summarized in 1 unique player is equivalent to a unique SELL order made at the very low price 11.71.

Now, if by absurd we should "wait" for ERX to reach that remote price to "recover" the entire loss in one shot (we have also said we don't want really trade the "vertical" moves), it's obvious that we would wait forever, as that is not going to happen, ever Note, in fact, that the historic chart of ERX is as follows (always above 46 in the last 2 years), and, in addition, it is the "inverse" of ERY, which has a strong drift downward (combined effect of futures contango and "daily rebalancing"):

So, how can we change the logic representation, so that at least "fractionally" we can still work on a recovery, assuming of course that, as an illustrative exercise, we want to insist with it (and not to move an equivalent investment on some other "correlated" instrument, for instance some futures, requiring less margin) ?

What we would need is somehow to "pull up" that player in a price "zone" which is at least tradable. In fact, we cannot possibly wait forever in the hope to make back the entire loss in one shot, but what we would like is to continue working gradually on it, so that, piece by piece, in "smaller bytes" , we can possibly make back, say in a fractional manner, some of the loss by continued scalping action.

In other words, we would like to change the "logic representation" of the stored information, putting it in a form which is more useful for a more gradual "recovery plan". Clearly, the "transformation" is not going to change anything about the "loss": what changes instead is the recovery logic. Since you don't want to sit there forever waiting for a complete recovery at an "impossible price", it is more meaningful to plan a possible "gradual", "fractional", recovery at more meaningful prices.

Now how do we then "pull up" that player ?

We have seen in the previous post that we could represent with a "lower" SELL 1 @ 71.36 a combination of buys/sell occurred at higher prices and with various size.

Now the situation is similar: we got a low sell and we want to transform it into 2 new player, 1 buy and 1 sell (which in case can be "split" further later) which are located at higher prices.

In other words, we need an equivalent transformation of this sell player into a pair of buy/sell players, in such a way that both the global position and global average remain unchanged. It is obvious that there exist infinite of such representations, however we just need one which makes sense to us and "pulls" the players up in a "workable" zone.

So what we can do, is define 2 parameters say:

R := size of the opposite-side player
d := distance of the opposite-side player

and work out the equivalent transformation.
Doing so for a SELL N @ P (where N := position, P := price), we obtain, as equivalent players:

BUY R @ (P + d)
SELL (N + R) @ ( P + d * R / (N+R) )

where, adjusting the parameters, R and d, we can locate the 2 new players suitably above the original sell player, in a zone which is at least in the trading range. In fact, note that the larger R and the larger d, the larger are the averages of the 2 new players.

Let's see this in action. For instance, by choosing: R = 4,000 and d=1,400% (there are infinite pairs we can choose from), we get:

Original player: CT_SELL 7,581 @ 11.71

Avg of new players: 11.71
Global signed position of new players: -7,581
Distance of new opposite player: 163.95 (1,400.00%)
Price range: [68.34, 175.66] 916.45%

CT_BUY 4,000 @ 175.66 (Reversing) Value: 702,651.31
CT_SELL 11,581 @ 68.34 (Reversing) Value: 791,431.30

obtaining the following (rather "extreme") "split":

and now, we can even further split these 2 players (by using the player splitter we already know on each player):

Focusing on the most recent price range, we see we have reduced that intractable position in something which can be worked on at the current price (well, one would need funds ):

This is of course just a random example, but I hope that the illustration is passing through: we can manipulate our trading information from a representation to an equivalent representation, in such a way that the new form may be more useful to our purposes.

As to the formulas for the splitting, they are not complicated and, in particular, for a sell:

for a buy:

(and you might verify that the global position and avg remain unchanged passing from a representation to another one).

 

[fullautotrading]

Yesterday was expiration day for the CL options. So I dropped a few puts at a far away strikes just to "steal away" a few thousands $$$ without risk (also because we have a substantial short position on the underlying). They have obviously expired OTM.

As usual, we add a "virtual fill" at price 0, comms 0, to "close" the position (adding this virtual position has the purpose to make the PNL computations precise to the penny (because a 0 price is never actually received); we have seen that before) and then remove the layer from the "trading monitor". Here is an example:

so we are perfectly aligned with IB trading report:

Then we can remove this layer, as we did for all the other profitable ones.

In the meantime we continue scalping to recover our large DD.
On the development side, I have been refining some "fundamental" concepts, and removed the notion of "player adjustment" (which was used, for instance, in rollovers or to "incorporate" the information about the closed scalps) , to replace it with the more powerful mechanisms provided by player "grouping" and "splitting". In a next post I will explain more in detail why this is necessary and how it works, with concrete applications.

 

[fullautotrading]

Oil (CL) moved a bit overnight, thus recovering some DD. ERX will probably open with a gap down today. I have moved practically all the funds on CL, since working with ERX there would not be enough funds for a recovery, and we would be experiencing the "short blanket" effect again (we still have the large loss essentially originating from the initial disproportionate ERY position), as ERX takes more resources than the futures. In the meantime, I am doing a partial recovery attempt on ERX using 2 opposite layers to scalp away as much horizontal component as possible. Our DD is still not looking good: about -40%.

Layers with "profits" have been "removed" from our trading monitor, as we don't need the relevant information, but it is instead crucial to carefully retain the information about the losing layers, because that is what can enable us to perform a precise recovery action just where it is needed. and gives us exactly the sense of where we are and how to plan our actions. In particular, the "removed" layers are the following 108 layers, totaling about 204K (scalped away in about 65 days, so far) :

This is somehow just the "opposite "of what the majority of people naturally do when trading: they (want to ) "forget the losses" (after all, stops are just a means of destroying information and removing, even at a psychological level, about the past losses. But we don't want that.)

CL FOP 201502 51 C NYMEX 1000            PNL: 700.76, comms: 9.24, fills: 4, Pos: 0
CL FOP 201502 52 C NYMEX 1000            PNL: 427.66, comms: 32.34, fills: 8, Pos: 0
CL FOP 201502 53 C NYMEX 1000            PNL: 293.74, comms: 106.26, fills: 16, Pos: 0
CL FOP 201502 54 C NYMEX 1000            PNL: 659.94, comms: 60.06, fills: 7, Pos: 0
CL FOP 201502 56 C NYMEX 1000            PNL: 696.84, comms: 83.16, fills: 13, Pos: 0
CL FOP 201502 60 C NYMEX 1000            PNL: 306.14, comms: 13.86, fills: 3, Pos: 0
CL FOP 201502 61 C NYMEX 1000            PNL: 736.90, comms: 23.10, fills: 4, Pos: 0
CL FOP 201502 62 C NYMEX 1000            PNL: 76.14, comms: 13.86, fills: 2, Pos: 0
CL FOP 20150217 47 C NYMEX 1000          PNL: 221.52, comms: 18.48, fills: 7, Pos: 0
CL FOP 20150217 60 C NYMEX 1000          PNL: 102.50, comms: 577.50, fills: 42, Pos: 0
CL FOP 20150217 63 C NYMEX 1000          PNL: 5.38, comms: 4.62, fills: 2, Pos: 0
CL FOP 20150217 65 C NYMEX 1000          PNL: 21.52, comms: 18.48, fills: 3, Pos: 0
CL FOP 20150217 67 C NYMEX 1000          PNL: 10.76, comms: 9.24, fills: 2, Pos: 0
CL FOP 201503 48 C NYMEX 1000            PNL: 1,166.84, comms: 83.16, fills: 32, Pos: 0
CL FOP 201503 49 C NYMEX 1000            PNL: 1,616.90, comms: 23.10, fills: 8, Pos: 0
CL FOP 201503 54 C NYMEX 1000            PNL: 109.18, comms: 50.82, fills: 14, Pos: 0
CL FOP 201503 55 C NYMEX 1000            PNL: 107.66, comms: 32.34, fills: 11, Pos: 0
CL FOP 201503 58 C NYMEX 1000            PNL: 109.15, comms: 80.85, fills: 9, Pos: 0
CL FOP 201503 60 C NYMEX 1000            PNL: 815.23, comms: 154.77, fills: 43, Pos: 0
CL FOP 201506 49 C NYMEX 1000            PNL: 1,643.04, comms: 36.96, fills: 16, Pos: 0
CL FOP 201506 54 C NYMEX 1000            PNL: 113.80, comms: 46.20, fills: 13, Pos: 0
CL FOP 201502 46 P NYMEX 1000            PNL: 176.90, comms: 23.10, fills: 2, Pos: 0
CL FOP 201502 49 P NYMEX 1000            PNL: 502.25, comms: 57.75, fills: 14, Pos: 0
CL FOP 201502 51 P NYMEX 1000            PNL: 499.97, comms: 30.03, fills: 4, Pos: 0
CL FOP 201502 52 P NYMEX 1000            PNL: 430.76, comms: 9.24, fills: 2, Pos: 0
CL FOP 201502 55 P NYMEX 1000            PNL: 310.76, comms: 9.24, fills: 2, Pos: 0
CL FOP 201502 57 P NYMEX 1000            PNL: 1,030.76, comms: 9.24, fills: 2, Pos: 0
CL FOP 201503 40 P NYMEX 1000            PNL: 16.14, comms: 13.86, fills: 6, Pos: 0
CL FOP 201503 41 P NYMEX 1000            PNL: 10.76, comms: 9.24, fills: 4, Pos: 0
CL FOP 201503 42 P NYMEX 1000            PNL: 148.42, comms: 41.58, fills: 11, Pos: 0
CL FOP 201503 44 P NYMEX 1000            PNL: 141.34, comms: 198.66, fills: 60, Pos: 0
CL FOP 201503 45 P NYMEX 1000            PNL: 59.06, comms: 170.94, fills: 56, Pos: 0
CL FOP 201503 46 P NYMEX 1000            PNL: 939.00, comms: 231.00, fills: 63, Pos: 0
CL FOP 201503 47 P NYMEX 1000            PNL: 1,859.88, comms: 120.12, fills: 40, Pos: 0
CL FOP 201503 48 P NYMEX 1000            PNL: 2,215.08, comms: 304.92, fills: 64, Pos: 0
CL FOP 201503 48 P NYMEX 1000_L1         PNL: 553.80, comms: 46.20, fills: 13, Pos: 0
CL FOP 201503 49 P NYMEX 1000            PNL: 259.88, comms: 120.12, fills: 33, Pos: 0
CL FOP 201503 51 P NYMEX 1000            PNL: 693.04, comms: 36.96, fills: 10, Pos: 0
CL FOP 201503 54 P NYMEX 1000            PNL: 8,670.58, comms: 189.42, fills: 49, Pos: 0
CL FOP 201503 55 P NYMEX 1000            PNL: 1,151.46, comms: 78.54, fills: 22, Pos: 0
CL FOP 201503 56 P NYMEX 1000            PNL: 1,525.32, comms: 64.68, fills: 15, Pos: 0
CL FOP 201503 57 P NYMEX 1000            PNL: 498.42, comms: 41.58, fills: 11, Pos: 0
CL FOP 201503 58 P NYMEX 1000            PNL: 815.32, comms: 64.68, fills: 24, Pos: 0
CL FOP 201503 60 P NYMEX 1000            PNL: 1,388.42, comms: 41.58, fills: 5, Pos: 0
CL FOP 201503 61 P NYMEX 1000            PNL: 4,626.90, comms: 23.10, fills: 4, Pos: 0
CL FOP 201504 56 P NYMEX 1000            PNL: 2,453.80, comms: 46.20, fills: 2, Pos: 0
CL FOP 201502 50 P NYMEX 1000            PNL: 342.28, comms: 27.72, fills: 4, Pos: 0
ERY OPT 20150116 18 P SMART 100          PNL: 341.52, comms: 158.48, fills: 4, Pos: 0
ERY OPT 20150116 20 P SMART 100          PNL: 36.48, comms: 3.52, fills: 2, Pos: 0
ERX STK SMART                            PNL: 2,897.87, comms: 451.11, fills: 365, Pos: 0
ERX STK SMART_L2                         PNL: 9,359.62, comms: 418.41, fills: 91, Pos: 0
ERY STK SMART_USD_L1                     PNL: 334.13, comms: 68.33, fills: 9, Pos: 0
ERY STK SMART_USD                        PNL: 3,164.86, comms: 2,979.85, fills: 709, Pos: 0
CL FUT 201512 NYMEX 1000                 PNL: 8,038.80, comms: 1,201.20, fills: 402, Pos: 0
ERY OPT 20150116 20 C SMART 100          PNL: 2,637.54, comms: 262.46, fills: 4, Pos: 0
ERY OPT 20150116 25 P SMART 100          PNL: 576.18, comms: 23.82, fills: 2, Pos: 0
ERY OPT 20150116 30 P SMART 100          PNL: 2,920.15, comms: 79.85, fills: 2, Pos: 0

continues...

 

[fullautotrading]

... other removed layers:

ERY OPT 20150417 25 P SMART 100          PNL: 1,930.15, comms: 69.85, fills: 2, Pos: 0
ERY OPT 20150116 21 C SMART 100          PNL: 13,529.79, comms: 160.21, fills: 9, Pos: 0
ERY OPT 20150116 24 C SMART 100          PNL: 5,293.60, comms: 56.40, fills: 5, Pos: 0
ERY OPT 20150116 25 C SMART 100          PNL: 2,899.64, comms: 25.36, fills: 2, Pos: 0
ERY OPT 20150116 27 C SMART 100          PNL: 2,708.61, comms: 246.39, fills: 9, Pos: 0
ERY OPT 20150417 25 C SMART 100          PNL: 12,027.15, comms: 142.85, fills: 3, Pos: 0
CL FUT 201512 NYMEX 1000_L2              PNL: 1,272.28, comms: 27.72, fills: 8, Pos: 0
CL FUT 201512 NYMEX 1000_L3              PNL: 1,389.94, comms: 60.06, fills: 13, Pos: 0
CL FUT 201512 NYMEX 1000_L4              PNL: 1,278.42, comms: 41.58, fills: 7, Pos: 0
CL FUT 201512 NYMEX 1000_L5              PNL: 2,658.42, comms: 41.58, fills: 10, Pos: 0
CL FUT 201512 NYMEX 1000_L6              PNL: 4,399.94, comms: 60.06, fills: 14, Pos: 0
CL FOP 201506 55 C NYMEX 1000            PNL: 133.04, comms: 36.96, fills: 16, Pos: 0
CL FOP 201506 53 C NYMEX 1000            PNL: 1,801.52, comms: 18.48, fills: 5, Pos: 0
CL FUT 201512 NYMEX 1000_L2              PNL: 653.01, comms: 66.99, fills: 17, Pos: 0
CL FOP 201506 55 C NYMEX 1000            PNL: 0.76, comms: 9.24, fills: 4, Pos: 0
CL FOP 201506 57 C NYMEX 1000            PNL: 411.52, comms: 18.48, fills: 6, Pos: 0
ERX STK SMART_L3                         PNL: 25,134.52, comms: 204.33, fills: 51, Pos: 0
CL FUT 201512 NYMEX 1000_L2              PNL: 1,158.42, comms: 41.58, fills: 11, Pos: 0
CL FOP 201506 59 C NYMEX 1000            PNL: 203.04, comms: 36.96, fills: 13, Pos: 0
CL FOP 201506 55 C NYMEX 1000            PNL: 201.52, comms: 18.48, fills: 7, Pos: 0
CL FOP 201506 58 C NYMEX 1000            PNL: 1,692.80, comms: 277.20, fills: 32, Pos: 0
CL FOP 201506 56 C NYMEX 1000            PNL: 276.90, comms: 23.10, fills: 7, Pos: 0
CL FOP 201503 45 P NYMEX 1000            PNL: 7,512.98, comms: 97.02, fills: 5, Pos: 0
CL FOP 201503 44 P NYMEX 1000            PNL: 6,255.20, comms: 184.80, fills: 7, Pos: 0
CL FOP 201503 43 P NYMEX 1000            PNL: 4,907.60, comms: 92.40, fills: 2, Pos: 0
CL FOP 201503 42 P NYMEX 1000            PNL: 3,707.60, comms: 92.40, fills: 2, Pos: 0
CL FOP 201506 58 C NYMEX 1000            PNL: 75.38, comms: 4.62, fills: 2, Pos: 0
CL FUT 201503 NYMEX 1000                 PNL: 746.90, comms: 23.10, fills: 7, Pos: 0
CL FOP 201503 48 P NYMEX 1000            PNL: 254.50, comms: 115.50, fills: 8, Pos: 0
CL FUT 201503 NYMEX 1000_L1              PNL: 1,230.76, comms: 9.24, fills: 2, Pos: 0
CL FOP 20150617 57 C NYMEX 1000          PNL: 362.28, comms: 27.72, fills: 4, Pos: 0
CL FOP 201503 41 P NYMEX 1000            PNL: 1,761.40, comms: 138.60, fills: 4, Pos: 0
CL FUT 201503 NYMEX 1000_L2              PNL: 780.76, comms: 9.24, fills: 4, Pos: 0
CL FOP 20150217 43 P NYMEX 1000          PNL: 120.76, comms: 9.24, fills: 3, Pos: 0
CL FOP 20150217 44 P NYMEX 1000          PNL: 85.38, comms: 4.62, fills: 2, Pos: 0
CL FOP 20150217 45 P NYMEX 1000          PNL: 864.38, comms: 235.62, fills: 8, Pos: 0
CL FOP 201503 50 P NYMEX 1000            PNL: 186.08, comms: 73.92, fills: 23, Pos: 0
CL FOP 201503 41 P NYMEX 1000            PNL: 353.80, comms: 46.20, fills: 2, Pos: 0
CL FOP 201503 42 P NYMEX 1000            PNL: 1,415.20, comms: 184.80, fills: 3, Pos: 0
CL FOP 20150617 57 P NYMEX 1000          PNL: 153.04, comms: 36.96, fills: 9, Pos: 0
CL FOP 20150217 48 P NYMEX 1000          PNL: 1,061.40, comms: 138.60, fills: 2, Pos: 0
CL FOP 20150217 50 P NYMEX 1000          PNL: 179.85, comms: 150.15, fills: 7, Pos: 0
CL FOP 20150217 47 P NYMEX 1000          PNL: 1,815.20, comms: 184.80, fills: 4, Pos: 0
CL FOP 20150217 46 P NYMEX 1000          PNL: 902.01, comms: 297.99, fills: 6, Pos: 0
CL FOP 20150217 45 P NYMEX 1000          PNL: 679.15, comms: 80.85, fills: 5, Pos: 0
CL FOP 20150217 50 P NYMEX 1000          PNL: 1,961.40, comms: 138.60, fills: 3, Pos: 0
ERX STK SMART_L4                         PNL: 12,408.08, comms: 2,652.64, fills: 476, Pos: 0
CL FUT 201504 NYMEX 1000                 PNL: 607.60, comms: 92.40, fills: 15, Pos: 0
CL FUT 201505 NYMEX 1000                 PNL: 1,255.32, comms: 64.68, fills: 10, Pos: 0
CL FUT 201504 NYMEX 1000                 PNL: 40.76, comms: 9.24, fills: 4, Pos: 0
CL FUT 201504 NYMEX 1000                 PNL: 56.90, comms: 23.10, fills: 9, Pos: 0

Count:    108
Sum:      204365.71
Avg:      1892.28
Median:   698.8
Std:      3450.17

In order to roll over the trading information for expired options or futures, we "transfer" the players to the new instruments and then we can proceed by "regrouping" them.

In this sense, the rollover procedure, in our context takes the meaning of an information transfer from a price reference system to another one. And, in principle, it can be applied to any instrument with target to any other one. For instance, we could transfer the players from a stock to another one (eg., even mixing AAPLs with "ORANGEs" ). A special case of that if the "forward" or "reverse split". Using the rollover of the information we can just move our players to the new "split version" of a stk. Clearly, we just add appropriate "virtual fills" to close and reopen the position on the different layers, and price levels, and then we can "regroup" them so that we can start playing on the new price reference system on the "new" instrument.

(The only "limitation" I can see at the moment is that the original instrument and the target must have the same multiplier (we could probably get around that technically, but at the moment I don't see a real point, for practical applications), for the rest, in principle, we can make transfer of information (and relevant funds) with a great deal of freedom.)

In a next post we will see a practical example of that.

 

[fullautotrading]

Oil did not move much so far and we are still in significant DD (-47%), but incessantly scalping away the due daily profit

The "removed" profitable layers amount now to (result of 81 days) about 220K :

*** PNLs in removed layers ***

Count:    134
Sum:      219,147.78
Avg:      1635.43
Median:   563.42
Std:      3173.57

We need to recover ERX, which has been beaten severely due to the problems with ERY. As to CL we have a good load on it and scalping intensively.

In the meantime, I have been revising the "rollover" facility so that now we can "transfer" the information to any other instrument at will, even if it has already position of its own (for now, with the technical constraint that the multiplier be the same: later, we will see if we can "relax" this, provided it makes sense).

This is very helpful because it allows an easy "transfer" of the trading information about losses even across different instruments and this can be very handy.

To make a concrete example, let's consider for instance this loss I have on this (expired on Feb 20) layer of CL (this is the "player-view" mode):

We have a "loss" (or, better, an "investment" ) of -10K on this layer.

Now, let's say I want to transfer this information on a "tradable" layer: for instance this one which I have been scalping just today "semiautomatically" (that is under my strict "supervision"):

(had some free time ). This is the "newer" contract expiring on March 20 (in "order-view" mode).

After the transfer, or "merge", we get:

while the old layer is removed (in this case the PNL info is not stored, as it happens for the layers removed by the user, because it is merged on an active layer).

So, all the past orders have been "transferred" (or "merged", if you prefer) on this new layer (carrying open players, PNL, commissions, etc.) so that we can continue tracking precisely the "investment" and therefore have the technical means to recover it. The orders of the previous instrument could be made "visible" on the screen by pushing back the time origin for this layer (which in the above picture is just 19 hours away. A dashed vertical blue line will signal where the information "merge" (a sort of "roll over" in this case) has been done.)

In fact, we could say, that a currently negative "investment" becomes actually a "loss" when you "forget" the relevant information (such as in the usual "stop loss" approach)
Keep in mind the above sentence, and think a bit about it.

[ Incidentally, I notice that it's relatively easier to "steal" a nice profit from the CL futures with the new features I have been adding and refining for "semi automated" scalping (in this case you want to choose the expiry with smallest spread). Also, very effective for the CL options. Not the same I can say for the ETFs, where I do not seem able to grab a profit as easily. The reason I think, is essentially the data microstructure, which for CL seems a lot more favorable for this kind of activity. I have tried to do the same for ERX, but with scarce success. Anyway, will keep trying. ]

 

[fullautotrading]

Sorry for some days of silence I have been occupied with various problems including a couple days with a nasty flu (a gift from my little son ) and, icing on the cake, internet outage, and corruption of session file and backup. Only yesterday (friday) I could finally resume trading.

I will be spending some time to clean up all the mess and restore everything back to normal to be ready to restart on when mkt re-opens, tomorrow. Apart the flu and the connectivity problems, which were fixed in a couple of days, the largest problem appears to be the corruption of the session file, that is the file which physically holds all the information about all the instruments being traded and their fills.

Actually, currently I do maintain multiple levels of backup, as follows:

- 1. current session file
- 2. previous saved copy of session file
- 3. copy of session file saved before restarting the current instance

Usually this seemed sufficient, as if there is for instance a power outage we can always resume from the session file, when the power is back and usually the worst thing that can happen is that the session file is unusable due to "truncation". Truncation may happen if the blackout occurred precisely at the very moment the file is being written.

Clearly, since I use laptops, a file "truncation" practically never happens due to the laptop batteries which keep it going.

This time, however, it seems Murphy got me and, I managed to corrupt not only 1) but even the previous copy 2) of the file. So what remained was copy 3 (saved before starting a new instance). This is better than nothing, of course, but there is a gap spanning from Fri 28 Feb 2015 5:27 edt (date of last write of the session file which has been open) to Thu 05 Mar 2015 07:20:54 edt (resume time).

This means that this session file is missing all the orders which were executed in the said period, and this causes the application and the PNL to go out of sync, respect to IB.

Let's see how I have solved this problem.

Clearly, one could have proceed by taking the orders from the IB trade reports and inserting them manually as "virtual fill". Clearly, while that is possible in principle, and it can be feasible for small gaps, in practice it would be too error prone and tedious to attempt it for a larger period. Further, we would never be sure we are perfectly in sync, at penny level, as we like to be.

So, what I did is to create a new feature which, for a given instrument, given 2 dates (a min date and a max date), and the "copy & paste" of the IB trade report, will automatically summarize and extract all the orders which we are missing and automatically insert them in the application as virtual fills. This seems quite practical and powerful and, in general, it is a very useful feature to have to deal with "disasters".

Let's see an example of recovery, step by step.

Let's start with one layer of CL, our preferred futures. Let's take this layer (CLJ5 contract):

First of all, we notice that there is a large gap in the tickdata. That is very simple to fix, as we already have a "backfiller" (we did this previously) which can fill any gap we indicate:

And that solves the issue with the visual appearance of the price curves (bid/ask).

Now, we need to take care of the position and PNL re-sync. First of all, we take a look in the "account info from IB" and the application tells us about what is out of sync.

We got:

============================================== NOT MATCHING PRESENT IN THIS INSTANCE ===============================================
Symb                                                                       This G-BOT             IB            Exp     Sync
CL                                 [CLZ5]                                         -17            -22    258.20 days     SELL 5 @ XXX
ERX                                [ERX]                                        -1331           -431              -     BUY 900 @ XXX
CL                                 [CLJ5]                                          -5            -24     13.20 days     SELL 19 @ XXX
CL                                 [LOM5 C5500]                                    -1             -2     68.20 days     SELL 1 @ XXX
CL                                 [LOM5 C5700]                                    -1             -5     68.20 days     SELL 4 @ XXX
CL                                 [LOM5 C5800]                                    -4             -8     68.20 days     SELL 4 @ XXX
ERY                                [ERY]                                        -4100          -4910              -     SELL 810 @ XXX
------------------------------------------------------------------------------------------------------------------------------------
Symb                                                                       This G-BOT             IB            Exp     Sync
CL                                 [LOM5 P4500]                                   -23              0     68.20 days     BUY 23 @ XXX
CL                                 [LOM5 P4400]                                   -24              0     68.20 days     BUY 24 @ XXX
CL                                 [LOM5 P4300]                                   -27              0     68.20 days     BUY 27 @ XXX
CL                                 [LOM5 P4200]                                   -26              0     68.20 days     BUY 26 @ XXX
------------------------------------------------------------------------------------------------------------------------------------

and in particular, for our instrument under examination CLJ5, we have:

CL                                 [CLJ5]                                          -5            -24     13.20 days     SELL 19 @ XXX

So there is a difference of 19 contracts between what we have stored in the application and what we have in our "physical" account.

Now, in order to re-sync the situation, we just take all the trades from the IB report (clearly, the report must include at least the days we are missing):

and copy paste in the dialog of the new feature I have just added for this kind of "bulk reconstruction":

the facility is telling us that it has found 135 orders in this period, algebraically totaling to -19. So we can proceed with a bulk insertion of those fills, by using directly the data from the report, obtaining the following nice perfect reconstruction:

(Of course now we can also transform all the virtual fills into players if we want to.)
Clearly, inserting those 135 orders one by one would have been a bit unpractical. Not to mention that we have other instruments to fix.

Let's now move to the next layer: CL FUT 201512 [ CLZ5 ]. In this case we are 5 out of sync. Running the reconstruction facility we find 74 orders (total signed position = -5). This fixes it.

Let's now move to ERX, which is 900 out of sync. In this case we found 53 orders, and signed sum of fills: 900:

Similarly we now do ERY. Then, we do all the mismatching options:

Symbol: CL JUN15 55.0 C
Orders found for symbol:29
Falling within the period: 13
Total signed qty: -1

Symbol: CL JUN15 57.0 C
Orders found for symbol:39
Falling within the period: 16
Total signed qty: -4

Symbol: CL JUN15 58.0 C
Orders found for symbol:26
Falling within the period: 11
Total signed qty: -4

Symbol: CL JUN15 42.0 P
Orders found for symbol:18
Falling within the period: 5
Total signed qty: 26

Symbol: CL JUN15 43.0 P
Orders found for symbol:19
Falling within the period: 5
Total signed qty: 27

Symbol: CL JUN15 44.0 P
Orders found for symbol:15
Falling within the period: 3
Total signed qty: 24

Symbol: CL JUN15 45.0 P
Orders found for symbol:13
Falling within the period: 3
Total signed qty: 23

and, finally, with little work, we achieve a perfect match:

============================================== NOT MATCHING PRESENT IN THIS INSTANCE ===============================================
None

------------------------------------------------------------------------------------------------------------------------------------

and we are ready to resume the battle tomorrow night.

So we have anyway managed to find a way to get something very useful in general, from relatively negative occurrences. It does not matter how hard we are hit, what matters is to always come back with a bigger hammer

(And btw I have also decided to increase the number of backup copies and revise the overall backup mechanism)

 

[fullautotrading]

Clearly, when we do a data "reconstruction" like this, we also need to check the instruments which do not "appear" to be "out of sync" based on the position only. In fact, we could anyway had, in the given period, a series of orders with signed total =0, and therefore, while position is not affected, the actual PNL would be.

In fact, it turns out, checking the other instruments in the report, that we have 3 options (CL JUN15 46/47/48 P ) which had orders totaling 0 in the given period:

SELL 1 @ 1.9800 (comms: -2.31) Mon 02 Mar 2015 08:45:29 +CL JUN15 46.0 P
SELL 1 @ 1.8500 (comms: -2.31) Mon 02 Mar 2015 12:06:06 +CL JUN15 46.0 P
BUY 1 @ 1.9000 (comms: -2.31) Tue 03 Mar 2015 10:01:02 +CL JUN15 46.0 P
SELL 1 @ 1.8100 (comms: -2.31) Tue 03 Mar 2015 11:27:02 +CL JUN15 46.0 P
SELL 1 @ 1.7200 (comms: -2.31) Tue 03 Mar 2015 13:20:53 +CL JUN15 46.0 P
SELL 1 @ 1.7300 (comms: -2.31) Tue 03 Mar 2015 14:27:56 +CL JUN15 46.0 P
BUY 4 @ 1.7000 (comms: -9.24) Wed 04 Mar 2015 12:28:43 -CL JUN15 46.0 P

Symbol: CL JUN15 46.0 P
Orders found for symbol:12
Falling within the period: 7
Total signed qty: 0

SELL 1 @ 2.1200 (comms: -2.31) Mon 02 Mar 2015 02:18:17 +CL JUN15 47.0 P
SELL 1 @ 1.9400 (comms: -2.31) Mon 02 Mar 2015 11:03:52 +CL JUN15 47.0 P
SELL 1 @ 2.0300 (comms: -2.31) Mon 02 Mar 2015 11:58:45 +CL JUN15 47.0 P
SELL 1 @ 2.0500 (comms: -2.31) Mon 02 Mar 2015 12:33:03 +CL JUN15 47.0 P
SELL 1 @ 2.2200 (comms: -2.31) Mon 02 Mar 2015 16:09:27 +CL JUN15 47.0 P
BUY 1 @ 2.1700 (comms: -2.31) Tue 03 Mar 2015 09:56:13 +CL JUN15 47.0 P
SELL 1 @ 2.0400 (comms: -2.31) Tue 03 Mar 2015 11:27:09 +CL JUN15 47.0 P
BUY 5 @ 1.9800 (comms: -11.55) Tue 03 Mar 2015 11:45:58 +CL JUN15 47.0 P

Symbol: CL JUN15 47.0 P
Orders found for symbol:13
Falling within the period: 8
Total signed qty: 0

SELL 1 @ 2.4500 (comms: -2.31) Mon 02 Mar 2015 02:40:03 +CL JUN15 48.0 P
SELL 1 @ 2.3400 (comms: -2.31) Mon 02 Mar 2015 10:37:53 +CL JUN15 48.0 P
SELL 1 @ 2.1900 (comms: -2.31) Mon 02 Mar 2015 11:03:58 +CL JUN15 48.0 P
BUY 3 @ 2.2200 (comms: -6.93) Mon 02 Mar 2015 11:10:09 +CL JUN15 48.0 P

Symbol: CL JUN15 48.0 P
Orders found for symbol:7
Falling within the period: 4
Total signed qty: 0

So we need to add those orders to be in perfect sync again.

 

[fullautotrading]

As promised I am also restructuring and reinforcing the session file "backup" logic, so hopefully we won't need to make these kind of reconstructions ever again.

I will explain here with some detail the changes I am doing, also giving the general logic in form of simplified code (I like to use C#, however here I am including a vb.net version which should be readable for a more general audience: do suggest any improvement. You can copy paste it in any online conversion tool if you need to modify it.)

So let's see what is the need. As we know this application allows restarting trading from any machine and any side of the world on a given account or multiple account. It also allows to "superpose" at will multiple instances on the same or multiple accounts. This is possible because all the information is stored and all the computations are done independently by the application. Clearly, we also have all the means to check our state of synchronization with the account(s) traded.

Now all the trading information is stored in the so called session file. Clearly, it is fundamental that we do not damage this file, or else, when we close (or interrupt by an accident) the application, then we are not able to restart just were we left it. Well, we can any any case, as we can now take the information directly from the official trade report, but of course, in general we want to avoid such occurrence.

Now, since the session file is so important for all our activities, I have set in place various mechanisms to ensure that we do not lose this central piece of information. In particular, this is the general working of the new backup mechanism:

I keep the "original" + 3 backup levels. Every time the last version of the session file is written to disk (which happens periodically or after significant events, like fills), the current version becomes the backup #1, the backup #1 becomes backup #2, and backup #2 becomes backup #3, and so on.

So we have a sort of "chain" or "cascade" backup like this:

 current session file -> Previous Version 1 -> Previous Version 2 -> Previous Version 3

(where the session file name is session-specific, and created as a combination of the account number, start date, port number and client number)

This way we should really be even beyond any reach of Murphy

In addition to this, I have another backup mechanism which consists in saving the last session file found at session resume and keeping untouched the backup during the session. This is actually a very useful mechanism in practice and it is, actually, the one which saved our data this time.

Here again, I am implementing multiple levels of backup. So, when we resume a session, if there is already a session file, it is saved according to the following chain:

current session file -> BeforeRestart1 -> BeforeRestart2 -> BeforeRestart3

Let's see now how this works in practice, and then I will also give a snippet, so that you can provide also some practical suggestions.
Assume for instance, that I am starting a session pointing to port 9999 and using client 1 (I have momentarily setup the TWS in demo mode to listen on port 9999 just for this example).

Once we start, the application automatically creates a folder in Documents, and place in that folder all the files of the running session. The folder name as well as the session file are a combination of: Account Name + Session start date + Port Number + Client number:

In the folder the application saves the session file and possible other accessory files (for instance the log file, etc.):

in a while the backup files also appear, and they keep being overwritten according to the " circular scheme" we have just described:

In addition to these backups, we have also the backups relative to the session resume and these backups appear when we resume a session. So, for instance let's interrupt this session and resume it, by reloading the session file:

This time, if we start and go look in the session folder we find a new backup file, which will not be touched during this entire run (and for the next 2 runs, actually):

Lastly I am still undecided whether for the backup file I should use a different extension, to avoid any possibility of confusion on session resume, since the need to use a backup should be relatively "rare"...

Yes, I will definitely change that: in fact, if by distraction, one should load a "backup session file" on resume, instead of the current session file, practically he would self-defy the purpose of having backups (and I think this actually happened in the past in at least one case). I will use something like "old" or... whatever better idea you have:

This way there no possibility of mix ups for the manager, since the .ses extension will be filtered by the "open file dialog" for the session file.

Here is a snippet of the basic idea for the functionality: let me know any ideas for improvements (this is just the general idea of the mechanism: in actuality the saving is done in a background process to avoid interfering with the trading activity, and surrounded by the appropriate exception handling constructs):

Private SessionSavedOnceAfterRestart As Boolean

Sub SaveSession()

        Dim CompleteNameBasedOnAccountPortAndClient As String = Me.FileSession_FullPath 'get the suggested name for the session file from the application

        Dim FilenamePortionOnly As String = Path.GetFileNameWithoutExtension(CompleteNameBasedOnAccountPortAndClient)
        Dim FolderOnly As String = Path.GetDirectoryName(CompleteNameBasedOnAccountPortAndClient)


        Dim NameOfCurrentSession As String = Path.Combine(FolderOnly, FilenamePortionOnly & ".Current.ses")

        Dim NamesForCurrentVersionBackups As String() = New String() {NameOfCurrentSession, _
                                                                      Path.Combine(FolderOnly, FilenamePortionOnly & ".Previous1.old"), _
                                                                      Path.Combine(FolderOnly, FilenamePortionOnly & ".Previous2.old"), _
                                                                      Path.Combine(FolderOnly, FilenamePortionOnly & ".Previous3.old")}

        Dim NamesForBeforeStartBackups As String() = New String() {NameOfCurrentSession, _
                                                                   Path.Combine(FolderOnly, FilenamePortionOnly & ".BeforeRestart1.old"), _
                                                                   Path.Combine(FolderOnly, FilenamePortionOnly & ".BeforeRestart2.old"), _
                                                                   Path.Combine(FolderOnly, FilenamePortionOnly & ".BeforeRestart3.old")}

        'First save on resume

        If Not SessionSavedOnceAfterRestart Then

            If File.Exists(NameOfCurrentSession) Then
                'chain backup
                For i As Integer = NamesForBeforeStartBackups.Count - 1 To 2 Step -1
                    If File.Exists(NamesForBeforeStartBackups(i)) Then File.Delete(NamesForBeforeStartBackups(i))
                    If File.Exists(NamesForBeforeStartBackups(i - 1)) Then File.Move(NamesForBeforeStartBackups(i - 1), NamesForBeforeStartBackups(i))
                Next
                File.Copy(NameOfCurrentSession, NamesForBeforeStartBackups(1))   'session file is copied only, not moved in this case
            Else
                'very first time: skip
            End If

            SessionSavedOnceAfterRestart = True
        End If


        'Ordinary save of session file

        'cascade backup
        For i As Integer = NamesForCurrentVersionBackups.Count - 1 To 1 Step -1
            If File.Exists(NamesForCurrentVersionBackups(i)) Then File.Delete(NamesForCurrentVersionBackups(i))
            If File.Exists(NamesForCurrentVersionBackups(i - 1)) Then File.Move(NamesForCurrentVersionBackups(i - 1), NamesForCurrentVersionBackups(i))
        Next

        'Save
        Using FileStream As New FileStream(NameOfCurrentSession, FileMode.Create)
            Dim b As New BinaryFormatter
            b.Serialize(FileStream, Me.Session)
        End Using

    End Sub

 

[fullautotrading]

We are now perfectly realigned with the IB account and well prepared for the next battles

As a brief summary of where we are, we have been suffering a significant DD, caused by a too large position initially taken on ERY options (initial "mistake"). Later we have recovered almost completely that instrument but at the expenses of ERX which, in turn, caused an even larger DD, beyond 50%.

Having noticed that the "blanket" was too short to make the recovery game with ETFs, we have switched most of the funds to CL. By using CL, I have been able to recover some good part of the DD, which is now around -33%:

I have been removing most of the layers which were positive, in order to get them out of the way. And actually, the fact of removing the "profitable" layer is not just useful to free application memory and focus our attention, but somehow having a positive target to reach for each layer helps actually a lot in the trading activity too. Also, the fact that one makes some profits and somehow "archive" them while focusing on losses, is something that sounds good on the psychological side, as it provides a tangible sense of growing income, even when we are still in drawdown.

In particular, at this moment the summary of "removed" layers (which include a lot of CL options) is as follows (93 days trading):

*** PNLs in removed layers ***

Count:    144
Sum:      212,657.51
Avg:      1476.79
Median:   527.65
Std:      3244.83

So, while we have been initially draw down significantly due to the initial error with ERY options (then propagated to ERX), in the meantime we have also managed to grab some 212K (mostly thanks to CL) in the last 93 days.

We have been paying almost 4K of short interest (not a small amount actually) and about 42K in commissions (looks like the broker is the only one getting rich so far ). Here is a piece (the NAV section) of the trade report from IB:

(once I am back in the green, I will link the entire report with details for all trades, which are now around 9,800.)

ERX, is practically where we left it (-400K). CL appears to be a good choice for several reason which we can list as follows:

1. ETFs take too much margin and was practically impossible to recover the loss of an ETF with another ETF ("short blanket effect"). Expensive as to commissions. Expensive as to short interests.

2. When we compare ERX to CL, while there is obviously correlation between the 2 instruments, it appears clear quite soon that CL has a tickdata microstructure which makes it much more favorable and much easier to use for the scalping/hedging activity. Also the CL options are good for the same purpose. The same thing cannot be said for ERX. [Btw, it seems that it's better to use the most recent CL contract for trading (I have been using the December contract, but it appears that for various reasons it's better to use the most recent one, or the one with smaller spread, in any case.) ]

So in essence, it's just thanks to a future like CL and some decent starting capital, that we can still fight to recover the initial nasty DD, or else at this time we would be done.

So far my personal opinion, based on experiences so far, is that if one has the right $$$, just focus on futures, and limit the use the ETFs to a minor role. Avoid most of the ETFs options, as way too problematic (illiquid, serious spread, execution issues), and be very careful with the ultrashort ETFs (these pose a lot of technical and executions challenges, and commissions + short interests + nonshortable constraints etc. can be very irritating).

If in the future I find something suitable for relatively small traders, of course, I will let you know even though, frankly, it probably does not exist at all but for the moment my recommendation for traders with with relatively small funds is: do not trade (with real $$$. Do paper trade instead, which never hurts, and build up precious experience useful for the future times when "adequate" funding is available)

On the development side, we have instead been doing giant's steps, and I am pretty happy about all the new features and concepts we have been adding and which are very valuable especially when it comes to facing the most difficult situations.