Automated folio trading - Role of Information and Hedging

[fullautotrading]

Choice of parameters (size, distance)

Clearly, the size and the spacing of these "split players" has to be chosen in such a way that they will occupy a meaningful portion of the price range (for instance you would not let them to arrive at a "too small" price, or the may remain there forever ). I have added a dialog to preview these aspects and make a choice.

I think, anyway, it's important to have come out with this feature, because now it allows to take up blatantly any abnormal situation (even "imported" from some other discretionary system) and "break it down" in a form that is more manageable for the algorithmic engine to try engage in a non-predictive scalping/hedging action before it's too late.


Technical details and alternate distributions

It is obvious that, once we enter this idea of "player splitting", the splitting itself could be done in infinite ways, obviously under the constraint that the global position and the avg remain unchanged. Clearly, the most "naive" way is to just break it down into players which are "uniformly distributed" around the average (Avg) of the original player:

Avg + h * Distance, Avg + (h-1) * Distance, ... , Avg - (h-1) * Distance, Avg - h * Distance

(where h is the number of the "split players" divided 2).

Of course, one could conceive different distribution of the split players. For instance, increasing the distance and decreasing the size or vice versa. (One could even arrive to conceive splitting the original player in a combination of buy and sell players, but that is probably too much to take for now )

For simplicity let's stick with the naive "uniform distribution". In case, later we can refine this aspect. (This can actually be a useful coding exercise and homework for those willing to play with code: determine the prices of the split players which keep unchanged the original position and avg, but whose distance/position increases/decreases or decreases/increases linearly from top to bottom, according to a given factor f:

What I have implemented, for now, is the most intuitive case of equal distances and sizes (apart divisibility problems), the case with f=1, which should be enough to start playing with the concept, and see how it works out in practice.

 

[fullautotrading]

I have been "splitting" all the big players on all layers, and actually the overall behavior seems significantly improved. Having to deal with small and well spaced players makes the whole action much more fluid and allows a detailed control.

We are paying in terms of bad DD for the "disproportionate" (relative to the available funds) load we put on ERY. Further, we could have spared ourselves a big chunk of DD if we had been allowing the large position to scalp immediately: in fact it had previously reached 94K, before spiking up in red territory. But we did not have the splitter . So almost 100K vanished just for that.

Anyway, lesson for next time, and most importantly it has been crucial to focus on the need of a player splitting feature, about which I am pretty happy The introduction of this called also for more conceptual refinements, which I will explain later.

Now ERY has returned to around 19, and if we had more funds, we would have actually made a boatload of money riding the fall. Instead, the necessity of hedging with ERX and CL, in order to recover funds to avoid liquidation of ERY at high price has in practice caused to move all the "investment" (or loss, depending on he perspective) on ERX and CL, in an essentially opposite position.

As you see, it's not enough to "guess" and be right about the direction (the "directional expectation" here was clearly ERY going down), but it's way more complicate than that in practice, and with a directional view it's likely funds eventually never suffice to be after some instruments like those 3X, and, possibly, any instrument. So it's definitely better to abandon a "directional" view of the market, and instead embrace a non-predictive approach, which focuses on the "horizontal components" of price moves.

Today, I also got a small time window where the infamous CL PUT 54 appeared to be tradable in this Dxxx account, and even positive (8K), so I closed it immediately, to avoid further issues with it (yes, it will continue to fall, but I did not want to take more chances on it). Same thing for DGAZ, which I closed with a small profit, and in case we can reopen then "unfinished business" later on . We need now to focus on the layers which need to be recovered.

Essentially now we need to get back CL and ERX, and being focused on just those two is actually quite convenient. I am not worried about CL, which appears still quite manageable with a short load and bearable DD. More difficult is the situation of ERX, that is badly oversized (this was necessary to allow ERY to close some positions and discharge the load, or else there would not be sufficient funds to close it.)

Normally you don't want to be short on and ultralong like ERX, and, least of all, with a real "monster" position like this one.


What's next ?

So this is the strategy to recover the situation.

I will split all the players on ERX layer and gradually transfer most of the load on CL, and at the same time slowly reverse ERX. An additional layer for ERX will help hedging it with long players, and reversing the position.


About CL I will proceed as follows:

- CL futures: full-auto, short-position constraint
- CL PUTs: OTM, full-auto, with short-only players
- CL CALLs: open with strikes a bit OTM, strike and expiry such that price not too small, inject long players for protection, full-auto, with short only players. As soon as the option goes in the money (ITM), leave the layer alone in full-auto mode, with a take profit enabled.


Interestingly enough, so far I have closed practically all the CL options in profit, which suggests that they are quite manageable instruments, deserving more attention.

Note as all the trading information remains self contained in all layers, and not even 1 penny is lost track of. This allows us to have a very precise view of where the "losses/investments" are, and carry out all the actions and adjustment to eventually and methodically recover each and every losing player.

 

[fullautotrading]

Today CL retraced a bit, after the recent rally, and so did ERX. This time, having "discharged" ERY, we have been able to ride a bit the move with the short players: we are not going to do the same mistake 2 times

Instead of letting ERX overload the short position, as the s/h game would tend to do, as said in the previous post, I have been gradually dropping buy players on another ERX layer in order to ("realize" and) gradually invert the ERX position.
Instead, I have let CL to load up at will, using the funds freed by the buys on the ERX layer.

So, currently am both long and short on ERX on 2 different layers, which I plan to close both in profits (clearly, at different times), while I have a significant short position on CL.

Simultaneously, I have been hedging CL with some PUTs, and as soon as they went ITM, I let them to full auto mode in short-only mode, and often they close profit even though they served to free funds and protect (I closed practically all the CL options in profit, except the 1 which expired OTM on January 14, and that I promised to use it to show an example of "option rollover": we will do that once I successfully unload the current layers and the the PNL looks better ).

If the move continues, I would prefer reverse completely ERX (to remain long), while loading up CL with short position. If it does not, I need to hedge both CL and ERX with long layers.
The player "splitting" and these hedging techniques have improved the situation, and currently the DD is about 26%. Recovered over 70K today.

Dealing with CL is way more comfortable, as possible move overnight can be dealt with, so it's better to have a bigger position with futures. Also, it is way more efficient as to margin usage. Clearly, ERX and CL aren't the same thing, however do they have some positive correlation.

Note that I will now remove all profitable layers (mostly options) from the trading monitor so that we can focus on the layers which are "under" and in need of care. (All the results of removed layers will anyway be kept in memory, and the PNL will at any time reflect their contribution too.)

 

[fullautotrading]

Not much progress on the test side: we have still a significant DD (or "investment" ). Most damage has been made by ERX, which was needed to free some funds and allow ERY to close the huge position. But with a “short blanket” effect , we ended up with an even worse situation with ERX, and therefore we need to change approach to find a way out of this "catch 22". I will explain later how I plan to do that.

However, today was a good day because I implemented a new feature which actually opens up a whole new world of possibilities, and somehow represents the "conceptual leap" which was necessary to complete the picture of the player management.

We have started by naively describing the "players" with the analogy of "multiple traders" playing on the same account, and therefore having the possibility to superpose the "trades" (the so called "player-superposition").

So, at first sight, those players looked like a sort of simple programmatic entries "equipped" with the ability to close automatically based on rules specified by the fund manager (the "game rules".)

A couple of days ago I introduced the concept of player "splitting", as consequence of dealing with the "lump of money" problem. If you think a moment about it, this introduction suddenly put the players in a different light. In fact, since you can replace a player with, say, N different players which are functionally equivalent, suddenly it becomes evident that these players are much more than automated entries and exit, but they become actually a "logic layer" which traps the trading information and governs the "underlying" physical layer (the actual orders). Since for each player there exist infinite equivalent representations, it is evident that also there are infinite logic representations of the same underlying orders.

This realization opens up a whole "new world", because we can start manipulating this logical representation itself, of a given order cloud, to achieve forms and configurations which are more useful and easier to manage. And this represent a great empowerment for the fund manager.

There is however a final crucial mechanism which was missing to this purpose. We have added the possibility to"split" a player. What is fundamental now is the possibility to group the players together into one "unique player". You would be wondering, why on hearth would you want that, once we established that the ideal situation is to distribute a large number of small orders over the price range and scalp as much as possible the horizontal components of price moves ?

Well, the answer is: just for this precise purpose. Let me make a practical concrete example taken from our demo test, so everything will be clear.
Take for instance this ERX layer (in "player-view" mode). This layer has several SELL players (red squares) and BUY players (blue squares). Now this was created mostly to allow ERY to close positions, but this is irrelevant. What matters is that we know that if we could close all those players as the price moves up and down, we would end up with a profit. That is just "by definition", because the open players are those which are losing and the players which close in profit are removed from this "player-view" mode.

Now, if we consider those players all together, it is possible to work out a position and a price which "represents" them all with a unique instance. This price and position is actually printed on the screen at any time (see cyan rectangle). In this particular case, that line is saying:

Open players: Buy 27,941 @ 54.41, Sell 58,817 @ 47.10, Equiv: -30,876 @ 40.49

In other words, it is saying that there are 27,941 shares in buy players, 58,817 shares in sell players, and if we take the all together, they, combined, are equivalent to a unique entry of -30,876 @ 40.49. In fact -30,876 is just the difference 27,941 (buys) - 58,817 (sell), and the price is just the one which makes "equivalent" the position.

Now, once we start looking at the players like a logical representation of the "underlying orders", we could also make the conceptual final jump in this new way of looking at things and say: since this is just a logical representation, and it is one of infinite possible, why not to change it to make more manageable for a fund manager?

So the idea is, let's "group" all these players in the equivalent position ("player re-grouping") and , then, as we already know, let's split them again so we get a simpler and more efficient representation, much more useful, for instance, for hedging with pairs of layers. The new representation could be the splitting we have already seen ("player splitting").

So I could for instance replace the entire set of players with an "equivalent unique player" (this is the feature I just added today):

and finally split it again (the one "big fat player" is not shown in the picture below, but it was approximately in the middle of the red pile of "split players", just before the splitting, where the red line is), to obtain a completely new, more understandable, representation (note that in this case the buy players have disappeared completely because "incorporated" in the new sell players):

Note that since the "split representations" are all equivalent (in the sense that they keep unchanged the orders' average price ad position), I am now left again with the same task of closing all the new players, but the new situation looks much more clear and efficiently manageable. It also offers a deeper insight on the extent of the "damage" and the feasibility of the effort we need to go through to recover all the open players (for instance we might decide to stop and give up ). All the trading information is there, unchanged.

"Grouping" and "splitting" we can go back and forth through different representations, whenever we like, with no loss of information, and spreading the players on different price ranges and different order sizes, depending on how it is most useful. And this opens up a fascinating new world of hedging possibilities.

For instance, I can use this on the highly "devastated" ERX layer, to get a more meaningful representation of the situation and more effectively work to try "repair" it. (For instance, I could open another ERX layer and place on it an approximately equal but opposite position, split it, and let the layers play one against the other, so that I could scalp the horizontal moves, through the smaller players I have created, with almost no funds, as at any time I will be keeping approximately the position equal but opposite on the 2 layers, but I will allow the to freely scalp each side.)

In other words, we can create purely "logical representation" of the underlying "physical" reality, which help us a lot in our scalping/hedging activity. It's just logic, but it's the useful logic which allows us to look inside, and more easily manage, possibly complex situations.

 

[fullautotrading]

I have been working on the application to reflect the latest conceptual developments and to perfect the management of the players. Now it's possible to have a complete control on the players, switching back and forth between alternate "logic" representations. This is a powerful enhancement will come handy in the future. Later I will explain in detail the changes.

On the trading side not much, with the large DD on ERX being the main problem. I have been closing and removing almost all the positive layers, to get them "out of the way" and focus only on the problematic instruments.

Since ERX cannot be actually recovered in short time due to the "short blanket effect" that we have already experienced with ERY, I am planning to try the following way.
As to ERX, I will just use 2 opposite layers which will trade one against the other (2 layers of the "same" instrument is equivalent to a pair of perfectly correlated instruments), using few funds, to slowly try to recover what is possible, while for the "main investment" I will move the funds on CL, which, working on margin and 24/7, offers way more power for a recovery (or to definitely blow up ).

[ For the future, it may actually be worth to explore also the approach of trading pairs of layers (it could be the "same" or correlated instruments, with opposite "direction constraints", or inverse correlated instruments with same "direction constraints". Clearly, keeping the drifts and decays/contango into careful consideration, and not trading against them.) ]

CL will be "protected", when needed, with CALLs ("just" OTM, 2-3 months expiration) and at the same time we will be placing short PUTs ("far" OTM, immediate expiration) to get paid for the time we remain "locked" in the CL position. (I will try to keep the protection "dynamic", doing and undoing it as necessary, because we need to overload as much as possible a possible favorable move, even if this means to take extra risk.)

(Long positions to a given strike of a protective CALL will be added as long as the option is OTM or slightly ITM: when it goes definitely ITM, it is abandoned to full-auto mode with a short-only constraint and a take profit (and any possible positive long player is immediately closed) to recover as much as possible of the time decay. The next strike which is just OTM will then be used for the protection.)

 

[fullautotrading]

The picture below shows our "plan" about ERX. We have, on top, the big losing layer which had to accumulate a huge short position in the effort to free funds to allow ERY to close. Below we have a new layer where we have placed an (approximately) equal position, to "stop" the bleeding.
On both layers I have split the global position into several smaller players (using the "split" device we just introduced), to allow a gradual process.

Clearly, the intent is to gradually "discharge" the short layer, which has gone way beyond we can tolerate, and "load up", instead, on the opposite long player. In fact we don't want to be short on ERX (which is an ultralong) for the "long term".

On the short layer, I have placed a constraint to close only the players, while in the long layer I am allowing both short and long entry, and using the "shield+" game (slightly biased towards long.)

Most of the money has now been moved on CL and its options which will also hedge the possibly growing position of ERX.

I have taken the habit to systematically "remove" (from view) the layers which show a profit and are closed, so that it's easier to focus on the rest. It also provides a more methodical way to proceed, a sort of divide et impera, which is always good, in general, to do great things , and helps to reassess the situation "between fluctuations". So far, I have removed a total of 88 layers for a combined PNL of about 177K. DD is still pretty large anyway: -37%, but at least we have plan .

 

[fullautotrading]

Ending the week with large DD. We are now at -53% all concentrated on ERX and CL. Several layers (precisely 98) were closed in profit and removed from view in the meantime, for a total of + 184K, but they could not do much against the irresistible rally of the oil

They are in particular :

CL FOP 201502 51 C NYMEX 1000            PNL: 700.76, comms: 9.24, fills: 4, Pos: 0
CL FOP 201502 52 C NYMEX 1000            PNL: 427.66, comms: 32.34, fills: 8, Pos: 0
CL FOP 201502 53 C NYMEX 1000            PNL: 293.74, comms: 106.26, fills: 16, Pos: 0
CL FOP 201502 54 C NYMEX 1000            PNL: 659.94, comms: 60.06, fills: 7, Pos: 0
CL FOP 201502 56 C NYMEX 1000            PNL: 696.84, comms: 83.16, fills: 13, Pos: 0
CL FOP 201502 60 C NYMEX 1000            PNL: 306.14, comms: 13.86, fills: 3, Pos: 0
CL FOP 201502 61 C NYMEX 1000            PNL: 736.90, comms: 23.10, fills: 4, Pos: 0
CL FOP 201503 48 C NYMEX 1000            PNL: 1,166.84, comms: 83.16, fills: 32, Pos: 0
CL FOP 201503 49 C NYMEX 1000            PNL: 1,616.90, comms: 23.10, fills: 8, Pos: 0

.... omissis ... (to comply with post limit length)

CL FOP 201503 54 P NYMEX 1000            PNL: 8,670.58, comms: 189.42, fills: 49, Pos: 0
CL FOP 201503 55 P NYMEX 1000            PNL: 1,151.46, comms: 78.54, fills: 22, Pos: 0
CL FOP 201503 56 P NYMEX 1000            PNL: 1,525.32, comms: 64.68, fills: 15, Pos: 0
CL FOP 201503 57 P NYMEX 1000            PNL: 498.42, comms: 41.58, fills: 11, Pos: 0
CL FOP 201503 58 P NYMEX 1000            PNL: 815.32, comms: 64.68, fills: 24, Pos: 0
CL FOP 201503 60 P NYMEX 1000            PNL: 1,388.42, comms: 41.58, fills: 5, Pos: 0
CL FOP 201503 61 P NYMEX 1000            PNL: 4,626.90, comms: 23.10, fills: 4, Pos: 0
CL FOP 201504 56 P NYMEX 1000            PNL: 2,453.80, comms: 46.20, fills: 2, Pos: 0
CL FOP 201502 50 P NYMEX 1000            PNL: 342.28, comms: 27.72, fills: 4, Pos: 0
ERY OPT 20150116 18 P SMART 100          PNL: 341.52, comms: 158.48, fills: 4, Pos: 0
ERY OPT 20150116 20 P SMART 100          PNL: 36.48, comms: 3.52, fills: 2, Pos: 0
ERX STK SMART                            PNL: 2,897.87, comms: 451.11, fills: 365, Pos: 0
ERX STK SMART_L2                         PNL: 9,359.62, comms: 418.41, fills: 91, Pos: 0
ERY STK SMART_USD_L1                     PNL: 334.13, comms: 68.33, fills: 9, Pos: 0
ERY STK SMART_USD                        PNL: 3,164.86, comms: 2,979.85, fills: 709, Pos: 0
CL FUT 201512 NYMEX 1000                 PNL: 8,038.80, comms: 1,201.20, fills: 402, Pos: 0
ERY OPT 20150116 20 C SMART 100          PNL: 2,637.54, comms: 262.46, fills: 4, Pos: 0
ERY OPT 20150116 25 P SMART 100          PNL: 576.18, comms: 23.82, fills: 2, Pos: 0
ERY OPT 20150116 30 P SMART 100          PNL: 2,920.15, comms: 79.85, fills: 2, Pos: 0
ERY OPT 20150417 25 P SMART 100          PNL: 1,930.15, comms: 69.85, fills: 2, Pos: 0
ERY OPT 20150116 21 C SMART 100          PNL: 13,529.79, comms: 160.21, fills: 9, Pos: 0
ERY OPT 20150116 24 C SMART 100          PNL: 5,293.60, comms: 56.40, fills: 5, Pos: 0
ERY OPT 20150116 25 C SMART 100          PNL: 2,899.64, comms: 25.36, fills: 2, Pos: 0
ERY OPT 20150116 27 C SMART 100          PNL: 2,708.61, comms: 246.39, fills: 9, Pos: 0
ERY OPT 20150417 25 C SMART 100          PNL: 12,027.15, comms: 142.85, fills: 3, Pos: 0
CL FUT 201512 NYMEX 1000_L2              PNL: 1,272.28, comms: 27.72, fills: 8, Pos: 0
CL FUT 201512 NYMEX 1000_L3              PNL: 1,389.94, comms: 60.06, fills: 13, Pos: 0
CL FUT 201512 NYMEX 1000_L4              PNL: 1,278.42, comms: 41.58, fills: 7, Pos: 0
CL FUT 201512 NYMEX 1000_L5              PNL: 2,658.42, comms: 41.58, fills: 10, Pos: 0
CL FUT 201512 NYMEX 1000_L6              PNL: 4,399.94, comms: 60.06, fills: 14, Pos: 0
CL FOP 201506 55 C NYMEX 1000            PNL: 133.04, comms: 36.96, fills: 16, Pos: 0
CL FOP 201506 53 C NYMEX 1000            PNL: 1,801.52, comms: 18.48, fills: 5, Pos: 0
CL FUT 201512 NYMEX 1000_L2              PNL: 653.01, comms: 66.99, fills: 17, Pos: 0
CL FOP 201506 55 C NYMEX 1000            PNL: 0.76, comms: 9.24, fills: 4, Pos: 0
CL FOP 201506 57 C NYMEX 1000            PNL: 411.52, comms: 18.48, fills: 6, Pos: 0
ERX STK SMART_L3                         PNL: 25,134.52, comms: 204.33, fills: 51, Pos: 0
CL FUT 201512 NYMEX 1000_L2              PNL: 1,158.42, comms: 41.58, fills: 11, Pos: 0
CL FOP 201506 59 C NYMEX 1000            PNL: 203.04, comms: 36.96, fills: 13, Pos: 0
CL FOP 201506 55 C NYMEX 1000            PNL: 201.52, comms: 18.48, fills: 7, Pos: 0
CL FOP 201506 58 C NYMEX 1000            PNL: 1,692.80, comms: 277.20, fills: 32, Pos: 0
CL FOP 201506 56 C NYMEX 1000            PNL: 276.90, comms: 23.10, fills: 7, Pos: 0
CL FOP 201503 45 P NYMEX 1000            PNL: 7,512.98, comms: 97.02, fills: 5, Pos: 0
CL FOP 201503 44 P NYMEX 1000            PNL: 6,255.20, comms: 184.80, fills: 7, Pos: 0
CL FOP 201503 43 P NYMEX 1000            PNL: 4,907.60, comms: 92.40, fills: 2, Pos: 0
CL FOP 201503 42 P NYMEX 1000            PNL: 3,707.60, comms: 92.40, fills: 2, Pos: 0
CL FOP 201506 58 C NYMEX 1000            PNL: 75.38, comms: 4.62, fills: 2, Pos: 0
CL FUT 201503 NYMEX 1000                 PNL: 746.90, comms: 23.10, fills: 7, Pos: 0
CL FOP 201503 48 P NYMEX 1000            PNL: 254.50, comms: 115.50, fills: 8, Pos: 0
CL FUT 201503 NYMEX 1000_L1              PNL: 1,230.76, comms: 9.24, fills: 2, Pos: 0
CL FOP 20150617 57 C NYMEX 1000          PNL: 362.28, comms: 27.72, fills: 4, Pos: 0
CL FOP 201503 41 P NYMEX 1000            PNL: 1,761.40, comms: 138.60, fills: 4, Pos: 0
CL FUT 201503 NYMEX 1000_L2              PNL: 780.76, comms: 9.24, fills: 4, Pos: 0
CL FOP 20150217 43 P NYMEX 1000          PNL: 120.76, comms: 9.24, fills: 3, Pos: 0
CL FOP 20150217 44 P NYMEX 1000          PNL: 85.38, comms: 4.62, fills: 2, Pos: 0
CL FOP 20150217 45 P NYMEX 1000          PNL: 864.38, comms: 235.62, fills: 8, Pos: 0
CL FOP 201503 50 P NYMEX 1000            PNL: 186.08, comms: 73.92, fills: 23, Pos: 0
CL FOP 201503 41 P NYMEX 1000            PNL: 353.80, comms: 46.20, fills: 2, Pos: 0
CL FOP 201503 42 P NYMEX 1000            PNL: 1,415.20, comms: 184.80, fills: 3, Pos: 0
CL FOP 20150617 57 P NYMEX 1000          PNL: 153.04, comms: 36.96, fills: 9, Pos: 0
CL FOP 20150217 48 P NYMEX 1000          PNL: 1,061.40, comms: 138.60, fills: 2, Pos: 0

Count:    98
Sum:      184459.44
Avg:      1882.24
Median:   698.8
Std:      3441.58

Clearly, the initial sizing mistake with ERY has caused a chain of actions and events on ERX and CL, which are causing nasty troubles. The scalping action has been working incessantly to hedge, actually sparing us a lot of more losses ("saved" about 152K on ERX, and 663K on CL), however the position too large relative to the account and the strong move of the instrument are evidently causing a relatively large DD. It is, on the other hand, almost surprising how much abuse a "portfolio account" can take when compared with RegT margin mode.

Unlike most people who show you only the winning instruments and layers ("cherry picking" or "selective reporting", http://rationalwiki.org/wiki/Selective_reporting), I show you only the losers . It is in fact important to see what are the actual risks of trading (even if this is just a test), the real world DD and how mistakes can escalate. (And this is especially important in the "trading world" where there are a lot of mystificators creating false illusions about easy gains with little work, while the reality is quite different.):

For the moment, I have grouped and re-split all players on the losing layers to reorganize them, and give to the "machine" the task to try recovering the situation, in full auto mode. After 50%, this is not, in general, a very doable task, anyway we will see how it behaves and what more we can learn from the process. These are the situations of most massive learning (and also most useful on the development side).

I have been adding a few other useful features, handy to deal with difficult or complex situations. In addition to the grouping into 1 unique player for further splitting, I have also added a grouping in 2 players, for the buy/sell side, which actually in practice turns out pretty useful.

Now that I have a full "separation" of the "physical" (orders) and the "logical" domain (players), I have also a few other ideas I want to implement during the weekend, like the separate grouping of only the players of just one side (buy or sell side).


In general, it is quite evident that we need to organize better our overall approach in order to avoid these large DD. Looks like scalping the options is not a bad idea after all, judging by the practical results. (Problem is that this is doable in practice only with a handful of instruments.)

Certainly, it is a bad idea, instead, to pursue "vertical" moves ("bets") especially with large sizes. Even if there is a sort of psychological resistance (this includes me too, evidently ), I think one should rather focus on the "horizontal moves" and actually visually picture the whole mkt as a series of horizontal moves spaced by vertical gaps, or "intermezzos" (rather then thinking in terms of "trends").

And to stress this intent, opposite "competing" layers could also be used. In any case we will naturally end up "leaning on one side" (because, in the long term, there is always one side which will close most of the scalps), but one thing is arriving to that ("near exhaustion" and against the instrument "drift") as result of a balanced scalping action, and another thing is to start with a large unilateral "bet". This might also require the introduction of a few specific features to somehow "synchronize" the behavior of these competing layers, (so that they keep re balancing as much as possible): this has to be explored.

 

[fullautotrading]

Splitting details

On the development side, it may be interesting to take a quick tour on some details of the player "splitting" process, to gain some more familiarity with the idea. These are usually details which I work out almost automatically, at the same time that I type in the code. However, it can be useful to later revise them, just in case something can be made better.

Let's consider the simplest case of "uniform distribution" of the split players.
Assume we have a player that we want to split, say a buy 32 @ 4.64. Denote

N := Position (e.g., 32)
P := Price (e.g., 4.64)

We want to split it using the following "preferred" size and distance:

s := "preferred" size
d := "preferred" distance

The reason why I use the adjective "preferred", is because whenever N is not divisible by s, obviously you cannot get exactly the same sizes for all players (because sizes can only be integers), or perfectly equal distances among the "split players".

Let's denote:

N(i) := Position of i-th split player
P(i) := Price of i-th split player

with i= 0, ..., k-1, and let k be the number of the "split players"


Case A: N is divisible by s
============================

This should be the "no-brainer" case. We have that the number of the "split players" is k = N / s and

N(i) := s
P(i) := P + ( (k-1) / 2 - i ) * d (i= 0, ..., k-1)

This works because the prices are chosen equispaced and forming symmetric deviations about the mean P, and, therefore, the average of the "split players" has to be equal to P. At the same time, the sum of the k sizes of the split players gives N.

If you are not convinced you can write it down :

Case B: N is not divisible by s
================================

In this case, there is not a "unique" meaningful way to do the splitting, and some arbitrary decisions can be made.

For instance, if N=32 and s=6, you have 5 players of size 6 but you need another one for the remainder 2.
Probably, the most immediate idea would be to just "adapt" the previous case. So one would take the k=6 split players and allocate s to all of them, except, for instance, for the last one, where we need to allocate the "residual" position. So:

N(i) := s for i= 0, ..., k-2
N(k-1) := P - (k-1) * s

where, in this case, the number k of the split players is:

k = floor(N/s) + 1 or, if you prefer, ceiling(N/s) .

As to the prices P(i), i= 0, ..., k-1, we might use the same prices as in the previous case, and simply adjust the last one, so that is set equal what is needed to make the average equal to P:

P(i) := P + ( (k-1) / 2 - i ) * d for i= 0, ..., k-2
P(k-1) := [ N * P - SumOfPrevious( P(i) * N(i)) ] / N(k-1)

Now this might sound ok. However, in practice it has some disadvantage. In fact, by keeping the same size s for the k-1 players and allocating the residual in the last one, it can happen that this residual position, which obviously (being a remainder) is always smaller than the preferred size s, is so different (say small) from the other players that to get back the the mean P, the player price P(k-1) gets "thrown off" significantly. And that does not look nice, nor practical for applications.

So, another solution, which we may want to prefer, is to take the residual and, instead of creating a new small player, add it up to the previous player. And, in case it is greater than or equal to the number of players k = floor(N/s), we redistribute the units evenly in the previous players. So we have:

k = floor(N/s) and denoting r = remainder of N/s

if r < k:

N(i) := s for i= 0, ..., k-2
N(k-1) := s + r


if r >= k:

N(i) := s + floor(r / k) for i= 0, ..., k-2
N(k-1) := r mod k

The effect of this "redistribution" is to have k-1 players with a size still uniform and close to the "preferred" size , while the last player will have, as size, the complement to N. Since the P(i)'s are the same as before and the N(i)'s have now increased, the new price for the last player has to increase due to the relationship:

P(k-1) := N * P - SumOfPrevious( P(i) * N(i))

because the sum is increased.

Let's see a concrete visual example. Assume we must split a player with:

N = 32
P = 4.64

With s = 6 and d = 2%. Using the first way we would get:

CT_BUY 6 @ 4.87
CT_BUY 6 @ 4.78
CT_BUY 6 @ 4.68
CT_BUY 6 @ 4.59
CT_BUY 6 @ 4.50
CT_BUY 2 @ 3.94

Note that the price 3.94 is too "far off", and we may not want that.

Using the second way, we instead get:

CT_BUY 6 @ 4.82
CT_BUY 6 @ 4.73
CT_BUY 6 @ 4.64
CT_BUY 6 @ 4.54
CT_BUY 8 @ 4.50

which looks nicer, because the latest price is "thrown" within the other ones when it has larger size.

In this second case if we use s=7 or 8, we get:

CT_BUY 8 @ 4.78
CT_BUY 8 @ 4.68
CT_BUY 8 @ 4.59
CT_BUY 8 @ 4.50

which "relaxes" the constraint on the "preferred" size and distance, but makes the sizes equal, which we clearly prefer.

For s=9

CT_BUY 10 @ 4.73
CT_BUY 10 @ 4.64
CT_BUY 12 @ 4.56

In other words, the second way increases the "homogeneity" of the sizes, and helps "smoothing" the last distance, and the player splitting works better.

In the picture below, a visual representations of the various variants in our example:

What I am adding now is the option to group separately the buy and sell side for further re-splitting, which turns out pretty useful, especially in situations which have turned out messed up and complex for whatever reason.

 

[fullautotrading]

Weekend: a good time to make some general reflections about what we have been learning or deducing so far from the various experiences and tests. Feel free to jump in, adding your contributions and experiences...


Margin method: "RegT" vs "Portfolio margin"

Let's start from the account setup. We have seen that in practice the RegT margin methods with STKs and ETFs is hardly viable as it places too "strict" restrictions. The consequence is that the already small funds (if one has large funds RegT is obviously meaningless) are quickly absorbed by the stk position, and one ends up being forced to take losses pretty soon, even if the trades turn out to be done correcly.

However, the porfolio margin method (https://www.interactivebrokers.com/en/?f=margin&p=pmar) has a min 110K requirement, so in practice if you don't have a good risk capital, to maintain that figure even in DD, better forget about it.

Clearly the alternative would be either trading FX (https://www.interactivebrokers.com/en/index.php?f=margin&p=forex) or futures (Span margin method: https://www.interactivebrokers.com/en/?f=margin). However, trading futures with small funds, may not be a brilliant idea, and one can end up pretty badly. For the small trader, it remains FX. Here the problem is the absence of options (one could use the futures options, but would probably be too big), and also the general dynamic of the price curve which tends to have longer waves and relatively smaller inner fluctuations. Thus, long discouraging DD with little or no profits.

So, the general consideration would be: do not trade, unless you have suitable risk capital. That is, funds that you can afford to lose, without it being a problem. If you don't have it, but still passionate about the field, the research and with money management, what I always suggest is to build up practical experience with a good trading tool (paper trading would also do fine, provided the methodology and approach is convincing) and then, when mature, look for wise people with suitable resources, and especially "balls" , to fund the venture and investment. Backtests are mostly a waste of time, or even worse, strongly misleading, when they favor curve-fitting activities (practically always), performed on a simple "projection" (the price dimension) of a complex multidimensional phenomenon. Trading is instead mostly a live performing activity. Just like aerial combat, or a live concert Much more useful and meaningful is instead a simulation activity, carried out to examine the deterministic aspects of the component represented by the non-predictive scalping/hedging game.


"Risk" capital

It is important to note that the funds used for trading, no matter if large or small, should be treated with complete emotional detachment. That is, they must not represent your pension or some "lifetime savings". Nor any aspect of your life must depend on it. The reason for that is, since there is no method which does not have some DD (well, raise your hand if you know one, that is not a backtest ), if you start feeling anxiety and stress when there is an "investment phase" and you are in DD, that is a sure way to lose. If you cannot remain emotionally detached from your capital being traded, you have actually already lost it, even before start trading. That is sure 100%. In fact, given enough time, with finite funds and emotional attachment, there will always be a DD which will "convince" you take an unrecoverable loss. It's only a matter of time and frequency.


Ideal trading setting

It seems that the most suitable environment for this kind of activity is, after all, a portfolio of liquid futures, with decent risk capital. Liquid instruments with liquid options should be the preferred choice. One or two FX pairs might also be included if there are large funds (we may be better off with the corresponding futures anyway). Most sectors are correlated so, in practice, the idea of "diversification", while appealing in principle, can be harder than expected to achieve in reality.


Never build a long term position against drift/decay/contango

It is important to carefully look at the "drift" that the instrument exhibits and make sure not to build a long term position against it. In fact, the decay would eat up any profit stemming from scalping activity, if the main position is systematically against it.


Watch the gaps and choose wisely the instruments

ETFs, as we have seen, offer a lot of challenges. Some of these are:

- Drifts/ decay etc.: do not trade a long term position against the possible drifts
- Non shortability: instruments are often non shortable causing great disruption in algorithmic trading and scalping activity. Also, for small price instruments, execution related issues may arise.
- Margin: take too much margin and eat up quickly your funds (with portfolio margin the situation is slightly better)
- Possible large overnight gaps: usually you don't want to go overnight with a large exposure, so you may need to "flatten" the position. Problem is that if you were short, the next day you may not be able to re-open (pretty typical with ultrashort ETFs) and remain "trapped" in a large loss after a long and painful "investment".
- Illiquid options: this can be a significant problem, since the spread is so large that practically once you make a transaction you are "trapped" in the position until expiration


FUTURES: best "bang for the buck" for speculative activity. Things that have to be watched:

- Contango / backardation: do not trade systematically against them long term, or you will end up being eaten alive
- Physical delivery (http://ibkb.interactivebrokers.com/node/992) and rollover: we have a few mechanisms in place to carry on the "past trading information" (that is the player "logic") from an instrument to the next contracts, so this should not represent a problem at conceptual level
- Very good "risk capital", because even just 1 contract has usually a pretty large multiplier for most instruments


Never make (large) "directional" bets: non-predictive scalping

The reason for that is, in hindsight , pretty obvious. If one goes about trading by making systematically large directional "bets" it is unavoidable that, given enough time, he will end up with some large loss beyond recovery, because sooner or later funds will not suffice to sustain an unfavorable move. And, in any case, the DD can last a discouraging length of time, so an anxious investor could anyway run out of patience and leave the money on the table, even if the bet turns out eventually to be a winner (I have seen this several times).

One could argue: what about relatively "high probability" trades and and "relatively small stops"? Well here the discussion could become heated. I might start by saying with De Finetti that "Probability does not exist" (https://faculty.fuqua.duke.edu/~rnau/definettiwasright.pdf).
Also, in a nutshell, wherever you place your stop or take profit, by symmetry, ignoring the trading costs, you will end up in the long term, with the following relationship:

Avg profit / Avg loss = Prob Loss / Prob Profit

that is, essentially an "asymptotic" PNL = 0, with the commissions and spread (and possible other forms of "decay") adding a strongly negative drift, which will be more evident as the trading frequency increases (with less frequent trades it easier to quickly "curve fit" by computer and get self-deceiving simple "strategy", which essentially ends up making more trades in the prevalent direction of the instrument, during the chosen backtest period: typical naive curve-fit with a 45-degree equity curve ).
(Actually when the stops get really "small", the relationship may actually break into a even more unfavorable one, giving rise to the so called "death by thousand cuts".)

This, in essence, is saying is that there is no real reason why the value of profitable trades will exceed the value of losing trades, just by effect of how you choose the take profit and stop distances. It is obvious that it is so, because, apart there is no reason for that to happen, if such a "regularity" existed in a mkt it would be readily discovered and exploited, leading to the extinction of the specific mkt, which is not the case, as instead who get extinct are rather the traders But, anyway, try that, and let me know

The truth is that you could be "guaranteed" to be profitable beyond any limit only if an instrument were trading within a "finite" (and "relatively" small) range, like for instance a stochastic process evolving continuously within "reflecting barriers" (e.g., http://en.wikipedia.org/wiki/Reflected_Brownian_motion). In that case, it is obvious that you could make unlimited profits just suitably buying and selling within that range (there would be no need of any "prediction"). Problem is that obviously such instruments do not exist, as if they existed the relevant mkt itself would quickly disappear. So, what we can do is to subdivide the price move into approximately "horizontal segments" (where a profit can be actually extracted through non-predictive scalping/hedging), while finding ways to hedge against the vertical component (which can carry either a "loss" or a "profit" on "reversion" to a previous "horizontal" level.)

[continues...]

 

[fullautotrading]

...

How do we focus on the horizontal components and at the same time watch for decay/contango and other drifts

Here is where the architecture based on "player superposition" and "layer overlay" becomes essential. The "player superposition" allows to scalp effectively the horizontal component in a "multithreading" mode and to efficiently keep track of the past trading information.

- Player superposition: this is necessary because if, instead, you go ahead with a sequence of "disjoint" consecutive trades, you are, in fact, essentially back to a "directional" betting framework, and that is not what we may want, for the reasons argued before. Also the "player architecture" is necessary to store the past trading information and allow "loss unwinding", as we have seen in practical examples, which is a powerful source of edge.

- Layer overlay: even with "player superposition" we cannot eliminate, in practice, the DD due to directional moves. The reason for that is, when there is a strong directional move, the players on one side will tend to close in order to take a profit, while most of the players on the other side will tend to remain open. This causes evidently the scalping/hedging system to be inherently "reversing" in nature. Not "mean reversing" obviously, but, in any case, needing the price to eventually retrace to a previous level to close the open players (the "hedging" ones). In fact, the whole logic of creating an edge by the player superposition is in the fact that, due to the erratic movements of the price, we end up, eventually, closing all the players which where used as "stops" or hedge, thus unbalancing the natural 50/50 % will/loss ratio to our favor. Problem is, in practice, that the funds may not suffice before there is globally enough reversion to previous price levels. especially if the scalping packet sizes are exaggerated.


Dealing with moderate "packets" and order sizes

Clearly, the scalping game can partially deal with the "directional" component too, because we also use "protective players", that is players which open along the current direction buying high and selling low, however they can only "capture" a fraction of the move (typically around 1/3, depending on the settings), and they also reduce (or even destroy if their size is exaggerated) the profits on the purely "horizontal" components.

One could naively argue: why don't you immediately close all the losing players when the price goes "against" you ? The problem with that is what we have called the "lump of money" issue. That is "stopping and reversing" would cause large orders which introduce a lot of "inefficiency" in the scalping process, and large losses concentrated in small segments of the price curve. Instead, what we want is a smooth and gradual action on a layer so that in due time, we can gradually recover all the losing players. The "stop and reverse" approach would in the long term have fatal consequences because you would have lethal series of large buy/sell orders with the buys above the sells, and therefore large losses concentrated in one spot, which will become practically impossible to "undo".

Layer superposition, drifts

So, since we cannot "abruptly" reverse a layer, at least in a continuous and systematic manner, a possible approach is to have another layer playing an almost symmetrical but mostly independent game, which could provide some further hedging actions.
A source of complexity is the correct "balancing" of the two layers, and the fact that you need, at the same time, to keep into consideration possible contango, or any "drift". For instance, if you have 2 CL layers you may want to keep them almost "balanced" to capture most of the "horizontal" component, but, at the same time, slightly unbalanced on the short side for the long term, to avoid being eaten alive by the contango.
Another source of hedging may be provided by options.
Options have also a strong time decay, in a known timeframe, which, for liquid instrument and suitable strikes, seem to be exploited with relative ease by automated scalping with an appropriately biased game.