Trading FUTURES with IB

hi

I have been working hard on some massive changes.
In the process i have also added (see picture) a synoptic Excel export of filled orders (just for traders not wiring to SQL server, as the SQL server export was already in place).

Having changed the PNL computation engine, i have done a vast revision of all performance metrics, and i hope this can lead to a more powerful and precise selection process of strategies.

Let see how i have modified the PNL computations. (What follows may correct/change some points of previous post).
First of all i define:

<b><center>PNL = REALIZED GAIN + REALIZED LOSS + UNREALIZED</center></b>
further, i define:
INVESTMENT = REALIZED LOSS + UNREALIZED

We all know how to compute the PNL.
Now the crucial point is to compute in a meaningful way the REALIZED GAIN and LOSS.
Consider, for instance, the sequence of orders (ignore multipliers and commissions) and assume that the <b>current ASK is 5000</b>:

SELL 1 @ 2000
SELL 1 @ 7000
SELL 1 @ 12000
BUY 1 @ 10000

If we used the FIFO rule (as for taxes), we would be matching:

SELL 1 @ 2000
BUY 1 @ 10000

So a Realized of -8,000. The current PNL (including the trade to close) is 1,000.
So we have PNL = Realized + Unrealized => 1,000 = -8,000 + 9,000

Now, while this is probably favorable (in this example) for taxes (because we "realize" a negative amount), it's not good for an intuitive understanding of how the strategy is doing.
In fact, as traders, we would prefer that the profitable scalps be "matched" first (instead of using the FIFO rule), because that is how the strategy works: we invest and trade.

Infact, we would like to fist "recognize" the good trade:
SELL 1 @ 12000
BUY 1 @ 10000 a $ 2,000 profit trade

and then allocate the rest as "unrealized". So we prefer to think about it as:

PNL = REALIZED GAIN + REALIZED LOSS + UNREALIZED

1,000 = 2,000 + 0 - 1,000

This way, I think we can follow much better the strategy and the evolution of the REALIZED GAIN, as any strategy will naturally strive to maximize that part. We know we have made a good trade of 2,000 and an investment of 1,000 is ongoing.

So the basic idea boils down to: <b>we match first the good trades (realized GAIN), then we match the bad trades (realized LOSS) and what remains is "unrealized".</b> Clearly this affects the algorithm to compute the PNL components (and not the PNL itself).

The "total investment" can be seen as the unrealized + realized LOSS, where actually the unrealized is the actual part of the "investment" which can still create wealth ("ongoing investment").

(If you dont like or agree with this terminology/definitions or you have better wording, please let me know, it can be changed. This is just ongoing work and thinking... so anything can be done... )

Tom

<img src="http://www.elitetrader.com/vb/attachment.php?s=&postid=3250616" width="800" />
 

Attachments

hi,

the last month has been one of the most volatile we have seen in a while, with a strongle bubble and instruments which have gone up over 5,000 ticks (like RB) !

A pretty catastrophic scenario, "good" to check how the bot behaves.

As i have anticipated in the previous post when we were still in "deep" drawdown, the bot has done actually pretty well, considering the scenario, keeping the total "investment" on a pretty large folio (13 instruments) within the 100K figure, and <b>not taking more than 5 contracts</b> per instrument.

As a result, it has now profited from all the past investments, and it is now its launching itself in a <b>new round of "volatility investment" with GC and CHF (about 60K-70K)</b>. When also this one will be over, we should see a nice profit.

Note that if we had applied stops somewhere, we would have never come back. As systematic <b>stops make the potential (and actual) loss unbounded</b>, while here we are bounded by the mkt volatility (as at certain prices there are obviously no many buyers or sellers for a given instrument).
(Clearly, the practical feasibility of this strategic plan, is due to the scalping action and the particular calibration of the investment that the bot does, which is kept as low as possible, without compromising the actual possibility of turning it into profit.)

You can also understand why all people testing this with real money have been so far profitable, just having to be patient with drawdowns ("investment") and clearly having to had the appropriate risk capital (compared to the traded folio). Simply the bot doesn't want to leave a penny to the mkt.

[Clearly it also helps a touch of "discretionality", allowed by the app, for instance by avoiding the bot to take long/short position on very low/high (which we have not done here because interested in looking at the worst scenarios). ]

[chart below is, as usual the most recent version]

[ As a side note, you can see in these picture an ugly "drop" of the "realized curve", which is merely due to rollovers.
This kind of ugly chart was actually one of the motivations for all the discussion and point i am making about the new PNL decompositions. Infact in the new bot release, we won't see anymore things like that, because i am separating gain from losses and placing them on separate curves, which allows a better understanding of the strategy and a better performances assessment (i will continue the discussion on PNL decomposition in next posts). ]

Tom

<img src="http://www.elitetrader.com/vb/attachment.php?s=&postid=3260263" width="1200" />
 

Attachments

as an example, these are the trades of CL.

The maximum position reached was (short) 3 contracts (on a range of almost 2000 ticks).

Here it is apparent how the bot distributed parsimoniusly the investment and scalped on drawdown, until the move reached exhaustion. This way it places itself on the right side of the mkt with a good position, and can ride the trend downward

[this chart is the newer version, the older one has more scalps. the trade on the center where there is the vertical red line is the rollover: the bot joined the 2 contracts into a unique price curve].


Tom

<img src="http://www.elitetrader.com/vb/attachment.php?s=&postid=3260577" width="1200" />
 

Attachments

As promised, while we wait that also our investment in GC and CHF volatility works, i will discuss here a few details about the new <b>PNL decomposition:

<center>PNL = Gain + Loss + Unrealized</center></b>
To make it easier to follow the discussion for traders and people who don't do much coding, i will be making several simplifyings assumption, and converting to VB.NET. In case you want to see the code in C#, just paste it here: http://www.developerfusion.com/tools/convert/vb-to-csharp/

<b>Assumptions</b> i make (to simplify the illustration):

- All Filled qty are equal to 1
- Commissions are ignored
- Multiplier (if any) is assumed = 1
- i assume you have already placed your BUY and SELL orders into 2 stacks, called, in this example, UnitOrders_BUY, and UnitOrders_SELL:

Code: 
    Public UnitOrders_BUY As Stack(Of UnitOrder)
    Public UnitOrders_SELL As Stack(Of UnitOrder)
in such a way that the <b>bigger prices are on top of the stacks</b>.

i assume that you have 2 variables holding the current bid and ask:
Code: 
    Public Current_BID As Decimal = 5000
    Public Current_ASK As Decimal = 5000
- i assume the Order class is this simple (filled qty assumed=1):
Code: 
Public Class UnitOrder

    Public Price As Decimal

    Sub New(ByVal Price As Decimal)
        Me.Price = Price
    End Sub

End Class
Now the idea is pretty simple. We do order matching considering first the positive scalp (<b>gain</b>), then we match the negative scalps (<b>loss</b>), and finally, the remaining is the <b>unrealized</b>:
Code: 
    Sub ComputePNLComponents_UnitOrders()

        Dim PositionSigned As Integer
        Dim Unrealized As Decimal

        Dim PermanentGain As Decimal           'amount of nonnegative scalps
        Dim PermanentLoss As Decimal           'amount of negative scalps

        Dim QtyInGoodScalps As Integer         'non-negative scalps
        Dim QtyInBadScalps As Integer          'negative scalps

        Dim PNL As Decimal

        '---------------
        'POSITIVE SCALPS
        '---------------

        Dim UnmatchedBuys As New Stack(Of UnitOrder)

        If Me.UnitOrders_BUY.Count > 0 AndAlso Me.UnitOrders_SELL.Count > 0 Then

            Do

                Dim UnitOrder_Buy As UnitOrder = Me.UnitOrders_BUY.Pop
                Dim UnitOrder_Sell As UnitOrder = Me.UnitOrders_SELL.Peek

                If UnitOrder_Buy.Price <= UnitOrder_Sell.Price Then
                    QtyInGoodScalps += 1
                    PermanentGain += UnitOrder_Sell.Price - UnitOrder_Buy.Price
                    Me.UnitOrders_SELL.Pop()  'matched: buy/sell can be thrown away
                Else
                    UnmatchedBuys.Push(UnitOrder_Buy)
                End If

                If Me.UnitOrders_BUY.Count = 0 OrElse Me.UnitOrders_SELL.Count = 0 Then Exit Do

            Loop

        End If

        'finish popping the buys if any remaining
        While Me.UnitOrders_BUY.Count > 0
            UnmatchedBuys.Push(Me.UnitOrders_BUY.Pop)
        End While

        '---------------
        'NEGATIVE SCALPS  (continue process for losses)
        '---------------

        Dim UnmatchedSells As New Stack(Of UnitOrder)

        If UnmatchedBuys.Count > 0 AndAlso Me.UnitOrders_SELL.Count > 0 Then

            Do
                Dim UnitOrder_Sell As UnitOrder = Me.UnitOrders_SELL.Pop
                Dim UnitOrder_Buy As UnitOrder = UnmatchedBuys.Peek

                If UnitOrder_Sell.Price < UnitOrder_Buy.Price Then
                    QtyInBadScalps += 1
                    PermanentLoss += UnitOrder_Sell.Price - UnitOrder_Buy.Price
                    UnmatchedBuys.Pop()
                Else
                    UnmatchedSells.Push(UnitOrder_Sell)
                End If

                If Me.UnitOrders_SELL.Count = 0 OrElse UnmatchedBuys.Count = 0 Then Exit Do

            Loop

        End If

        'finish popping the sells if any remaining
        While Me.UnitOrders_SELL.Count > 0
            UnmatchedSells.Push(Me.UnitOrders_SELL.Pop)
        End While

        'Now only unmatched orders are remaining, either buy or sell

        '-----------------------
        'Unrealized and Position
        '-----------------------

        If UnmatchedBuys.Count > 0 Then

            Dim AmountBuy As Decimal
            Dim Qty_Buy As Integer
            For Each UnitOrder_Buy As UnitOrder In UnmatchedBuys
                AmountBuy += UnitOrder_Buy.Price
                Qty_Buy += 1
            Next

            PositionSigned = Qty_Buy
            Dim ValueSigned As Decimal = PositionSigned * Me.Current_BID 'will be a sell
            Unrealized = -AmountBuy + ValueSigned

        Else

            Dim AmountSell As Decimal
            Dim Qty_Sell As Integer
            For Each UnitOrder_Sell As UnitOrder In UnmatchedSells
                AmountSell += UnitOrder_Sell.Price
                Qty_Sell += 1
            Next

            PositionSigned = -Qty_Sell
            Dim ValueSigned As Decimal = PositionSigned * Me.Current_ASK
            Unrealized = AmountSell + ValueSigned

        End If

        PNL = PermanentGain + PermanentLoss + Unrealized


        'Print results

        With Me.RichTextBoxResults
            .AppendText(vbCrLf)
            .AppendText(vbCrLf & "----------------------------------------")
            .AppendText(vbCrLf & "PositionSigned:              " & PositionSigned)
            .AppendText(vbCrLf & "QtyInGoodScalps:             " & QtyInGoodScalps)
            .AppendText(vbCrLf & "QtyInBadScalps:              " & QtyInBadScalps)
            .AppendText(vbCrLf)
            .AppendText(vbCrLf & "PermanentGain:        " & PermanentGain)
            .AppendText(vbCrLf & "PermanentLoss:        " & PermanentLoss)
            .AppendText(vbCrLf & "Unrealized:           " & Unrealized)
            .AppendText(vbCrLf & "                      ----")
            .AppendText(vbCrLf & "PNL:                  " & PNL)
        End With

    End Sub
A word of caution. The above code can be run to understand my illustration, but it's not obviously fit for a real project.

In a "production" program you will:

- Make adjustment to consider orders with <b>any filled qty</b>
- Add commissions and apply the multiplicator
- Don't use structure sizes such as UnmatchedBuys.Count in the unrealized computations (which does not carry to the more general case) but the actual qty count
- Bring some variables out of the procedure, in order to update the PNL computations only when necessary (when there are no fills only the unrealized needs to be updated obviously)

These are all straightforward changes for anyone, except the first one (which requires just some little refinements for quantity matching).

If you run the above program having filled the 2 stacks of orders with the same prices we used in our example (SELL @ 2000, 7000, 12000), (BUY @ 10000 ) and current ASK = 5000:

Code: 
        Me.UnitOrders_SELL = New Stack(Of UnitOrder)
        With Me.UnitOrders_SELL
            .Push(New UnitOrder(2000))
            .Push(New UnitOrder(7000))
            .Push(New UnitOrder(12000))  'top of stack (0)
        End With

        Me.UnitOrders_BUY = New Stack(Of UnitOrder)
        With Me.UnitOrders_BUY
            .Push(New UnitOrder(10000))
        End With

you get the following results:
Code: 
----------------------------------------
PositionSigned:              -2
QtyInGoodScalps:             1
QtyInBadScalps:              0

PermanentGain:        2000
PermanentLoss:        0
Unrealized:           -1000
                      ----
PNL:                  1000
which we already know, from our "manual" example.

So the new version of the application will have this new logic concerning the decomposition of PNL.
I have been doing massive changes and completely revised the testing facilities and all the performance indicators, to explore better the impact of volatility.

Motivated, by the high volatility of the last month, I have done also many <b>strategic changes</b> (i will discuss these in next posts) to reinforce the hedging on vertical moves. The main goal is to "neutralize" as much as possible the "first round" of any vertical move, and continue profiting, as we have done thus far on scalps and retracements (which can also be seen as "backward trend riding").

Thanks so much for all the so many people helping with this project and testing my work as: i will be updating soon.

Tom
 
Here is what may look like the above decomposition of PNL:

Gain = thick dotted green
Loss = dotted red
Unrealized = dotted blue/green
PNL = solid cyan

<img src="http://www.elitetrader.com/vb/attachment.php?s=&postid=3268289" width="1200" />
 

Attachments

Hi Tom,

Great work, the new version looks good! After looking at G-Bot trading over the past few days I can see that your new PnL decomposition makes more sense than your actual one.
It is good to see while in drawdown that the bot is still scalping and making money trying to fight his way to being profitable again!

What about adding the realized pnl on top of these measures? It might be useful too (maybe through an option)? Or do you think that the chart would be too crowded with it?

Anyway the new version looks quite promising, thanks for your hard work!

Laurent
 
Thank you Laurent!

This PNL decomposition is actually:

PNL = (Realized) Gain + (Realized) Loss + Unrealized

where i use the words "Gain" and "Loss" are short for "Realized gain" and "Realized loss" (please correct me if you have better wording).

So we have that:

Realized = (Realized) Gain + (Realized) Loss

One of the reasons to split the "Realized" is to separate gain from losses, so that we can see the gain grow.

If we plot the realized too, we have what we saw in the previous charts that we tried to improve (realized may go up and down simply due to manual orders, rollover, stop etc. )

Clearly, this decomposition is for us to help trading, improve performance metrics and and also provide a feeling of the actual bot scalping action (which is also a tremendous psychological help). For the definition of Realized / Unrealized which is needed for <b>tax purposes</b> one has to make reference to the <b>Broker official statement</b>.

As we noted, for taxes they use criterions such as first in first out (FIFO) matching, or the so called "specific share identification", see for instance:
http://www.irs.gov/pub/irs-pdf/p550.pdf
clearly these bureaucratic definitions are also good to know, as they have a nonnegligible impact on taxes. [ It would also be nice if anyone would provide more info about this point. It would also be nice to see the actual code used for tax computations. ]

Order matching is however a little "arbitrary" and just the consequence a convention. Our new matching process works instead by first matching the positive scalps starting with the higher prices. This removes the ambiguity of the matching process and above all allows us to actually see how the scalping is going and effectively separate the permanent "losses".
For instance, if we make a manual losing trade (stop), it will go (separately) in the (realized) loss curve.

This approach isn't completely perfect, because as the trading evolves we might occasionally have (temporary) order matching which are somehow different from what we expected at first glance (because the higher prices tend to be matched first). [I will show this effect in a next post] However, among the various ideas, this seems at the moment the one which is more helpful for the Trader and the Quant. Clearly, if we find a better way or some refinement we will readily adopt it!

Tom


Quote from ryker:

Hi Tom,

Great work, the new version looks good! After looking at G-Bot trading over the past few days I can see that your new PnL decomposition makes more sense than your actual one.
It is good to see while in drawdown that the bot is still scalping and making money trying to fight his way to being profitable again!

What about adding the realized pnl on top of these measures? It might be useful too (maybe through an option)? Or do you think that the chart would be too crowded with it?

Anyway the new version looks quite promising, thanks for your hard work!

Laurent
More...
 
So, just as promised, we came out with a profit from the worst period in several years, juggling a large folio of 13 instruments (without stops).

The 2 significant drawdown were caused 2 large cycles of investment (mainly RB first, and GC). In real trading, these would have dealt with by opening a manual clone and trading against the bot. Anyway we "took the blow" (as Franks says), just to see as far it could go.

This just to remember to "stop fans", that using <b>systematically</b> (and let me repeat, systematically) stops is the way to <b>give up the only thing that makes trading fair</b>, for an investor with <b>relatively</b> good capital.

[We never let a penny in the mkt, and ended up profitable each and every session so far.]

Ok now that i have given evidence to support this point, i will switch to another strategy version, which i have been working during holidays and that i am currently refining. The aim of the new variation is to address the most unpleasant aspect of this approach, which are clearly drawdowns. Trading this way, besides decent capital, requires a lot of "balls" from the trader side, and balls are even harder to find than capital. ;-)

Tom

<img src="http://www.elitetrader.com/vb/attachment.php?s=&postid=3281485" width="1200" />
 

Attachments

So about the new strategy.

First of all have been distributing to all my friends helping me with testing and to the traders, the new version. I will redistribute soon a another build with further strategic improvements.

In this new version i am totally focused on the dd issue, especially under high volatility, which is obviously the only thing which may limit usability to a small circle of investors and fund managers.

We can't expect to make dd disappear by magic, as it is often the source itself of profits (investment), however, i will do all i can to make it as low and improbable as it is possible, without, clearly losing profitability.

If you look at the screenshot below (new strat), you will notice how i have modified the exit set of rules to allow quicker exits.
For instance the session shown below is running since 3 days and i have a position of only 1 !

This gives you a first feeling of how careful this approach is aimed to be. CL made a total of 22 orders, and the column chart shows the number of orders of the other instruments.

These quicker exits, turn out to help the bot riding the big moves, trying rather to focus on scalping than investing.

I have done several other improvements, as adding <b>volume, volatility</b> (see bottom line of trading info) and cointegration computations (which may turn useful in future, if we ever add pair traders).

[The entire backtesting system has been recoded, and now the GBM have also the possibility to be run with randomized volatility, which also indicates my current focus.]

Tom

<img src="http://www.elitetrader.com/vb/attachment.php?s=&postid=3281498" width="1200" />
 

Attachments

You must log in or register to reply here.
Back
Top