Robotrading: CT + Trending Strategy on folios of futures

fullautotrading · May 21, 2011

Ok let's take the discussion from where we left it.

We have defined ABS_SPEED as the number of scalps crossed per unit of time. Say that, for instance, SI is crossing averagely 2.5 scalp lines per hour, so its ABS_SPEED is 2.5. [Or we might say 60 scalps per day, if we prefer.] This is immediately understood by a trader, much better than a volatility figure. Also it helps understanding any current danger for potential drawdowns, immediately in terms of actual money.

Now, for risk management, we also need to know if this "absolute speed" is mostly expressing itself SIDEWAYS or DIRECTIONALLY.
So we split the ABS_SPEED in 2 components:

ABS_SPEED = S * ABS_SPEED + D * ABS_SPEED

with S + D = 1, being the 2 "coefficients" that express the Sideways (S) and Directional (D) components, respectively. [Clearly, we are talking of realtime measures]

Now we need to define at least D. Then, we can get: S = 1 - D.

What is D ? We want to separate the "directional component" from the total abs speed.
Here is my proposal. (Feel free to correct or make alternative proposals.)

I define the coefficient D as the "normalized number of different scalp lines crossed".

Why ? Imagine the price moving. If the price has a strong directional component, for instance a violent rally up with no retracements, we will have all prices, in the hit scalp lines, different. On the contrary, if the price is directionless, it will keep hitting the same prices.
Therefore, i propose to derive the Directional component, based on the number of different Hits, say d.
To find, D, we need to "normalize" this number d, to have it in the range [0,1].

So we proceed this way. Assume that the number of Hits, say N, used for speed computation is even. The maximum for D is clearly equal to the number of Hits, as they can actually be occurring at all different prices:
max d = N

About the minimum of D, we have to notice that it is not 0. This is because we have the constraint that adjacent Hits must have different price. Therefore, we conclude that the minimum for D is equal to the half of the Hits:
min d = N / 2

With this information, we can obtain D, by "normalizing" d, in order to make its range in the interval [0,1]. We get:

D = ( d - min d ) / ( max d - min d )
= ( d - N / 2 ) / ( N - N / 2 )
= 2 * d / N - 1

So we have:

D = 2 * d / N - 1
S = 1 - D = 2 * ( 1 - d / N )

and the decomposition:

ABS_SPEED = S * ABS_SPEED + D * ABS_SPEED

This should be interesting, as it allows us to quantify separately the 2 components: "directional" and "sideways".

So we can define 2 distinct abs speeds:

ABS_SPEED_Directional = ABS_SPEED * ( 2 * d / N - 1 )
ABS_SPEED_Sideways = ABS_SPEED * 2 * ( 1 - d / N )

where:
 ABS_SPEED = ( N - 1 ) / ( Total time between hits ) 

It is now interesting to derive another metric for a direct comparison of Directionality Vs Sideways components.

I will propose that in a next installment, after possibly hearing also your thoughts ;-))

Tom

PocketChange · May 21, 2011

Tom,

Your speed indicator is explicitly tied to a fixed scalp size.
If you want to change the scalp size from $250 to $400 for example you will need to reprocess 2 days of price history.

You can calculate from your scalp line price history any even increments of your scalp size. ie. $250, $500, $750 etc.

You may want to consider capturing the minimum common denominator scalp size for purposes of these metrics. ie. $50 then you can analyze and adjust your scalp size for trading to any increment of $50.

As part of your folio selector process different scalp increments and have the tool suggest a folio and recommend a scalp size.

fullautotrading · May 21, 2011

Quote from PocketChange:

Tom,

Your speed indicator is explicitly tied to a fixed scalp size.
If you want to change the scalp size from $250 to $400 for example you will need to reprocess 2 days of price history.

You can calculate from your scalp line price history any even increments of your scalp size. ie. $250, $500, $750 etc.

You may want to consider capturing the minimum common denominator scalp size for purposes of these metrics. ie. $50 then you can analyze and adjust your scalp size for trading to any increment of $50.

As part of your folio selector process different scalp increments and have the tool suggest a folio and recommend a scalp size.

Right pocketchange. Very insightful observation! :-)

There is an even simpler solution than contemplating all possible subdivisions.

As i stated at the beginning i have made the simplifying assumption the scalp sizes were the same. This just to present the idea.

Let's see how we can deal with the general case of unequal scalp sizes.
Now, under the fixed scalp hypothesis, the ABS_SPEED could be defined as number of scalps per hour.

With different scalps, a more meaningful definition would: (Value of scalps per hour)
So we have, in the general case, denoting by A the amount of the scalp (for instance $250)

ABS_SPEED = ( N - 1 ) * A / ( Total time between hits )

This generalizes the question, and all the other devepments and decomposition coefficient can remain unchanged.

This would allow speed comparison across multiple instruments (if scalp sizes are different) or across time for the same instrument (if its scalp size is modified).

Furthermore, the intuitive meaning is even more meaningful (money per hour) [ a sort of "$ speed", as one of my French friends likes to refer to the concept ;-)) ].

Tom

PocketChange · May 21, 2011

As you increase scalp size you will have to parse the data for unique adjacents of the larger scalp increments otherwise you will have erroneous counts.

You may have a count of 20 at $250 but only 0 or 1 at $750.

Quote from fullautotrading:

Let's see how we can deal with the general case of unequal scalp sizes.
Now, under the fixed scalp hypothesis, the ABS_SPEED could be defined as number of scalps per hour.

With different scalps, a more meaningful definition would: (Value of scalps per hour)
So we have, in the general case, denoting by A the amount of the scalp (for instance $250)

ABS_SPEED = ( N - 1 ) * A / ( Total time between hits )

This generalizes the question, and all the other devepments and decomposition coefficient can remain unchanged.

This would allow speed comparison across multiple instruments (if scalp sizes are different) or across time for the same instrument (if its scalp size is modified).

Furthermore, the intuitive meaning is even more meaningful (money per hour) [ a sort of "$ speed", as one of my French friends likes to refer to the concept ;-)) ].

Tom [/B]

fullautotrading · May 21, 2011

Quote from PocketChange:

As you increase scalp size you will have to parse the data for unique adjacents of the larger scalp increments otherwise you will have erroneous counts.

You may have a count of 20 at $250 but only 0 or 1 at $750.

The logic here is mainly to allow a comparison across instruments which may be working with different scalp sizes.

It would be very uncommon changing the scalp size while trading, as a trading strategy is usually created with reference to a given scalp size (as it affects both profits and drawdown).

Speed and directionality are not absolute concepts, but relative to the scalp size. In other words, what may look like a "trending market" for a trader with a smaller scalp size, might look like a sideways market to a trader with using a larger scalp (which has a "larger" perspective).

The purpose of the methodology is just to make the concept of speed and directionality relative to the scalp size, and not to a generic timeframe (which would not be useful for strategic purposes).

Tom

fullautotrading · May 21, 2011

hi

As anticipated, I introduce now another interesting metric.
So we have defined: ABS_SPEED = ( N - 1 ) * A / ( Total time between hits ) and we might perhaps call it expressively "$ SPEED", as my friend L. would call it ;-))

Now, assume I want also to make a direct comparison between the 2 components: "Directional" and "Sideways" and i want to know which one is prevailing.

So i define a "DIRECTIONALITY index" working as follows. The index will vary in the interval [-1 , 1] where 1 = maximum directionality (trendiness) while -1 means maximum sideways behaviour. I like it to possibly be positive and negative, because also visually it's quite expressive to have 2 different signs for the 2 components.

Now, how can i derive this directionality index ?

I will show you 2 ways, and how they both lead to the same result ;-)

Method 1.
=========

We can see it as a relative difference between the 2 components we just derived:

DIRECTIONALITY index = ( ABS_SPEED_Directional - ABS_SPEED_Sideways ) / ABS_SPEED
= ( ABS_SPEED * ( 2 * d / N - 1 ) - ABS_SPEED * 2 * ( 1 - d / N ) ) / ABS_SPEED
= ( ( 2 * d / N - 1 ) - 2 * ( 1 - d / N ) )
= 2 * d / N - 1 - 2 + 2 * d / N
= 4 * d / N - 3

So we get:
DIRECTIONALITY index = 4 * d / N - 3


Method 2.
========

We use a similar reasoning (linear transformation) used to derive the D coefficient.
The only difference is that now we want it to vary in the interval [-1, 1] instead of the interval [0,1].

So we define
DIRECTIONALITY index = a * d + b

where we must have:
a * d + b = 1 when d = max d
a * d + b = -1 when d = min d

so we get:
a * N + b = 1
a * N/2 + b = -1

hence:
b = 1 - a * N
a * N/2 + 1 - a * N = -1

and finally:
a = 4 / N
b = -3

(another simpler way to look at it, could have been taking the D coefficient and doing the transformation 2 * D - 1 to double the range and recenter on the origin. In fact, D = 2 * d / N - 1 and 2 * D - 1 = 4 * d / N - 3)

Therefore:
DIRECTIONALITY index = a * d + b = d * 4 / N - 3
Which is the same result as in Method 1.

So we have now a nice (unitless) index which can tell us whether "directionality" (POSITIVE SIGN) is prevailing against "SIDEWAYS" behavior (NEGATIVE SIGN). And the index varies from -1 to 1.

Clearly, the index could be "reversed" if we liked the positive sign to signify "sideways": 3 - 4 * d / N (being "centered" on the origin, it sufficies to change the sign).

And, actually, i think it's preferable to us having the positive sign on "sideways", as this is the most profitable situation for our games. So let's rather consider:

SYDEWAYS index = 3 - 4 * d / N 

Tom

LeeD · May 22, 2011

Hi Tom,

Quote from fullautotrading:

I think i have already mentioned this before.
( see for instance http://www.elitetrader.com/vb/showthread.php?s=&threadid=208265&perpage=6&pagenumber=17 )

while i keep the discussion of ideas and the project public, my app is distributed on a strictly private basis, in free * personal * copies to the people who wish to have it.

I figured out that much from Gbot Web-page http://www.datatime.eu/public/gbot/default.htm My question was regarding the source code. Thanks for unswering it at length!

Quote from fullautotrading:

Furthermore, i work on it and make substantial improvements on a daily basis, as you can also see from this thread, and therefore the concept itself of "source" would remain undefined and make actually no sense, as it changes and grows on a daily basis. I make a point to increase its performances everyday (even of an infinitesimal amount).

[Not to mention that we are talking here of millions code lines of a multithreading async app, with a complex overlay of multiple, multi-agent, strategies, which would probably not be really understandable to many people, even if they were commented line by line ;-)) ]

Personally, i don't even believe in the open source myth, and i instead consider it a coward commercial approach, of selling services, instead of being fully responsible for high quality applications, but that's just my opinion ...

The basic philosophy behind giving software away for free is simple. Compared to tangible goods, the cost of replicating software (i.e. copying a file) is negligible. So, if you have written a piece for personal use, you may want to give it away because it doesn't cost you anything... and it may save someone else substantial effort.

Once you have given it away, users may start requesting bug fixes and features. However, if you've given away the source code, these same peopel may actually fix the bugs or implement new features and share their improvements with you. This way you as the original author may receive benefits you wouldn't have seen if you didn't give the source away.

I take the claim regarding "millions code lines" with a grain of salt. For comparison, TradeLink (a perfect example fo what you call "coward commercial approach") looks like much larger a project and contains in the range of 80,000 lines. However, there is now doubt Gbot is a large project and as such is bound to contain bugs. So, the more people are using it, the more likley they encounter and report any issues. You actually do achieve this with offering the binaries "for private use".

Originally, I was under impression Gbot was what people call a "platform"... and futures trading strategies were separate from it. So, the platform could be "open source" and the trading decision algorthms private... With trading decisions there is a liquidity issue. So, you don't want too many people doing the same trades as you (unless you plan front-running those using the "public" version). With opening the "platform", the author doesn't loose anything except for opportunity to sell (license) it later.

Quote from fullautotrading:

There are currently funds and investors currently trusting it to trade real money and it is responsible to deal with these capitals in an efficient and reserved way: clearly, nobody would use anymore an open source application, and that is clearly in contrast with my wish.

Clearly, if you plan to monetise Gbot in some way (either by publishing academic research or by evolving into an asset manager), there is little incentive to go open-source.

fullautotrading · May 22, 2011

Quote from LeeD:

Hi Tom,
I figured out that much from Gbot Web-page http://www.datatime.eu/public/gbot/default.htm My question was regarding the source code. Thanks for unswering it at length!
...

The basic philosophy behind giving software away for free is simple. Compared to tangible goods, the cost of replicating software (i.e. copying a file) is negligible. So, if you have written a piece for personal use, you may want to give it away because it doesn't cost you anything... and it may save someone else substantial effort.
...

> it may save someone else substantial effort.

Why on the hearth should i allow other people save "substantial effort" ? Even as an educator, that would be a bad thing to do.
I am already expounding many ideas here. Should i also serve cooked source code ? ;-))
There is only one way to learn and progress. And it is through hard work, and not by copying and pasting other people ideas.
I have already given tons of code (in other areas) and scientifical papers to the community (and actually i did not see any real good coming).

The app has now enough users working with it, that even the smallest bug would be immediately caught and immediately fixed. And when i say immediately, it's immediately, not the "next release". When people money is concerned there is <n>no room even for the smallest slip.

The app is to me only a way to ship strategies. It's much different from other projects, where they ship the "platform" without strategic ideas (and it's not hard to figure out why).

>"it doesn't cost you anything?".

Ah ah. Do you think that i am working 24/7, not having vacations, not sleeping, and divorcing from wife ;-))) to code for the some monkey who would hopelessly copy and paste my work without even having the slightest idea of what it signifies ?
I don't do charity. People have to work hard, to achieve something. When trading is concerned everything, ideas and success is to be measure in actual results.

Tom

fullautotrading · May 22, 2011

Let me recap the results so far.

I have been discussing with my friend J. from Chicago and he has made a very nice suggestion, to create a signed unitless index which works like that:

-1 (negative trend) ------- 0 (sideways) ------- +1 (positive trend)

This is a beautiful idea. And we can't help pursuing it. ;-)

Let me also fix something in the previous development in the derivation of the coefficients D and S. I think i mixed up the maximum of distinct equal values (N/2) with the minimum of different values, so the correct formulas are:

S = 2 * DE / (N + r)
D = 1 - S

where DE is the number of the distinct duplicated (equal) values in the Hit Point list, and r is 0/1 according to N being even or odd.

All the other relationships, which i recap below are fine:

1. ABS_SPEED or "$ SPEED" = ( N - 1 ) * A / ( Total time between hits )

2. ABS_SPEED = ABS_SPEED_Directional + ABS_SPEED_Sideways = D * ABS_SPEED + S * ABS_SPEED, D + S = 1

3. ABS_SPEED_Sideways = S * ABS_SPEED
4. ABS_SPEED_Directional = D * ABS_SPEED

5. SYDEWAYS index = (ABS_SPEED_Sideways - ABS_SPEED_Directional ) / ABS_SPEED = S - D
6. DIRECTIONALITY index = (ABS_SPEED_Directional - ABS_SPEED_Sideways ) / ABS_SPEED = D - S

where, in particular:

S = 2 * DE / (N + r)
D = 1 - S

1. 3. 4 are expressed in $ per hour
5. 6. are unitless indexes in the range [-1, 1]

Following the suggestion by J., let's create another indicator which is near 0 in sideways markets and near -1, 1 in directional markets. Let's call it SIGNED_DIRECTIONALITY index.

Having done the previous development, the task might have a simple solution:

Let's denote by "Sign" the sign (1/-1) of the difference: LastHitPoint - FirstHitPoint

SIGNED_DIRECTIONALITY = Sign * ( D - S + 1 ) / 2

infact, it sufficies to translate the previous directionality index and reduce it a unitary range.

(This can be seen as a normalized form of signed "speed".)

I guess that smarter ways to compute the "Sign" could and should be devised. Maybe you have suggestions ...

Tom

<img src="http://www.elitetrader.com/vb/attachment.php?s=&postid=3191621" width="1200" />

PS.
(picture shows simulated instruments, being now the mkt closed. As soon as it opens we can study and tune better the various indexes and find possible bugs ... )

fullautotrading · May 22, 2011

btw,

i think there is another detail, i did not explain. You see i added the r variable to deal with both even and odd N. In fact, at a certain point we had previously assumed an even number of Hits in our list. This is an assumption we would think we can make because we can decide the cardinality of the scalp line set where we carry out the computation. For instance, we could decide to compute the metrics on the most recent N = 10 or N = 20 hit points.

... * * * * * * * * * *

everytime a new hit point is added to our list when it has size, say 10, we drop the first (and oldest) hit point of the list, thus keeping the list "trailing" the last price and updated with most recent prices.

Before the N Hit Points are collected, we might still be wanting to start computing the metrics, however we specify a minimum number of Hit Points to do so, because if the number is too
small the computation wont make actually sense. So we could say, for speed computation min N >= 2.

For the sideways/directionality index, however N = 2 is still too small, because adjacent Hit Points have different prices by design. So we might say that min N >= 4, for all the metrics to start making sense (in practice, we would pick an higher threshold).

If we allow computations before arriving to the target size of the Hit point set, or if we dinamically allow changing it (to the trader), we have to consider also an odd number of points N.
So wee need to generalize our expressions. So, in brief, this is the reason why i added the remainder of the division by 2 (in the S formula):

r = N mod 2.

Tom

Robotrading: CT + Trending Strategy on folios of futures

Attachments