Both open and close
Fully automated futures trading
- Thread starter globalarbtrader
- Start date
Both open and close
About 4K was Oil, but there was more to it than that.
Here are yesterdays moves, fifth column is move in standard deviation terms, next column is my signal:
Code:
Headline chanages: (norm price change>1 sigma)
code last_price datetime_last_price last_norm_return rawsignal rawsigchange multisignal multisigchange
28 MXP 0.055 2016-01-15 19:01:25 -2.46 -1.21 0.063 -12.6 -0.273
24 AUD 0.684 2016-01-15 18:53:32 -2.30 -1.51 -0.135 -15.8 -0.062
34 CRUDE_W 36.335 2016-01-15 18:10:44 -2.22 -3.00 0.000 -37.6 0.535
20 CAC 4179.000 2016-01-15 16:15:39 -2.21 -1.37 -0.145 -9.4 -0.005
19 AEX 399.450 2016-01-15 16:14:14 -2.17 -1.23 -0.218 -8.5 -0.007
26 GBP 1.426 2016-01-15 18:57:16 -2.13 -2.52 -0.061 -26.3 0.071
37 EUROSTX 2934.500 2016-01-15 16:17:01 -2.13 -0.68 -0.034 -0.0 0.000
22 NASDAQ 4100.375 2016-01-15 17:48:25 -2.10 -1.56 -0.214 -10.6 0.249
2 LIVECOW 117.375 2016-01-15 18:18:59 -2.01 -0.92 -0.047 -9.6 0.168
21 SMI 8009.000 2016-01-15 16:18:26 -1.99 -0.23 -0.235 -1.6 -0.004
23 SP500 1857.875 2016-01-15 17:50:05 -1.94 -1.19 -0.201 -8.1 0.199
18 KOSPI 228.575 2016-01-15 04:42:30 -1.79 -1.74 -0.214 -11.8 -0.124
30 COPPER 1.943 2016-01-15 19:04:54 -1.10 -2.53 -0.069 -31.6 0.119
8 BTP 138.445 2016-01-15 16:02:19 1.07 1.69 0.225 15.0 -0.253
10 OAT 151.690 2016-01-15 16:05:05 1.15 2.09 0.209 18.6 0.037
7 BOBL 131.515 2016-01-15 16:00:55 1.21 0.75 0.044 3.3 -0.001
11 SHATZ 111.642 2016-01-15 16:06:27 1.28 1.07 -0.001 0.0 -0.000
15 US5 119.996 2016-01-15 18:46:53 1.36 1.76 0.139 3.9 0.011
12 US10 128.445 2016-01-15 18:41:25 1.46 1.37 0.132 3.0 0.005
14 US20 159.266 2016-01-15 18:45:02 1.48 1.00 0.213 2.2 0.007
36 EDOLLAR 98.303 2016-01-15 19:14:24 1.52 1.84 0.040 22.9 0.025
16 V2X 27.750 2016-01-15 15:19:34 1.97 0.38 0.003 3.0 -0.007
13 US2 109.090 2016-01-15 18:43:09 2.06 1.93 0.003 4.2 0.011
Wrong
25 EUR 1.096 2016-01-15 18:55:23 1.13 -0.11 0.325 -1.1 -0.434
17 VIX 23.725 2016-01-15 18:51:46 1.82 -0.53 0.055 -4.2 -0.002
1 LEANHOG 77.975 2016-01-15 18:17:23 -1.07 0.23 0.011 2.4 -0.041So it was more a case that all the stars were aligned in the right direction. Of course this could equally go the other way...
GAT
I've now added the ability to estimate forecast scalars using a rolling window to the code for pysystemtrade.
GAT
Hi GAT,
Looking at your above formula, it is not clear to me what you mean by L/4 smooth - can you elaborate? Why 4 and where does the L/4 smooth enter the EWMA? I assume you then volatility standardise this EWMA.
Also, in order to obtain forecast scalars that are on average equal to the absolute value of 10, you mention that 'you need to backtest the behaviour of the rule with real data to find average forecasts'. Is this backtesting conducted for each individual instrument in isolation of the others or across all instruments? Depending on what it is, the outcome at the instrument level wil be different:
i)if it is at the instrument level in isolation of other instruments, each instrument will have a forecast scalar equal to the average absolute value of 10, which ultimately leads to all instruments having an average absolute forecast scaler of 10
ii) if it is at the portfolio level across all instruments, the average across all instruments will equal 10 but the average absolute value for individual instruments will not necessarily equal 10
Last edited:
It's probably easier to show you my code.
Code:
def breakout(x, ws):
max_x=pd.rolling_max(x, ws, min_periods=min(len(x),int(ws/2)))
min_x=pd.rolling_min(x, ws, min_periods=min(len(x), int(ws/2)))
sig=[breakout_one_row(idx, x, max_x, min_x) for idx in range(len(x.index))]
sig=pd.TimeSeries(sig, index=x.index)
sig=pd.ewma(sig, span=int(ws/4.0), min_periods=int(ws/8.0))
return sig
def breakout_one_row(idx, x, max_x, min_x):
r_px=x[idx]
r_min=min_x[idx]
r_max=max_x[idx]
return 4*(r_px - mean([r_min, r_max]))/(r_max - r_min)I'd do it by pooling across instruments (by coincidence just blogged on this http://qoppac.blogspot.co.uk/2016/01/pysystemtrader-estimated-forecast.html)
Hi, you write on your blog about scalar estimation 'The odd one's out are V2X (with a very low scalar) and Eurostoxx (very high) - both have only a year and a half of data - not really enough to be sure of the scalar value.'
I downloaded data for the Eurex Eurostoxx 50 from Quandl (https://www.quandl.com/collections/futures/eurex-euro-stoxx-50-index-futures) going back to Sep 1998. Are you talking about a different contract?
This thing shows some peculiarities. Here is a plot of it from my system:
Settle is the panama adjusted settle, OSettle the 'original' settle price.
Carry shows the difference to the next contract, adjusted to annual (In this case multiplied by four, since ESTX50 has quarterly contracts).
Do you have any idea why the March contract shows those regular spikes?
Seems to be really consistent over the years.
Other contracts like DAX, SMI don't show this, the CAC40 also looks strange.
I downloaded data for the Eurex Eurostoxx 50 from Quandl (https://www.quandl.com/collections/futures/eurex-euro-stoxx-50-index-futures) going back to Sep 1998. Are you talking about a different contract?
This thing shows some peculiarities. Here is a plot of it from my system:
Settle is the panama adjusted settle, OSettle the 'original' settle price.
Carry shows the difference to the next contract, adjusted to annual (In this case multiplied by four, since ESTX50 has quarterly contracts).
Do you have any idea why the March contract shows those regular spikes?
Seems to be really consistent over the years.
Other contracts like DAX, SMI don't show this, the CAC40 also looks strange.
Over time Quandl have been backfilling their data. When I first tried to get this from quandl, a couple of years ago, they didn't have any Eurostoxx data. So I had to just use the IB data.
At some point I'll add this extra data.
The carry 'bumps' are due to the different timing of dividend payments. It affects different contracts to different degrees. This is a well known effect and not a problem with the data.
GAT
Thank you! Do you have some more information why ESTX50 is so strongly affected by this?
The DAX is a performance index (includes dividends), is this why it doesn't show that pattern?
There is probably no easy way to use this for trading?
Another thing: Your code for the breakout rule above uses Python list comprehensions along the series axis.
I think this horrendously slow and not really needed.
You can write it fully vectorized (loops in pandas/numpy C code) like this:
(stochastic here is scaled to -0.5 .. 0.5)
The DAX is a performance index (includes dividends), is this why it doesn't show that pattern?
There is probably no easy way to use this for trading?
Another thing: Your code for the breakout rule above uses Python list comprehensions along the series axis.
I think this horrendously slow and not really needed.
You can write it fully vectorized (loops in pandas/numpy C code) like this:
(stochastic here is scaled to -0.5 .. 0.5)
Code:
def stochastic(data, column, length, smooth):
roll_max = pd.rolling_max(data[column], length, min_periods=length // 2)
roll_min = pd.rolling_min(data[column], length, min_periods=length // 2)
stoch = (data[column] - roll_min) / (roll_max - roll_min) - 0.5
if smooth > 1:
return pd.ewma(stoch, span=smooth)
else:
return stochYes the DAX includes dividends; so the effect doesn't apply. I think the yield on the Eurostoxx is quite high. I guess also there are more firms paying semi-annual dividends (quarterly divis rarer in europe than the US).
Thanks for the code tip. Speed isn't a concern with a daily system, but it's always nice to have more efficient code even for the sake of it.
GAT
About forecast scalars:
I am using artificial random data to estimate the scalars and correlations between rules.
For example, I created 1000 different random walks, each 100000 bars('days') long and applied trading rules to them:
The 50% value is the median along the 1000 tries (data is from the pandas describe() function).
ewmac256 means the EWMA 256/64 rule and so on.
The medians are lower than the values that you give in your book (1.87, 2.65, 3.75, 5.3), probably reflecting the fact that the artificial data has no trends.
But even with 1000 tries of 100000 days, i.e. about 400 years, you can see the error in the values is still significant.
I wonder if it is really possible to get stable values from real data where we only have maybe 40 futures and 50 years of data.
I am using artificial random data to estimate the scalars and correlations between rules.
For example, I created 1000 different random walks, each 100000 bars('days') long and applied trading rules to them:
Code:
count mean std min 25% 50% 75% max
scalar_ewmac256 1000 1.628219 0.070967 1.420118 1.581897 1.626521 1.678534 1.827633
scalar_ewmac128 1000 2.301430 0.071873 2.092129 2.253581 2.300945 2.349642 2.524300
scalar_ewmac64 1000 3.262073 0.073323 3.039892 3.211492 3.261055 3.313205 3.511254
scalar_ewmac32 1000 4.639525 0.075807 4.417496 4.589838 4.636334 4.694308 4.875523The 50% value is the median along the 1000 tries (data is from the pandas describe() function).
ewmac256 means the EWMA 256/64 rule and so on.
The medians are lower than the values that you give in your book (1.87, 2.65, 3.75, 5.3), probably reflecting the fact that the artificial data has no trends.
But even with 1000 tries of 100000 days, i.e. about 400 years, you can see the error in the values is still significant.
I wonder if it is really possible to get stable values from real data where we only have maybe 40 futures and 50 years of data.