Update time.
Performance wise, nothing to write home about eithier way.
For the calendar year I'm up 4.4%, so a smidgen on my last update. For the financial year I'm up 6.3%.
For context, gere's some extracted bullet points from my last couple of updates:
New markets
As of now there are 77 markets with price data in my database, with another four for which I've downloaded data but have not yet set up, plus one that I'm currently in the middle of downloading data for.
One unforseen consequence of doing this is that my system process timing got out of whack, because it was taking much longer to get all the prices, which meant my other processes weren't running until after midnight (when I start trading again - I was operating with a 'closed period' between 8pm and midnight to do all my daily stuff, and that wasn't long enough).
As a quick fix I've moved the start time for daily price collection earlier (to 3pm, which means I won't be getting closing prices for that day), but I have a better fix in mind which involves only collecting prices for currently trading instruments on a daily basis, and getting a full update at weekends.
Replacement markets
After some dilly dallying (see this thread for discussion), I replace the following markets with their micro or mini relations:
- CRUDE_W
- GAS_US
- GOLD
- KOSPI
- NASDAQ
- SP500
I'm not 100% sure about the KOSPI mini, since I've had a couple of bad fills when trading it which make it look expensive, but for now I'm going to monitor the situation.
"New" system
So today I implemented my 'new' system. I say, 'new' but it's very similar to the old strategy. Here are all the changes I've made, most of which are discussed above:
Notice that there are quite a few ideas from above that got binned, basically because I decided not to implement the complex dynamic optimisation methodology for dealing with small capital constraints. I've tried to avoid making the system more complex; sure I've added a few things but also removed things, so overall it's not that different from before.
Instrument selection
I've got a more rigorous approach to instrument selection with some better diagnostics.
First of all there are the instruments I don't consider at all or calculate forecasts for. These are the large contracts for which I currently trade small contracts, listed above. Including them in the system would result in double counting for cross sectional forecasts and messed up correlation matrices. I continue to sample prices however, in case I decide to revert back to the larger contracts for some reason.
There are also some instruments I can't or don't want to trade (but which I can calculate forecasts for, so they will enter into cross sectional calculations). Right now this includes US real estate (rule 871), and the European sector futures (which I'm reserving for a future cross sectional system).
That brings us to instruments which I could trade, but which are too expensive or illiquid. I've added some new functionality to help here. Firstly, I've added a new method which samples bid/ask prices every hour for all instruments which are currently open. There are a few possible uses for this data, eg monitoring liquidity throughout the day and deciding when it's best to trade, but it's current use is in a new costs report. This report compares the median bid/ask spread from 3 sources:
(I can also use 1 vs 2 to evaluate the success of my execution algo but that's for another day)
Once I'm happy with the configured spread, the next part of the costs report shows me the current calculation of costs in SR terms per trade. I use this to exclude any instruments with a cost of greater than 0.01 SR units per trade.
The other new report I have is a liquidity report, that shows liquidity sorted by contracts per day (<100 is a problem for me), and daily trades as units of annualised risk (I'm unhappy with less than $1.5 million).
So now I've got a list of instruments that I will exclude from consideration (but I will still calculate forecasts for):
Finally I generate instrument weights in the normal way. I will also do one last check to make sure that I can currently take at least one contract in every instrument at maximum forecast.
Here's the new list of instrument weights (zeros are for excluded markets). I've added a * to new instruments which I haven't been trading, but for which I've now got a positive instrument weight.
Some of these instrument weights are surprising; they are certainly a little different from what 'manual' handcrafting would have come up with. In particular there is the inclusion of both 5 year bonds and 5 year swaps. For now I decided to let them ride, since I'm planning to do RV models with swaps and I wanted to get some trading history for them.
One question is how often I go through this exercise. For example, what if an instrument I'm currently trading suddenly becomes too expensive or illiquid? I'm currently leaning towards doing this annually, with the obvious time being at fiscal year end when I evaluate my performance anyway. By year end I'll certainly have a few more instruments I can trade with prices, and there could be instruments I want to trade with spread models that I don't want to also trade here (more later).
Some thoughts about implementing system changes
There are two dangers with system changes. The first is that you'll screw something up. The second is that they cost money in trading costs.
The first defense against screwing up is having good diagnostics and staring at them a lot. So I checked and rechecked every calculated position, and in most cases they're not that different from what I currently have on. I'm reasonably confident I haven't screwed something up. However there are still steps you can take; rather than just flipping a switch and updating the configuration files, and saying 'let's go!':
But that isn't a risk I'm bothered about, so in the end I (very carefully!) did the changeover in one go, over the last hour or so.
Apart from occasional changes to instruments traded, I have made no significant changes to my system since 2014 (the implementation of pysystemtrade was designed to be a like for like replacement). I don't believe in making regular changes to a system, except where there is good reason (and changes in instrument costs and liquidity come into this category); why put yourself through unneccessary stress and pay extra trading costs?
I also feel I've rather flogged to death this particular part of the trading ecosystem (I've done a lot of research, and mostly concluded that it works just fine!), which means from now on I'll most probably be adding entirely new strategies rather than modifying my workhorse carry/trend model.
Which brings me on to...
What's next: New systems, New book
So that's where I am today and good timing, as I plan to go on holiday shortly, having rashly decided to implement my changes a few days in advance - fingers crossed. I'm quite mentally exhausted, firstly with the rabbit hole of small system optimisation, then with the less cerebral but mind numbing effort to build up and test the revised trading strategy.
Hopefully after a rest I'll be refreshed and ready to attack my next couple of projects: new trading systems and a new book. I've talked at length about the new trading systems, which fall into two main categories: a faster univariate system (principal tech challenge: using conditional limit orders and netting with my existing strategy), and a series of relative value systems (principal tech challenge: trading eithier baskets of instruments or calendar spreads rather than just outrights as I do now).
As for the book, of the half dozen or so ideas I have permanently bubbling around my brain the lowest risk one is a book specifically about trading futures, and with much more on actual trading rules than I usually include (since cynically that's what people want! And there's only so many times you can explain what position and risk management people should be doing).
I haven't yet decided what I will start on first; to an extent the book is dependent on the systems since I plan to write about my new systems in the book. However I can do them in parallel to an extent, since most of the content will relate to stuff I already know. It might make sense for me to get a book plan and deadline in place before I do anything else; it's always better to have tangible goals, plus I feel a bit 'coded out' and it would be nice to do a bit of writing in English rather than Python for a bit.
Hope everyone enjoys their summer...
GAT
Performance wise, nothing to write home about eithier way.
For the calendar year I'm up 4.4%, so a smidgen on my last update. For the financial year I'm up 6.3%.
For context, gere's some extracted bullet points from my last couple of updates:
- Right now I'm working on adding lots of markets, as per the recent blog post.... I'm continuing to add markets but clearly this is going to take a while.
- The other thing I'm doing, also flagged up in the post, is to substitute several of my existing markets for the relevant micro or mini contract. I decided to add the micro/mini instruments as additional markets rather than replacing existing ones.
- I'm going to look into my idea of modifying the system to make better use of limited capital.
- I have a bunch of other changes and potential trading signals I want to look at putting into the core model. A lot of this is stuff I've blogged about over the last couple of years but not implemented... . I added some 'fast' cross sectional momentum (I already had slow cross sectional mean reversion within asset classes, so this was just a case of reversing the sign and shortening the window). I added the skew and kurtosis rules. This also means I can take out the 'short vol bias' rule. A 'feature' that isn't strictly a new signal is the idea of attenuating momentum and carry style signals.
- The next step will be to 'build' the new system as a traditional model. So that means generating forecast scalars, and then fitting forecast weights. I have a method for fitting forecast weights that I will probably blog about. It involves fitting on each instrument individually, then clustering and fitting clusters; finally fitting across the whole set of instruments. The final forecast weights will be a blend of the weights from each of the three methods, the blend depending on the amount of data an instrument or cluster. Then it's down to calculate the FDM, and fit instrument weights (something I'll have to do regularly as I add new instruments).
- Once the system is working in the traditional model, it's a case of adding the additional optimisation stage that accounts for discrete positions. As part of the optimisation I will be including my idea for dynamically targeting risk according to average forecast strength (this is why an IDM is no longer needed).
- At this stage I also want to include my new blended vol estimate that incorporates mean reverting vol. Another thing I will have is a 'don't trade' list of instruments that is updated automatically according to my criteria for adding new markets (in practice we can trade, but it won't allow new positions to be opened); in particular volume and cost constraints (the 'too large to trade' will be dealt with by the opimisation).
- I will make some simplifications. I can remove the hacky forecast mapping that I use right now for larger contract size. And the exogenous risk overlay will also be redundant. Finally buffering will no longer be required, since the optimisation will include a cost penalty.
- Some completely new systems: fast mean reversion, intermarket RV eg spreads, intramarket RV spreads (calendar spreads).
- The next book...
New markets
As of now there are 77 markets with price data in my database, with another four for which I've downloaded data but have not yet set up, plus one that I'm currently in the middle of downloading data for.
One unforseen consequence of doing this is that my system process timing got out of whack, because it was taking much longer to get all the prices, which meant my other processes weren't running until after midnight (when I start trading again - I was operating with a 'closed period' between 8pm and midnight to do all my daily stuff, and that wasn't long enough).
As a quick fix I've moved the start time for daily price collection earlier (to 3pm, which means I won't be getting closing prices for that day), but I have a better fix in mind which involves only collecting prices for currently trading instruments on a daily basis, and getting a full update at weekends.
Replacement markets
After some dilly dallying (see this thread for discussion), I replace the following markets with their micro or mini relations:
- CRUDE_W
- GAS_US
- GOLD
- KOSPI
- NASDAQ
- SP500
I'm not 100% sure about the KOSPI mini, since I've had a couple of bad fills when trading it which make it look expensive, but for now I'm going to monitor the situation.
"New" system
So today I implemented my 'new' system. I say, 'new' but it's very similar to the old strategy. Here are all the changes I've made, most of which are discussed above:
- Adding new trading rules as detailed above. I now have 41 trading rule variations in the system, covering twelve different trading rules.
- Removing the 'short bias' rule for vol (effectively replaced by skew)
- Adding attenuation for momentum and carry signals
- Replacing the vol estimate with one that uses both recent and a long run historical average
- Removed thresholding and the exogenous risk overlay
- Forecast weights are fitted using an average of instrument specific, asset class pooled, and globally pooled data.
- All instruments with data are included in the system, but only ~30 will have non zero instrument weights. This means that cross sectional signals within asset classes will be more accurate.
- Instruments are selected from the possible universe using the static optimisation technique I found to be the best in my series of posts on this topic.
Notice that there are quite a few ideas from above that got binned, basically because I decided not to implement the complex dynamic optimisation methodology for dealing with small capital constraints. I've tried to avoid making the system more complex; sure I've added a few things but also removed things, so overall it's not that different from before.
Instrument selection
I've got a more rigorous approach to instrument selection with some better diagnostics.
First of all there are the instruments I don't consider at all or calculate forecasts for. These are the large contracts for which I currently trade small contracts, listed above. Including them in the system would result in double counting for cross sectional forecasts and messed up correlation matrices. I continue to sample prices however, in case I decide to revert back to the larger contracts for some reason.
There are also some instruments I can't or don't want to trade (but which I can calculate forecasts for, so they will enter into cross sectional calculations). Right now this includes US real estate (rule 871), and the European sector futures (which I'm reserving for a future cross sectional system).
That brings us to instruments which I could trade, but which are too expensive or illiquid. I've added some new functionality to help here. Firstly, I've added a new method which samples bid/ask prices every hour for all instruments which are currently open. There are a few possible uses for this data, eg monitoring liquidity throughout the day and deciding when it's best to trade, but it's current use is in a new costs report. This report compares the median bid/ask spread from 3 sources:
- The spread captured just before an order is created
- The actual spread paid for an order (twice the mid versus fill price)
- The spread from the daily sampling process
(I can also use 1 vs 2 to evaluate the success of my execution algo but that's for another day)
Once I'm happy with the configured spread, the next part of the costs report shows me the current calculation of costs in SR terms per trade. I use this to exclude any instruments with a cost of greater than 0.01 SR units per trade.
The other new report I have is a liquidity report, that shows liquidity sorted by contracts per day (<100 is a problem for me), and daily trades as units of annualised risk (I'm unhappy with less than $1.5 million).
So now I've got a list of instruments that I will exclude from consideration (but I will still calculate forecasts for):
- Can't trade
- Won't trade
- Too expensive to trade
- Too illiquid to trade
Finally I generate instrument weights in the normal way. I will also do one last check to make sure that I can currently take at least one contract in every instrument at maximum forecast.
Here's the new list of instrument weights (zeros are for excluded markets). I've added a * to new instruments which I haven't been trading, but for which I've now got a positive instrument weight.
Code:
AEX: 0.0
AUD: 0.015453375106911324
BBCOMM: 0.0
BITCOIN: 0.022023075020260716
BOBL: 0.022316032042555146
BONO: 0.0
BRE: 0.0194706222681167 *
BTP: 0.0
BUND: 0.0
CAC: 0.03631698944675614
CAD: 0.014391724118600534 *
CHF: 0.0
COPPER: 0.0
CORN: 0.0
CRUDE_W_mini: 0.040003011151262376
EDOLLAR: 0.05045629150461343
EU-AUTO: 0.0
EU-BASIC: 0.0
EU-DIV30: 0.06683805046890984 *
EU-FOOD: 0.0
EU-HEALTH: 0.0
EU-OIL: 0.0
EU-RETAIL: 0.0
EU-TECH: 0.0
EU-TRAVEL: 0.0
EU-UTILS: 0.0
EUR: 0.0
EUROSTX: 0.03631697826517361 *
FEEDCOW: 0.0
GASOILINE: 0.040003011151262376 *
GAS_US_mini: 0.0
GBP: 0.013266567131044219 *
GOLD_micro: 0.01821250887431691
HEATOIL: 0.0
JPY: 0.03066560714009915
KOSPI_mini: 0.024131621339922613
KR10: 0.0
KR3: 0.052541406749157
LEANHOG: 0.034485508922167
LIVECOW: 0.034485508922167
MILK: 0.0
MXP: 0.0194706222681167
NASDAQ_micro: 0.04099135200960326
NOK: 0.0
NZD: 0.015453375106911324 *
OAT: 0.02231603171026421
OATIES: 0.0
PALLAD: 0.0
PLAT: 0.0
REDWHEAT: 0.0
RICE: 0.0
RUR: 0.0
SEK: 0.0
SHATZ: 0.0
SILVER: 0.01821250887431691 *
SMI: 0.0
SOYBEAN: 0.0
SOYMEAL: 0.0
SOYOIL: 0.0
SP500_micro: 0.04099135200960326
US-REALESTATE: 0.0
US10: 0.02707276334871268
US2: 0.0
US20: 0.0
US5: 0.02707276334871268
USIRS10: 0.0
USIRS5: 0.09227765188468574 *
V2X: 0.0
VIX: 0.0735667522716838
WHEAT: 0.05106532368159405Some of these instrument weights are surprising; they are certainly a little different from what 'manual' handcrafting would have come up with. In particular there is the inclusion of both 5 year bonds and 5 year swaps. For now I decided to let them ride, since I'm planning to do RV models with swaps and I wanted to get some trading history for them.
One question is how often I go through this exercise. For example, what if an instrument I'm currently trading suddenly becomes too expensive or illiquid? I'm currently leaning towards doing this annually, with the obvious time being at fiscal year end when I evaluate my performance anyway. By year end I'll certainly have a few more instruments I can trade with prices, and there could be instruments I want to trade with spread models that I don't want to also trade here (more later).
Some thoughts about implementing system changes
There are two dangers with system changes. The first is that you'll screw something up. The second is that they cost money in trading costs.
The first defense against screwing up is having good diagnostics and staring at them a lot. So I checked and rechecked every calculated position, and in most cases they're not that different from what I currently have on. I'm reasonably confident I haven't screwed something up. However there are still steps you can take; rather than just flipping a switch and updating the configuration files, and saying 'let's go!':
- Turning off the automatic trading (what I call the 'stack handler')
- Adding especially tight position or trade limits for new instruments
- Manually controlling the trades especially in new instruments
But that isn't a risk I'm bothered about, so in the end I (very carefully!) did the changeover in one go, over the last hour or so.
Apart from occasional changes to instruments traded, I have made no significant changes to my system since 2014 (the implementation of pysystemtrade was designed to be a like for like replacement). I don't believe in making regular changes to a system, except where there is good reason (and changes in instrument costs and liquidity come into this category); why put yourself through unneccessary stress and pay extra trading costs?
I also feel I've rather flogged to death this particular part of the trading ecosystem (I've done a lot of research, and mostly concluded that it works just fine!), which means from now on I'll most probably be adding entirely new strategies rather than modifying my workhorse carry/trend model.
Which brings me on to...
What's next: New systems, New book
So that's where I am today and good timing, as I plan to go on holiday shortly, having rashly decided to implement my changes a few days in advance - fingers crossed. I'm quite mentally exhausted, firstly with the rabbit hole of small system optimisation, then with the less cerebral but mind numbing effort to build up and test the revised trading strategy.
Hopefully after a rest I'll be refreshed and ready to attack my next couple of projects: new trading systems and a new book. I've talked at length about the new trading systems, which fall into two main categories: a faster univariate system (principal tech challenge: using conditional limit orders and netting with my existing strategy), and a series of relative value systems (principal tech challenge: trading eithier baskets of instruments or calendar spreads rather than just outrights as I do now).
As for the book, of the half dozen or so ideas I have permanently bubbling around my brain the lowest risk one is a book specifically about trading futures, and with much more on actual trading rules than I usually include (since cynically that's what people want! And there's only so many times you can explain what position and risk management people should be doing).
I haven't yet decided what I will start on first; to an extent the book is dependent on the systems since I plan to write about my new systems in the book. However I can do them in parallel to an extent, since most of the content will relate to stuff I already know. It might make sense for me to get a book plan and deadline in place before I do anything else; it's always better to have tangible goals, plus I feel a bit 'coded out' and it would be nice to do a bit of writing in English rather than Python for a bit.
Hope everyone enjoys their summer...
GAT