Fully automated futures trading
- Thread starter globalarbtrader
- Start date
I think there's also a problem with mini-crude: it has a good liquidity but not in the Dec contract, all volume is concentrated in the 2 front months. I guess it's still possible to trade it but not with Dec contract.. How critical is avoiding seasonality in this case by trading only December ?
We do trade GAS_US monthly even though it should also probably have seasonality..
Btw, in this file "https://github.com/robcarver17/pysystemtrade/blob/master/sysinit/futures/config/rollconfig.csv" X and V months in the roll and hold parameters are flipped for GAS, does it have any special meaning or it doesn't matter?
Also, regarding mini-GAS, not sure if there's enough volume in the second contract if we use front as carry and trade second., e.g. we would probably need to roll into the Jul contract around April 20th to April 27th (when May, which was Carry for Jun expired), and Jul's volume only started reaching 100 contracts on April 26..
Sorry for too many annoying questions
Last edited:
And another thingI think there's also a problem with mini-crude: it has a good liquidity but not in the Dec contract, all volume is concentrated in the 2 front months. I guess it's still possible to trade it but not with Dec contract.. How critical is avoiding seasonality in this case by trading only December ?
View attachment 258857
We do trade GAS_US monthly even though it should also probably have seasonality..
Btw, in this file "https://github.com/robcarver17/pysystemtrade/blob/master/sysinit/futures/config/rollconfig.csv" X and V months in the roll and hold parameters are flipped for GAS, does it have any special meaning or it doesn't matter?
View attachment 258858
Also, regarding mini-GAS, not sure if there's enough volume in the second contract if we use front as carry and trade second., e.g. we would probably need to roll into the Jul contract around April 20th to April 27th (when May, which was Carry for Jun expired), and Jul's volume only started reaching 100 contracts on April 26..
Sorry for too many annoying questions
MINI-KOSPI seems to be monthly, not quaterly:
http://global.krx.co.kr/contents/GLB/02/0201/0201040204/GLB0201040204.jsp
"The quaterly month : The settlement price of KOSPI200 futures
The non-quaterly month : The current settlement price methodology is applied"
I was just looking at the volume of the HMUZ months and there's a volume gap, i.e. if I tried to roll from Mar to Jun I would have to do it before Mar 11th (expiry day) and the Jun contract has almost zero volume around Mar 11, so people are clearly rolling into April and May first..
It's not critical and there is probably more value to be gained from greater granularity than from having the seasonal match.
Action: change mini crude to monthly rolls. This will involve rebuilding the multiple and adjusted price series, and needs to be done before the next roll occurs.
No the order is irrelevant, pretty sure they are sorted when used.
Action: change mini gas carry offset to +1. This will involve rebuilding the multiple and adjusted price series, and needs to be done before the next roll occurs.
And another thing
MINI-KOSPI seems to be monthly, not quaterly:
http://global.krx.co.kr/contents/GLB/02/0201/0201040204/GLB0201040204.jsp
"The quaterly month : The settlement price of KOSPI200 futures
The non-quaterly month : The current settlement price methodology is applied"
I was just looking at the volume of the HMUZ months and there's a volume gap, i.e. if I tried to roll from Mar to Jun I would have to do it before Mar 11th (expiry day) and the Jun contract has almost zero volume around Mar 11, so people are clearly rolling into April and May first..
This is a bit trickier. I can't just rebuild the multiple and adjusted price series, because I don't have monthly contracts for KOSPI in the past, only quarterly (and I don't have them for barchart). At the moment I'd be looking to buy June, which would be fine right now anyway. But then the forward contract in the multiple adjusted price series is shown as September. It's probably possible to hack this, but it strikes me that a lot can go wrong.
What I think I will do is delete the cloned prices, see how much recent historic data I can get from IB, and effectively start again with micro KOSPI as a new contract. I'll lose some data history, but that is a lot simpler than torturing my code and possible introducing bugs.
Action (for now): override positions on KOSPI-mini
Action: Get prices from scratch for KOSPI-mini
Sorry for too many annoying questions
No problem! In fact I should really have put these changes up on here before implementation as a sense check. Stupid to waste the free resource of people looking at your stuff for you.
GAT
What is IB's symbol code for this KOSPI-mini? I am using K200 as KOSPI and have daily OHLC price files since about middle of 2018.
It's K200M. If you have monthly rolled K200 prices though that would be perfect as well.
GAT
I only know of K200 being a quarterly contract using March, June, September and December. I am not aware of any other expiry dates. IB only shows me these when I use reqContractDetails().
Edit: added a screen shot from IB's contract search tool:
Only the K200M has monthly contracts:
Last edited:
I'm glad to help
Onboarding new instruments is very annoying and it's always good to have multiple eyeballs on the details., and I'm naturally encountering these issues as I'm trying to onboard these contracts into my system, especially because I have automatic rolls I'm forced to check all these things when I generate new contracts and roll-schedules..Yeah, I think the full KOSPI has only quaterly contracts (http://global.krx.co.kr/contents/GLB/02/0201/0201040201/GLB0201040201.jsp) so no way to use full KOSPI prices for all monthly contracts of mini-KOSPI..
Last edited:
Micro WTI futures are coming soon. If it's like the other micros, it should have good liquidity, but only in the front month. https://www.cmegroup.com/media-room...e_group_to_launchmicrowtifuturesonjuly12.htmlI think there's also a problem with mini-crude: it has a good liquidity but not in the Dec contract, all volume is concentrated in the 2 front months. I guess it's still possible to trade it but not with Dec contract.. How critical is avoiding seasonality in this case by trading only December ?
View attachment 258857
We do trade GAS_US monthly even though it should also probably have seasonality..
Btw, in this file "https://github.com/robcarver17/pysystemtrade/blob/master/sysinit/futures/config/rollconfig.csv" X and V months in the roll and hold parameters are flipped for GAS, does it have any special meaning or it doesn't matter?
View attachment 258858
Also, regarding mini-GAS, not sure if there's enough volume in the second contract if we use front as carry and trade second., e.g. we would probably need to roll into the Jul contract around April 20th to April 27th (when May, which was Carry for Jun expired), and Jul's volume only started reaching 100 contracts on April 26..
Sorry for too many annoying questions
Feels like it's time for a progress report, as it's my birthday and I'm feeling introspective.
Performance has been good; a quick check on the tape reveals my drawdown is down to less than 4% and I'm up more than 10% for the tax year (perhaps 8% for the calendar year because the first two months were down).
pysystemtrade is in a pretty stable condition, with both the production and sim code pretty decently refactored to the point where I'm confident I can keep it well maintained. Today I decided to release 'version 1.0'; about 5.5 years after it's first release - incredible how time flies!
Right now I'm working on adding lots of markets, as per the recent blog post. This is pretty tedious stuff which involves backfilling data from barchart, running a load of scripts, etc etc etc. I have another 75 (!) markets to do in the current batch (markets I can get data for, and which pass my trading filters). Then there another 100 or so markets for the next batch (markets I can get data for, but which don't currently pass my trading filters).
The main constraint is the 100 tickers a day I can download from barchart (eg two instruments with 12 years of quarterly rolls), and the fact I'd kill myself if I spent more than a couple of hours on this a day. I'm probably going to take at least 2 months to get through the curent batch, and another 3 months for the next batch (it could be quicker, as Barchart doesn't always have more than a few years of data, and it's possible there are markets that aren't on barchart at all). I could speed things up by looking at buying the data elsewhere, but the manual process means I actually check everything quite carefully.
The other thing I'm doing, also flagged up in the post, is to substitute several of my existing markets for the relevant micro or mini contract. This is slightly easier, but you just need to be a bit careful. First, I make sure I close my positions in the relevant contracts by setting my trade control to 'reduce only'. I now have to wait for all the positions to be closed. If it gets close to a roll, and they aren't closed, I'll have to close them manually.
Then once I have no active positions in the relevant instrument, I change the IB and instrument configuration so that it is using the new ticker and multiplier. I then get the prices to pull in manually and make sure there is no jump. This is also a good time to do a futures roll if one is due. Finally I turn off the 'reduce only' flag and check that the new positions are opened up okay. Obviously this means historic positions can't be compared, but that doesn't bother me.
Running alongside that I need to do some profiling and optimisation work (just got a copy of https://www.oreilly.com/library/view/high-performance-python/9781492055013/). With around 230 instruments pysystemtrade will be very slow indeed even on my 32GB/i7 trading servers, if it runs at all! Even so, I'd probably never run a fully blown backtest (up to 35 years of daily data plus 7 years of hourly data, 230 instruments, rolling optimisations...). Instead, I'll do things like run single instrument systems with one period optimisation to get the forecast weights and FDM; then with those fixed do a single period optimisation to get the instrument weights and IDM. In production I'll only pull in say the last two years of data for the nightly backtest.
Once I've sped things up a bit, but not neccessarily before I've finished eithier batch one or batch of adding new instruments, I'm going to look into my idea of modifying the system to make better use of limited capital. At the moment I have the very hacky solution I wrote about here. I probably get asked about this more than anything, and I'm as excited as anyone to see if my idea works, but the profiling has to come first!
Other projects to follow that are, in no particular order:
- I have a bunch of other changes and potential trading signals I want to look at putting into the core model. A lot of this is stuff I've blogged about over the last couple of years but not implemented.
- Some completely new systems: fast mean reversion, intermarket RV eg spreads, intramarket RV spreads (calendar spreads).
- The next book...
GAT
Progress has been quite swift over the last couple of weeks.
I'm continuing to add markets but clearly this is going to take a while. I've put all the markets for both the first and second batches into my configuration file and there is a sobering total of 227 rows; of which 40 or so were my legacy markets, and where I've added another 16 of the remaining 187 so far (including the new micro and mini markets). This is going to be a long haul!
For some markets where there isn't Barchart data I created some code that will 'seed' the database with as much IB data as it can get (again thanks to @HobbyTrading for the heads up about expired contracts).
I decided to add the micro/mini instruments as additional markets rather than replacing existing ones. That means things like risk and position reports in the past will still make sense, and overall just seemed a less risky approach. That also means for the time being I'm still sampling the prices of the original markets; given the number I'm adding to my system another half dozen is neither here or there.
I did some profiling work and was able to get some quick wins quite easily. Broadly speaking I did this by removing use of pandas apply (on rows) using my own functions, and instead did the relevant operation on the entire dataframe in situ.
A particular time sink was the calculation of the carry signal; in particular working out what the distance between two expiries is in a fraction of a year. Now I could assume the distance is always what it is specified in the roll configuration, but to be safe I'd rather not do that.
I was doing this with apply and also working with the string "20060300" converting it into a datetime, and working out the exact number of days. This is crazy since it assumes the expiry is on the first of the month, so we're doing a massive approximation, so it doesn't really matter if (for example) we account for the fact that February is 28 days long. Instead I changed the contract series to floats so the operation could be done on the entire dataframe, assuming each month was exactly 1/12 of a year.
There were also little things like "".join(["x","y"]) being faster than "x"+"y", introducing a new 'simple' contractDate object where both inputs are strings (since the full object is very flexible and therefore slow).
Then there is this little beauty, run every time a contractDate is initialised (since I specify spread contracts as eg '20200100_20200200':
Code:
MISSING_STR = -1
def list_of_items_separated_by_underscores(this_str, result = ()):
find_underscore = this_str.find("_")
if find_underscore is MISSING_STR:
result_as_list = list(result)
result_as_list.append(this_str)
return result_as_list
partial_str = this_str[:find_underscore]
result = result + tuple([partial_str])
remaining_str = this_str[find_underscore+1:]
return list_of_items_separated_by_underscores(remaining_str, result=result)Which I replaced with the surprisingly quick:
Code:
date_str_as_list = date_str.split("_")WTF was I thinking?! Anyway, moving on, nothing to see here...
In a change from the plan above, I decided to circle back to potential trading signals and changes to the core model before I went ahead with anything else.
When I sat down and went through the list, there weren't actually that many things that I intended to add in terms of new signals (which wouldn't be more suited to a newer seperate strategy, such as spread trading). I added some 'fast' cross sectional momentum (I already had slow cross sectional mean reversion within asset classes, so this was just a case of reversing the sign and shortening the window). I added the skew and kurtosis rules. This also means I can take out the 'short vol bias' rule. That's basically it. A 'feature' that isn't strictly a new signal is the idea of attenuating momentum and carry style signals.
The next step will be to 'build' the new system as a traditional model. So that means generating forecast scalars, and then fitting forecast weights. I have a method for fitting forecast weights that I will probably blog about. It involves fitting on each instrument individually, then clustering and fitting clusters; finally fitting across the whole set of instruments. The final forecast weights will be a blend of the weights from each of the three methods, the blend depending on the amount of data an instrument or cluster. Quite slow to do, I'll probably do it as a single in sample fit once I've tested it, and quite heuristic, but I would like to introduce some more instrument specific weightings where that is justified. I expect this will be in my next blog post.
Then it's down to calculate the FDM, and fit instrument weights (something I'll have to do regularly as I add new instruments). One consequence of the new methodology, by the way, is that no IDM is needed (but I'll probably set it to the maximum 2.5 just for old times sake).
Once the system is working in the traditional model, it's a case of adding the additional optimisation stage that accounts for discrete positions. As part of the optimisation I will be including my idea for dynamically targeting risk according to average forecast strength (this is why an IDM is no longer needed).
At this stage I also want to include my new blended vol estimate that incorporates mean reverting vol. Another thing I will have is a 'don't trade' list of instruments that is updated automatically according to my criteria for adding new markets (in practice we can trade, but it won't allow new positions to be opened); in particular volume and cost constraints (the 'too large to trade' will be dealt with by the opimisation).
I will make some simplifications. I can remove the hacky forecast mapping that I use right now for larger contract size. And the exogenous risk overlay will also be redundant. Finally buffering will no longer be required, since the optimisation will include a cost penalty.
GAT