I also have about 5 trades per day. However, I added a number of instruments with a low notional value and low value volatility. This may result in more trading then when you would only use instruments with larger notional value (and corresponding value volatility).
Fully automated futures trading
- Thread starter globalarbtrader
- Start date
Btw, I just checked, my system started rolling into the next Korean bond (Sept 2020) on Wednesday (fill @112.11) and sold last 2 June contracts on Thursday (June 11) @112.15.
Looking at the settings for the corresponding predefined roll cycle, "SoftRoll" started on June 9, and "HardRoll" on June 11. So the system did what it was told, but just not sure maybe it was too early, next Korean bonds don't have much volume till the very end. Can't check the exact volumes right now, but the system recorded that the ratio of recent price to forward contract volumes was 0.015, and the trigger for starting soft roll is 1%..
Looking at the settings for the corresponding predefined roll cycle, "SoftRoll" started on June 9, and "HardRoll" on June 11. So the system did what it was told, but just not sure maybe it was too early, next Korean bonds don't have much volume till the very end. Can't check the exact volumes right now, but the system recorded that the ratio of recent price to forward contract volumes was 0.015, and the trigger for starting soft roll is 1%..
I don't use volume triggers but I would probably have set the system to hard roll on Thursday (not worth the soft roll period since the possible roll period for Korean bonds is so short). Fridays relative volumes were 27% for 3 year and 13% for 10 year. Let me show off my new system:
Code:
rob@boromir:~/pysystemtrade/sysproduction/linux/scripts$ . interactive_roll_info
sysproduction.interactive_roll_info.interactive_roll_info:
Print information about whether futures contracts should be rolled
:param instrument_code: The instrument code, for example 'AUD', 'CRUDE_W'. Specify ALL for everything
:return: None, but print results
Arguments:
['instrument_code']
Argument instrument_code (default: 'ALL') (type: str)?KR3
********************************************************************************
Roll status report produced on 2020-06-13 08:09:48.018925
********************************************************************************
================================================
Status and time to roll in days
================================================
Status Roll_exp Prc_exp Crry_exp
KR3 Force_Outright 18 2 93
Roll_exp is days until preferred roll set by roll parameters. Prc_exp is days until price contract rolls, Crry_exp is days until carry contract rolls
===================================================
List of contracts
===================================================
C0 C1 C2 C3 C4 C5
KR3 20200300 20200600p 20200900f 20201200
Suffix: p=price, f=forward, c=carry
=======================================
Positions
=======================================
Pos0 Pos1 Pos2 Pos3 Pos4 Pos5
KR3 0.0 2.0 0.0 0.0 0.0 0.0
==================================
Relative volumes
==================================
V0 V1 V2 V3 V4 V5
KR3 0.0 1.0 0.27 0.0 0.0 0.0
Contract volumes over recent days, normalised so largest volume is 1.0
===================================
Prices
===================================
P0 P1 P2 P3 P4 P5
KR3 112.08 112.02 NaN NaN NaN NaN
********************************************************************************
END OF ROLL REPORT
********************************************************************************Notes:
- It looks like my preferred roll date parameter is wrong (18 days from now, should be -1 since as I said I should have rolled on Friday).
- Yes all my computers are named after Lord of the rings characters, I am indeed a proper old-school nerd...
GAT
Looking at my system I don't have many positions on (14), vols are high so most positions are small or zero, and I don't have positions on in the markets that are actually moving a lot (and also have high vol): Crude, Vol and the equities.
Positions (no report for this yet so here's an interactive session):
Code:
rob@boromir:~/pysystemtrade/sysproduction/linux/scripts$ . interactive_order_stack
sysproduction.interactive_order_stack.interactive_order_stack:
None
Arguments:
[]
0: View
1: Create orders
2: Fills and completions
3: Modify/cancel
4: Delete
Your choice? <RETURN for EXIT> 0
0: View specific order
1: View instrument order stack
2: View contract order stack
3: View broker order stack (stored local DB)
4: View IB orders and fills
9: View positions
Your choice? <RETURN for Back> 9
Strategy positions
strategy_name instrument_code position
0 medium_speed_TF_carry BOBL 2.0
1 medium_speed_TF_carry GAS_US -3.0
2 medium_speed_TF_carry EDOLLAR 10.0
3 medium_speed_TF_carry OAT 1.0
4 medium_speed_TF_carry KR3 2.0
5 medium_speed_TF_carry SOYBEAN -4.0
6 medium_speed_TF_carry AUD 1.0
7 medium_speed_TF_carry BTP 1.0
8 medium_speed_TF_carry WHEAT -4.0
9 medium_speed_TF_carry PLAT 1.0
10 medium_speed_TF_carry LEANHOG -1.0
11 ETFHedge EUROSTX -2.0
12 medium_speed_TF_carry JPY -1.0
13 medium_speed_TF_carry MXP 1.0
14 medium_speed_TF_carry CORN -5.0I don't yet have a nice report for forecast strength, so here's some ugly code:
Code:
system.combForecast.get_combined_forecast("BOBL").tail(1)
index
2020-06-12 20.0
Freq: B, dtype: float64
system.combForecast.get_combined_forecast("GAS_US").tail(1)
index
2020-06-11 -19.521301
Freq: B, dtype: float64
system.combForecast.get_combined_forecast("EDOLLAR").tail(1)
index
2020-06-11 15.948758
Freq: B, dtype: float64
system.combForecast.get_combined_forecast("OAT").tail(1)
index
2020-06-12 19.506551
Freq: B, dtype: float64
system.combForecast.get_combined_forecast("KR3").tail(1)
index
2020-06-12 17.655809
Freq: B, dtype: float64
system.combForecast.get_combined_forecast("SOYBEAN").tail(1)
index
2020-06-11 -10.364216
Freq: B, dtype: float64
system.combForecast.get_combined_forecast("AUD").tail(1)
index
2020-06-11 8.392098
Freq: B, dtype: float64
system.combForecast.get_combined_forecast("BTP").tail(1)
index
2020-06-12 20.594738
Freq: B, dtype: float64
system.combForecast.get_combined_forecast("WHEAT").tail(1)
index
2020-06-11 -17.735054
Freq: B, dtype: float64
system.combForecast.get_combined_forecast("PLAT").tail(1)
index
2020-06-11 9.489023
Freq: B, dtype: float64
system.combForecast.get_combined_forecast("LEANHOG").tail(1)
index
2020-06-11 -5.717445
Freq: B, dtype: float64
system.combForecast.get_combined_forecast("JPY").tail(1)
index
2020-06-11 -4.444731
Freq: B, dtype: float64
system.combForecast.get_combined_forecast("MXP").tail(1)
index
2020-06-11 8.414867
Freq: B, dtype: float64
system.combForecast.get_combined_forecast("CORN").tail(1)
index
2020-06-11 -12.531132
Freq: B, dtype: float64GAT
boromir eh?
I remember at the university where I studied CS they did the same thing, but just LOTR wouldn't be enough for the number of machines they had, so they had to tap into Silmarillion - that has enough cool names of people and places to name a sworm of nano-robots You don't use volume triggers, but you do look at the volume ratios when deciding when to roll right ?
My roll algo doesn't use just volume to decide when to start SoftRoll, it has to be both greater than the predefined SoftRoll date and the volume ratio above 1%. For HardRoll it's just the date, so it will start rolling on HardRoll date no matter what.
for that KR3 cycle I had these settings:
So HardRoll did start on Thursday.
I agree that having only 2 days of SoftRoll probably doesn't do much, it's just I have the same roll mechanics for all instruments, and for some of them SoftRoll periods are longer and can be beneficial., but for 3KTB I just squished it to a couple of days, and also it's waiting for at least 1% of volume to start SoftRoll, and if that volume never comes it will never enable SoftRoll and will go from NoRoll straight to HardRoll..
Last edited:
boromir eh?
You don't use volume triggers, but you do look at the volume ratios when deciding when to roll right ?
My roll algo doesn't use just volume to decide when to start SoftRoll, it has to be both greater than the predefined SoftRoll date and the volume ratio above 1%. For HardRoll it's just the date, so it will start rolling on HardRoll date no matter what.
for that KR3 cycle I had these settings:
View attachment 230480
So HardRoll did start on Thursday.
I agree that having only 2 days of SoftRoll probably doesn't do much, it's just I have the same roll mechanics for all instruments, and for some of them SoftRoll periods are longer and can be beneficial., but for 3KTB I just squished it to a couple of days, and also it's waiting for at least 1% of volume to start SoftRoll, and if that volume never comes it will never enable SoftRoll and will go from NoRoll straight to HardRoll..
Yes I look at the volume ratios, but it's a manual decision without any fixed rules.
GAT
Hey folks, here's an update from me as well, haven't been posting for a while.
My last update was https://www.elitetrader.com/et/threads/fully-automated-futures-trading.289589/page-209#post-5055797.
After that stunning start, things proceeded to go exactly as you would expect, meaning, I'm down 5.7% since April 1st, which brings the total to a profit of 20.26% since inception (February 1st).
Here's the money shot:
I'm putting a bunch of stuff in this post, so I'll add some headings, feel free to skip the parts that are not interesting.
# Performance
At the very end of that podcast that Rob mentioned a few posts back, he said that there are folks that consider themselves systematic traders but fiddle with the trades before posting them manually, so naturally I felt called out, so now I'm automating everything, no more discretionary input.
Joke
Well, the part that I felt called out is a joke, however, I am automating execution. I mentioned in the previous post that I'm fiddling in the system by curating the execution, meaning, I would delay it a bit, or sometimes enter execute that the system would say. I did some number since mid April (which is from when I have precise numbers for this) and it turns out that if I didn't do that, and just executed everything the system wanted right when it said so, the loss would have been 2% less. I realize it's not a big sample size (I've had about 120 orders in that period) but looking closely at some specific examples when I interveened, I think it's not just noise. So I decided to go fully automated.
Lastly, the burden of proof should be on the discretionary element, not the systematic portion - in other words, the default should be not to fiddle with it, and only fiddle if the numbers prove that it actually produces better results.
That said, I can see that's going to be painful for me For example, right now, the system is saying I should be short soybeans, but looking at the chart, I can't for the love of everything see a single reason to go short (well, besides carry, which is a big portion of my system, so I guess that's the reason).
For reference, I am now trading 12 markets.
# Infrastructure
Most of the last 2 months I've spent on improving the infrastructure and quality of life in general (code quality, automated testing, better reporting, server, etc.) so here's a few words about that.
Rob mentioned that he wants to set up a new PC, so here's a shot of my server in its natural habitat:
It's a Raspberry Pi 4, 4GB model. On it, I run IB Gateway with IBController. Distribution is Raspbian Lite. There was some hacking needed to get the IB Gateway to work, because IB bundles ancient Java in the installer (literally a binary within the .sh file you download to install), which didn't like the fact that I wanted to run it on the ARM CPU, but it wasn't too complex. I also run a minimalistic web UI on that, written in Python, using Flask framework. As mentioned in the previous post, I use Google Cloud SQL as my datastore, and the system talks to that. Periodic tasks (eg. price collection, end-of-day accounting and forecast run, etc.) are executed via systemd timers. It's connected via UTP cable partly because I didn't want to fiddle with setting up wireless (it's really not hard though) and partly because I'd rather not expose this to WiFi if not necessary. If you're thinking "gee, 4GB model sure is an overkill for that", you're absolutely right. After a week of running all of the above, it consumes about 450MB of RAM. That is partly because I'm anal about not running any GUI, so it has none. Since IB Controller needs some GUI though, to click around and put in the username and password for the IB Gateway, I simulate that with a virtual screen, xvfb. I have all the ports closed to the traffic from anywhere, except ssh and http ports which are allowed from my local network.
So far it's working flawlessly. Rob has once said (either on his blog or here) that he spends about 100 pounds per server for his system, this was $55 for the RPi and $15 for the case. Beat that Rob!
# Upcoming work
Few words about what I'm going to be working on over the next few months. First, wrap up the automated execution work. I have all the code in place, but doing some more testing to get more confidence that this won't cause big trouble. As part of this work, I'm also planning to implement carry smoothing, something I still don't have, and I think it's necessary if moving to automated execution, mainly because of things like V2TX, where nearer contract can fluctuate so wildly to flip my position from long to short or other way around in one day.
Then, I'd like to automate rolling, probably not fully, but to a large degree, for sure for instruments where I don't have a position on.
After that, better reporting, now that I have some data gathered, I'd like to get better visibility into how the system is doing.
Alright, that's it for this update!
I wanted to close with a question, is anybody familiar with a Slack room or something similar for systematic traders? I specifically don't mean those chat rooms where people share day trading tips, more like a place to chat about system development, perhaps trends in the markets and similar, in a very asynchronous manner.
Thanks!
My last update was https://www.elitetrader.com/et/threads/fully-automated-futures-trading.289589/page-209#post-5055797.
After that stunning start, things proceeded to go exactly as you would expect, meaning, I'm down 5.7% since April 1st, which brings the total to a profit of 20.26% since inception (February 1st).
Here's the money shot:
I'm putting a bunch of stuff in this post, so I'll add some headings, feel free to skip the parts that are not interesting.
# Performance
At the very end of that podcast that Rob mentioned a few posts back, he said that there are folks that consider themselves systematic traders but fiddle with the trades before posting them manually, so naturally I felt called out, so now I'm automating everything, no more discretionary input.
Joke
Well, the part that I felt called out is a joke, however, I am automating execution. I mentioned in the previous post that I'm fiddling in the system by curating the execution, meaning, I would delay it a bit, or sometimes enter execute that the system would say. I did some number since mid April (which is from when I have precise numbers for this) and it turns out that if I didn't do that, and just executed everything the system wanted right when it said so, the loss would have been 2% less. I realize it's not a big sample size (I've had about 120 orders in that period) but looking closely at some specific examples when I interveened, I think it's not just noise. So I decided to go fully automated.
Lastly, the burden of proof should be on the discretionary element, not the systematic portion - in other words, the default should be not to fiddle with it, and only fiddle if the numbers prove that it actually produces better results.
That said, I can see that's going to be painful for me
For example, right now, the system is saying I should be short soybeans, but looking at the chart, I can't for the love of everything see a single reason to go short (well, besides carry, which is a big portion of my system, so I guess that's the reason).For reference, I am now trading 12 markets.
# Infrastructure
Most of the last 2 months I've spent on improving the infrastructure and quality of life in general (code quality, automated testing, better reporting, server, etc.) so here's a few words about that.
Rob mentioned that he wants to set up a new PC, so here's a shot of my server in its natural habitat:
It's a Raspberry Pi 4, 4GB model. On it, I run IB Gateway with IBController. Distribution is Raspbian Lite. There was some hacking needed to get the IB Gateway to work, because IB bundles ancient Java in the installer (literally a binary within the .sh file you download to install), which didn't like the fact that I wanted to run it on the ARM CPU, but it wasn't too complex. I also run a minimalistic web UI on that, written in Python, using Flask framework. As mentioned in the previous post, I use Google Cloud SQL as my datastore, and the system talks to that. Periodic tasks (eg. price collection, end-of-day accounting and forecast run, etc.) are executed via systemd timers. It's connected via UTP cable partly because I didn't want to fiddle with setting up wireless (it's really not hard though) and partly because I'd rather not expose this to WiFi if not necessary. If you're thinking "gee, 4GB model sure is an overkill for that", you're absolutely right. After a week of running all of the above, it consumes about 450MB of RAM. That is partly because I'm anal about not running any GUI, so it has none. Since IB Controller needs some GUI though, to click around and put in the username and password for the IB Gateway, I simulate that with a virtual screen, xvfb. I have all the ports closed to the traffic from anywhere, except ssh and http ports which are allowed from my local network.
So far it's working flawlessly. Rob has once said (either on his blog or here) that he spends about 100 pounds per server for his system, this was $55 for the RPi and $15 for the case. Beat that Rob!
# Upcoming work
Few words about what I'm going to be working on over the next few months. First, wrap up the automated execution work. I have all the code in place, but doing some more testing to get more confidence that this won't cause big trouble. As part of this work, I'm also planning to implement carry smoothing, something I still don't have, and I think it's necessary if moving to automated execution, mainly because of things like V2TX, where nearer contract can fluctuate so wildly to flip my position from long to short or other way around in one day.
Then, I'd like to automate rolling, probably not fully, but to a large degree, for sure for instruments where I don't have a position on.
After that, better reporting, now that I have some data gathered, I'd like to get better visibility into how the system is doing.
Alright, that's it for this update!
I wanted to close with a question, is anybody familiar with a Slack room or something similar for systematic traders? I specifically don't mean those chat rooms where people share day trading tips, more like a place to chat about system development, perhaps trends in the markets and similar, in a very asynchronous manner.
Thanks!
Hey folks, here's an update from me as well, haven't been posting for a while.
My last update was https://www.elitetrader.com/et/threads/fully-automated-futures-trading.289589/page-209#post-5055797.
After that stunning start, things proceeded to go exactly as you would expect, meaning, I'm down 5.7% since April 1st, which brings the total to a profit of 20.26% since inception (February 1st).
Here's the money shot:
View attachment 230612
I'm putting a bunch of stuff in this post, so I'll add some headings, feel free to skip the parts that are not interesting.
# Performance
At the very end of that podcast that Rob mentioned a few posts back, he said that there are folks that consider themselves systematic traders but fiddle with the trades before posting them manually, so naturally I felt called out, so now I'm automating everything, no more discretionary input.
Joke
Well, the part that I felt called out is a joke, however, I am automating execution. I mentioned in the previous post that I'm fiddling in the system by curating the execution, meaning, I would delay it a bit, or sometimes enter execute that the system would say. I did some number since mid April (which is from when I have precise numbers for this) and it turns out that if I didn't do that, and just executed everything the system wanted right when it said so, the loss would have been 2% less. I realize it's not a big sample size (I've had about 120 orders in that period) but looking closely at some specific examples when I interveened, I think it's not just noise. So I decided to go fully automated.
Lastly, the burden of proof should be on the discretionary element, not the systematic portion - in other words, the default should be not to fiddle with it, and only fiddle if the numbers prove that it actually produces better results.
That said, I can see that's going to be painful for me
For reference, I am now trading 12 markets.
# Infrastructure
Most of the last 2 months I've spent on improving the infrastructure and quality of life in general (code quality, automated testing, better reporting, server, etc.) so here's a few words about that.
Rob mentioned that he wants to set up a new PC, so here's a shot of my server in its natural habitat:
View attachment 230610
It's a Raspberry Pi 4, 4GB model. On it, I run IB Gateway with IBController. Distribution is Raspbian Lite. There was some hacking needed to get the IB Gateway to work, because IB bundles ancient Java in the installer (literally a binary within the .sh file you download to install), which didn't like the fact that I wanted to run it on the ARM CPU, but it wasn't too complex. I also run a minimalistic web UI on that, written in Python, using Flask framework. As mentioned in the previous post, I use Google Cloud SQL as my datastore, and the system talks to that. Periodic tasks (eg. price collection, end-of-day accounting and forecast run, etc.) are executed via systemd timers. It's connected via UTP cable partly because I didn't want to fiddle with setting up wireless (it's really not hard though) and partly because I'd rather not expose this to WiFi if not necessary. If you're thinking "gee, 4GB model sure is an overkill for that", you're absolutely right. After a week of running all of the above, it consumes about 450MB of RAM. That is partly because I'm anal about not running any GUI, so it has none. Since IB Controller needs some GUI though, to click around and put in the username and password for the IB Gateway, I simulate that with a virtual screen, xvfb. I have all the ports closed to the traffic from anywhere, except ssh and http ports which are allowed from my local network.
So far it's working flawlessly. Rob has once said (either on his blog or here) that he spends about 100 pounds per server for his system, this was $55 for the RPi and $15 for the case. Beat that Rob!
# Upcoming work
Few words about what I'm going to be working on over the next few months. First, wrap up the automated execution work. I have all the code in place, but doing some more testing to get more confidence that this won't cause big trouble. As part of this work, I'm also planning to implement carry smoothing, something I still don't have, and I think it's necessary if moving to automated execution, mainly because of things like V2TX, where nearer contract can fluctuate so wildly to flip my position from long to short or other way around in one day.
Then, I'd like to automate rolling, probably not fully, but to a large degree, for sure for instruments where I don't have a position on.
After that, better reporting, now that I have some data gathered, I'd like to get better visibility into how the system is doing.
Alright, that's it for this update!
I wanted to close with a question, is anybody familiar with a Slack room or something similar for systematic traders? I specifically don't mean those chat rooms where people share day trading tips, more like a place to chat about system development, perhaps trends in the markets and similar, in a very asynchronous manner.
Thanks!
Brilliant! If I may be indulged by quoting from my own book: "... you can run quite sophisticated strategies on $30 Raspberry Pi micro computers" (ST page 4).
I don't know of any slack channel or similar. I'm not sure the more cerebral nature of systematic trading is suited to a platform like that, since normal slack trading channels tend to be full of people constantly telling you what they are trading and what economic news is coming (and that's the good ones!). In fact there is a general dearth of good quality discussion on the subject (dare I say this thread is one of the better places for talking about it?). There are the quantopian forums but they are biased towards their platform, a reddit group thats sometimes quite interesting and the occasional thread on nuclear phynance, but that's about it.
GAT
Agree.
The latest episode popped up on my iTunes yesterday while I was walking the dog
Here are some comments on using signal strength for position sizing (that's what I do for my strategies). It has good and bad sides to it
- it is more suitable to reversion/normalization strategies since you increase your signal as your expected alpha becomes larger even if it has not been working
- pnl/tradevalue usually improves since you do not have fixed size round-trips as you do with fixed thresholds, it's also much easier to think about transaction cost control in this setting
- you have to have some sort of min and max signal parameter (min so you don't trade the noise and max so you don't go overboard), so it's yet another parameter to fit (and possibly curve fit)