Fully automated futures trading

[rosalux]

so I've got another pysystemtrade noob question (let me know if this isn't the right place...)

say I've got one asset and one trading rule. The resulting accountCurve then remains fairly consistent through adding stages to the system (scale, combine, sizing)...

... that is, until the final, Portfolio, stage. For some reason, the accountCurve produced by portfolio() looks a bit different and has different (worse) stats. I tried various values for capital and vol target, no change. No idea what causes it, other than portfolio() calling pandl_for_instrument() which is where the accountCurve gets built.

to replicate this, you can start e.g. from simplesystem.ipynb and do

my_config = Config(
    dict(
        trading_rules=dict(ewmac8=ewmac_8),
        instrument_weights=dict(US10=1),
        instrument_div_multiplier=1,
        forecast_scalars=dict(ewmac8=5.3),
        forecast_weights=dict(ewmac8=1),
        forecast_div_multiplier=1.))
my_system = System([
    Account(), Portfolios(), PositionSizing(), ForecastCombine(),
    ForecastScaleCap(), Rules()
], data, my_config)

my_system.accounts.portfolio().curve().plot()
print(my_system.accounts.portfolio().percent().stats())

for the portfolio() curve, and for the manual accountCurve you can do

price = data.get_raw_price('US10')
from syscore.accounting import accountCurve
account = accountCurve(price, forecast = my_system.portfolio.get_actual_position("US10"))
print(account.percent().stats())
account.curve().plot()

here's the two resulting curves:

what am I missing?

 

[globalarbtrader]

my_system.accounts.portfolio().curve().plot()
print(my_system.accounts.portfolio().percent().stats())[/CODE]
for the portfolio() curve, and for the manual accountCurve you can do

price = data.get_raw_price('US10')
from syscore.accounting import accountCurve
account = accountCurve(price, forecast = my_system.portfolio.get_actual_position("US10"))
print(account.percent().stats())
account.curve().plot()

here's the two resulting curves:
View attachment 223425 View attachment 223426
what am I missing?

This is the killer line:

account = accountCurve(price, forecast = my_system.portfolio.get_actual_position("US10"))

You're passing a position into an accounting function as a forecast. They're not the same thing at all. What you probably want to is something like this:

forecast=my_system.combForecast.get_combined_forecast("US10")
account = accountCurve(price, forecast = forecast)

or

position = my_system.positionSize.get_subsystem_position("US10")
account = accountCurve(price, positions = position)

It's also generally better if you use system.accounts to get the p&l of part of the system (as you did to get the whole curve). There are dozens of options: read the documentation (carefully, as it's quite complex). And they may produce slightly different results because of rounding, capping and various other effects. They will also use different levels of capital, which will change the scaling.


GAT

 

[wopr]

Hey folks,
I've been lurking around for quite a while, so decided to do a short writeup of what I'm doing, for those who are interested. This is a bit lengthy, so I'll put the important stuff at the top

I've been trading my system for 2 full months (first trade was placed 1/29) so this is as good time as any. Performance so far 26.97% up (IB somewhere shows that, somewhere 27.99%). I do realize that large part of that is luck, and second large part is recent market conditions.

(I don't have proper PnL and accounting implemented yet, so here are some screenshots from IB.)

Current positions:

TL;DR - I'm doing pretty much exactly Rob described in Leveraged Trading book, except that I only trade futures (book talks about other leveraged products as well).

Some highlights:
- 3 types of trading rules, moving average crossover (6 variations), breakout (6 variations) and carry. About 3 weeks ago, I turned off carry for instruments where I'm holding the front contract
- 10 instruments, though I started with 3, and as I added capital, I onboarded more instruments
- not fully automated, I trigger the trades by running a script (more on this below)
- written in Python, currently ~1900 lines of code, plus 280 lines of TOML config
- I run it from my Linux desktop, once a day, though I might run price collection more often. I don't have any cron jobs for anything yet
- Interactive Brokers as a broker, with ib_insync library
- I use PostgreSQL as a database, including storing prices. It is hosted on Google Cloud
- yearly target risk: 25%. instrument risk target: 55%. (all values as suggested by Rob in Leveraged Trading)


As of January, this is what I do full time. I'm a software engineer by trade and after 10 years in the industry (last 6 in the big tech companies) I decided to take a break from that. To be clear, I didn't quit my day job to live from this, this is just something I'm really interested in and I decided to try it while I'm on my sabbatical.

I've read Rob's first book Systematic Trading back in 2015 and that's what got me started. I recall Rob somewhere said (I think it was on one podcast where the question was "what's your secret" or something similar, or maybe here somewhere in this thread) that if you read his first book, his blog and this thread on ET, you can replicate 95% of what he does, so that's what I did
First system I implemented was based off ST, back in 2016-2017. I traded it for a bit, but realized I have huge gaps in understanding anything finance related. I'm ok with software side of things, especially Python, but I had no idea what a future is, how can I buy something worth $150K with a fraction of that in my account and so on.

So when LT came out, I picked that up, and implemented this current system based on that, meanwhile learning more about markets/financial side of the problem.

The reason I'm running my system manually is that I'd like to get more confidence and see what I can expect from it once I fully automate it. For example, thins like: calculating PnL, how futures settlement works, what's a first notice date, how little liquidity do I need to trade and how to do a spread trade to roll. I'm quite new to all of those, so this is a great learning experience.

Currently the biggest issue I have is the human factor, which is a nice way to say that I mess with it. I've been doing discretionary trading (US stocks) for a few years now with some success, and that might be a problem here. By mess with it I mean that when a system says I should make a trade, I would wait until there would be a good setup to make the trade. That might be delaying it few hours to a day. Likewise, if I see a position I'm holding is not looking good (say I'm long, and there's a bear flag with some additional confirmation setting up) I would exit the position. In hindsight, this has paid of quite nicely over the last 3-4 weeks of high volatility, meaning, looking back, vast majority of those "interruptions" saved me good chunk of money. Rob nailed it in this Tweet:

https://twitter.com/x/status/1242409206424240128

That's something I just couldn't do.

I'm still thinking what to do about that, there are couple of options. First one is to just stop doing it, and trust in the backtest that things play out well in the long term. Another option is to see when and why do I make these adjustments and see if I can automate that, backtest it and try that. Lastly, I can just continue doing this, and be some sort of a semi-automated trader. For now, I'm writing these down and monitoring the outcomes.

Other than that, some future plans:
- tests. I'm really big believer in software testing to drive good architecture and enable rapid iterations, so adding these now. Some parts of the system are not well designed - main reason for that is my lack of understanding of the domain. For example, I took Rob's execution algo post
https://qoppac.blogspot.com/2014/10/the-worlds-simplest-execution-algorithim.html read it, and implemented my execution in one big function, mainly to flesh out unknowns. Now that I've been running that for a few weeks, I can begin to decompose that.
- PnL and accounting. Not much to say here, I'm currently going off of IB reports
- I did one stream on Twitch where I worked on the system, I might do some more of those

That's it, I'm happy to answer any questions!

Lastly, big thanks to Rob for all his work, I'm really not sure how I would do
any of this was it not for his books and stuff he gives for free. When I'm in
London, I owe you (more than) a few pints.

 

[globalarbtrader]

Hey folks,
I've been lurking around for quite a while, so decided to do a short writeup of what I'm doing, for those who are interested. This is a bit lengthy, so I'll put the important stuff at the top

I've been trading my system for 2 full months (first trade was placed 1/29) so this is as good time as any. Performance so far 26.97% up (IB somewhere shows that, somewhere 27.99%). I do realize that large part of that is luck, and second large part is recent market conditions.

(I don't have proper PnL and accounting implemented yet, so here are some screenshots from IB.)

View attachment 223628

View attachment 223629

Current positions:
View attachment 223630


TL;DR - I'm doing pretty much exactly Rob described in Leveraged Trading book, except that I only trade futures (book talks about other leveraged products as well).

Some highlights:
- 3 types of trading rules, moving average crossover (6 variations), breakout (6 variations) and carry. About 3 weeks ago, I turned off carry for instruments where I'm holding the front contract
- 10 instruments, though I started with 3, and as I added capital, I onboarded more instruments
- not fully automated, I trigger the trades by running a script (more on this below)
- written in Python, currently ~1900 lines of code, plus 280 lines of TOML config
- I run it from my Linux desktop, once a day, though I might run price collection more often. I don't have any cron jobs for anything yet
- Interactive Brokers as a broker, with ib_insync library
- I use PostgreSQL as a database, including storing prices. It is hosted on Google Cloud
- yearly target risk: 25%. instrument risk target: 55%. (all values as suggested by Rob in Leveraged Trading)


As of January, this is what I do full time. I'm a software engineer by trade and after 10 years in the industry (last 6 in the big tech companies) I decided to take a break from that. To be clear, I didn't quit my day job to live from this, this is just something I'm really interested in and I decided to try it while I'm on my sabbatical.

I've read Rob's first book Systematic Trading back in 2015 and that's what got me started. I recall Rob somewhere said (I think it was on one podcast where the question was "what's your secret" or something similar, or maybe here somewhere in this thread) that if you read his first book, his blog and this thread on ET, you can replicate 95% of what he does, so that's what I did
First system I implemented was based off ST, back in 2016-2017. I traded it for a bit, but realized I have huge gaps in understanding anything finance related. I'm ok with software side of things, especially Python, but I had no idea what a future is, how can I buy something worth $150K with a fraction of that in my account and so on.

So when LT came out, I picked that up, and implemented this current system based on that, meanwhile learning more about markets/financial side of the problem.

The reason I'm running my system manually is that I'd like to get more confidence and see what I can expect from it once I fully automate it. For example, thins like: calculating PnL, how futures settlement works, what's a first notice date, how little liquidity do I need to trade and how to do a spread trade to roll. I'm quite new to all of those, so this is a great learning experience.

Currently the biggest issue I have is the human factor, which is a nice way to say that I mess with it. I've been doing discretionary trading (US stocks) for a few years now with some success, and that might be a problem here. By mess with it I mean that when a system says I should make a trade, I would wait until there would be a good setup to make the trade. That might be delaying it few hours to a day. Likewise, if I see a position I'm holding is not looking good (say I'm long, and there's a bear flag with some additional confirmation setting up) I would exit the position. In hindsight, this has paid of quite nicely over the last 3-4 weeks of high volatility, meaning, looking back, vast majority of those "interruptions" saved me good chunk of money. Rob nailed it in this Tweet:

https://twitter.com/x/status/1242409206424240128

That's something I just couldn't do.

I'm still thinking what to do about that, there are couple of options. First one is to just stop doing it, and trust in the backtest that things play out well in the long term. Another option is to see when and why do I make these adjustments and see if I can automate that, backtest it and try that. Lastly, I can just continue doing this, and be some sort of a semi-automated trader. For now, I'm writing these down and monitoring the outcomes.

Other than that, some future plans:
- tests. I'm really big believer in software testing to drive good architecture and enable rapid iterations, so adding these now. Some parts of the system are not well designed - main reason for that is my lack of understanding of the domain. For example, I took Rob's execution algo post
https://qoppac.blogspot.com/2014/10/the-worlds-simplest-execution-algorithim.html read it, and implemented my execution in one big function, mainly to flesh out unknowns. Now that I've been running that for a few weeks, I can begin to decompose that.
- PnL and accounting. Not much to say here, I'm currently going off of IB reports
- I did one stream on Twitch where I worked on the system, I might do some more of those

That's it, I'm happy to answer any questions!

Lastly, big thanks to Rob for all his work, I'm really not sure how I would do
any of this was it not for his books and stuff he gives for free. When I'm in
London, I owe you (more than) a few pints.

That's a fantastic post @wopr
I'm very touched And I won't forget the offer of pints!

GAT

 

[ValeryN]

Hey folks,
I've been lurking around for quite a while, so decided to do a short writeup of what I'm doing, for those who are interested. This is a bit lengthy, so I'll put the important stuff at the top

I've been trading my system for 2 full months (first trade was placed 1/29) so this is as good time as any. Performance so far 26.97% up (IB somewhere shows that, somewhere 27.99%). I do realize that large part of that is luck, and second large part is recent market conditions.
...

Great start! Good luck on your journey.

If automation is a goal - keep a track of what 100% automated version of your system would do compared to you, as well as a detailed record of your discretionary overrides/whys. Then you can see if removing you from its execution would actually improve things over time.

Val

 

[Kernfusion]

Hey folks,
I've been lurking around for quite a while, so decided to do a short writeup of what I'm doing, for those who are interested. This is a bit lengthy, so I'll put the important stuff at the top

I've been trading my system for 2 full months (first trade was placed 1/29) so this is as good time as any. Performance so far 26.97% up (IB somewhere shows that, somewhere 27.99%). I do realize that large part of that is luck, and second large part is recent market conditions.

(I don't have proper PnL and accounting implemented yet, so here are some screenshots from IB.)

View attachment 223628

View attachment 223629

Current positions:
View attachment 223630


TL;DR - I'm doing pretty much exactly Rob described in Leveraged Trading book, except that I only trade futures (book talks about other leveraged products as well).

Some highlights:
- 3 types of trading rules, moving average crossover (6 variations), breakout (6 variations) and carry. About 3 weeks ago, I turned off carry for instruments where I'm holding the front contract
- 10 instruments, though I started with 3, and as I added capital, I onboarded more instruments
- not fully automated, I trigger the trades by running a script (more on this below)
- written in Python, currently ~1900 lines of code, plus 280 lines of TOML config
- I run it from my Linux desktop, once a day, though I might run price collection more often. I don't have any cron jobs for anything yet
- Interactive Brokers as a broker, with ib_insync library
- I use PostgreSQL as a database, including storing prices. It is hosted on Google Cloud
- yearly target risk: 25%. instrument risk target: 55%. (all values as suggested by Rob in Leveraged Trading)


As of January, this is what I do full time. I'm a software engineer by trade and after 10 years in the industry (last 6 in the big tech companies) I decided to take a break from that. To be clear, I didn't quit my day job to live from this, this is just something I'm really interested in and I decided to try it while I'm on my sabbatical.

I've read Rob's first book Systematic Trading back in 2015 and that's what got me started. I recall Rob somewhere said (I think it was on one podcast where the question was "what's your secret" or something similar, or maybe here somewhere in this thread) that if you read his first book, his blog and this thread on ET, you can replicate 95% of what he does, so that's what I did
First system I implemented was based off ST, back in 2016-2017. I traded it for a bit, but realized I have huge gaps in understanding anything finance related. I'm ok with software side of things, especially Python, but I had no idea what a future is, how can I buy something worth $150K with a fraction of that in my account and so on.

So when LT came out, I picked that up, and implemented this current system based on that, meanwhile learning more about markets/financial side of the problem.

The reason I'm running my system manually is that I'd like to get more confidence and see what I can expect from it once I fully automate it. For example, thins like: calculating PnL, how futures settlement works, what's a first notice date, how little liquidity do I need to trade and how to do a spread trade to roll. I'm quite new to all of those, so this is a great learning experience.

Currently the biggest issue I have is the human factor, which is a nice way to say that I mess with it. I've been doing discretionary trading (US stocks) for a few years now with some success, and that might be a problem here. By mess with it I mean that when a system says I should make a trade, I would wait until there would be a good setup to make the trade. That might be delaying it few hours to a day. Likewise, if I see a position I'm holding is not looking good (say I'm long, and there's a bear flag with some additional confirmation setting up) I would exit the position. In hindsight, this has paid of quite nicely over the last 3-4 weeks of high volatility, meaning, looking back, vast majority of those "interruptions" saved me good chunk of money. Rob nailed it in this Tweet:

https://twitter.com/x/status/1242409206424240128

That's something I just couldn't do.

I'm still thinking what to do about that, there are couple of options. First one is to just stop doing it, and trust in the backtest that things play out well in the long term. Another option is to see when and why do I make these adjustments and see if I can automate that, backtest it and try that. Lastly, I can just continue doing this, and be some sort of a semi-automated trader. For now, I'm writing these down and monitoring the outcomes.

Other than that, some future plans:
- tests. I'm really big believer in software testing to drive good architecture and enable rapid iterations, so adding these now. Some parts of the system are not well designed - main reason for that is my lack of understanding of the domain. For example, I took Rob's execution algo post
https://qoppac.blogspot.com/2014/10/the-worlds-simplest-execution-algorithim.html read it, and implemented my execution in one big function, mainly to flesh out unknowns. Now that I've been running that for a few weeks, I can begin to decompose that.
- PnL and accounting. Not much to say here, I'm currently going off of IB reports
- I did one stream on Twitch where I worked on the system, I might do some more of those

That's it, I'm happy to answer any questions!

Lastly, big thanks to Rob for all his work, I'm really not sure how I would do
any of this was it not for his books and stuff he gives for free. When I'm in
London, I owe you (more than) a few pints.

Yeah, maybe there is something to that "manual\discretionary trading" thing after all I started to watch some day-traders on youtube (quite a contagious thing) and it seems they make money.. What they do makes little sense sometimes, like one guy seems to be trading a very quick break-out-like strategy (like minutes to tens of minutes trades) on cheap (~2$) stocks at the market open when the stock gaps up in pre-market and then pulls back a little or something like that.. You never know of course if they show everything, good and bad trades, and there's really no way to systematically check these things because they have so many subjective rules, like "after a loosing day I will only trade half of my usual capital because I'll be in a wrong emotional state"..
I'm not touching my system at all, only looking at daily reports, sometimes not even that (I'm also a developer and I spent a lot of quality after-work hours writing it, so I refuse to put any more work into it, also I wouldn't know what to do manually anyway ), and I'm only up like 4% since the beginning (at the worst market-crash time I was up like 12%, then gave back almost all of it, now it's a little up again). So if I stopped V2X and MXP manually I'd be up additional 20k (or 20%)..
But what's interesting is that my paper system, which is doing the same thing but on 31 instruments and 310k (instead of 14 instruments and 100k in PROD) made ~50k in the last month, and didn't really give much back. So there's a hope that it can be much better if only I could find another 200k of capital somewhere

Btw, I'm also not a professional trader, I learned everything myself (a good chunk of it from Rob's material ), so I have gaps in my knowledge. For example the question I wanted to understand for a while is how exactly margin requirements work in a regular Reg-T account (NOT portfolio margin) when trading both futures and stocks?
@globalarbtrader you mentioned a couple of times, that you financed your account with storks\ETFs, but I thought if I buy(hold) stocks on my account it reduces my available margin.. (or you can only do it when you're on portfolio margin, not Reg-T?)

So here's my understanding, of how Reg-T margin account works (Interactive Brokers), please correct me if I'm wrong:
Say I have 100k cash, Reg-T account means I can hold overnight maximum 2x or 200k worth of stocks\ETFs or alternatively I could hold 10 Futures contracts with 10k overnight margin requirement each.
So if I for example hold 5 of these futures that leaves me with 50k of cash, and I can borrow up to 50k more from the broker and buy and hold overnight maximum 100k of stocks\ETFs in addition to my 5 futures contracts. (of course that leaves no room for positions going against me).
So with 100k cash I could either have 200k in stocks or 10 futures with 10k margin each or 5 futures and 100k in stocks.
Or is this not how it works ?

 

[HobbyTrading]

So here's my understanding, of how Reg-T margin account works (Interactive Brokers), please correct me if I'm wrong:
Say I have 100k cash, Reg-T account means I can hold overnight maximum 2x or 200k worth of stocks\ETFs or alternatively I could hold 10 Futures contracts with 10k overnight margin requirement each.
So if I for example hold 5 of these futures that leaves me with 50k of cash, and I can borrow up to 50k more from the broker and buy and hold overnight maximum 100k of stocks\ETFs in addition to my 5 futures contracts. (of course that leaves no room for positions going against me).
So with 100k cash I could either have 200k in stocks or 10 futures with 10k margin each or 5 futures and 100k in stocks.
Or is this not how it works ?

This is indeed how it works, at IB at least. For a while I combined an ETF trading strategy and the futures trading strategy in one account. Sometimes a futures trade was blocked by IB because my total margin usage would exceed the limit. What I then used to do was to either throttle back the ETF system, resulting in more margin available for futures. Or look at those futures instruments which use a lot of margin per contract (yes, I was looking at you V2TX) and throttle back those instruments.
In the end I noticed that my ETF trading system was not adding much profit, so I decided to close it. Since then I hardly ever reach the margin limitation.

 

[Kernfusion]

This is indeed how it works, at IB at least. For a while I combined an ETF trading strategy and the futures trading strategy in one account. Sometimes a futures trade was blocked by IB because my total margin usage would exceed the limit. What I then used to do was to either throttle back the ETF system, resulting in more margin available for futures. Or look at those futures instruments which use a lot of margin per contract (yes, I was looking at you V2TX) and throttle back those instruments.
In the end I noticed that my ETF trading system was not adding much profit, so I decided to close it. Since then I hardly ever reach the margin limitation.

Thanks @HobbyTrading I see.
I'm actually also running another stock\ETF strategy on the same account, in fact I started with it. It's a pairs-trading mean reversion thing, which enters when spread between 2 related things widens on a moving average with standard deviation bands around it and exits when it closes back. it was doing OK for a first half a year (maybe 5% annualised return), but then it pretty-much started to continuously loose because the number of unhinged spreads making big sudden losses was overwhelming small gains from the still working spreads. I then introduced a stop-loss based on what Rob described in the last book, that didn't help much, so I closed most of the pairs and only trading 4 or 5. Mostly because I can't justify shutting down a system that I put so much work in, even if it's not really making anything

 

[tradrjoe]

Thanks @HobbyTrading I see.
I'm actually also running another stock\ETF strategy on the same account, in fact I started with it. It's a pairs-trading mean reversion thing, which enters when spread between 2 related things widens on a moving average with standard deviation bands around it and exits when it closes back. it was doing OK for a first half a year (maybe 5% annualised return), but then it pretty-much started to continuously loose because the number of unhinged spreads making big sudden losses was overwhelming small gains from the still working spreads. I then introduced a stop-loss based on what Rob described in the last book, that didn't help much, so I closed most of the pairs and only trading 4 or 5. Mostly because I can't justify shutting down a system that I put so much work in, even if it's not really making anything

Have you done any backtesting on that mean reverting strategy? I would be surprised if it would actually be profitable in today's environment as I've read that generic pairs MV had not worked since the early 2000s.

 

[Kernfusion]

Have you done any backtesting on that mean reverting strategy? I would be surprised if it would actually be profitable in today's environment as I've read that generic pairs MV had not worked since the early 2000s.

yes, I did back-testing, that's how I selected the pairs to trade so of course they were all crazy-profitable in the backtest (that's why I selected them), and out of sample - not so much I mean I know it's overfitting, and I didn't expect them to continue to be as profitable as the best pairs in backtest, but what other criteria do I have for selecting the pairs? (I also ran cointegration test of course..) I can't just assume that all stock pairs mean-revert and trade them all as we do with futures ("all futures trend")., and selecting them by "logical relationship", like GM vs Ford, didn't look good even in the backtest..
I actually saw a strategy on Quantpedia that if you repeat selections of which pairs to trade every half a year it works, but haven't tried that..