Funny one. I may have lived too long in Japan though.
Fully automated futures trading
- Thread starter globalarbtrader
- Start date
So it's basically the end of the year. If I still had a proper job, this would be the last day that anyone with any seniority would be in the office until January 4th. When I worked in IB as a junior trader I did have to work in that weird period between Christmas and New years eve when markets are open but nothing is happening. Of course back in those days you could get away with a little lunchtime drinking courtesy of your friendly neighborhood broker, or knocking off at 3pm (okay twenty years before you could do that *every single day* rather than just in late December).
Now because of COVID very few people will be in the office next week anyway, and I suspect that even for quite senior bankers it will become more appealing to 'just going to the study to "check my emails" darling' by December 27th than spend any more time with your family (since until 2020 you probably spent more time with your secretary than your wife - nudge nudge - and it's been a bit of a shock to the system), or your in laws (assuming they've made the cut into your Christmas bubble).
As I can now do whatever I bleedin' like, it will be no surprise that today is going to be my last day of 'work' for a couple of weeks. Right now I'm mostly refactoring code (see discussion later), and since I never test everything properly I don't want to make any changes right now so that I can rely on the system running smoothly for the next couple of weeks (more on that later as well). Everything ran last night and today without any issues so the changes I made yesterday will be the last I do until January.
So it seems like an appropriate time to do some kind of 'review of the year'
First performance. I don't normally do a proper full on calendar year peformance review, but people like that sort of thing so here are some figures, for the period up to yesterday (for my trading account, not my long only stuff):
Overall 13.4%
Core futures trading 16.6%
ETFs, hedge, FX movements -3.2%
I paid around 0.5% of account value in commissions, which is around 1.6% in SR units. Of course there's slippage as well, which will probably double that number (my trade report code crashed when trying to do the entire year, and I can't be assed to fix it in case I introduce a bug: see earlier comments).
Here's the tail of the tape in a bit more detail (numbers on left are pandas indices, ignore):
Apart from banging my usual drum about diversification, it's difficult to know what to say about the above.
I would do you a nice chart, but I haven't written the code. I promise my April review will be charts galore. Suffice to say that looking at Fundseeder (which I see isn't updating again; I fear it will be like my other favourite fintech firm Quantopian and vanish, killed by a market that doesn't like people who try and do things properly) I basically made all this money in January, and I've since run flat; up a bit, down a bit. That means the p&l for the financial year, which I actually care about, is -4% (incidentally, I've never lost money in a financial year on my entire trading account, though I came close in 2017. It's looking like this will be the year). Current d/d is around 11.5%.
OK that's the boring stuff over with. How has 2020 been for everyone? Stupid question, perhaps. I had a quick look at my new years resolutions- mostly stuff about spending less time on social media and exercising more- and they haven't worked out so well.
I also had a bit of fun looking back to a post I made almost exactly a year ago, which I quote from here:
"Since the summer I've rethought my plan to shut down my existing trading system and pysystemtrade in favour of a completely new platform. Instead I've gone back to plan A: keep the existing system running whilst adding production logic to pysystemtrade so it will be able to replace the legacy code. I'm also going to implement my new ideas on pysystemtrade, although this will require some tweaking to the current system it's worth doing.
....
My initial goal is to get FX and futures price collection running, and in preparation for that I have been copying across my legacy data and rebuilding the adjusted price curves for each market. To help with that process I've taken out a one year subscription to another price service. It cost $300, but it means I've been able to backfill some more data history. I also want to add price collection for some futures I don't currently trade, since the new system (which exists only in my head at the moment).
As you may have noticed, I've had an early new years resolution of writing one new blog post a month, which I've so far managed to stick to.
I've had a lot of conference requests for 2020, and so far I've said yes to all of them. Definites are New York, possibles are Australia, Hamburg and Istanbul. I'm also doing some talks at local universities. I start teaching in January, so it's looking pretty busy for next year."
OK we can all have a good laugh about the fact I didn't end up flying all over the world talking to people; in fact I haven't been on an aircraft since a skiing trip to Northern Italy in February that may well have made me a localised patient zero for all I know. But otherwise the plan has gone pretty well. I did close down my existing system, albeit I was forced to in the end. Pysystemtrade is now a fully fledged trading system that runs pretty well, and I'm very comfortable to let it run for two weeks with no intervention. As well as the daily emails it sends me, I even have a little web page now which monitors the status of every process on the machine (the first HTML I have done in over 20 years):
(yeah I know, I should have spent months learning the docker/kubernetes/puppet/chef/ansible stack and do this properly, instead of a few hours writing my own process manager)
So what does 2021 look like (ignoring, again, the pandemic sized pachyderm in the parlour). Much to my surprise my teaching contract was renewed, so I have to prep my course which starts in late January. That will be a bit more work than usual since it's going to be online, and I need to create a lot more resources (not easy, as there is a lot of confusion about exactly how this is going to be delivered, plus a beweildering array of advice when I just want to be told what to do).
The first of my new trading PCs should be delivered on Monday, so I'll be configuring that, and perhaps even building another one (see previous posts). I'm about halfway through a refactoring/tidying up of pysystemtrade to ensure everything is reasonably bullet proof and also extendable. Then there's a relatively short list of features I want to add to it before heading into my first piece new trading system research, what I've called 'stage 5' of the 'project'. Most of these are things to make the system easier to run more easily on a day to day basis; the only one that might cause some pain is upgrading pandas to 1.0
Stage 5 basically involves an optimiser that utilities small amounts of capital more efficiently, and can thus 'trade' (or at least generate signals for) a very large number of markets (including markets I can't actually trade, because the undelayed data is too expensive or the lot size too big). I'm fairly confident I can finish stage 5 and get something implemented fairly quickly. My other major goal is to get a proposal written for my next book, and start writing it. Working backwards, if I can get a proposal done by April then the book will come out in October 2022. So there will probably be some overlap; I find I'm happier if I'm not 100% on one project at a time (context switching in weekly blocks works pretty well).
Anyway, good wishes to all and all that stuff, and let's hope things look generally better in a years time.
GAT
Now because of COVID very few people will be in the office next week anyway, and I suspect that even for quite senior bankers it will become more appealing to 'just going to the study to "check my emails" darling' by December 27th than spend any more time with your family (since until 2020 you probably spent more time with your secretary than your wife - nudge nudge - and it's been a bit of a shock to the system), or your in laws (assuming they've made the cut into your Christmas bubble).
As I can now do whatever I bleedin' like, it will be no surprise that today is going to be my last day of 'work' for a couple of weeks. Right now I'm mostly refactoring code (see discussion later), and since I never test everything properly I don't want to make any changes right now so that I can rely on the system running smoothly for the next couple of weeks (more on that later as well). Everything ran last night and today without any issues so the changes I made yesterday will be the last I do until January.
So it seems like an appropriate time to do some kind of 'review of the year'
First performance. I don't normally do a proper full on calendar year peformance review, but people like that sort of thing so here are some figures, for the period up to yesterday (for my trading account, not my long only stuff):
Overall 13.4%
Core futures trading 16.6%
ETFs, hedge, FX movements -3.2%
I paid around 0.5% of account value in commissions, which is around 1.6% in SR units. Of course there's slippage as well, which will probably double that number (my trade report code crashed when trying to do the entire year, and I can't be assed to fix it in case I introduce a bug: see earlier comments).
Here's the tail of the tape in a bit more detail (numbers on left are pandas indices, ignore):
Code:
codes pandl
0 V2X -4.41
1 SOYBEAN -1.96
2 KR10 -1.48
3 VIX -1.42
4 MXP -1.42
5 COPPER -0.89
6 CAC -0.84
7 SP500 -0.78
8 JPY -0.72
9 AEX -0.67
10 GOLD -0.27
11 KR3 -0.08
12 EUR -0.03
13 US5 0.00
14 GBP 0.05
15 US10 0.07
16 OAT 0.09
17 US2 0.20
18 KOSPI 0.33
19 GAS_US 0.53
20 NZD 0.56
21 BOBL 0.56
22 PLAT 0.73
23 NASDAQ 0.73
24 LIVECOW 0.87
25 CRUDE_W 0.97
26 BUND 1.12
27 CORN 1.36
28 LEANHOG 1.53
29 AUD 3.08
30 BTP 3.68
31 WHEAT 4.14
32 EDOLLAR 4.84
33 PALLAD 6.11
==================
P&L by asset class
==================
codes pandl
Vol -5.83
Equity -1.17
OilGas 1.50
FX 1.52
Bond 4.17
STIR 4.84
Metals 5.68
Ags 5.93Apart from banging my usual drum about diversification, it's difficult to know what to say about the above.
I would do you a nice chart, but I haven't written the code. I promise my April review will be charts galore. Suffice to say that looking at Fundseeder (which I see isn't updating again; I fear it will be like my other favourite fintech firm Quantopian and vanish, killed by a market that doesn't like people who try and do things properly) I basically made all this money in January, and I've since run flat; up a bit, down a bit. That means the p&l for the financial year, which I actually care about, is -4% (incidentally, I've never lost money in a financial year on my entire trading account, though I came close in 2017. It's looking like this will be the year). Current d/d is around 11.5%.
OK that's the boring stuff over with. How has 2020 been for everyone? Stupid question, perhaps. I had a quick look at my new years resolutions- mostly stuff about spending less time on social media and exercising more- and they haven't worked out so well.
I also had a bit of fun looking back to a post I made almost exactly a year ago, which I quote from here:
"Since the summer I've rethought my plan to shut down my existing trading system and pysystemtrade in favour of a completely new platform. Instead I've gone back to plan A: keep the existing system running whilst adding production logic to pysystemtrade so it will be able to replace the legacy code. I'm also going to implement my new ideas on pysystemtrade, although this will require some tweaking to the current system it's worth doing.
....
My initial goal is to get FX and futures price collection running, and in preparation for that I have been copying across my legacy data and rebuilding the adjusted price curves for each market. To help with that process I've taken out a one year subscription to another price service. It cost $300, but it means I've been able to backfill some more data history. I also want to add price collection for some futures I don't currently trade, since the new system (which exists only in my head at the moment).
As you may have noticed, I've had an early new years resolution of writing one new blog post a month, which I've so far managed to stick to.
I've had a lot of conference requests for 2020, and so far I've said yes to all of them. Definites are New York, possibles are Australia, Hamburg and Istanbul. I'm also doing some talks at local universities. I start teaching in January, so it's looking pretty busy for next year."
OK we can all have a good laugh about the fact I didn't end up flying all over the world talking to people; in fact I haven't been on an aircraft since a skiing trip to Northern Italy in February that may well have made me a localised patient zero for all I know. But otherwise the plan has gone pretty well. I did close down my existing system, albeit I was forced to in the end. Pysystemtrade is now a fully fledged trading system that runs pretty well, and I'm very comfortable to let it run for two weeks with no intervention. As well as the daily emails it sends me, I even have a little web page now which monitors the status of every process on the machine (the first HTML I have done in over 20 years):
Code:
Last update 2020-12-18 10:29:07.064857
run_capital_update Started 12/18 01:00:00 ended 12/17 19:00:28 Status GO PID 107046.0 running
run_daily_prices_updates Started 12/17 20:05:18 ended 12/17 21:40:48 Status GO PID 0.0 not running
run_systems Started 12/17 21:40:48 ended 12/17 22:13:25 Status GO PID 0.0 not running
run_strategy_order_generator Started 12/17 22:13:25 ended 12/17 22:17:00 Status GO PID 0.0 not running
run_stack_handler Started 12/18 00:30:23 ended 12/17 19:30:18 Status GO PID 106943.0 running
run_reports Started 12/17 22:55:27 ended 12/17 23:23:59 Status GO PID 0.0 not running
run_cleaners Started 12/17 22:00:18 ended 12/17 22:00:53 Status GO PID 0.0 not running
run_backups Started 12/17 22:05:21 ended 12/18 00:02:29 Status GO PID 0.0 not running
Status of run_reports changed from running to not running at 12/16 23:27:17
Status of run_backups changed from running to not running at 12/17 00:07:19
Status of run_stack_handler changed from not running to running at 12/17 00:32:20
Status of run_capital_update changed from not running to running at 12/17 01:02:21
Status of run_capital_update changed from running to not running at 12/17 19:03:20
Status of run_stack_handler changed from running to not running at 12/17 19:33:21
Status of run_daily_prices_updates changed from not running to running at 12/17 20:08:23
Status of run_daily_prices_updates changed from running to not running at 12/17 21:43:27
Status of run_systems changed from not running to running at 12/17 21:43:27
Status of run_backups changed from not running to running at 12/17 22:08:28
Status of run_systems changed from running to not running at 12/17 22:13:28
Status of run_strategy_order_generator changed from not running to running at 12/17 22:13:28
Status of run_strategy_order_generator changed from running to not running at 12/17 22:18:28
Status of run_reports changed from not running to running at 12/17 22:58:30
Status of run_reports changed from running to not running at 12/17 23:28:32
Status of run_backups changed from running to not running at 12/18 00:03:33
Status of run_stack_handler changed from not running to running at 12/18 00:33:34
Status of run_capital_update changed from not running to running at 12/18 01:03:36(yeah I know, I should have spent months learning the docker/kubernetes/puppet/chef/ansible stack and do this properly, instead of a few hours writing my own process manager)
So what does 2021 look like (ignoring, again, the pandemic sized pachyderm in the parlour). Much to my surprise my teaching contract was renewed, so I have to prep my course which starts in late January. That will be a bit more work than usual since it's going to be online, and I need to create a lot more resources (not easy, as there is a lot of confusion about exactly how this is going to be delivered, plus a beweildering array of advice when I just want to be told what to do).
The first of my new trading PCs should be delivered on Monday, so I'll be configuring that, and perhaps even building another one (see previous posts). I'm about halfway through a refactoring/tidying up of pysystemtrade to ensure everything is reasonably bullet proof and also extendable. Then there's a relatively short list of features I want to add to it before heading into my first piece new trading system research, what I've called 'stage 5' of the 'project'. Most of these are things to make the system easier to run more easily on a day to day basis; the only one that might cause some pain is upgrading pandas to 1.0
Stage 5 basically involves an optimiser that utilities small amounts of capital more efficiently, and can thus 'trade' (or at least generate signals for) a very large number of markets (including markets I can't actually trade, because the undelayed data is too expensive or the lot size too big). I'm fairly confident I can finish stage 5 and get something implemented fairly quickly. My other major goal is to get a proposal written for my next book, and start writing it. Working backwards, if I can get a proposal done by April then the book will come out in October 2022. So there will probably be some overlap; I find I'm happier if I'm not 100% on one project at a time (context switching in weekly blocks works pretty well).
Anyway, good wishes to all and all that stuff, and let's hope things look generally better in a years time.
GAT
So here's my charts (they're generated automatically every week, so it's no trouble to copy them here). As I mentioned, I'm running 2 systems in parallel: PROD with 13 instruments on 100k base capital and PAPER with 36 instruments and currently 500k USD of virtual money. Same rules and rule-weights, same vol target (25%), different instruments and base capital, also the paper system was running longer and had more adjustments like adding more instruments and virtual capital.
Here's PROD graphs:
Here's PAPER:
So I guess things look as expected - higher realized %-Volatility in PROD because of smaller number of instruments but same IDM cap(2.5) on both systems, more profits in paper, which mostly came from 2 long-term bonds (BTP and ZB) which I couldn't include in PROD. The biggest losers are Vix\V2tx, MXP, JPY(PAPER only), Wheat(PAPER only), ESTX50(PAPER only) ..
How's the retirement in general Rob? in terms of the necessity to "have something to do" or "have obligations" to feel happy?
I remember Hemmingway said somewhere (paraphrasing) "If I don't write at least several pages of a book every day I can't enjoy the rest of my life" (Meaning I guess drinking 25 daiquiris in the local Havana bar in the evening
). Is that really true that even after retirement people need to have some kind of "work" for themselves otherwise it just gets tremendously boring just sitting around doing nothing, or maybe it's different for individual people? Buddhists seem to have found another way - to simply get rid of all desires and passions so one can want nothing and do nothing and be happy...
To me I guess doing something is still more fun than not doing anything, especially when you can choose what to do, not that it's a pressing issue, as for many more years I still have to continue doing things that I must instead of want or become hungry and homeless
Anyway, happy holidays everyone!
Financially: thus far extremely well. I plan to write a (rare) update to the journal I'm having here at ET at the end of the year.
Otherwise: very boring year due to limitations in movement caused by the pandemic. I can't travel international as the country where I reside won't allow me to return once I leave. So I have to stay here. Traveling domestically is not fun in the current situation. Hopefully all this will improve in 2021.
Hello Robert,
I've ran into some issues with regards to calculating carry.
I'll try to show them with a couple of examples: WTI Crude Oil (CL) and Natural Gas (NG). Both of them have 12 active months in a year, so a monthly roll over (I know you use different scheme for your trading, like only trading December contract for CL).
If I can trade the front month and use the method of calculating the carry by: (price of front month - price of the next month) x 12, for NG and CL examples, I get the following net expected annualised returns (in points, no volatility normalisation applied at this point) over the last 6 years, smoothed with a simple 30 day moving average.
For Crude Oil:
looks quite OK
However for Natural Gas there seems to be some kind of seasonality in the series:
does this look OK? The sign of carry is changing quite often, which is raising some suspicions. Should the measurement of carry for certain markets be done differently than simply taking the difference between two adjacent contracts perhaps? The term structure for Nat Gas looks abnormal:
At the beginning it goes slightly downward but in the longer term it goes upwards.
I've ran into some issues with regards to calculating carry.
I'll try to show them with a couple of examples: WTI Crude Oil (CL) and Natural Gas (NG). Both of them have 12 active months in a year, so a monthly roll over (I know you use different scheme for your trading, like only trading December contract for CL).
If I can trade the front month and use the method of calculating the carry by: (price of front month - price of the next month) x 12, for NG and CL examples, I get the following net expected annualised returns (in points, no volatility normalisation applied at this point) over the last 6 years, smoothed with a simple 30 day moving average.
For Crude Oil:
looks quite OK
However for Natural Gas there seems to be some kind of seasonality in the series:
does this look OK? The sign of carry is changing quite often, which is raising some suspicions. Should the measurement of carry for certain markets be done differently than simply taking the difference between two adjacent contracts perhaps? The term structure for Nat Gas looks abnormal:
At the beginning it goes slightly downward but in the longer term it goes upwards.
It is known that there is quite a seasonal influence as gas consumption in the US is much higher in the winter than in the summer. Therefore are prices in the winter higher than during summer. This might be what you are seeing in the chart?
You'd expect seasonality for energy futures, which is why for Crude I only trade Z. Sadly that isn't an option for NG. That's also why the term structure looks like that; and it will vary depending on when during the year you are looking at it.
GAT
How's the retirement in general Rob? in terms of the necessity to "have something to do" or "have obligations" to feel happy?
I remember Hemmingway said somewhere (paraphrasing) "If I don't write at least several pages of a book every day I can't enjoy the rest of my life" (Meaning I guess drinking 25 daiquiris in the local Havana bar in the evening
Buddhists seem to have found another way - to simply get rid of all desires and passions so one can want nothing and do nothing and be happy...
To me I guess doing something is still more fun than not doing anything, especially when you can choose what to do, not that it's a pressing issue, as for many more years I still have to continue doing things that I must instead of want or become hungry and homeless
Anyway, happy holidays everyone!
I think it depends on what kind of person you are. I am rubbish at doing nothing. Sometimes (when my wife is out obviously) I sit and watch TV at lunchtime, but I'm still multitasking, eating and reading at the same time.
I'm extremely lucky to have a thing that I'm really interested in doing which occupies my time (really a set of interconnected things), some of which I get paid for, others of which may in the long run earn some monetary reward. I think retirement would be impossible otherwise, at least for me.
If I lived in Havana, well maybe I'd feel differently (and maybe I'd already be dead from liver failure). But I don't think I am a very good Buddhist. Definitely more of the protestant work ethic (though I'm actually a Catholic...)
Harder is explaining to people exactly what it is I do. The answer depends on who I'm talking to. If it's a builder, I say I trade my own money. If it's a city pro I'm trying to make jealous, I say I'm retired. If it's an intellectual left wing type, I say I'm a writer and university lecturer (I might chuck in consultancy, to explain the disparity between my minimal working hours and fairly comfortable lifestyle) In practice, it's a combination of all of those...
GAT
How do you go about measuring carry for nat gas to generate the signal?
As for crude oil, do you simply look at the difference between CLZ21 and CLZ22?
It definitely has a seasonality pattern. Same goes for other commodities like grains. The problem is of course how to do carry trading, as per the book it is defined as a simple differential of two adjacent contracts, however this doesn't feel right for such commodities.