If the NASDAQ stays up, it won't be
Fully automated futures trading
- Thread starter globalarbtrader
- Start date
I'm close to HWM right now, but not quite there, I started 4.5 months ago, up 9%, but it's changing quite a bit, several days ago it was like 4%. 3KTB and Corn made some money in the last couple of days, I'm currently long 3KTB and short Corn.. (hopefully it's not a price glitch, I'm getting that for 3KTB sometimes..)
Just checked my NASDAQ, I'm currently flat, the carry forecasts are all around zero, all trend variations are around 7-9 (out of max 20), the system's raw target position is around 0.2 contracts, I guess the still high estimated volatility and lack of capital are not allowing me to have a position in it..
Just checked my NASDAQ, I'm currently flat, the carry forecasts are all around zero, all trend variations are around 7-9 (out of max 20), the system's raw target position is around 0.2 contracts, I guess the still high estimated volatility and lack of capital are not allowing me to have a position in it..
Last edited:
an idea about automation: if it's really possible to put the whole system on a small computer like wopr did, it should surely be possible to put it on an average-performance modern android phone\tablet, and then just carry this thing around with you everywhere, let it connect to cell-network internet to do it's thing, especially if it's not event driven tick-by-tick system but a "several times a day on timer" one. It's becoming like a third option of doing it, the first 2 are a home computer under the table and a rented cloud server., and the benefit of this one is that you're always holding your physical trading computer in your hands, even when you're on vacation or away from home, so you can shut it down, fix it, etc., and there's no chance that in a critical moment you will not have a connection between you and it because the internet channel is down somewhere in between..
A similar option was available before though, by just putting the system on a laptop with a cellular internet connection through the phone..
On the other hand, you still want to be able to sleep while the system is trading at certain times of the day for some markets, so you can't always baby-sit your trading computer, so there should be a good reliability and trust in the system anyway..
A similar option was available before though, by just putting the system on a laptop with a cellular internet connection through the phone..
On the other hand, you still want to be able to sleep while the system is trading at certain times of the day for some markets, so you can't always baby-sit your trading computer, so there should be a good reliability and trust in the system anyway..
Last edited:
This weeks update. TLDR: I have made money and I have written some code, but also deleted some.
Let's start with performance, like most people this has been a pretty good few weeks. Here are some graphs, zooming in on more recent periods of time (total, rolling year, YTD)
https://photos.app.goo.gl/B9zXmRfTJq27mpgm7
https://photos.app.goo.gl/eDCzbrs7QXNuJm5u8
https://photos.app.goo.gl/mxJNVG1FyG3xnZCCA
I've now also got decent P&L reporting so here's a more granular picture for the last 6 months.
Code:
********************************************************************************
P&L report produced on 2020-06-26 11:40:24.401362 from 2019-12-29 11:38:17.779093 to 2020-06-26 11:38:17.779085
********************************************************************************
Total p&l is 21.575%
====================================
P&L by instrument for all strategies
====================================
instrument pandl
0 V2X -4.30
1 MXP -3.77
2 VIX -1.93
3 PLAT -1.44
4 OAT -1.01
5 JPY -0.97
6 COPPER -0.89
7 CAC -0.84
8 SP500 -0.78
9 AEX -0.67
10 GOLD -0.27
11 EUR -0.03
12 KR10 -0.00
13 US5 0.00
14 GBP 0.05
15 US2 0.20
16 KOSPI 0.33
17 WHEAT 0.34
18 BOBL 0.38
19 NZD 0.56
20 US10 0.57
21 BTP 0.66
22 KR3 0.68
23 NASDAQ 0.73
24 CRUDE_W 0.97
25 BUND 1.37
26 GAS_US 1.51
27 LEANHOG 1.80
28 SOYBEAN 2.13
29 AUD 2.13
30 LIVECOW 2.55
31 CORN 3.31
32 EDOLLAR 5.27
33 PALLAD 6.11
34 EUROSTX 13.61
Total futures p&l is 28.410%
Residual p&l is -6.835%
********************************************************************************
END OF P&L REPORT
********************************************************************************For interpretation, my futures trading p&l will be equal to 'total futures p&l 28.4%' less Eurostoxx 13.6% (only traded as a hedge) which is 14.8%. Residual is mostly long only ETF positions (though will also include FX movements and interest payments) so effectively my ETF hedge strategy made 13.6% + -6.8% = 6.8%.
In the future I will do something that works out the hedge P&L properly, but that isn't a priority right now.
As far as the rebuild goes I've made pretty decent progress towards my goal of being safely and fully automated in around a month from now.
First small thing I did was get the current inside spread before trading, so I can accurately measure my slippage. Also made sure the market was already open before submitting any trades, which is probably a good idea!
Then some large stuff: I've built a little process runner that allows you to schedule the different processes needed, set start/stop times, maximum iterations and frequency of running, any dependencies (other processes that need to finish first). This is all a bit overkill since most things only need to run once a day, with the possible exception of the 'order stack handler' (which manages orders, duh), however it's future proofing for possible intraday systems.
It also means my crontab doesn't have to be as large since I can consolidate dozens of 'methods' that need running into a few 'processes' (to be clear, each process is kicked off by the crontab, and the process runner manages the sub-processes). I also don't need to fine tune the crontab start/stop times to make sure, for example, that the optimal positions do not start being generated until I've updated the prices. I can just set everything to kick off at midnight every day, and then the process runner will start the optimal position generation once the prices are done. Also, process control is managed from a YAML file which is much nicer than a crontab.
I've also built the system monitor report that checks when things last run and made sure they ran when they were supposed to.
Code:
********************************************************************************
Status report produced on 2020-06-26 12:27:09.522865
********************************************************************************
=====================================================================================================================================================
Status of processses
=====================================================================================================================================================
name run_capital_update run_daily_prices_updates run_stack_handler run_systems run_strategy_order_generator
running False False True False False
start 06/26 08:41 06/26 08:41 06/26 11:23 06/26 09:51 06/26 09:57
end 06/26 08:41 06/26 09:50 06/26 11:23 06/26 09:56 06/26 09:57
status GO GO GO GO GO
finished_in_last_day True True False True True
start_time 01:00:00 01:00:00 01:00:00 01:00:00 01:00:00
end_time 19:50:00 23:50:00 19:45:00 23:50:00 23:50:00
required_machine None None None None None
right_machine True True True True True
time_to_run True True True True True
previous_required None None run_strategy_order_generator run_daily_prices_updates run_systems
previous_finished True True True True True
time_to_stop False False False False False
=============================================================================================
Status of methods
=============================================================================================
process_name last_run_or_heartbeat
method_or_strategy
safe_stack_removal run_stack_handler 06/25 16:28
update_total_capital run_capital_update 06/26 08:41
strategy_allocation run_capital_update 06/26 08:41
update_fx_prices run_daily_prices_updates 06/26 09:50
update_sampled_contracts run_daily_prices_updates 06/26 09:50
update_historical_prices run_daily_prices_updates 06/26 09:50
update_multiple_adjusted_prices run_daily_prices_updates 06/26 09:50
medium_speed_TF_carry run_systems 06/26 09:56
medium_speed_TF_carry run_strategy_order_generator 06/26 09:57
check_external_position_break run_stack_handler 06/26 12:23
spawn_children_from_new_instrument_orders run_stack_handler 06/26 12:24
generate_force_roll_orders run_stack_handler 06/26 12:24
create_broker_orders_from_contract_orders run_stack_handler 06/26 12:24
process_fills_stack run_stack_handler 06/26 12:24
handle_completed_orders run_stack_handler 06/26 12:24
==============================================
Status of adjusted price / FX price collection
==============================================
last_update
name
NASDAQ 2020-06-25 23:00:00
PALLAD 2020-06-25 23:00:00
SP500 2020-06-25 23:00:00
NZD 2020-06-25 23:00:00
US10 2020-06-25 23:00:00
MXP 2020-06-25 23:00:00
LIVECOW 2020-06-25 23:00:00
LEANHOG 2020-06-25 23:00:00
JPY 2020-06-25 23:00:00
GOLD 2020-06-25 23:00:00
PLAT 2020-06-25 23:00:00
GAS_US 2020-06-25 23:00:00
GBP 2020-06-25 23:00:00
EUR 2020-06-25 23:00:00
EDOLLAR 2020-06-25 23:00:00
CRUDE_W 2020-06-25 23:00:00
CORN 2020-06-25 23:00:00
COPPER 2020-06-25 23:00:00
US20 2020-06-25 23:00:00
US5 2020-06-25 23:00:00
VIX 2020-06-25 23:00:00
WHEAT 2020-06-25 23:00:00
AUD 2020-06-25 23:00:00
US2 2020-06-25 23:00:00
SOYBEAN 2020-06-25 23:00:00
KOSPI 2020-06-26 07:00:00
KR10 2020-06-26 07:00:00
KR3 2020-06-26 07:00:00
AEX 2020-06-26 08:00:00
BOBL 2020-06-26 08:00:00
BTP 2020-06-26 08:00:00
CAC 2020-06-26 08:00:00
BUND 2020-06-26 08:00:00
SMI 2020-06-26 09:00:00
V2X 2020-06-26 09:00:00
SHATZ 2020-06-26 09:00:00
OAT 2020-06-26 09:00:00
EUROSTX 2020-06-26 09:00:00
JPYUSD 2020-06-26 23:00:00
AUDUSD 2020-06-26 23:00:00
CADUSD 2020-06-26 23:00:00
CHFUSD 2020-06-26 23:00:00
EURUSD 2020-06-26 23:00:00
GBPUSD 2020-06-26 23:00:00
HKDUSD 2020-06-26 23:00:00
KRWUSD 2020-06-26 23:00:00
=====================================================
Status of optimal position generation
=====================================================
last_update
name
medium_speed_TF_carry/AEX 2020-06-26 09:56:20.699
medium_speed_TF_carry/AUD 2020-06-26 09:56:20.910
medium_speed_TF_carry/BOBL 2020-06-26 09:56:21.057
medium_speed_TF_carry/BTP 2020-06-26 09:56:21.245
medium_speed_TF_carry/BUND 2020-06-26 09:56:21.437
medium_speed_TF_carry/CAC 2020-06-26 09:56:21.631
medium_speed_TF_carry/COPPER 2020-06-26 09:56:21.797
medium_speed_TF_carry/CORN 2020-06-26 09:56:22.054
medium_speed_TF_carry/CRUDE_W 2020-06-26 09:56:22.229
medium_speed_TF_carry/EDOLLAR 2020-06-26 09:56:22.431
medium_speed_TF_carry/EUR 2020-06-26 09:56:22.585
medium_speed_TF_carry/EUROSTX 2020-06-26 09:56:22.652
medium_speed_TF_carry/GAS_US 2020-06-26 09:56:22.854
medium_speed_TF_carry/GBP 2020-06-26 09:56:23.093
medium_speed_TF_carry/GOLD 2020-06-26 09:56:23.333
medium_speed_TF_carry/JPY 2020-06-26 09:56:23.570
medium_speed_TF_carry/KOSPI 2020-06-26 09:56:23.641
medium_speed_TF_carry/KR10 2020-06-26 09:56:23.707
medium_speed_TF_carry/KR3 2020-06-26 09:56:23.772
medium_speed_TF_carry/LEANHOG 2020-06-26 09:56:23.995
medium_speed_TF_carry/LIVECOW 2020-06-26 09:56:24.230
medium_speed_TF_carry/MXP 2020-06-26 09:56:24.384
medium_speed_TF_carry/NASDAQ 2020-06-26 09:56:24.518
medium_speed_TF_carry/NZD 2020-06-26 09:56:24.642
medium_speed_TF_carry/OAT 2020-06-26 09:56:24.730
medium_speed_TF_carry/PALLAD 2020-06-26 09:56:24.979
medium_speed_TF_carry/PLAT 2020-06-26 09:56:25.322
medium_speed_TF_carry/SHATZ 2020-06-26 09:56:25.434
medium_speed_TF_carry/SMI 2020-06-26 09:56:25.512
medium_speed_TF_carry/SOYBEAN 2020-06-26 09:56:25.732
medium_speed_TF_carry/SP500 2020-06-26 09:56:25.871
medium_speed_TF_carry/US10 2020-06-26 09:56:26.074
medium_speed_TF_carry/US2 2020-06-26 09:56:26.196
medium_speed_TF_carry/US20 2020-06-26 09:56:26.436
medium_speed_TF_carry/US5 2020-06-26 09:56:26.628
medium_speed_TF_carry/V2X 2020-06-26 09:56:26.711
medium_speed_TF_carry/VIX 2020-06-26 09:56:26.818
medium_speed_TF_carry/WHEAT 2020-06-26 09:56:27.056
********************************************************************************
END OF STATUS REPORT
********************************************************************************There was a bit of a fiddle with getting that working; essentially I have these 'data' objects that I pass around freely that abstract where all the various data is coming from, and I also store logging objects within them. These logging objects have attributes, some state that determines how log entries are labelled, eg here is an order being filled:
Code:
2020-06-26:1123.58 {'type': 'stack_handler', 'component': 'mongoInstrumentOrderStackData', 'strategy_name': 'medium_speed_TF_carry', 'instrument_code': 'EDOLLAR', 'instrument_order_id': 30185} Changed fill qty from 0 to 1.0 for order (Order ID:30185) For medium_speed_TF_carry/EDOLLAR, qty 1.0 fill 0, Parent:no parent Child:[30165]Everything before 'Changed' is the attributes of the log entry. This makes it very handy to find the logs for a particular trade, or instrument_code.
I thought it would be a fine idea to use these log entries to determine when a particular thing ran, in this case the 'stack_handler'. However I had problems in that the data objects were being rather freely shared between processes, which made the logs very misleading. For example, because I get FX data before futures data, and the last FX rate I get is Korea, all my futures data log entries had the attribute 'currency_code="KRWUSD". That is just annoying, but more seriously the 'type' attribute that I use to label sub-processes would have different values depending on whether a given object had retained state from an earlier sub-process or not, hence I don't know for sure when a sub-process actually ran.
I eventually solved this by generating seperate data objects for each sub-process thus ensuring there was no danger of log attributes being polluted (the data objects are 'pipelines' that have almost no state apart from the log attributes, so it's not like having multiple versions around is especially memory intensive).
As I've said before, I don't want to run fully automated code without a lot of safety valves. The main safety feature in my code is something that stops things being traded if my internal position database is out of line with what IB is sending back. Normally, this should only happen for a few seconds, in the period between a fill being received and my order stack, and subsequently position database being updated.
However a genuine difference of opinion is a terrible thing to happen, because you might for example keep trying to buy because you hadn't realised you had already bought, and before you know it you are long 10,000 Eurodollar futures and the nice man from IB would like a word with you. Because my old system ran hourly (a bizzare decision in retrospect, but it does mean at least I've collected hourly price data), this was very important. Again it is something that will be very important for intraday trading.
In my old system these locks had to be cleared manually, now this happens automatically when the positions are back in line again. Doesn't sound much but an example of the kind of thing I wished dearly I had put in the first version of my trading system.
(I've mostly managed to ignore the https://en.wikipedia.org/wiki/Second-system_effect by being very strict about what I include in each iteration of the project since I had to start using this for production: remember the first stage was the minimum for manual trading, this is the minimum for auto trading. Stage three will be the minimum to get me back to the functionality of my previous system, and then bells and whistles will be in subsequent stages).
Other safety features I have now implemented are a limit on the maximum number of trades that can be done over a given period, the ability to start/stop processes, and the ability to apply 'overrides' to positions (an override can be; multiply position by {0,1}, close position, do not open trades in this instrument, or only allow reducing trades).
As well as writing code, I did take out some code that would allow orders to be modified in the order stack, as this made things way too complicated. A quick explanation: the order stack has 3 levels: instrument orders for a particular strategy (eg buy 2 US 10 years), contract orders (eg buy 2 US 10 year Z20 futures), and broker orders (which are effectively the orders that have actually been submitted to the broker). This abstraction means that trading multiple strategies wll be much simpler (as I can do netting for example), and also for trading spread strategies (eg an 'instrument' could be the US2 year - US 10 year spread, which then gets resolved to the contract stack as two trades), and also for trading algos that break up larger orders into several broker orders, or the use of conditional orders (when you've bought the 2 year, buy the 10 year).
However, it does mean that order creation and fills have to be propogated up and down the stack. To do the same with modifications was a nightmare, since you need to keep track of what state the modification request is at every level. There must have been 1000 lines of code involved.
(This doesn't cover when orders are being executed by an execution algo, something I am not doing yet as I'm just using market orders. An execution algo would want to modify limit prices and potentially quantity, however this will only be happening at the bottom broker level)
The use case for modifications was that intraday strategies might change their mind about what position they wanted to put on; perhaps in the morning they wanted to buy, but before the order was executed in the US market something changes and they increase or reduce their order. However I came up with a reasonably elegant solution to this. Firstly, a strategy will take into account any orders that are already on the stack when placing their order (so for example, if they want to buy 2, and there is an unfilled buy 3 order on the stack already, they will issue a sell 1 order).
Secondly, I clear the order stack at the end of every day. So for daily strategies this code will have no effect anyway. Finally, once I have multiple strategies I will be writing an internal order netter, which will work both within and across strategies. If a strategy changes it's mind, then the original buy 3 and sell 1 order will be netted to buy 2.
Next on the agenda for the next couple of weeks is... well it's probably easier to do this in bullet points:
- Reports
- Roll report (done, see previous posts)
- P&l report (done, but I need versions for different time periods and a graphical output)
- Status report- done, see above
- Reconcile report: to do (check status of order book, reconcile positions and trades with IB)
- Trades report: to do
- Overrides and controls: to do (these are discussed above)
- All reports to be put inside a 'process' as some will be run multiple times a day
- Backups
- Backup time series data to csv
- Backup mongoDB files
- Cleaning
- Clean historical log files
- Clean historical echo files
- Implementation
- Create crontab
- Setup new machine
GAT
Last edited:
This weeks update. TLDR: I have made money and I have written some code, but also deleted some.
Let's start with performance, like most people this has been a pretty good few weeks. Here are some graphs, zooming in on more recent periods of time (total, rolling year, YTD)
https://photos.app.goo.gl/B9zXmRfTJq27mpgm7
https://photos.app.goo.gl/eDCzbrs7QXNuJm5u8
https://photos.app.goo.gl/mxJNVG1FyG3xnZCCA
I've now also got decent P&L reporting so here's a more granular picture for the last 6 months.
Code:******************************************************************************** P&L report produced on 2020-06-26 11:40:24.401362 from 2019-12-29 11:38:17.779093 to 2020-06-26 11:38:17.779085 ******************************************************************************** Total p&l is 21.575% ==================================== P&L by instrument for all strategies ==================================== instrument pandl 0 V2X -4.30 1 MXP -3.77 2 VIX -1.93 3 PLAT -1.44 4 OAT -1.01 5 JPY -0.97 6 COPPER -0.89 7 CAC -0.84 8 SP500 -0.78 9 AEX -0.67 10 GOLD -0.27 11 EUR -0.03 12 KR10 -0.00 13 US5 0.00 14 GBP 0.05 15 US2 0.20 16 KOSPI 0.33 17 WHEAT 0.34 18 BOBL 0.38 19 NZD 0.56 20 US10 0.57 21 BTP 0.66 22 KR3 0.68 23 NASDAQ 0.73 24 CRUDE_W 0.97 25 BUND 1.37 26 GAS_US 1.51 27 LEANHOG 1.80 28 SOYBEAN 2.13 29 AUD 2.13 30 LIVECOW 2.55 31 CORN 3.31 32 EDOLLAR 5.27 33 PALLAD 6.11 34 EUROSTX 13.61 Total futures p&l is 28.410% Residual p&l is -6.835% ******************************************************************************** END OF P&L REPORT ********************************************************************************
For interpretation, my futures trading p&l will be equal to 'total futures p&l 28.4%' less Eurostoxx 13.6% (only traded as a hedge) which is 14.8%. Residual is mostly long only ETF positions (though will also include FX movements and interest payments) so effectively my ETF hedge strategy made 13.6% + -6.8% = 6.8%.
In the future I will do something that works out the hedge P&L properly, but that isn't a priority right now.
As far as the rebuild goes I've made pretty decent progress towards my goal of being safely and fully automated in around a month from now.
First small thing I did was get the current inside spread before trading, so I can accurately measure my slippage. Also made sure the market was already open before submitting any trades, which is probably a good idea!
Then some large stuff: I've built a little process runner that allows you to schedule the different processes needed, set start/stop times, maximum iterations and frequency of running, any dependencies (other processes that need to finish first). This is all a bit overkill since most things only need to run once a day, with the possible exception of the 'order stack handler' (which manages orders, duh), however it's future proofing for possible intraday systems.
It also means my crontab doesn't have to be as large since I can consolidate dozens of 'methods' that need running into a few 'processes' (to be clear, each process is kicked off by the crontab, and the process runner manages the sub-processes). I also don't need to fine tune the crontab start/stop times to make sure, for example, that the optimal positions do not start being generated until I've updated the prices. I can just set everything to kick off at midnight every day, and then the process runner will start the optimal position generation once the prices are done. Also, process control is managed from a YAML file which is much nicer than a crontab.
I've also built the system monitor report that checks when things last run and made sure they ran when they were supposed to.
Code:******************************************************************************** Status report produced on 2020-06-26 12:27:09.522865 ******************************************************************************** ===================================================================================================================================================== Status of processses ===================================================================================================================================================== name run_capital_update run_daily_prices_updates run_stack_handler run_systems run_strategy_order_generator running False False True False False start 06/26 08:41 06/26 08:41 06/26 11:23 06/26 09:51 06/26 09:57 end 06/26 08:41 06/26 09:50 06/26 11:23 06/26 09:56 06/26 09:57 status GO GO GO GO GO finished_in_last_day True True False True True start_time 01:00:00 01:00:00 01:00:00 01:00:00 01:00:00 end_time 19:50:00 23:50:00 19:45:00 23:50:00 23:50:00 required_machine None None None None None right_machine True True True True True time_to_run True True True True True previous_required None None run_strategy_order_generator run_daily_prices_updates run_systems previous_finished True True True True True time_to_stop False False False False False ============================================================================================= Status of methods ============================================================================================= process_name last_run_or_heartbeat method_or_strategy safe_stack_removal run_stack_handler 06/25 16:28 update_total_capital run_capital_update 06/26 08:41 strategy_allocation run_capital_update 06/26 08:41 update_fx_prices run_daily_prices_updates 06/26 09:50 update_sampled_contracts run_daily_prices_updates 06/26 09:50 update_historical_prices run_daily_prices_updates 06/26 09:50 update_multiple_adjusted_prices run_daily_prices_updates 06/26 09:50 medium_speed_TF_carry run_systems 06/26 09:56 medium_speed_TF_carry run_strategy_order_generator 06/26 09:57 check_external_position_break run_stack_handler 06/26 12:23 spawn_children_from_new_instrument_orders run_stack_handler 06/26 12:24 generate_force_roll_orders run_stack_handler 06/26 12:24 create_broker_orders_from_contract_orders run_stack_handler 06/26 12:24 process_fills_stack run_stack_handler 06/26 12:24 handle_completed_orders run_stack_handler 06/26 12:24 ============================================== Status of adjusted price / FX price collection ============================================== last_update name NASDAQ 2020-06-25 23:00:00 PALLAD 2020-06-25 23:00:00 SP500 2020-06-25 23:00:00 NZD 2020-06-25 23:00:00 US10 2020-06-25 23:00:00 MXP 2020-06-25 23:00:00 LIVECOW 2020-06-25 23:00:00 LEANHOG 2020-06-25 23:00:00 JPY 2020-06-25 23:00:00 GOLD 2020-06-25 23:00:00 PLAT 2020-06-25 23:00:00 GAS_US 2020-06-25 23:00:00 GBP 2020-06-25 23:00:00 EUR 2020-06-25 23:00:00 EDOLLAR 2020-06-25 23:00:00 CRUDE_W 2020-06-25 23:00:00 CORN 2020-06-25 23:00:00 COPPER 2020-06-25 23:00:00 US20 2020-06-25 23:00:00 US5 2020-06-25 23:00:00 VIX 2020-06-25 23:00:00 WHEAT 2020-06-25 23:00:00 AUD 2020-06-25 23:00:00 US2 2020-06-25 23:00:00 SOYBEAN 2020-06-25 23:00:00 KOSPI 2020-06-26 07:00:00 KR10 2020-06-26 07:00:00 KR3 2020-06-26 07:00:00 AEX 2020-06-26 08:00:00 BOBL 2020-06-26 08:00:00 BTP 2020-06-26 08:00:00 CAC 2020-06-26 08:00:00 BUND 2020-06-26 08:00:00 SMI 2020-06-26 09:00:00 V2X 2020-06-26 09:00:00 SHATZ 2020-06-26 09:00:00 OAT 2020-06-26 09:00:00 EUROSTX 2020-06-26 09:00:00 JPYUSD 2020-06-26 23:00:00 AUDUSD 2020-06-26 23:00:00 CADUSD 2020-06-26 23:00:00 CHFUSD 2020-06-26 23:00:00 EURUSD 2020-06-26 23:00:00 GBPUSD 2020-06-26 23:00:00 HKDUSD 2020-06-26 23:00:00 KRWUSD 2020-06-26 23:00:00 ===================================================== Status of optimal position generation ===================================================== last_update name medium_speed_TF_carry/AEX 2020-06-26 09:56:20.699 medium_speed_TF_carry/AUD 2020-06-26 09:56:20.910 medium_speed_TF_carry/BOBL 2020-06-26 09:56:21.057 medium_speed_TF_carry/BTP 2020-06-26 09:56:21.245 medium_speed_TF_carry/BUND 2020-06-26 09:56:21.437 medium_speed_TF_carry/CAC 2020-06-26 09:56:21.631 medium_speed_TF_carry/COPPER 2020-06-26 09:56:21.797 medium_speed_TF_carry/CORN 2020-06-26 09:56:22.054 medium_speed_TF_carry/CRUDE_W 2020-06-26 09:56:22.229 medium_speed_TF_carry/EDOLLAR 2020-06-26 09:56:22.431 medium_speed_TF_carry/EUR 2020-06-26 09:56:22.585 medium_speed_TF_carry/EUROSTX 2020-06-26 09:56:22.652 medium_speed_TF_carry/GAS_US 2020-06-26 09:56:22.854 medium_speed_TF_carry/GBP 2020-06-26 09:56:23.093 medium_speed_TF_carry/GOLD 2020-06-26 09:56:23.333 medium_speed_TF_carry/JPY 2020-06-26 09:56:23.570 medium_speed_TF_carry/KOSPI 2020-06-26 09:56:23.641 medium_speed_TF_carry/KR10 2020-06-26 09:56:23.707 medium_speed_TF_carry/KR3 2020-06-26 09:56:23.772 medium_speed_TF_carry/LEANHOG 2020-06-26 09:56:23.995 medium_speed_TF_carry/LIVECOW 2020-06-26 09:56:24.230 medium_speed_TF_carry/MXP 2020-06-26 09:56:24.384 medium_speed_TF_carry/NASDAQ 2020-06-26 09:56:24.518 medium_speed_TF_carry/NZD 2020-06-26 09:56:24.642 medium_speed_TF_carry/OAT 2020-06-26 09:56:24.730 medium_speed_TF_carry/PALLAD 2020-06-26 09:56:24.979 medium_speed_TF_carry/PLAT 2020-06-26 09:56:25.322 medium_speed_TF_carry/SHATZ 2020-06-26 09:56:25.434 medium_speed_TF_carry/SMI 2020-06-26 09:56:25.512 medium_speed_TF_carry/SOYBEAN 2020-06-26 09:56:25.732 medium_speed_TF_carry/SP500 2020-06-26 09:56:25.871 medium_speed_TF_carry/US10 2020-06-26 09:56:26.074 medium_speed_TF_carry/US2 2020-06-26 09:56:26.196 medium_speed_TF_carry/US20 2020-06-26 09:56:26.436 medium_speed_TF_carry/US5 2020-06-26 09:56:26.628 medium_speed_TF_carry/V2X 2020-06-26 09:56:26.711 medium_speed_TF_carry/VIX 2020-06-26 09:56:26.818 medium_speed_TF_carry/WHEAT 2020-06-26 09:56:27.056 ******************************************************************************** END OF STATUS REPORT ********************************************************************************
There was a bit of a fiddle with getting that working; essentially I have these 'data' objects that I pass around freely that abstract where all the various data is coming from, and I also store logging objects within them. These logging objects have attributes, some state that determines how log entries are labelled, eg here is an order being filled:
Code:2020-06-26:1123.58 {'type': 'stack_handler', 'component': 'mongoInstrumentOrderStackData', 'strategy_name': 'medium_speed_TF_carry', 'instrument_code': 'EDOLLAR', 'instrument_order_id': 30185} Changed fill qty from 0 to 1.0 for order (Order ID:30185) For medium_speed_TF_carry/EDOLLAR, qty 1.0 fill 0, Parent:no parent Child:[30165]
Everything before 'Changed' is the attributes of the log entry. This makes it very handy to find the logs for a particular trade, or instrument_code.
I thought it would be a fine idea to use these log entries to determine when a particular thing ran, in this case the 'stack_handler'. However I had problems in that the data objects were being rather freely shared between processes, which made the logs very misleading. For example, because I get FX data before futures data, and the last FX rate I get is Korea, all my futures data log entries had the attribute 'currency_code="KRWUSD". That is just annoying, but more seriously the 'type' attribute that I use to label sub-processes would have different values depending on whether a given object had retained state from an earlier sub-process or not, hence I don't know for sure when a sub-process actually ran.
I eventually solved this by generating seperate data objects for each sub-process thus ensuring there was no danger of log attributes being polluted (the data objects are 'pipelines' that have almost no state apart from the log attributes, so it's not like having multiple versions around is especially memory intensive).
As I've said before, I don't want to run fully automated code without a lot of safety valves. The main safety feature in my code is something that stops things being traded if my internal position database is out of line with what IB is sending back. Normally, this should only happen for a few seconds, in the period between a fill being received and my order stack, and subsequently position database being updated.
However a genuine difference of opinion is a terrible thing to happen, because you might for example keep trying to buy because you hadn't realised you had already bought, and before you know it you are long 10,000 Eurodollar futures and the nice man from IB would like a word with you. Because my old system ran hourly (a bizzare decision in retrospect, but it does mean at least I've collected hourly price data), this was very important. Again it is something that will be very important for intraday trading.
In my old system these locks had to be cleared manually, now this happens automatically when the positions are back in line again. Doesn't sound much but an example of the kind of thing I wished dearly I had put in the first version of my trading system.
(I've mostly managed to ignore the https://en.wikipedia.org/wiki/Second-system_effect by being very strict about what I include in each iteration of the project since I had to start using this for production: remember the first stage was the minimum for manual trading, this is the minimum for auto trading. Stage three will be the minimum to get me back to the functionality of my previous system, and then bells and whistles will be in subsequent stages).
Other safety features I have now implemented are a limit on the maximum number of trades that can be done over a given period, the ability to start/stop processes, and the ability to apply 'overrides' to positions (an override can be; multiply position by {0,1}, close position, do not open trades in this instrument, or only allow reducing trades).
As well as writing code, I did take out some code that would allow orders to be modified in the order stack, as this made things way too complicated. A quick explanation: the order stack has 3 levels: instrument orders for a particular strategy (eg buy 2 US 10 years), contract orders (eg buy 2 US 10 year Z20 futures), and broker orders (which are effectively the orders that have actually been submitted to the broker). This abstraction means that trading multiple strategies wll be much simpler (as I can do netting for example), and also for trading spread strategies (eg an 'instrument' could be the US2 year - US 10 year spread, which then gets resolved to the contract stack as two trades), and also for trading algos that break up larger orders into several broker orders, or the use of conditional orders (when you've bought the 2 year, buy the 10 year).
However, it does mean that order creation and fills have to be propogated up and down the stack. To do the same with modifications was a nightmare, since you need to keep track of what state the modification request is at every level. There must have been 1000 lines of code involved.
(This doesn't cover when orders are being executed by an execution algo, something I am not doing yet as I'm just using market orders. An execution algo would want to modify limit prices and potentially quantity, however this will only be happening at the bottom broker level)
The use case for modifications was that intraday strategies might change their mind about what position they wanted to put on; perhaps in the morning they wanted to buy, but before the order was executed in the US market something changes and they increase or reduce their order. However I came up with a reasonably elegant solution to this. Firstly, a strategy will take into account any orders that are already on the stack when placing their order (so for example, if they want to buy 2, and there is an unfilled buy 3 order on the stack already, they will issue a sell 1 order).
Secondly, I clear the order stack at the end of every day. So for daily strategies this code will have no effect anyway. Finally, once I have multiple strategies I will be writing an internal order netter, which will work both within and across strategies. If a strategy changes it's mind, then the original buy 3 and sell 1 order will be netted to buy 2.
Next on the agenda for the next couple of weeks is... well it's probably easier to do this in bullet points:
Stay well, and be lucky.
- Reports
- Roll report (done, see previous posts)
- P&l report (done, but I need versions for different time periods and a graphical output)
- Status report- done, see above
- Reconcile report: to do (check status of order book, reconcile positions and trades with IB)
- Trades report: to do
- Overrides and controls: to do (these are discussed above)
- All reports to be put inside a 'process' as some will be run multiple times a day
- Backups
- Backup time series data to csv
- Backup mongoDB files
- Cleaning
- Clean historical log files
- Clean historical echo files
- Implementation
- Create crontab
- Setup new machine
GAT
Reading back, there is a bit of 'second system effect' going on.
I feel that the 3 tier order stack was justified, since it would have been very hard to re-engineer afterwards if I had started with something simpler (would have probably ended up throwing it away and starting again).
The modification code was certainly not neccessary, and I could have avoided the pain by asking myself if it was really something that had to be done now.
The process runner is a bit more of a grey area, since I could have stuck to my current approach of one massive crontab with a carefully designed schedule. I would still have needed to write some code to get the stack handler to finish by a certain time, so ultimately I don't think it was too much of a scope creep.
GAT
@Kernfusion
I switched recently to the MNQ and MES (as well as to mini aud and eur and crude), so that I could have at least some exposure without requiring a huge forecast in my smaller account.
That does sound interesting and maybe I should do it.,
But isn't there some kind of problem with these contracts? like low volume, hence large bid-ask spread, so they are more expensive to trade than the regular ones ? Or maybe it's harder to roll from one to the next because of low volume? I mean, we all here (I think?) have this problem with limited capital and large contracts, there must be a reason why people still trade the big ones when these smaller versions are available?
(although, I remember someone else in this thread started trading smaller contracts before..)
I think that's definitely true with some, including crude. I think it is insignificant with currencies and equity indices. With regard to crude, I am making the assumption that the exposure benefit is worth the lower liquidity. They all roll at the same time as their bigger brothers, as far as I can tell. I actually even use the price histories of the big brothers and just adjust the point value in the instrumentconfig file.
For what it's worth, the way I'm doing it is that basically I have 2 layers in that area: the strategy-layer that decides the required target position and invokes the broker-interaction-layer, which accepts the target position and decides which orders to place to achieve it.
The broker-interaction-layer obviously knows the current broker's position and any currently active(unfilled) orders (only one active order at a time per contract(specific-instrument) is allowed to exist )., if the target position matches CurrentPosition+ActiveOrderTargetQty it does nothing (except of maybe adjusting the limit-price, because it also acts as a simple execution algo), if they don't match, then if the active order is not filled, it cancels it and does nothing, if it's partially-filled, it does nothing.
The 2 layers can also communicate backwards: the strategy layer can ask the broker-interaction layer "what is the current position of this contract including all the placed orders" (strategy-layer might need this info e.g. to decide not change it's position if the difference is too small or for other things.). So if the strategy layer wanted a target position of 2, then it decided to change it to 3, but the active order was already partially-filled at that time, the request will be ignored, the order will be allowed complete, and the strategy will basically have to "try again next time".
In reality it's more complicated, and includes order-limit protection, IB-API interaction code etc., but in principle it's like that "broker layer will complete what it's currently doing, and the strategy will have to bug it again next time when it doesn't have an active order on this contract, if it decided it wants something else"..
(I actually went back and double-checked my several-year old code after reading this "...before you know it you are long 10,000 Eurodollar futures..." - very scary stuff
)
Last edited:
In a pretty impressive display of speed I managed to get 99% of this done in one week, plus write a blog post on non binary forecasts.
I finished writing the code by Tuesday evening, then spent Wednesday setting up all three of my trading servers (live, backup, development). Yesterday and today I've been grappling with getting everything to run automatically off the crontab, which should be done by COB today.
[Crontab format: minutes then hours. WHY?!]
Assuming this is succesful, after a few days catching up with some other tasks I'll be moving on to phase 3:
- Allow a manual trade to be created for the system to execute (sometimes useful)
- Create a manual FX trade (required to maintain positive balance across currencies)
- Rolling using spread orders (this will require a bit of work as there are a lot of assumptions that orders only have a single leg)
- Use an execution algo to trade rather than market orders
- Properly seperate out the hedge strategy
- Create an error 'stack' (at the moment the system emails me if a critical error is generated that requires user input; these emails could stack up in your inbox, so the idea is to only send one email and then allow the user to manually read the other critical emails; only once the stack is cleared will they get more emails)
Oh and I lost some money this week. I've also been working on a new podcast series with these guys (as well as my regular monthly slot with them).
GAT