Oh really? I don't keep a check on those reports and was not aware of it.
Fully automated futures trading
- Thread starter globalarbtrader
- Start date
Hey Kernfusion, how is this effort going? Did you move to production and have you tried snap to market?
I'm asking because I moved back to Europe, and the country I'm living now requires me to register an LLC for full time trading. I did that, but then IB classifies me as professional wrt market data fees, so what used to be $120 per month in subscriptions is now $780. So I'm looking at alternatives
While doing research, I found these folks https://firstratedata.com that offer intraday data, down to minutely. Usually 10-15 years of history, depending on the instrument. Posting here in case anybody needs it, and I know @globalarbtrader was asking for data providers a few months back.
ohh, that's a bummer, so like it's not allowed to "trade" without an LLC? how about "investing" and where's the dividing line exactly?Hey Kernfusion, how is this effort going? Did you move to production and have you tried snap to market?
I'm asking because I moved back to Europe, and the country I'm living now requires me to register an LLC for full time trading. I did that, but then IB classifies me as professional wrt market data fees, so what used to be $120 per month in subscriptions is now $780. So I'm looking at alternatives
While doing research, I found these folks https://firstratedata.com that offer intraday data, down to minutely. Usually 10-15 years of history, depending on the instrument. Posting here in case anybody needs it, and I know @globalarbtrader was asking for data providers a few months back.
or maybe it's about futures specifically..But yeah, I was looking into trading through a corp. entity for tax purposes, but IB\Exchanges will charge the full price for data in that case, which kills the benefit (or at least spoils it..).
I've been trading in PROD these ICE\no-data products for about a month now(some more some less) : Cocoa, cotton no. 2, coffee, FC orange juice, Canola, Sugar no. 11, European Rapeseed Future. And will be adding more. They're a pain to onboard, because they all have different expiration rules and I have to study every contract specs to correctly preprogram the roll-timings for my automatic rolls, so can't onboard them in-bulk as I could for example with some CME FX an Bond contracts..
So far I think it's going good, I'm using OrderType = "SNAP MID"; (in PAPER I use that order type for everything at the moment, in PROD only for the onboarded ice futures) and according to my report all fills happen within the initial bid-ask spread (or actually between min(bid) and max(ask) within +\- 4 sec interval around the time I place the order). I realize that report wouldn't make sense for delayed products, because the bid\ask I see when I'm submitting the order is from 15 min ago, and when my snap-order hits the exchange it can be anything and might look to me as if the fill happened outside of the spread even though it was perfectly within the actual spread at that time.
But basically I think the snap order works fine.
I had another issue with collecting EOD prices.. IB doesn't make delayed Last price available when the market is closed, so for a while I was only getting the Delayed Close, which is not today's close but yesterday's, and you also have to be careful when you collect it over the weekend or on Monday, because the delayed close on Monday is actually Thursday close, but it also depends on the time of day..
So what I did - I started to run the EOD collecting job 2 times a day, in the morning when all markets are open and in the evening.. Still need to keep monitoring my EOD price history collection process to make sure there's no gaps.. If you're planning to use another EOD data provider (not sure why you'd need their minute data, because it'll still be delayed, and you get 15-min delayed from IB anyways..), then that part will be easier.. Though, more work to integrate with them, and how reliable and long-lasting are they is also a question..
Btw, IB's TWS actually gives EOD price history through the UI for unsubscribed contracts, and there's an option to save it in CSV, even though it's not available through the API, so that can be used for backfilling..
Overall, this seems workable, but don't trust this completely as I'm pretty lazy with monitoring things and might be missing some issues
Here's some PROD execution statistics on delayed instruments., Turns out I only had orders on sugar so far, the bid\ask in the table are as the system saw them at the time of order placement, i.e. 15-min delayed. Seems ok I guess, in some cases I got even better fills than hoped for..
Never try to trade futures with delayed data. Bad juju.
Thanks for both replies Kernfusion. I'll put some of my thoughts here as well.
At the moment, I've subscribed to almost all markets (still not trading anything from ICE, reason being I don't want to pay even more), and with sales tax, it comes out to over $900 per month. As Rob would say in his books, costs are the only thing that's certain - broker will take their fee every time, and charge for market data every month, so I'll be focusing on reducing this ASAP. I've now bought it so I can backfill some data and get the trading going, but I don't expect to run like this for more than 2 months.
Looking at your last post - if I've summed up the prices correctly, just in those executions you posted you've "lost" $408 in slippage, counting from the midpoint. I'm not sure if we can extrapolate much from that - I realize it's only a few trades. Perhaps it would even out over the long run, and over more instruments. I'm also not sure having real time data would necessarily help here, unless you have some sort of an execution algo. Rob posted a few times about his "primitive" execution algo, which does help though and is what I used before I shut the system down for moving few months back.
I turned the system back on this week and until I get some infra work underway, I've been executing manually. I had 18 orders to execute on the first day, to establish initial positions. I sent them all in at once, all as limit orders on my side of the spread and was surprised to see 16 of them filled within 45 min or so. I only had to move my limit on 2. Liquid stuff is usually filled within seconds or minutes. This was around 11-12 AM UTC, so not even the peak liquidity hours.
My current guidelines for execution are these two: 1) try to capture the spread, 2) executing during liquid hours is better than executing during non-liquid hours.
I'm planning on doing a small project to determine most liquid hours for all instruments (IB does provide "liquid hours" values, but I'm sure you've realized that that is useless for all but the most liquid US based futures), and will be aiming to do executions in those hours. I'll just get minutely spread data from IB for the last month or so and do some averaging. That's what I currently use to estimate slippage for the purposes of calculating costs. Current plan is then, to place all the orders as soon as DO runs, which is around 4.30AM UTC for me, and place them all as limit on my side of the spread. Then as the liquid hours for each instrument arrive, I'll switch the remaining unfilled orders to SNAP MID. That way I'm giving it a few hours to try and capture the spread and if I don't get it, I fill as best as I can.
I realize that I'm ignoring the price movement in general. I'd be interested to know how the prices move intraday when compared to my forecast, eg. if my forecast is in top 20% range, what do prices usually do the next day. Perhaps then I could be a bit smarter on the execution, however, I don't have the data to do that experiment, and it does seem like a lot of work.
Also, those are totally DO trades for Sugar, flipping back and forth every few days, I have that too
At the moment, I've subscribed to almost all markets (still not trading anything from ICE, reason being I don't want to pay even more), and with sales tax, it comes out to over $900 per month. As Rob would say in his books, costs are the only thing that's certain - broker will take their fee every time, and charge for market data every month, so I'll be focusing on reducing this ASAP. I've now bought it so I can backfill some data and get the trading going, but I don't expect to run like this for more than 2 months.
Looking at your last post - if I've summed up the prices correctly, just in those executions you posted you've "lost" $408 in slippage, counting from the midpoint. I'm not sure if we can extrapolate much from that - I realize it's only a few trades. Perhaps it would even out over the long run, and over more instruments. I'm also not sure having real time data would necessarily help here, unless you have some sort of an execution algo. Rob posted a few times about his "primitive" execution algo, which does help though and is what I used before I shut the system down for moving few months back.
I turned the system back on this week and until I get some infra work underway, I've been executing manually. I had 18 orders to execute on the first day, to establish initial positions. I sent them all in at once, all as limit orders on my side of the spread and was surprised to see 16 of them filled within 45 min or so. I only had to move my limit on 2. Liquid stuff is usually filled within seconds or minutes. This was around 11-12 AM UTC, so not even the peak liquidity hours.
My current guidelines for execution are these two: 1) try to capture the spread, 2) executing during liquid hours is better than executing during non-liquid hours.
I'm planning on doing a small project to determine most liquid hours for all instruments (IB does provide "liquid hours" values, but I'm sure you've realized that that is useless for all but the most liquid US based futures), and will be aiming to do executions in those hours. I'll just get minutely spread data from IB for the last month or so and do some averaging. That's what I currently use to estimate slippage for the purposes of calculating costs. Current plan is then, to place all the orders as soon as DO runs, which is around 4.30AM UTC for me, and place them all as limit on my side of the spread. Then as the liquid hours for each instrument arrive, I'll switch the remaining unfilled orders to SNAP MID. That way I'm giving it a few hours to try and capture the spread and if I don't get it, I fill as best as I can.
I realize that I'm ignoring the price movement in general. I'd be interested to know how the prices move intraday when compared to my forecast, eg. if my forecast is in top 20% range, what do prices usually do the next day. Perhaps then I could be a bit smarter on the execution, however, I don't have the data to do that experiment, and it does seem like a lot of work.
Also, those are totally DO trades for Sugar, flipping back and forth every few days, I have that too
$408 looked kinda scary to me, so went ahead I calculated all that slippage in USD myself (thanks a bunch for the extra work )
and this is what I got (removed unfilled orders):
So yeah, counting from mid I lost $414.4 and from the "worst side" $324.8.
Looking at the individual losses, it's mostly down to the -$291.2 from Jul 27.
But all these Bid\Ask prices in the table are 15-min delayed, i.e. the true exchange price at the moment was different., So I went to my RtPrice table for this contract and checked the saved quotes from the time of the order to +15min (because that would actually be the true price as I was submitting it):
So, the order was placed around '03:45', and the system thought that the spread was 0.1842/0.1844, but the true spread at that time (which I received 15 min later) was 0.1829\0.183, and my fill was 0.183 which actually means I captured the full spread and "made money" because it was a sell order.
Now, I don't know what's the "right" way to look at this., from one perspective, yes, compared to what my system "thought" the true price was I lost money., but in reality the SNAP MID order worked perfectly and I didn't loose anything on execution.. Another way to look at it is that my signal was generated incorrectly, because it was based on the stale price, but my execution was fine.. I.e. I don't know if these losses should be counted as slippage or not.. Next time the price might actually move the other way and I might make money..
The way I deal with illiquid market-times is I take about 1 week of recorded quotes after onboarding a new instrument, compute median bid-ask spread over this time, check if I can live with it (i.e. it's not greater that 50$ or so) and then say that I will only issue orders when the current spread is not more than 2x-3x of that median spread. So I don't preprogram liquid times ahead of time. (I also use that estimated median spread in the DO cost calculations..)
I still need to look again at my trading costs on the regular-data markets where I use Rob's algo, and on the costs overall to see how DO affected it, it's on the to-do list
Good to know you're seeing similar behavior for sugar, maybe what my system is doing isn't totally random
)and this is what I got (removed unfilled orders):
So yeah, counting from mid I lost $414.4 and from the "worst side" $324.8.
Looking at the individual losses, it's mostly down to the -$291.2 from Jul 27.
But all these Bid\Ask prices in the table are 15-min delayed, i.e. the true exchange price at the moment was different., So I went to my RtPrice table for this contract and checked the saved quotes from the time of the order to +15min (because that would actually be the true price as I was submitting it):
So, the order was placed around '03:45', and the system thought that the spread was 0.1842/0.1844, but the true spread at that time (which I received 15 min later) was 0.1829\0.183, and my fill was 0.183 which actually means I captured the full spread and "made money" because it was a sell order.
Now, I don't know what's the "right" way to look at this., from one perspective, yes, compared to what my system "thought" the true price was I lost money., but in reality the SNAP MID order worked perfectly and I didn't loose anything on execution.. Another way to look at it is that my signal was generated incorrectly, because it was based on the stale price, but my execution was fine.. I.e. I don't know if these losses should be counted as slippage or not.. Next time the price might actually move the other way and I might make money..
The way I deal with illiquid market-times is I take about 1 week of recorded quotes after onboarding a new instrument, compute median bid-ask spread over this time, check if I can live with it (i.e. it's not greater that 50$ or so) and then say that I will only issue orders when the current spread is not more than 2x-3x of that median spread. So I don't preprogram liquid times ahead of time. (I also use that estimated median spread in the DO cost calculations..)
I still need to look again at my trading costs on the regular-data markets where I use Rob's algo, and on the costs overall to see how DO affected it, it's on the to-do list
Good to know you're seeing similar behavior for sugar, maybe what my system is doing isn't totally random
Hi all
Sorry for the radio silence, been taking my usual extensive summer holidays (which haven't finished yet!)
My system basically didn't run for the last 3 weeks, as I wasn't around for the Sunday IB resets, and there was also a power cut. But positions were small with low risk so I was quite chilled. In fact my p&l over those 3 weeks was just -0.5%. It's turning out to be a fairly classic trend following year; profits racked up quite quickly until mid June, then some give back, risk taken off, and now just bumbling along preserving capital.
The nice thing about the system is it's self correcting nature; everything is designed to cope with short or longer periods without activity. I turned the computer back on and manually ran through the stages of getting prices, running the back-test and generating orders. I had to do some rolls, and also deal with the EUREX ticker change, but otherwise it all ran smoothly. My only fear in these situations is database corruption, which can sometimes happen if the computer is powered down with active connections, but touch wood all is okay. The DO system is naturally self correcting of course; it did have quite a few trades to do, but not that many considering it was a 3 week gap. This morning the cron started everything fine so I'm back to normal.
If someone here is a pysystemtrade user, please note that I had an issue logging into the github website as it didn't like my 2FA device; although I can still check code in and out. So the backlog of pull requests and issues won't be looked at until I get my github credentials sorted. Perhaps someone can post an announcement to the discussion forum to that effect? I must say I find the whole github security an absolute nightmare. It's like matplotlib - there always seems to be at least two ways of doing it, I work out a hacky way of doing it using stack overflow search, but I never feel I really understand what's going on.
I finished the first draft of my new book in mid July as planned, so that's jolly good. I'm now waiting for the editor to get back to me with his review. It's a tedious process, but such is life. At some point in the next few months I also need to pull together the spreadsheets and code snippets. Once I start back properly in September, I will have another look at my new markets and intraday trading projects.
Rob
Sorry for the radio silence, been taking my usual extensive summer holidays (which haven't finished yet!)
My system basically didn't run for the last 3 weeks, as I wasn't around for the Sunday IB resets, and there was also a power cut. But positions were small with low risk so I was quite chilled. In fact my p&l over those 3 weeks was just -0.5%. It's turning out to be a fairly classic trend following year; profits racked up quite quickly until mid June, then some give back, risk taken off, and now just bumbling along preserving capital.
The nice thing about the system is it's self correcting nature; everything is designed to cope with short or longer periods without activity. I turned the computer back on and manually ran through the stages of getting prices, running the back-test and generating orders. I had to do some rolls, and also deal with the EUREX ticker change, but otherwise it all ran smoothly. My only fear in these situations is database corruption, which can sometimes happen if the computer is powered down with active connections, but touch wood all is okay. The DO system is naturally self correcting of course; it did have quite a few trades to do, but not that many considering it was a 3 week gap. This morning the cron started everything fine so I'm back to normal.
If someone here is a pysystemtrade user, please note that I had an issue logging into the github website as it didn't like my 2FA device; although I can still check code in and out. So the backlog of pull requests and issues won't be looked at until I get my github credentials sorted. Perhaps someone can post an announcement to the discussion forum to that effect? I must say I find the whole github security an absolute nightmare. It's like matplotlib - there always seems to be at least two ways of doing it, I work out a hacky way of doing it using stack overflow search, but I never feel I really understand what's going on.
I finished the first draft of my new book in mid July as planned, so that's jolly good. I'm now waiting for the editor to get back to me with his review. It's a tedious process, but such is life. At some point in the next few months I also need to pull together the spreadsheets and code snippets. Once I start back properly in September, I will have another look at my new markets and intraday trading projects.
Rob
The solution which I'm using for the resets during the weekend: before going on a trip which includes a weekend I start TeamViewer on my trading computer. TeamViewer is available for both Ubuntu as well as iPad. So I can use my iPad while on the road to provide the credentials on Sunday to re-login to IB's Gateway.
For the risk of a power cut do I only have a partial solution: my trading computer is running off of a UPS, which can keep the computer "alive" for a couple of hours before it runs out of power. You would need to analyze the risk of having a power cut which lasts longer than the capacity of your UPS (or buy a larger size one). Alternatively you could consider to move your trading system to the cloud, but that comes with considerable cost. It is much more complicated to determine whether this is worth it.
Are you referring to monetary cost or something else? A decently powerful VPS is less than $10.
Did the TeamViewer solution myself many years ago, it was a pain because either end could've had a bad connection and power cuts are a real headache as you've stated.