I'm currently flat copper. Made some money on it, then gave back about a third, then the system closed the position..
Fully automated futures trading
- Thread starter globalarbtrader
- Start date
@wopr it sounds like that you need to have separate monitor software which monitors for any trade confirmation message 24/7. And place that information in your database.
My case is different: I place an order with a limit price at the midpoint between bid and ask. If it isn't filled within one minute I change the limit price to cross the spread. It happens rarely that then the order isn't filled immediately. The result is that I don't need to monitor for 24/7 whether any open orders are filled.
My case is different: I place an order with a limit price at the midpoint between bid and ask. If it isn't filled within one minute I change the limit price to cross the spread. It happens rarely that then the order isn't filled immediately. The result is that I don't need to monitor for 24/7 whether any open orders are filled.
Thanks for the explanation!
To make sure I understood, when your system places an order, whatever process placed the order stays up and running until the order is filled? Or you have another process that's constantly up and is listening for events from the broker to update the DB?
Also, how is the IBOrderID generated?
I've found that using the permId IB generates wasn't good enough because it doesn't show up in Flex Reports, and from casual observation, doesn't seem to be unique - I haven't been able to find out how they generate that.
Have you ever had a situation where you place an order, it gets filled, but for some reason you didn't update the DB (maybe a bug), is that what you then go in and fill manually?
Yep, sure looks like that. I was really hoping that I wouldn't have to run a service all the time to monitor this, as I don't need the information real time - I was hoping I could have a cron job that runs every few hours and scoops executions and updates my DB, but IB exposes no reliable way of doing that.
I should also mention, I've been using an execution algo similar to yours (I legit ripped off Rob's "world's simplest execution algo") for about a year. But I saw that about 60% of executions were "Aggressive" meaning I crossed the spread or even moved it further away from my side of the spread. Then I implemented this async execution, where I put the limit on my side of the spread and go away. Pros are: ~95% of the times I get filled and capture the spread.
Cons are the bookkeeping I'm mentioning above, since I no longer have a process that sits and waits for the order to fill, I gotta update it asynchronously somehow.
So far, this new execution proves to be only added complexity, I'll have to see if getting the spread is worth it.
- My whole system is a single process which is always up (except of the special maintenance windows like 2hour in the morning and 2 hour in the evening Mon-Fr and for the whole weekend it's down as well, starts again on Sunday evening). So the default state of the system is up., it's receiving price-ticks, makes trading decisions, places orders, receives updates on these orders, saves stuff to the db and so on, all of this is happening all the time while the process is up (for about maybe 20h a day in total). I.e. I'm not doing trading x-number of times a day on schedule, I trade on every tick (if that tick requires trading that is..).
(I do have a couple other 'helper' processes which run for a short time, but these are secondary, e.g. for refreshing EOD data which I run once a day while the main process is down).
- I think you control IBOrderID..
Just checked my code, actually you give them that ID every time you place the order in this call "ClientSocket.placeOrder(newOrdID, contract, order);" so that newOrdID is at first my max(IbOrderID) from my database and then I increment it every time I palce a new order, and provide to IB in that call, and when they return order-updates back to me they use that ID to tell me which order it's for. But they don't increment next order ID for you, you have to do it yourself every time (and do it synchronously if you're doing any funny multi-threading business there, I remember I had some problems when that ID was updated from multiple threads, because I was subscribing to 2 IB gateways simultaneously from the same process..).
There's also this callback "public virtual void nextValidId(int orderId)" in which IB gives you the next valid order id you're supposed to use when you connect., so in fact I'm doing nextOrderID = max(MyLastDBIbOrderID, nextValidorderIdReportedByIB). So in placeOrder you can't use an orderID smaller than they gave you in "nextValidId" and the maxIbOrderID you already have in your database.
I don't think I used their "Flex Reports"..
- yes, I had these situations, both because of bugs and because of technical glitches (e.g. network goes down at the wrong moment, or something got weirdly messed up in my database
). I want to say that I fixed at least all my bugs in that area (hopefully ), and such mess-ups only happen when e.g. I place an order and the network goes temporary down and I don't receive the fill-update., or IB just misbehaves and doesn't send me the fill-update at all for some reason.. I think the last time it happened was a couple of months ago, and on average I'd guess it happens once every 3-5 month.. In such cases my system will think that the order wasn't filled, and the next time IB sends me back my positions (I think I request them every 20 minutes or so) my system will detect position mismatch and will place that instrument into "don't trade" state and send me an angry email. (well, the email will be normal, it's me who will be angry that I have to go and update some db-tables manually and restart the system )
Last edited:
So basically, you're now stubbornly waiting for the fill on your side of the spread no matter what happens to the price? What happens if the price goes away from you and never comes back, do you cancel and resubmit the order after a while (next day)?
Similar to @Kernfusion my execution algo is blocking in that it waits for an order to be completed (with an appropriate message coming back from IB). All orders are stored in the database and I check their status as IB objects and update the database once an update (usually a fill) has occured; updating my order and position databases at the same time (they aren't formally linked a la SQL).
There are time outs however, so it's possible if IB stops talking to me that the code will abort leaving the order in an unfilled state on the database. So I do have seperate code that periodically checks to see if an order that is unfilled in the database but filled with IB. And I have code that checks that my positions are in line with IBs, which of course they won't be if a fill has been missed, and it will lock the instrument and prevent further trading until the mess has been sorted out.
Anyway by the end of the day I should have everything reconciled and so no need to know what yesterdays orders were from IB.
But there are times when it would have been handy to go back further than a single day. If the whole system dies on say Friday afternoon, then if I don't fix it quickly I end up on Monday morning with trades missed and positions out of line. Or when I've missed an expiry, and IB closes out the position for me. As it is now I have to enter these trades manually.
And I'd *really* like the ability to get say a year of trade statements through the API, rather than the current hacky XML parser that I use when I need to do my tax returns...
GAT
Thanks for the detailed explanations Kernfusion and Rob!
If folks are interested I can post my entire setup for getting those, it's quite simple.
My background is in software engineering at big tech co's, and my general mantra is that if I can avoid running something always-on as a service, I should. Every time you have that, you have to worry about keeping it on, providing some alerting and monitoring, and most importantly handling cases when it's down. And it will be down, for any reason - bugs, unhandled exeptions, deployments, OS upgrades, hardware issues, etc. If that happens you probably missed handling some requests while it was down, so some sort of a "backfill" is needed.
Since I'm (and I think most of us here) in a really relaxed environment, where things happen every few hours and not thousands of times per second like in a typical service, I like to use that to my advantage.
Yep. To be honest, that (price going away) only happened twice so far. Current behavior I have is that the order is cancelled if not filled after X hours and retried on the next day (if the new forecast still says we need the order). As with a few of you here, my system isn't impacted big time if execution is delayed by a day. Longer term, I was planning on implementing some sort of price moving on open orders, but I'd like to collect more data to see what's the best way to do that. I also have to run some calculations to see if all this is worth it - if me capturing the spread in majority of executions offsets those few where a price moves away from me.
Yes, this is the problem I'm trying to address in an automated fashion. Though, from comments here, I'm starting to think I'm shaving some yaks.
I'm sure you tried this and I'm missing something, but why can't you use Flex Reports for this? You can specify a full year, define fields that you want and it also supports CSV for output. It's really easy to get them as well, both via API or from IB's admin portal. To clarify, API here is not the socket-based API that IB Gateway or TWS expose, it's the Flex Report HTTP based API, IMHO even easier to use as you can just use requests to fire off a request.
If folks are interested I can post my entire setup for getting those, it's quite simple.
My background is in software engineering at big tech co's, and my general mantra is that if I can avoid running something always-on as a service, I should. Every time you have that, you have to worry about keeping it on, providing some alerting and monitoring, and most importantly handling cases when it's down. And it will be down, for any reason - bugs, unhandled exeptions, deployments, OS upgrades, hardware issues, etc. If that happens you probably missed handling some requests while it was down, so some sort of a "backfill" is needed.
Since I'm (and I think most of us here) in a really relaxed environment, where things happen every few hours and not thousands of times per second like in a typical service, I like to use that to my advantage.
Last edited:
Yeah, probably.. for this strategy you can do things once a day which will simplify things a lot..
But in the beginning I started my system as stock pairs-trading (start arb), and I thought I'll be trading quite often (i.e. if you get a noisy spread you can make several$$ ten times a day on it, so you need to trade it "in real time"), that didn't quite work.. Many spreads had infrequent large losses which mostly offset frequent small gains, and even these that made money didn't make a lot because you can't leverage properly with stocks. Also stocks are a very annoying instrument to trade algorithmically, perhaps even more than futures, because futures only have rolls, which complicate things, but are at least "100%" predictable, stocks have splits and dividends (which IB doesn't give properly in their API), also they get merged\acquired all the time and if you trade like 50 spreads then probably several times a week some crap like an acquisition or a missing dividend happens, and you have to go and fix something..
Here's some examples of the things I traded:
But also, I admit, I just like the idea of event-based system more
I mean to me it seems more pure and "correct"., i.e. if we look at this problem in general, what is the natural trigger for us to do anything - it's a price-change!. i.e. why do we choose to do things once a day at a certain arbitrarily-chosen time when really it's the new price-tick (change in bid or ask) is the natural trigger for us to do anything, so that's what we should be reacting too. And a more natural way to address noise-overtrading is not to "only look at the price once a day" but by using position-change thresholds and forecast-smoothing (and daily trading-limits as the emergency fail-safe).I Realize that it's mostly my inner-geek talking
because of course the individual price-ticks don't change your target position most of the time, i.e. 99.9% of ticks I process don't trigger any trading, and it's totally fine to do things once a day with a slow system.. But well, I just wanted to write something I liked myself the way I liked, and not what some faceless "collective" of managers, analysts and others at work wants me to..@"No work is ever done collectively, by a majority decision. Every creative job is achieved under the guidance of a single individual thought."
Regarding updates and hardware failures, again, it's not a corporation, where I have 5 departments which all control different things (e.g. these guys are only in charge of the operation system., VMs and hardware is another team in a different outsourced company in another country., and those guys are in charge of networking, but not the firewall settings, as it's "security" and it's done buy that other group of people.. Oh I hate my job so much
).I mean here I know when\how\if I update my operating system, I control the hardware, I don't have a bloody "dev ops team" that deploys my software (my dev-ops team is literally 2 script files 12 lines of code each, which only I execute, and it works like a charm in seconds! ). So my always-running service rarely fails (actually I don't think it ever failed, only failed to start), something weird happens maybe once in every 3-5 months and mostly because IBGateway gets stuck and can't open properly or something like that (which I could probably also address with some additional automation, but it's just not worth the effort).
Sorry for too many words, too much coffee +a weird desire to socialize during lockdown
Last edited: