[IBrokers] From 0 to Algorithmic Trading

J

jarjar

I am starting this journal to document progress on using Interactive Brokers to algorithmic trade. I will be starting from 0 and going through the steps. This took about 3-6 months of planning, learning different brokers and APIs, and finally it seems IB is the path of least effort.

The initial strategy will be to buy low and sell high, hopefully it can be improved as we go along ;)

The roadmap:

Stage 0: The Setup
Stage 1: The Configuration
Stage 2: Order entry and logging

Stage 3: The Data
  • Streaming: Try IBrokers
  • EndOfDay: Try IBrokers
  • Better solutions: Do tell?
Stage 4 -> ∞: The analysis
  • KISS and try a simple one
  • Abnormal option volume
  • Pairs trading (hard to find the pairs)
  • Gap and go
  • ∞
Stage 5: Paper trade


Let us start with Stage 0.
 
Last edited:
Stage 0 - 50% Complete:

Status screenshot:
t3gzGtQ.jpg
 
Last edited:
Stage 0 - 100% Complete. It seems the gateway uses port 4001 and the TWS uses 7496.

Stage 1 - In progress.

https://github.com/robcarver17/ibswigsystematicexamples is really helpful. The examples on the swigibpy page are OOD or something / dont work.

Running the example on swigibpy I get:

TWS call ignored - managedAccounts(DU15198)
TWS call ignored - nextValidId(1)

A valid theory is historical data is not available for edemo.

So I made some progress and now I am getting:

IB error id 999 errorcode 162 string Historical Market Data Service error message:HMDS query returned no data: GOOG@SMART Trades

Using

contract = swigibpy.Contract()
contract.exchange = "SMART"
contract.symbol = "GOOG"
contract.secType = "STK"
contract.currency = "USD"

tws.reqHistoricalData(999, contract, today.strftime("%Y%m%d %H:%M:%S %Z"),
"1 W", "1 day", "TRADES", 1, 1, None)

I think something is wrong with the exchange/symbol combination?

If I replace GOOG with AMD, it works. But IB throttles it to shit (maybe because demo account), ActiveTick was so much faster.

Also the AMD results make no sense:


History 21 Feb 2016 - Open: 1.87, High: 1.89, Low: 1.84, Close: 1.85, Volume: 34567
History 22 Feb 2016 - Open: 1.85, High: 1.88, Low: 1.82, Close: 1.86, Volume: 34569
History 23 Feb 2016 - Open: 1.87, High: 1.89, Low: 1.83, Close: 1.84, Volume: 34568
History 24 Feb 2016 - Open: 1.85, High: 1.87, Low: 1.83, Close: 1.86, Volume: 34568
History 25 Feb 2016 - Open: 1.86, High: 1.93, Low: 1.85, Close: 1.89, Volume: 34569
History 26 Feb 2016 - Open: 1.88, High: 1.9, Low: 1.82, Close: 1.84, Volume: 25665

That does not coincide at all with the volume (6m+ daily) and the price..
 
Last edited:
Stage 1 - 100% Complete.

Stage 2 - 20%

I can place an order using Erlang, now I just need to log and organize the data so I can keep track of what is open, what is closed etc.

We can see upon login EWrappers nextValidId(self, orderId) gives us the ID we should use for the next request/order, so we do not need to manage our own counter. Or so I thought. When you place an order it does not automatically give you the reqId to use lol. So now you need to either manage your own counter, or increment whenever you get an openOrder msg AND wherever else the behavior is not consistent.

Upon login to IBGateway we get all openOrders that we currently have.

V2uwFZd.png
 
Stage 2 - 50%

Designing the order system is a little complex as we need to ensure we don't even have a duplicate order placement sent to IB, a lost order or a data race condition.

This is subject to change but I currently opted for an event driven order system. Where any component (Strategy, Human override, kill all switch) writes to one place, then event subscribers get notified, this ensures we have decoupled components thus if one component crashes or goes down while calling into another, we don't lose our state (and our money).

bt2c3AY.png
 
jarjar said:
Stage 3: The Data
  • Streaming: Try IBrokers
  • EndOfDay: Try IBrokers
  • Better solutions: Do tell?
More...
At some point you might want to consider a third-party, tick-level data source, such as NxCore (which runs well under wine), as the broker data feeds (at least the retail ones) are generally not quite at that level. These types of feeds stream an entire market (e.g., the BBO and trades for all NMS stocks) at a fixed price (hundreds to thousands of $per month). Of course there may be no point to spending money on the more expensive data sources until you have a better picture of your ultimate system's needs.
 
Occam said:
At some point you might want to consider a third-party, tick-level data source, such as NxCore (which runs well under wine), as the broker data feeds (at least the retail ones) are generally not quite at that level. These types of feeds stream an entire market (e.g., the BBO and trades for all NMS stocks) at a fixed price (hundreds to thousands of $per month). Of course there may be no point to spending money on the more expensive data sources until you have a better picture of your ultimate system's needs.
More...


Yes, I was going to post a status update yesterday but skipped it, I am currently at Stage 3 integrating IQFeed as I found they where pretty decent for the price, NxCore is like 1.5k a month. IB data feed from what is written + the edemo just seems bad. I still would like to connect directly to NASDAQ and CBOE at some point, I mean its not that expensive compared to NxCore, I dont see the point, why have your data hop from NASDAQ to NxCore, have NxCore wrap it up 1 extra time, then ship it off to you.
 
jarjar said:
Yes, I was going to post a status update yesterday but skipped it, I am currently at Stage 3 integrating IQFeed as I found they where pretty decent for the price, NxCore is like 1.5k a month. IB data feed from what is written + the edemo just seems bad. I still would like to connect directly to NASDAQ and CBOE at some point, I mean its not that expensive compared to NxCore, I dont see the point, why have your data hop from NASDAQ to NxCore, have NxCore wrap it up 1 extra time, then ship it off to you.
More...

If you're speaking of some GUI platform that Nasdaq offers, I think we're comparing apples to oranges. When you speak of "algorithmic trading", I assume that (optimally at least) you really want full-feed products -- e.g., the entire US equity market fed to you all the time, rather than requesting one symbol at a time. If I'm wrong, then apologies in advance, as the rest of the post may not be relevant to your case.

I have no idea about CBOE, but to get the full feed (TotalView ITCH, http://www.nasdaqtrader.com/Trader.aspx?id=Totalview2) direct from Nasdaq is far more than NxCore -- several thousand a month (pushing $5k-10k, all-in -- it's MUCH more than just the list price, as you're required to buy a bunch of other stuff from them as well), and that's just Nasdaq quotes; you (probably) need all the other NMS exchanges as well, either through the SIP or direct from each exchange (big $, obviously; I've seen estimates of $600,000 per year, and that's without all the headaches/staff salaries you'd have to incur in administering several servers scattered around northern New Jersey).

NxCore is the least expensive and most convenient way I know of to get full feeds for North American exchanges, and a major reason for this is that NxCore has a very efficient, proprietary compression algorithm. There are a number of other vendors that offer decent solutions as well, but none as cheap or simple, IMO. What you get with some of the more expensive vendors is "ultra low latency" (although still not as low as direct feeds), and broader coverage e.g. of full book feeds for each exchange (although NxCore gives you Nasdaq Level II at least). On the downside, they generally don't deliver data over the Internet, so you need to either colocate with them or get some sort of private connection that may be quite expensive.
 
I must of misunderstood then, I thought buying streaming data for the entire market from NASDAQ was a few hundred dollars a mon, and a 650k one time fee if I want to pass this data onto my customers (say if I was IQFeed). NxCore is most likely the best then, they seem to be the only one I have seen so far that does not have a plaintext protocol nor one padded with twice as many zero bytes as the payload.

I am currently having great pains getting IQFeed to work, its so annoying. I am finalizing getting history data to work, then I still have streaming data and news feeds to get working, am not even considering options atm. I am happy though that IQFeed has pre/post market data and a simple symbol system.

With historical data I could try a gap and go strategy, with streaming ticks I could try some swing trading.


Stage 2 - same 50% untouched.
Stage 3 - 20% Can get day history now on linux with IQFeed.

dljuMoT.png


Getting the last 3 days of this POS :)

Nice tuple (maybe record in future) for insertion into mnesia for safekeeping.

For now I will format the numbers as floats because they are easier to read for the eyes when debugging, but probably would need to store them as int+decimal prec in future, I am not sure yet. Off by a cent on a 5$+ stock is not that significant
 
Last edited:
You must log in or register to reply here.
Back
Top