Platform-Independent Data Feed/IB Data?

T

TheWanderer

Hello,

This is my first post here, so please bear with me.

I need to download historical and real-time (not delayed) intraday data for, primarily, U.S. stocks and options, at different time intervals (daily, 60 minute, 10 minute, etc.) into text files throughout the day. I do not need streaming data (at least not yet). I have my own software that will scan and analyze OHLCV data in ASCII files.

After digging around here, searching the internet, and inquiring with various data providers, I can not find a reasonably priced, platform-independent data provider where I can download data to Mac OS X or Unix either through HTTP or a Java or C/C++ API. Does anyone know of any service?

Linnsoft's InvestorRT allows for data downloads on Windows and Mac OS X, but I would prefer to not have to work through an application.

It looks like IB's data may be able to suit my needs. However, I had some questions about IB's service.

First, the API documentation for requestHistoricalData says, "durationStr - set the query duration up to one week, using a time unit of seconds, days or weeks. Valid values include any integer followed by a space and then S(seconds), D (days) or W (week). If no unit is specified, seconds is used." Does this mean only 1 week of data can be downloaded at a time?

Second, I thought I read somewhere that IB's historical data is only available up through the current market open. Is this correct? If so, does that mean, say 60 minute, real-time bars can not be downloaded throughout the day?

Any help anyone can give would be greatly appreciated.

Thanks.
 
Thanks for the link. So close. It looks like they are only going to offer EOD data - no real time. I am really looking for real-time data.
 
Quote from TheWanderer:


First, the API documentation for requestHistoricalData says, "durationStr - set the query duration up to one week, using a time unit of seconds, days or weeks. Valid values include any integer followed by a space and then S(seconds), D (days) or W (week). If no unit is specified, seconds is used." Does this mean only 1 week of data can be downloaded at a time?
More...

For shorter bar lengths ( < 5 min) you get up to 1 days data. For longer bars you get 1 week. If you want more you need to issue multiple requests, each with the appropriate date.


Second, I thought I read somewhere that IB's historical data is only available up through the current market open. Is this correct? If so, does that mean, say 60 minute, real-time bars can not be downloaded throughout the day?
More...

No this is not correct. Up to date data can be obtained.
 
dcraig, thanks for the help with IB.

TruthTeller, thanks for the tip on Track Data. I haven't tried them yet. How are you accessing their data?
 
I called Track Data and, unfortunately, was told they were only compatible with Windows. I have found a couple new sources that I am checking into and will report back.
 
I have a ddfplus backup feed. It is not cheap, about $110/month, but I get what I need, http/tcp access, it is always on, etc. I too have been looking for something RELIABLE, http/socket based, NO DLL/COM b.s., and therefore x-platform, realtime, etc. However, they have fewer symbols available than IB. No FOREX (that I know of, for example). It is hard to find at a reasonable price... I use them mainly for historical data access, not their live feed, btw. Their 'history' data is up to the moment, so I can call it up as needed (i.e., if IB stops).

IB's data feed *should* be OK, but the reality is, you need to create an intermediate tool (IMHO) to manage the data stream and provide the data you want/need. You can use Medved's QuoteTracker as a possible solution.

IB's feed HOWEVER, is 'consolidated' and every 300ms or so TWS gets the aggregate changes for the items you monitor. Then it passes the data on to your app. This means that you sometimes lose a high/low price since if the price jumps up and then back down you only get the final value.

Additionally, IB's feed has NO TIMESTAMP! So, you have to inject/apply/sample your own clock and live with the network latency/machine latency variations that causes. This just means, the data bars come out a little different than a true tick-based, time-stamped feed.

Also, there is (basically) a 100 simultaneous realtime symbol limit. You can get 'more' if you trade several hundreds of dollars a month in commissions. But you get at least a minimum of 100.

All-in-all, IB's data is fine for 'approximate' price/data... and their history data seems to be tick-accurate. History data goes back 6 months from the present and resolution down to 1 second bars.

One thing IB does NOT do is manage spikes. That is, occasionally, there is a super high/low value in the bid/ask/last stream. Like for aapl, a 7000.00 High tick. Now, that is just not real, not even close. So, I run a clip filter to watch for those.

IB also has an annoying 'reset' issue. Every day their TWS application shuts down. There are several tools to work around this.

Additionally, IB is down from Midnight Friday until noon Sunday. No history data. No trading. No fun. Sure, most markets are closed, but why not allow History access during that time!? Oh well...

I use it because TWS runs as JAVA on the Macintosh. For the record, I think JAVA blows. But, the TWS data feed seems to run OK. TWS has issues, the charting does not produce much more than a crude view of the data, the book trader is well below 'standard', they have been adding 'features' lately that are seemingly bloating the app with relatively little value to the end-users of the API, there are no scripting tools-you have to use the API, the DDE interface is cantankerous at best, etc., ... but as a simple order entry tool, it works I guess. And they have low commissions.

OK, that is enough... :-) Just wanted to let you know about it... I use it, but there are tradeoffs... and it is 'free' if you trade a few times a month and keep an account with them.

If anyone out there has a good tip on a reliable, reasonable cost, HTTP/TCP, REALTIME + Historical down to the minute (at least!), tick-based/minute-based (prefer sub-minute), feed, I'd be glad to hear about it as well! And, again, NO WINDOWS DLL's! :-)
 
scottorama,

Thanks for all the information! I would be willing to pay the $110/month, but their API fee looks a little steep, $1500/year. Did you have to pay the $1500/year? Is this fee just for support or for API?

For anyone that is interested:

I'm trying to get something working with Linnsoft InvestorRT with DTNMA for downloading histroical data throughout the day.

Prophet.net's Gold subscription offers an HTTP download of historical data on their web site, but does not allow "screen scraping" in their Terms of Use without written permission. They restrict the number of downloads per day to 100. I have contacted them to try to work something, but no one has contacted me.
 
OK... here we go... been there... well... here is what I know...

1) I don't think I had to pay any fees up front for DDFPLUS, just started with the monthly developer fee. I intend(ed) to create an application (for sale), and if you do sell your app, then the fees drop to $39/user or something like that... and that would be AWESOME if it was a great feed, etc. I wound up implementing a 1-min history capture routine, so I can get my data for my app one way or another. NASD/AMEX/NYSE. Ask for Andrew, he can give you the rundown. Tell him I sent you... maybe he'll charge you more! (Just kidding... :-)

2) I have also spent about 8 months on Linnsoft / Investor R/T. Bill and Chad are great guys. It is a little pricey, and... you have to have a data feed. Ugh. I tried MyTrack with that situation. I wasn't real happy with it. So I quit that about 6/8 months ago. The IRT code is interesting, but still a bit buggy on the Mac. Well, not really buggy per se, but it is(was) written in the older Mac API (Carbon, not the new Mach-O OS X stuff). And so it had issues. I saw (the other day) that they have a new feed available, and I was considering re-trying it... but frankly... I just couldn't bring myself to do it. Too much money. IRT is quite complex, and while I like what they have, I still want to write my OWN code, so why pay so much for a 'conduit'. So, I haven't gone that route.

3) I also used Prophet.Net for charting. Great tool. Reasonable price. I am a Gold member. They had some cool 'auto trend' type stuff coming online. Then... INVESTools bought them. Crap. They raised the price, removed the trend tools and now you have to be a member of their cult (about $5000) to access the trend tools. They have the ability to download 1-min data to a csv/Excel file. Cool! But... they had a 100 download/day limit! Crap. I needed +/- 500 symbols per day. Well, NOW, they reduced the downloads to 5 and charge $1 EACH after that for their Excel plug-in when you go over the limit of 5!!!!! Holy cow! I am afraid to try going over 100 on the basic d/l! Plus it is EOD, so you can't use it for intraday charting. They have something called ProphetStream, and it works with QuoteTracker, but I have not used/tried it, as again, it adds QuoteTracker into the middle of things. It has some API, etc., but like you, I just want a simple, direct access feed!!!
https://www.prophet.net/forms/subscribe.jsp?product=PSS&preselect=PSS

It just seems like too much to ask for a simple feed, doesn't it?

If IB provided a GUI-less access to their API, that would be great. JAVA seems to be OK at doing the basic socket stuff and data management ... but the GUI is just too slow - typical JAVA. However, the damn memory footprint of the JAVA VM is pretty huge too. About 1GB on my Mac OS X. Of course it is virual memory and paged out, yada, yada, yada... but it is still way slow at startup and I bet it is no where near as efficient at processing the data stream. We're talking about a few buffers and some loops here people... not rocket science. Why load a whole VM for that!

With a bit of C code (NOT C++!) and some effort, a GOOD, stable, FAST data streamer could be made that could be ported to just about everything on the planet. NO BLOAT. There are web servers on PIC controllers, how cool would it be to have a stock trader like that!!! http://www-ccs.cs.umass.edu/~shri/iPic.html

OK, maybe not a chip that weak, but close! :-)

-Scott
 
You must log in or register to reply here.
Back
Top