Yahoo Historical Data - Did they change the URL recently?

[ET180]

tried IB's daily. no go. couple hours coding just to see this after first requests:

what a world we are living in...no free, decent,reliable source of simple daily data for publicly traded securities...fuck y'all

Well...for it to be free means that someone has to do the work for nuttin.

But that's quite irrelevant as the Yahoo data is unusable as of now, the adjustments are all over the place - so if you're okay with trading with charts that have 50%, 100%, 400% gaps on it that didn't actually take place, be my guest.

There are two pieces of data returned in the yahoo CSV data. One is adjusted and one is un-adjusted. I was using the un-adjusted as the adjusted seems to include dividend yield or something else in addition to correction for the splits. I agree with you that it's less than ideal, but I augment it with the IB data (use as much IB data as possible, but fill in more history with yahoo data). I don't have a problem with pacing for historical EOD quotes. But to pull the interesting part of a few option chains can take longer than 30 seconds. Threading helps a little, but seems that there's a hard limit on how much data can be returned over a period of time. Not a big issue for me. I just let it run over night. I'm more interested in the historical implied vol, but I like to overlay a few option deltas on my charts.

 

[HobbyTrading]

Another unanswered question that puzzles me is a huge difference in volume reported by IB and yahoo (yahoo and google match between each other). Anyone Can explain it? It's almost ×2 in many cases on liquid securities like QQQ
Thanks!

I'm not sure but recall reading somewhere that it was related to counting odd lots: some say that the IB data does not contain the volume of odd lots.

 

[Bob111]

I'm not sure but recall reading somewhere that it was related to counting odd lots: some say that the IB data does not contain the volume of odd lots.

Thanks!

question to IB experts. if I understand things correctly, after one collect daily data from IB, updating them by the end of the day shouldn't be a problem (talking about 15 seconds time restrictions between requests), if I request real time data for each stock RTH only. It should return all the data for the day? And I can request 100 tickets per shot? Im wrong?

Thank you!

 

[HobbyTrading]

Thanks!

question to IB experts. if I understand things correctly, after one collect daily data from IB, updating them by the end of the day shouldn't be a problem (talking about 15 seconds time restrictions between requests), if I request real time data for each stock RTH only. It should return all the data for the day? And I can request 100 tickets per shot? Im wrong?

Thank you!

What do you mean by "15 seconds time restrictions between requests"?
If you are requesting live market data you can do so for 100 tickers in parallel, at the same time (unless your account size is large, or paid for a larger quantity of tickers, then you can do more). If you are asking for historical data you can request 60 items per 10 minute period, is what their documentation states. However, lately some people reported that they were able to download more than that limitation.

 

[d08]

Thanks!

question to IB experts. if I understand things correctly, after one collect daily data from IB, updating them by the end of the day shouldn't be a problem (talking about 15 seconds time restrictions between requests), if I request real time data for each stock RTH only. It should return all the data for the day? And I can request 100 tickets per shot? Im wrong?

Thank you!

Keep in mind that when you append to the previous data, it won't be adjusted for splits.

Well...for it to be free means that someone has to do the work for nuttin.

There are two pieces of data returned in the yahoo CSV data. One is adjusted and one is un-adjusted. I was using the un-adjusted as the adjusted seems to include dividend yield or something else in addition to correction for the splits. I agree with you that it's less than ideal, but I augment it with the IB data (use as much IB data as possible, but fill in more history with yahoo data). I don't have a problem with pacing for historical EOD quotes. But to pull the interesting part of a few option chains can take longer than 30 seconds. Threading helps a little, but seems that there's a hard limit on how much data can be returned over a period of time. Not a big issue for me. I just let it run over night. I'm more interested in the historical implied vol, but I like to overlay a few option deltas on my charts.

Yahoo data isn't adjusted for dividends any longer. The unadjusted close is completely useless now, it's actually multiplied by the wrong multiplier, *2 instead of *0.5 for a 2:1 split. Amateur hour over at Yahoo. On top of that, many splits are missing as I've recently discovered.

 

[jtrader33]

That's an unix timestamp, it's literally the seconds from 1st January 1970.



I looked at one of those examples and it's not wrong. 1/12/17 AKP close is lower because that's the closing auction which happened at 16:02, the low prices are only recorded up to 16:00. That's actually the correct way to do things. If it makes you feel better, you can adjust the low to that close.

@d08 Are you saying that if you request historical data on a stock with contract.m_exchange = "SMART" and contract.m_primaryExch = 'primary exchange' that it will give you:

1) open = primary exchange opening auction
2) high = consolidated high
3) low = consolidated low
4) close = primary exchange closing auction

That's great if so, but my understanding was that the only way to get the primary open/close was set both contract.m_exchange and contract.m_primaryExch = 'primary exchange'.

 

[d08]

The opening price appears to be the first print after 9:30, not the auction price necessarily (except for NASDAQ where it typically is). Yet now it seems the closing auction is included in the daily bar, I don't remember it being so before. Best to just verify by looking at the data and comparing to time and sales.

 

[sysdevel99]

Yahoo data isn't adjusted for dividends any longer.

Man - that was one of the nice things about it. I also have TOS/IB and Google as an alternative but neither are dividend adjusted either. Are there any providers out there that are?

 

[d08]

Man - that was one of the nice things about it. I also have TOS/IB and Google as an alternative but neither are dividend adjusted either. Are there any providers out there that are?

I don't think so. It's not that hard to adjust for the data yourself but there's obviously some programming involved.

 

[ET180]

What do you mean by "15 seconds time restrictions between requests"?
If you are requesting live market data you can do so for 100 tickers in parallel, at the same time (unless your account size is large, or paid for a larger quantity of tickers, then you can do more). If you are asking for historical data you can request 60 items per 10 minute period, is what their documentation states. However, lately some people reported that they were able to download more than that limitation.

Yes, I am able to download more than the limitation although I'm only requesting a couple days usually since I run the same request every night and keep a history. Historical bars update quick, but pulling options chains really slows things down. My pulls are threaded, but the threading does not seem to improve performance, I still only get one symbol at a time. Maybe there's a bottleneck. Need to look into this more, but haven't found the need yet.