I've been using yahoo historical quotes and bringing it into excel using vba for years. Over the last couple of days, the historical quotes stop loading after about 30 quotes. To get it working again, I have to wait 10 minutes or so or change my VPN to another proxy. Anyone else experiencing this lately?
Yahoo historical data
- Thread starter joeactuary
- Start date
Yes, it may work.
No, I meant it stopped working before VPN. After about 30 calls, It only works now when I use a VPN proxy. It's like they are now tracking IP address and limiting it.
Are you storing the historical data, or redownloading almost the same historical data everyday?
Redownloading.
It would require updating/adding code and the knowledge to do so, but consider making your flow more free-data-supplier-friendly. Doing so may actually solve your problem, in this case (and help to keep free suppliers ... free).
More "friendly" as in the historical data is saved to the harddrive, etc. (csv, xlxs, etc.); and only new data is downloaded daily.
Until then, and if you are automating the downloading of multiple instruments; adding an appropriate delay between instruments might result in the 10 min. lockout no longer appearing. You'd have to experiment to find such a delay, if it even exists. Coding this is simpler than adding code to automate dealing with the 10 min. delay. (I assume you currently manually restart the process.)
Thank you for above tips. I will try putting in a delay to help cure problem. I like your suggestion and would like to be a good scraping citizen but for someone with my limited programming experience, it would be a massive undertaking for me to store all that data, figure out how to link it all back in(I currently just use excel) in or update my excel calculations and automate the real time error checking in manipulating the data.
Has anyone else recently started experiencing this from Yahoo? I'll like to know if they started ISP tracking or changed something in their system necessitating me to change how I handle cookies and crumbs?
Has anyone else recently started experiencing this from Yahoo? I'll like to know if they started ISP tracking or changed something in their system necessitating me to change how I handle cookies and crumbs?
I understand. (btw, it can all be done via Excel and Excel VBA)
www.tiingo.com also has a free tier and api.
I don't think this has changed for about four years, but one thing that you could do is refresh the cookie crumb every 10-20 minutes. For example do the equivalent of,
https://www.mapleprimes.com/posts/208409-Downloading-Historical-Stock-Quotes
Another thing that might help is have your requests alternate between the different IP addresses for yahoo finance.
Code:
$ nslookup finance.yahoo.com
Server: 1.1.1.1
Address: 1.1.1.1#53
Non-authoritative answer:
finance.yahoo.com canonical name = edge.gycpi.b.yahoodns.net.
Name: edge.gycpi.b.yahoodns.net
Address: 69.147.86.12
Name: edge.gycpi.b.yahoodns.net
Address: 69.147.86.11
Name: edge.gycpi.b.yahoodns.net
Address: 2001:4998:20:800::1000
Name: edge.gycpi.b.yahoodns.net
Address: 2001:4998:20:800::1001And to be friendly, your code should sleep a little between requests (e.g., 100 milliseconds -- more before a retry to recover from a failed request).