Looks like Yahoo Download API is now Dead - 11/1/2017

hayman said:
Yeah, have a backup source is good. I will probably build a backup using my IB or TD Ameritrade feeds.

FYI, there's an old command line utility called WebFetch.exe, which you can probably hunt around for, which could assist in the automation within a command line script.
More...

To use curl to download the export file it needs three things:

1. A copy of the cookies that Yahoo uses. Best way is to install a fresh copy of Firefox Portable and add the Export Cookies add-on, then log into Yahoo. Then right click on the title bar, select Menu Bar then select Tools/Export Cookies from the menu bar and save as say cookies.txt

2. The url that downloads the export file. In Firefox it can be copied with a right click on the 'Export' link and selecting Copy Link Location

3. Installing curl and running it on the command line. Grab a copy from https://curl.haxx.se/, install, copy the cookies file into the same directory, open a command line and change to the the directory. Then run something like:

curl "the-magic-url" --cookie cookies.txt --output yahooquotes.csv

Hopefully that should do it! :)

I guess Yahoo don't want to allow anonymous download as it makes it more open to abuse and denial of service attacks, even hacking.
 
Last edited:
I've just produced a python script that can handle multi ticker yahoo data acquisition and thought I'd share incase this can help anyone. **The script requires the following packages which can be attained through pip: pandas / pandas_datareader / fix_yahoo_finance **

The script saves each tickers data in a csv file in a folder named YahooData which is created in the same directory you place the python file. I have currently configured it to attain yahoo data for all S&P 500 companies but this can easily be modified to your own requirements by changing the list in line 42 such that it contains the tickers you require and finally comment out line 43.

To run the script simply type "python AttainYahooData.py" in your command line ensuring that you are in the directory the python script is located.

I've multi-threaded the process which speeds things up significantly.

The data acquisition isn't always successful first time round so to retry type y when prompted by the command line. You may need to retry three or four times if you're downloading hundreds of tickers.

The python script can be accessed on my google drive here:


I have just modified the script so it will repeatedly try to fetch the tickers, without the need for user input, until tickers have been downloaded successfully. This will be better suited to automated systems based on yahoo data. Make sure you type the tickers correctly or else you will be stuck in a never ending loop :) (this version is named AttainYahooData_v1.py therefore use command "python AttainYahooData_v1.py" to run.

 
Last edited:
and such programs:
YLoader
HSQuote / HSQuote Plus
it is not sufficient?
The creator of this HSQ is very polite and always helps with problems.
try.
 
...
The script saves each tickers data in a csv file
More...

Would you consider to create a modified script that creates a single csv file with one ticker data per line? Of course, if it is too much trouble, forget about it. I would do that myself but I don't have the skill needed.

Thanks.
 
fabiospark said:
Would you consider to create a modified script that creates a single csv file with one ticker data per line? Of course, if it is too much trouble, forget about it. I would do that myself but I don't have the skill needed.

Thanks.
More...

I can take a look into this when I have some spare time but could you please help me in understanding exactly what you require. Am I correct in thinking you want the script to fetch the most recent data available for a number of defined tickers and then compile it in a csv in the format below?

ticker, open, low, high, close, adj close, volume
A, xxx, xxx, xxx, xxx, xxx, xxx
B, xxx, xxx, xxx, xxx, xxx, xxx

If not could you give an example of what it is you want.
 
I would definitely be interested in something like that.
Ideally I would love to be able to pull Price, Dividend, and Upcoming Payment Date

If this is something I could pull from a web query into excel automatically that would be great.
I'm currently looking at https://iextrading.com/developer/docs/#iex-api to try and do this, but my knowledge is limited. I was using VBA code in excel to pull from Yahoo.
 
TauLepton said:
I can take a look into this when I have some spare time but could you please help me in understanding exactly what you require. Am I correct in thinking you want the script to fetch the most recent data available for a number of defined tickers and then compile it in a csv in the format below?

ticker, open, low, high, close, adj close, volume
A, xxx, xxx, xxx, xxx, xxx, xxx
B, xxx, xxx, xxx, xxx, xxx, xxx

If not could you give an example of what it is you want.
More...

Thank for your attention and for your answer.

In the code box below you can see the actual content of the csv file I used to download from Yahoo every few minutes with Free Download Manager.

Code: 
"ETFMIB.MI",22.504,"11/1/2017","5:35pm",+0.181,22.400,22.576,22.400,585835
"DJSC.MI",35.835,"11/1/2017","3:31pm",+0.195,35.910,35.930,35.850,520
"CASAMO.MI",N/A,N/A,N/A,N/A,N/A,N/A,N/A,N/A
"IDVY.MI",23.705,"11/1/2017","5:11pm",+0.095,23.625,23.775,23.625,12711
"VAL.MI",125.94,"11/1/2017","2:15pm",+0.80,125.55,126.00,125.55,518
"GWT.MI",130.26,"11/1/2017","2:15pm",+0.60,130.51,130.61,130.31,315
"DAXX.MI",128.66,"11/1/2017","5:27pm",+1.79,127.84,128.65,127.41,34393
"^STOXX",396.77,"11/1/2017","5:50pm",+1.55,395.91,398.05,395.91,0
"AUTP.MI",73.06,"11/1/2017","5:11pm",+1.27,72.63,73.10,72.63,284
"BRES.MI",54.60,"11/1/2017","5:20pm",+1.37,54.14,54.99,54.14,13883
"CHM.MI",111.21,"10/31/2017","5:11pm",+1.49,109.40,109.40,109.40,0
"FOO.MI",76.32,"11/1/2017","5:21pm",-0.09,76.20,76.60,76.20,3183
"OIL.MI",41.73,"11/1/2017","5:21pm",+0.40,41.48,41.91,41.48,1063
"PHG.MI",100.23,"11/1/2017","5:19pm",-0.19,100.78,100.79,100.21,218
"TELE.MI",37.445,"11/1/2017","4:59pm",+0.240,37.350,37.530,37.290,27490
"UTI.MI",41.18,"11/1/2017","3:01pm",+0.04,41.30,41.32,41.28,473
"INRG.MI",4.6875,"11/1/2017","4:16pm",+0.0225,4.6950,4.6950,4.6700,4266
"IUSA.MI",22.1225,"11/1/2017","5:16pm",+0.0925,22.0700,22.1675,22.0700,35953
"ITKY.MI",23.765,"11/1/2017","5:26pm",+0.525,23.300,23.840,23.300,603
"TUR.MI",45.26,"11/1/2017","5:29pm",+1.05,44.29,45.50,44.29,10776
"CINA.MI",145.50,"11/1/2017","5:20pm",+1.69,145.00,145.72,145.00,2123
"INDI.MI",16.765,"11/1/2017","5:19pm",+0.265,16.540,16.790,16.540,35262
"XMIN.MI",13.26,"11/1/2017","2:57pm",+0.07,13.16,13.31,13.16,10466
"JPN.MI",131.33,"11/1/2017","5:20pm",+1.15,130.50,131.43,130.50,59908
"IBZL.MI",27.355,"11/1/2017","5:28pm",-0.030,27.525,27.715,27.310,17607
"BRA.MI",17.49,"11/1/2017","5:18pm",+0.05,17.62,17.67,17.49,7918
"XWCD.MI",20.925,"4/19/2016","2:59pm",+0.055,20.910,20.910,20.910,450
"XWSD.MI",26.545,"4/14/2016","3:54pm",+0.095,26.665,26.665,26.550,371
"SAFRI.MI",35.29,"11/1/2017","5:21pm",+0.29,35.09,35.38,35.09,155
"COFF.MI",0.9835,"11/1/2017","5:17pm",-0.0195,1.0130,1.0160,0.9780,1192134
"SOYB.MI",18.235,"11/1/2017","4:47pm",+0.180,18.160,18.235,18.160,213
"PHAU.MI",105.50,"11/1/2017","5:26pm",+0.85,105.07,105.62,105.07,18688
"OIL3L.MI",0.252,"11/1/2017","5:35pm",+0.002,0.251,0.263,0.249,5536247
"NEX.PA",56.36,"11/1/2017","5:35pm",+0.16,56.20,57.07,56.01,212169
"UG.PA",20.805,"11/1/2017","5:35pm",+0.440,20.500,20.860,20.470,1958008
"VOW.DE",167.85,"11/1/2017","5:35pm",+8.45,160.65,168.20,160.40,192917
"BAN.MI",3.524,"11/1/2017","5:35pm",+0.004,3.530,3.608,3.524,23748
"ISP.MI",2.878,"11/1/2017","5:36pm",-0.008,2.894,2.902,2.874,67290043
"ISPR.MI",2.706,"11/1/2017","5:35pm",-0.002,2.714,2.718,2.698,2124166
"PRY.MI",30.00,"11/1/2017","5:35pm",+0.40,29.75,30.08,29.67,718499
"333129.MI",N/A,N/A,N/A,N/A,N/A,N/A,N/A,N/A
"333130.MI",N/A,N/A,N/A,N/A,N/A,N/A,N/A,N/A
"X0729K.TI",N/A,N/A,N/A,N/A,N/A,N/A,N/A,N/A
"347024.MI",N/A,N/A,N/A,N/A,N/A,N/A,N/A,N/A
"148058.MI",N/A,N/A,N/A,N/A,N/A,N/A,N/A,N/A
"B3ET.MI",N/A,N/A,N/A,N/A,N/A,N/A,N/A,N/A
"EU3L.MI",14.575,"11/1/2017","5:35pm",+0.170,14.505,14.665,14.505,8683
"XSX6.MI",81.13,"11/1/2017","2:16pm",+0.27,81.13,81.33,81.12,6457
"LEVMIB.MI",8.92,"11/1/2017","5:29pm",+0.15,8.80,8.97,8.80,467896
"82EA.L",N/A,N/A,N/A,+0.00,N/A,N/A,N/A,0
"X32348.TI",N/A,N/A,N/A,N/A,N/A,N/A,N/A,N/A
"475499.MI",N/A,N/A,N/A,N/A,N/A,N/A,N/A,N/A
"I78137.mi",N/A,N/A,N/A,N/A,N/A,N/A,N/A,N/A
"T1NV11.MI",N/A,N/A,N/A,N/A,N/A,N/A,N/A,N/A
"TLG13.MI",N/A,N/A,N/A,N/A,N/A,N/A,N/A,N/A
"764806.MI",N/A,N/A,N/A,N/A,N/A,N/A,N/A,N/A
"PIST14.MI",N/A,N/A,N/A,N/A,N/A,N/A,N/A,N/A
"PIST19.MI",N/A,N/A,N/A,N/A,N/A,N/A,N/A,N/A
"PIST21.MI",N/A,N/A,N/A,N/A,N/A,N/A,N/A,N/A
"PIST26.MI",N/A,N/A,N/A,N/A,N/A,N/A,N/A,N/A
"PIST35.MI",N/A,N/A,N/A,N/A,N/A,N/A,N/A,N/A
"POE00K.TI",N/A,N/A,N/A,N/A,N/A,N/A,N/A,N/A
"XS0935881853.TI",94.60,"11/1/2017","5:07pm",-0.17,N/A,94.81,94.58,70000
"XS0995130712.TI",95.87,"11/1/2017","1:26pm",-0.26,N/A,96.14,95.55,198000
"XS1139474206.TI",92.59,"11/1/2017","4:08pm",-0.08,N/A,92.66,92.58,90000
"XS1167524922.TI",97.40,"11/1/2017","5:15pm",+0.04,N/A,97.42,97.18,1430000
"764806.MI",N/A,N/A,N/A,N/A,N/A,N/A,N/A,N/A
"FR0010135103.PA",655.6900,"10/30/2017","9:00pm",-0.8000,N/A,N/A,N/A,0
"IE00B1Z6D669.IR",15.4100,"10/27/2017","8:00pm",+0.0100,N/A,N/A,N/A,0
"IE00B11XZB05.IR",19.7100,"10/30/2017","8:00pm",+0.0000,N/A,N/A,N/A,0
"AH8M.MU",118.30,"11/1/2017","6:47pm",+0.11,118.44,118.44,118.30,0
"FJRZ.MU",23.18,"11/1/2017","7:55pm",+0.32,23.11,23.24,23.11,30
"13716156.SW",N/A,N/A,N/A,N/A,N/A,N/A,N/A,N/A
"EURGBP=X",0.8774,"11/1/2017","7:58pm",+0.0006,0.8767,0.8784,0.8732,0
"EURAUD=X",1.5150,"11/1/2017","7:58pm",-0.0063,1.5213,1.5224,1.5115,0
"EURNZD=X",1.6883,"11/1/2017","7:58pm",-0.0033,1.6916,1.6919,1.6792,0
"EURZAR=X",16.3509,"11/1/2017","7:58pm",-0.0968,16.4477,16.4943,16.3088,0
"EURTRY=X",4.4381,"11/1/2017","7:58pm",+0.0220,4.4148,4.4564,4.4069,0
"EURUSD=X",1.1623,"11/1/2017","7:58pm",-0.0025,1.1647,1.1657,1.1605,0
"USDJPY=X",114.1230,"11/1/2017","7:58pm",+0.4730,113.6580,114.2820,113.5800,0
"XWD.TO",48.135,"11/1/2017","3:41pm",+0.005,48.300,48.350,48.080,42001
"DISW.MI",245.40,"11/1/2017","11:30am",+1.25,244.99,245.99,244.99,21
"STAW.MI",274.18,"11/1/2017","5:21pm",+1.16,274.39,274.71,274.39,11

This was the URL that created the csv from Yahoo:
Code: 
http://download.finance.yahoo.com/d/quotes.csv?s=ETFMIB.MI,DJSC.MI,CASAMO.MI,IDVY.MI,VAL.MI,GWT.MI,DAXX.MI,%5ESTOXX,AUTP.MI,BRES.MI,CHM.MI,FOO.MI,OIL.MI,PHG.MI,TELE.MI,UTI.MI,INRG.MI,IUSA.MI,ITKY.MI,TUR.MI,CINA.MI,INDI.MI,XMIN.MI,JPN.MI,IBZL.MI,BRA.MI,XWCD.MI,XWSD.MI,SAFRI.MI,COFF.MI,SOYB.MI,PHAU.MI,OIL3L.MI,NEX.PA,UG.PA,VOW.DE,BAN.MI,ISP.MI,ISPR.MI,PRY.MI,333129.MI,333130.MI,X0729K.TI,347024.MI,148058.MI,B3ET.MI,EU3L.MI,XSX6.MI,LEVMIB.MI,82EA.L,X32348.TI,475499.MI,I78137.mi,T1NV11.MI,TLG13.MI,764806.MI,PIST14.MI,PIST19.MI,PIST21.MI,PIST26.MI,PIST35.MI,POE00K.TI,XS0935881853.TI,XS0995130712.TI,XS1139474206.TI,XS1167524922.TI,764806.MI,FR0010135103.PA,IE00B1Z6D669.IR,IE00B11XZB05.IR,AH8M.MU,FJRZ.MU,13716156.SW,EURGBP=X,EURAUD=X,EURNZD=X,EURZAR=X,EURTRY=X,EURUSD=X,USDJPY=X,XWD.TO,DISW.MI,STAW.MI,&f=sl1d1t1c1ohgv&e=.csv

Then I have an auto updating query on external data in an Excel file that will get the data from the csv so every five minuts I can see the values updated.

Now that Yahoo feature is dead, being able to recreate a similar csv in some other way would be awesome.

Thanks.

fabio
 
barrett7 said:
I would definitely be interested in something like that.
Ideally I would love to be able to pull Price, Dividend, and Upcoming Payment Date

If this is something I could pull from a web query into excel automatically that would be great.
I'm currently looking at https://iextrading.com/developer/docs/#iex-api to try and do this, but my knowledge is limited. I was using VBA code in excel to pull from Yahoo.
More...

To get the data into Excel I first created a "Query on external data" manually importing from the csv into a sheet named "tests2", then I went and see the query properties, gave it a name (in my case "PF13") and saved it and then I created a button on a sheet ("WEB" for me) that triggers this VBA sub routine:
Code: 
Private Sub UpdateButton_Click()

Application.ScreenUpdating = False
Sheets("tests2").Activate
    With Sheets("tests2")
         .Select
          .QueryTables("PF13").Refresh BackgroundQuery:=True
    End With
    Sheets("WEB").Activate
Application.ScreenUpdating = True

End Sub

Apart Yahoo errors every now and then, it's been working like a charm.

Thanks.

f.
 
You must log in or register to reply here.
Back
Top