How to crawl historical intraday charts

N

ndtrader14a

I have a utility that can take the snapshot/picture of a webpage and save it. I am interested in using this utility to build crawlers that would download historical intraday charts of certain scrips. I looked at several options (Yahoo, Google finance and several others), but since they do not assign a unique URL to specific days during which the scrip(s) traded, I could not retrieve that charts.

I was just wondering if someone know of any websites that would allow me to accomplish this task. Ideally, I would need the intraday charts for a scrip to be listed on separate page for each day it traded. That way my crawler can fetch these separate webpages easily.

Any other suggestions are welcome too. Thanks in advance.
 
I am interested in printing out charts for a few scrips going back a few weeks at least and studying them closely.

Doesn't look like iqfeed or esignal are free though.
 
ndtrader14a said:
I have a utility that can take the snapshot/picture of a webpage and save it. I am interested in using this utility to build crawlers that would download historical intraday charts of certain scrips. I looked at several options (Yahoo, Google finance and several others), but since they do not assign a unique URL to specific days during which the scrip(s) traded, I could not retrieve that charts.

I was just wondering if someone know of any websites that would allow me to accomplish this task. Ideally, I would need the intraday charts for a scrip to be listed on separate page for each day it traded. That way my crawler can fetch these separate webpages easily.

Any other suggestions are welcome too. Thanks in advance.
More...

stock marker data are stored in a database and not on a web site. You may crawl HTML pages and source code of java scripts but it will give you nothing. You will not be able to get server side scripts and you will not be able to run scripts you got. You may get chart building java scripts but it will not give you much without intraday data which you will not be able to get by using a crawl. You are not the first one who is trying to get historical stock market charts for free. You will just lose a lot of time and you will end up nowhere. The only thing you may get is that your IP address could be blocked for sometime from web-site you crawl - web sites do not like when somebody (besides search engines) crawl them as it creates heavy traffic for them which may affect their subscribers.

I would recommend to find a cheap online provider of historical charts which would satisfy your requirement. Some of hem offer free trial and if you do not need regular access to historical intraday charts, you may subscribe to a free trial only when you need it. It will be free and no headache...

I am paying $143 per year ($12 per month) and I have charts which I can scroll back in history any time. I do not think it is a lot by taking into account I have access to other stuff as well. As an example , see below snapshot of the S&P 500 1-minute intraday volume chart on January 4th of 2011

20110104sp500.png

chart courtesy of http://www.marketvolume.com
 
viktor_k67 said:
stock marker data are stored in a database and not on a web site. You may crawl HTML pages and source code of java scripts but it will give you nothing. You will not be able to get server side scripts and you will not be able to run scripts you got. You may get chart building java scripts but it will not give you much without intraday data which you will not be able to get by using a crawl. You are not the first one who is trying to get historical stock market charts for free. You will just lose a lot of time and you will end up nowhere. The only thing you may get is that your IP address could be blocked for sometime from web-site you crawl - web sites do not like when somebody (besides search engines) crawl them as it creates heavy traffic for them which may affect their subscribers.

I would recommend to find a cheap online provider of historical charts which would satisfy your requirement. Some of hem offer free trial and if you do not need regular access to historical intraday charts, you may subscribe to a free trial only when you need it. It will be free and no headache...

I am paying $143 per year ($12 per month) and I have charts which I can scroll back in history any time. I do not think it is a lot by taking into account I have access to other stuff as well. As an example , see below snapshot of the S&P 500 1-minute intraday volume chart on January 4th of 2011

20110104sp500.png

chart courtesy of http://www.marketvolume.com
More...

I am very open to paying for access. But that still does not really meet my needs. I need to download, and print on paper intraday charts so that I can do stuff with it. I have a training plan that I put in place, and being able to print specific days (not several days at once) is part of it.
 
1. I can print the chart, I posted above. These stock charts have print option.

2. You can print anything you see on your monitor even if charts do not have print option - use print screen button on your keyboard and paste the screenshot into any redactor which allows you to print...

I see no problem. You need to print several charts each for a specific day. You scroll a chart to a specific day you print it, then you scroll to other specific date and you print another chart and so on...
 
viktor_k67 said:
1. I can print the chart, I posted above. These stock charts have print option.

2. You can print anything you see on your monitor even if charts do not have print option - use print screen button on your keyboard and paste the screenshot into any redactor which allows you to print...

I see no problem. You need to print several charts each for a specific day. You scroll a chart to a specific day you print it, then you scroll to other specific date and you print another chart and so on...
More...

Hmm let me chew on this a bit more. I do need to print hundreds, or possibly thousands of charts, depending on how my training goes. But I get the feeling that I could automate it if I find a suitable vendor.
 
Print Automation could be a problem. Never heard about anything like that. The only close thing comes to my mind is something like

1. Open a Microsoft Word file

2. scroll a chart to a specific day in history

3. press print screen button on a keyboard to take a chart's snapshot

4. Go to Word file and press Ctrl+V keyboard combination to paste a chart's snapshot

5. Repeat 2-4 step thousand times (or how many charts you need)

6. Start printing from the Word file all charts' snapshots at once

7. Go coffee while it's printing

Not fully automatic but if you have no time to do it, I guess a kid next door may do it for $20 if you set a specific instruction.
 
Last edited:
viktor_k67 said:
Print Automation could be a problem. Never heard about anything like that. The only close thing comes to my mind is something like

1. Open a Microsoft Word file

2. scroll a chart to a specific day in history

3. press print screen button on a keyboard to take a chart's snapshot

4. Go to Word file and press Ctrl+V keyboard combination to paste a chart's snapshot

5. Repeat 2-4 step thousand times (or how many charts you need)

6. Start printing from the Word file all charts' snapshots at once

7. Go coffee while it's printing

Not fully automatic but if you do have no time to do it, I guess a kid next door may do it for $20 if you set a specific instruction.
More...

Print automation would be a piece of cake really if and when I am able to automate the download of the hundreds of the charts. My utility will save charts (the entire webpage actually) as jpeg or png. I have an image processing module that would extract just the chart from the snapshot of the webpage. I will then programmatically generate LaTeX code to include these images in a pdf file (4 images per page say) obtained by compiling the LaTeX code. Then I can issue the print command and go get that coffee :)

Not as difficult as it sounds if you have the right background.
 
ndtrader14a said:
if and when I am able to automate the download of the hundreds of the charts.
More...

I guess you have to buy intraday historical data, create (if you have the right background) a charting soft which will auto-build historical charts and auto-save them as an image. You still will have to set an instruction (input data) for this soft. Then you still have to "enerate LaTeX code to include these images in a pdf file (4 images per page say) obtained by compiling the LaTeX code. "

I still would go with a "kid next door" who would do 6 steps I describe above for a small payment and there should not be problem to resize images in Ms Word to fit 4 charts on 1 page.
 
You must log in or register to reply here.
Back
Top