how to do web scrapping

Quote from slacker:

You might take a look at the Coursera course from Georgia Tech on "Computational Investing Part I". The first few weeks covers getting data from yahoo/google. The last part of the course was using Python scripts to build a portfolio. There is also a lot of information on the class discussion forum....

https://www.coursera.org/course/compinvesting1

Good luck
More...

Thanks looks interesting. I have never any python before. I might just enroll.

As an aside, Coursera says all courses are free on it.

How does coursera make money?
Why do professors upload their courses on coursera, if they are not getting paid anything? I know few people will do it for a good cause. But many professors at my university were "extremely interested" in keeping their intellectual property private and wanted to monetize their IP in as many ways as possible.

Hell - one professor never gave us powerpoints rather just printed slides and gave them to us so that we couldn't pass-on the powerpoints electronically to others.
 
Quote from gmst:

Thanks, but I haven't used C#. I can do VB though. I also have never done any parsing before. I will see if I can get some resource on net to learn it.

But the message I am getting from various posters is that it is not something that will take too much time to learn and implement.
More...

which one? VB6 or VBA?
it's fairly simple in VB6

just drop Internet Transfer Control on form and do this:

http://msdn.microsoft.com/en-us/library/aa733648(v=vs.60).aspx

what you trying to do anyway? someone might have coded already
 
Quote from Bob111:

which one? VB6 or VBA?
it's fairly simple in VB6

just drop Internet Transfer Control on form and do this:

http://msdn.microsoft.com/en-us/library/aa733648(v=vs.60).aspx

what you trying to do anyway? someone might have coded already
More...

I mainly use vba with excel but since vb6 is not too different. I will see into your link and try to work.

Just trying to learn this skill first. Once I pick up the skill, I have a lot of ideas that I would like to test going forward. Before I can test any idea, I will need some data for it.

In the beginning, it will be mostly shooting in the dark and hopefully pick up few things in the process. Following are some of the specific ideas that I am planning to collect data from:

Basic Stuff Mostly about market internals and some news
1) http://online.wsj.com/mdc/public/page/2_3022-mfgppl-moneyflow.html
2) collect information from google/yahoo finance

Advanced Stuff This will be a long time thing.
1) collect information from twitter on a particular stock
2) Characterize this information/news on a sentimental basis (bullish/bearish)
3) Try to build some kind of regression model to see if tweets can help me in my trading.
 
Quote from gmst:

Thanks looks interesting. I have never any python before. I might just enroll.

As an aside, Coursera says all courses are free on it.

How does coursera make money?
Why do professors upload their courses on coursera, if they are not getting paid anything? I know few people will do it for a good cause. But many professors at my university were "extremely interested" in keeping their intellectual property private and wanted to monetize their IP in as many ways as possible.

Hell - one professor never gave us powerpoints rather just printed slides and gave them to us so that we couldn't pass-on the powerpoints electronically to others.
More...

http://en.wikipedia.org/wiki/Coursera
 
Quote from gmst:

I mainly use vba with excel but since vb6 is not too different. I will see into your link and try to work.

Just trying to learn this skill first. Once I pick up the skill, I have a lot of ideas that I would like to test going forward. Before I can test any idea, I will need some data for it.

In the beginning, it will be mostly shooting in the dark and hopefully pick up few things in the process. Following are some of the specific ideas that I am planning to collect data from:

Basic Stuff Mostly about market internals and some news
1) http://online.wsj.com/mdc/public/page/2_3022-mfgppl-moneyflow.html
2) collect information from google/yahoo finance

Advanced Stuff This will be a long time thing.
1) collect information from twitter on a particular stock
2) Characterize this information/news on a sentimental basis (bullish/bearish)
3) Try to build some kind of regression model to see if tweets can help me in my trading.
More...

yahoo data -not a problem,goog-if you place too many requests too fast-you going to get an error. same with yahoo's YQL
you can use this-

http://www.gummy-stuff.org/Yahoo-data.htm

as for page from wsj-you can use web query

play with this internet transfer control. it's really simple. PM me,if you need examples for VB6 for yahoo. i have some pieces here and there. but from my practice-it's sometimes easier to code from scratch,than figure out someone's code :)

remember-with this control-you gong to get a string with HTML code. same stuff that you see,if you go on any page and see page source.then you have to go thru this data and pick pieces you need.

you can use this link for excel's web querry(link on page):

http://online.wsj.com/mdc/public/npage/2_3045-mfgppl-mfxml2csv.html
 
Thanks Bob, very helpful of you. :)

Let me see what I can do. I will get back if I need some example codes from you.
 
You must log in or register to reply here.
Back
Top