Data Scraping Barchart.com

M

MoneyMatthew

I wrote some software myself to data scrape intraday data from barchart.com. On November 1st data will no longer be contained in the html. There is a totally new build of the site coming soon.

I wrote a program to scrape the new site but it is slow as hell and crude to say the least. I guess I should just be happy that I have a solution.

I was wondering if anyone could tell me a better way to scrape https://beta.barchart.com ????

All of my methods were proprietary, I have no knowledge of the more mainstream ways to data scrape with Beautiful Soup, Selenium,Mapit.PY ect..ect..

I am just curious if any of you wizards out there can pull data from it? Thanks.
 
ibapilessons said:
The best for this is Nadaq... I think.

But, yes I think is better if you pay for it. Interactive Brokers data is not so expensive. You can use this method:

Reqmktdata method: https://www.interactivebrokers.com/en/software/api/apiguide/java/reqmktdata.htm

More...

Thanks for the advise everyone. Buying data is the best solution but I am so used to getting data for free :-) I need to update my tech skills. I should be using APIs instead of building everything from scratch. I was basically just treating the website as one big "string" and parsing the unstructured data from there.

BTW...getting EOD data for free is no big deal. I was scraping intraday one minute OHLC data. I think the one minute data is harder to get for free.
 
Yeah you are right. I was not hitting the site with HTTP requests non-stop. I was grabbing everything in a single pass. I don't feel like it was "abuse" but I understand that there is a fine line between high usage and abuse.

I guess any form of automation is a form of abuse. They intend for you to view ads ect...
 
You must log in or register to reply here.
Back
Top