How about using selenium for the web-crawling in IB webpage?

oly · Mar 23, 2016

It's what I am using. I read key statistics pages from Yahoo and pull data out of it. The bs4 bit ends up looking like this:

return soup.find("span", {"class": "rtq_exch"}).string

This grabs the span element where class is rtq_exch and gives me the contents.

I don't submit forms via python, though. It would be simple enough to make a post request and "send a form" that way. The downside is if they add some new form fields you have to tweak. Though, you probably have to tweak anyway if that kind of change happens.

An example for order sending with IB (though as you say you have more than one, but maybe this would be easier at least for the IB bit).

https://www.quantstart.com/articles...he-Interactive-Brokers-API-to-Automate-Trades

jk90029 · Mar 23, 2016

oly said:
An example for order sending with IB (though as you say you have more than one, but maybe this would be easier at least for the IB bit).

https://www.quantstart.com/articles...he-Interactive-Brokers-API-to-Automate-Trades

I did not buy this online book yet. If I buy this book, is it possible to use Python to

1) get real-time data from IB server
2) get holding stocks and ordered info
3) send new order to IB server?

Please tell me if you have read this book.

For IB only, this one book might be helpful. However, thing changes all the time and we may need to change IB to other broker or other countires.

oly · Mar 23, 2016

That link is to a page showing python code to submit orders to IB. It's not the book from earlier.

All he does there is send orders to IB. Unfortunately, that is all I do too (I am too cheap to pay for IB data). I'm 99% sure you can also connect and get price updates for a list of tickers but I haven't done that myself. Same goes for looking at your existing orders or positions.

jk90029 · Mar 23, 2016

oly said:
That link is to a page showing python code to submit orders to IB. It's not the book from earlier.

All he does there is send orders to IB. Unfortunately, that is all I do too (I am too cheap to pay for IB data). I'm 99% sure you can also connect and get price updates for a list of tickers but I haven't done that myself. Same goes for looking at your existing orders or positions.

IC. By help of the book, only 3) can be done.

By use selenium 1) and 2) were successsful to other broker (possibly many brokers), to scrap the webpage.
My helper with knowledge of C# and selenium may work 3) next week.

Best scenario is to program to trade to many brokers in many countries, since selenium scrap is one setup to different web page.

Worst scenario happens when brokers change web page very frequently so that one need to change the code each time.

oly · Mar 23, 2016

This might help

http://stackoverflow.com/questions/...access-and-store-a-variable-from-api-response

d08 · Mar 23, 2016

jk90029 said:
Agree, but there is some advance in selenium, compared with 90's technology such as PERL, if I heard correctly.

For example, even if the webpage is changed a little bit by webmaster, selenium can detect and adjust it by use of DOM.
Back in 90's, there was no concept of DOM.

perl isn't 90s technology as it's still used quite a bit by Amazon for example. I scrape using perl and it works fine. There are methods/modules to grab fields in html tables not unlike Selenium's DOM locators. I think you're making DOM locators as magic but you'll soon discover they can err as well.
But like oly said, this discussion isn't going anywhere.

jk90029 · Mar 23, 2016

d08 said:
perl isn't 90s technology as it's still used quite a bit by Amazon for example. I scrape using perl and it works fine. There are methods/modules to grab fields in html tables not unlike Selenium's DOM locators. I think you're making DOM locators as magic but you'll soon discover they can err as well.
But like oly said, this discussion isn't going anywhere.

If you are successful to use PERL to do the three things mentioned above, I can trade C# code for the three thing by scrapping, in exchange of your PERL code.

Did you do it for the IB only? I like to see code WITHOUT use of DOM.

Of course, I agree DOM is not magic. But it is a new technology as well as new HTML5, if I heard correctly.
(a lot different from old version of HTML1)

d08 · Mar 24, 2016

I'm not using it to interact with IB or a broker, I'm using perl to scrape data from various websites. To interact with IB I use IbPy (Python wrapper for IB's native Java API) as was suggested in this thread earlier.