[IB] web-crawling in IB webtrader

[jk90029]

Dear-

When I login IB webtrader in the chrome browser, I can see main menus

1) balance
2) Market value
3) Position
4) Portfolio
5) Order Management

as you can see also in your screen.

Now, is it possible to download those 5 data into corresponding 5 separate csv files in my local PC? For example IB1.csv , ... , IB5.csv

I heard webcrawling does work for me automatically with some programming, such as selenium.
If you do not know difference between crawling and scraping, you are a beginners like me.

PS) Probably IB API can does it too. But the above might be easier than API programming since API's are different in every brokers.

PS2) The aboves are for downloading from webpage (already seen in OUR 2 EYES) to local machine.
If it can be done, probably uploading, such as delivering orders to IB server, might be possible WITHOUT use of API.

 

[2rosy]

there's an API for a reason. It allows you to do all this easier than implementing work arounds. Just use the api

 

[jk90029]

Before attempting (easy and automatic) crawling as above, can we download the five files into csv in local machine, MANUALLY by clinking right-button in our mouse?

 

[jk90029]

there's an API for a reason. It allows you to do all this easier than implementing work arounds. Just use the api

Good point.
However, APIs are all different in every brokers so that we should study every individual API whenever we change broker.

However, web is pretty much standard all over the world and it will last next 100 years.
Therefore it is easy to change broker, without addicting ONE BROKER long time.

Fortunately, most broker provides platform such as IB TWS, as well as webtrader.

Is there any broker (in the world!!!) which does NOT provide web platform?

 

[jk90029]

Furthermore, studying individual API takes lot of my time.

Also, if I find helper to save my time, usually it costs four digits of valuable my dollars.
Please tell me now, if anyone gladly help me API work, for three digits of my dollars.

There are a few books on web crawling, in python, http://www.amazon.com/s/ref=nb_sb_n...g+in+python&rh=i:aps,k:web+crawling+in+python

 

[globalarbtrader]

I used python beautiful soup to scrape the trading and position management reports from IB

https://github.com/robcarver17/python-uk-trading-tax-calculator/blob/master/shredIBfiles.pyhttps://github.com/robcarver17/python-uk-trading-tax-calculator/blob/master/shredIBfiles.py

So it's possible. But what are you planning to do with the .csv files - this might not make sense?

GAT

 

[jk90029]

I used python beautiful soup to scrape the trading and position management reports from IB

https://github.com/robcarver17/python-uk-trading-tax-calculator/blob/master/shredIBfiles.py

So it's possible. But what are you planning to do with the .csv files - this might not make sense?

GAT

I like to fix my trading tool in ONE format, since I have two different accounts in different country.
Instead of adjusting each different layout that is ALMOST IMPOSSIBLE, hope to use my own SINGLE trading setup.

1) download a few screens (five as above) into csv file in my local PC.
2) design my down program, to make order matrix with 100 rows.
3) deliver the 100 rows of order, to server automatically by ONE CLICK

Step 2 requires thousand of line to program which is not an issue here.
Step 1 and step3 are to be disussed here, by use of web-crawling, hopefully instead of API.

 

[globalarbtrader]

I like to fix my trading tool in ONE format, since I have two different accounts in different country.
Instead of adjusting each different layout that is ALMOST IMPOSSIBLE, hope to use my own SINGLE trading setup.

1) download a few screens (five as above) into csv file in my local PC.
2) design my down program, to make order matrix with 100 rows.
3) deliver the 100 rows of order, to server automatically by ONE CLICK

Step 2 requires thousand of line to program which is not an issue here.
Step 1 and step3 are to be disussed here, by use of web-crawling, hopefully instead of API.

You have to use the API anyway to deliver your orders for (3). So why not use it for (1)?

If you have to write 1000's of lines of code for (2) why not write a few hundred to get the API working? I've tried to help you with this before I seem to remember...

GAT

 

[jk90029]

You have to use the API anyway to deliver your orders for (3). So why not use it for (1)?

If you have to write 1000's of lines of code for (2) why not write a few hundred to get the API working? I've tried to help you with this before I seem to remember...

GAT

Still not quite sure to understand API for the (1).
Why broker does NOT explain to a customer like me? Hope them to clearly show their API code in R/Python.
Sometimes customer RUN away forever, for lack of API understood.

BTW, are you SURE that (3) can be implemented by API, not by web?
If we can download web contents in any website, such as yahoo prices and IB web, then it might be possible to upload my order TO THE WEBTRADER too.

Furthermore, someone (with API experience) have offered me a help with exchange of thousand of my valuable dollars and he refused my counter-offer of hundreds of dollars, as stated above.
Please note that API is ONLY for a particular broker like IB, but web is (standard) for all the brokers in the world.

What a great advantage, if I change broker many times in the future, with NO MORE HARD WORK !!!!

 

[globalarbtrader]

Still not quite sure to understand API for the (1).
Why broker does NOT explain to a customer like me? Hope them to clearly show their API code in R/Python. Sometimes customer run away forever, for lack of API provided.

BTW, are you sure that (3) can be implemented by API, not by web?
If we can download web contents in any website, such as yahoo prices, then it might be possible to upload my order TO THE WEBTRADER.

Furthermore, someone (with API experience) offered me a help with exchange of thousand of my valuable dollars and he refused my offer of hundreds of dollars, as stated above.
Please note that API is ONLY for a particular broker like IB, but web is for all the brokers in the world.

What a great advantage, if I change broker many times in the future, with NO MORE HARD WORK !!!!

Speaking from experience, it's much harder to try and communicate with brokers (or indeed any organisation) through web interfaces than through provided API's.

So for example to scrape the positions from IB activity reports is much more involved than making a single API call. Also it's not a supported method. If a broker makes a slight change to their web interface you are screwed. I speak from bitter experience of writing data scrapers - days writing ugly code that worked for about a month before a website upgrade killed it. If they make a slight change to their API they will ensure it's kept backwardly compatible.

Trying to submit orders via the webtrader will be even harder. Quite apart from the fact that IB don't want you to do this, they want you to use the API, and will make it very very hard to do it any other way.

If you change brokers, it will be a lot quicker to write new API code (assuming it's properly isolated) than to write new web scrapers and order submitters.

Seriously, this is insane.

GAT