Customizing the sample Python IBAPI code provided in the testbed

[Aron]

Was mucking around last time. Wonder if this helps. Under strategy class, define the method. Then below, add a prefix of req (check against official API page). I'm unfamiliar with order book depth stuff; you have to explore.

# Import necessary libraries
from ibapi.client import EClient
from ibapi.wrapper import EWrapper
from ibapi.contract import Contract
from threading import Timer
import pandas as pd
import time

# Define strategy class - inherits from EClient and EWrapper
class Strategy(EClient, EWrapper):

# Initialize the class - and inherited classes
def __init__(self):
EClient.__init__(self, self)
self.df = pd.DataFrame(columns=['Time', 'Open', 'Close'])

# Receive historical bars from TWS
def historicalData(self, reqId, bar):
dictionary = {'Time':bar.date,'Open': bar.open, 'Close': bar.close}
self.df = self.df.append(dictionary, ignore_index=True)
print(f'Time: {bar.date}, Open: {bar.open}, Close: {bar.close}')

# Display a message once historical data is retreived
def historicalDataEnd(self, reqId, start, end):
print('\nHistorical Data Retrieved\n')
print(self.df.head())
# elf.df.to_csv('Historical_data.csv')

# -------------------------x-----------------------x---------------------------

# Create object of the strategy class
app = Strategy()

# Connect strategy to IB TWS
app.connect(host='127.0.0.1', port=7497, clientId=2)
print('Is application connected to IB TWS:', app.isConnected())

# Wait for sometime to connect to the server
time.sleep(1)

# Create object for contract
eurusd_contract = Contract()
eurusd_contract.symbol = 'EUR'
eurusd_contract.currency = 'USD'
eurusd_contract.secType = 'CASH'
eurusd_contract.exchange = 'IDEALPRO'

# Request for historical data
app.reqHistoricalData(reqId=33,
contract=eurusd_contract,
endDateTime='',
durationStr='300 S',
barSizeSetting='1 secs',
whatToShow='MIDPOINT',
useRTH=0,
formatDate=1,
keepUpToDate=False,
chartOptions=[])

# Invoke another thread that will disconnect the strategy from TWS
Timer(10, app.disconnect).start()

# Run the strategy
app.run()

Works like a charm !! Awesome !! I used this to modify the original program.py... many more questions on the way !! Thanks again !

 

[sef88]

Works like a charm !! Awesome !! I used this to modify the original program.py... many more questions on the way !! Thanks again !

Have fun!

Although debugging and integrating error handling stuff is a nightmare to handle in TWS API async framework.

 

[Aron]

Have fun!

Although debugging and integrating error handling stuff is a nightmare to handle in TWS API async framework.

That's absolutely true, it has slowed me down significantly. The next big challenge: getting over the pacing violations, now I have 150 Instruments for which I need 11 bar sizes each, for a period of 5 years. So given the pacing violations:

• Making identical historical data requests within 15 seconds.
• Making six or more historical data requests for the same Contract, Exchange and Tick Type within two seconds.
• Making more than 60 requests within any ten minute period.

How do I plan this, doing this manually will be a nightmare, considering all contracts are defined and resolved, how do we use the timer function iteratively.

In addition to this I will need to download the High resolution historical ticks with the following limitation: 1000 ticks per request, which is roughly 5 mins of historical ticks (given that there are more than one ticks per second).

Thanks in advance. I am learning a lot!

 

[Quiet1]

Seems more like a side project than a plan to actually build some proper data infrastructure? Sounds like you could save weeks of work (at least) by using something like IQfeed.

 

[Aron]

I see the flaw in this approach, specially the amount of time/Effort it takes, what would be your suggestion for a proper data infrastructure given that I would need APAC exchanges such as NSE and BSE (level II/III) as well, for which there are barely any vendors coverage. Thanks !!

 

[HobbyTrading]

That's absolutely true, it has slowed me down significantly. The next big challenge: getting over the pacing violations, now I have 150 Instruments for which I need 11 bar sizes each, for a period of 5 years. So given the pacing violations:

• Making identical historical data requests within 15 seconds.
• Making six or more historical data requests for the same Contract, Exchange and Tick Type within two seconds.
• Making more than 60 requests within any ten minute period.

How do I plan this, doing this manually will be a nightmare, considering all contracts are defined and resolved, how do we use the timer function iteratively.

In addition to this I will need to download the High resolution historical ticks with the following limitation: 1000 ticks per request, which is roughly 5 mins of historical ticks (given that there are more than one ticks per second).

Thanks in advance. I am learning a lot!

Luckily you only need to download all this historical data once. Once you have it on your computer you can add newer data to it, resulting in less requests to IB's data servers.
My guess is that you are not in a hurry to receive the old data. If you were in a hurry, you would have started this work multiple months ago. So you need to write software to submit only 60 requests per batch. Once the batch has been completed you wait until the 10 minute period has finished before you submit the next batch of 60 requests.

 

[sef88]

Luckily you only need to download all this historical data once. Once you have it on your computer you can add newer data to it, resulting in less requests to IB's data servers.
My guess is that you are not in a hurry to receive the old data. If you were in a hurry, you would have started this work multiple months ago. So you need to write software to submit only 60 requests per batch. Once the batch has been completed you wait until the 10 minute period has finished before you submit the next batch of 60 requests.

For stocks, you have to account for splits and dividends. That's another set of problem.

 

[HobbyTrading]

For stocks, you have to account for splits and dividends. That's another set of problem.

Indeed, a whole other set of problem. And I don't think that IB provides any support on this topic, so another source of data may be necessary.