Connecting to API (IQ Feed) to retrieve historical/live data VS loading data from QCollector?

[Howard]

Hi all,

In summary, I have a custom made trading application which loads 1-minute data from a *.csv file generated by QCollector. QCollector in turn extracts data from IQFeed.

I want to eventually re-design/re-build my application from scratch and one of the things I need to adress is how I retrieve and load data.

The way things work now ain't half bad to be honest. I update my database/software by EOD, but I can also update it live intraday. In short, my custom application recognizes when the *.csv file is updated and responds automatically by instantly updating my database.

I can set Q-Collector to load data on second-intervals which is more than fast enough for my purposes. Obviously, I'm not doing low-latency HFT strategies, nor do I aim to, but increasing speed is always of interest. I don't see myself using tick data at any point, but maybe I could increase granularity and move towards second bars.

The problem with my set-up is that my custom application is too slow to load/reload. In other words, the bottleneck is my application, not QCollector.

Still, I'm wondering if now that I'm re-designing my application if I should drop QCollector altogether and connect directly to an API instead? Or is using QCollector (or similar) to extract data an acceptable solution?

Does anyone have any advice to offer on this? Has anyone done this themselves?

Thanks in advance.

Howard

 

[nooby_mcnoob]

Hi all,

In summary, I have a custom made trading application which loads 1-minute data from a *.csv file generated by QCollector. QCollector in turn extracts data from IQFeed.

I want to eventually re-design/re-build my application from scratch and one of the things I need to adress is how I retrieve and load data.

The way things work now ain't half bad to be honest. I update my database/software by EOD, but I can also update it live intraday. In short, my custom application recognizes when the *.csv file is updated and responds automatically by instantly updating my database.

I can set Q-Collector to load data on second-intervals which is more than fast enough for my purposes. Obviously, I'm not doing low-latency HFT strategies, nor do I aim to, but increasing speed is always of interest. I don't see myself using tick data at any point, but maybe I could increase granularity and move towards second bars.

The problem with my set-up is that my custom application is too slow to load/reload. In other words, the bottleneck is my application, not QCollector.

Still, I'm wondering if now that I'm re-designing my application if I should drop QCollector altogether and connect directly to an API instead? Or is using QCollector (or similar) to extract data an acceptable solution?

Does anyone have any advice to offer on this? Has anyone done this themselves?

Thanks in advance.

Howard

What is the problem if it is too slow, what negative issue do you experience?

 

[zwangerz]

Direct API connection would be the cleanest, fastest and least error prone solution. Last time I checked IQfeed wanted like 200$ to get their API docs lol. I went with IB instead and never regretted it. It seems odd that your app is too slow to load. Recognizing a file change and loading/processing an ASCII file should take 2 seconds max. if the size is <50MB on a standard pc.

 

[Howard]

What is the problem if it is too slow, what negative issue do you experience?

The problem is that I'm using the application in a live, moving market. Speed is of the essence. At EOD it's less of a problem, although I wouldn't mind speeding things up there either.

Also, my application seems to hog my system resources. It's simply not very efficient and I suspect the code is looping through a LOT of steps that would/should not be necessary on subsequent calculations.

So, my application is the bottleneck for sure. Question is if it's worth connecting directly to an API when I'm making changes.

Of course, I can re-design the entire thing and leave data connection for a later upgrade.

 

[Howard]

Direct API connection would be the cleanest, fastest and least error prone solution. Last time I checked IQfeed wanted like 200$ to get their API docs lol. I went with IB instead and never regretted it. It seems odd that your app is too slow to load. Recognizing a file change and loading/processing an ASCII file should take 2 seconds max. if the size is <50MB on a standard pc.

Thanks.

Oh, I wasn't aware they charged that. Anyway, 200$ isn't a big deal.

Well, QCollector seems to upload data instantly. But it's the calculation steps inside my application which is the problem. For one, I suspect it's re-calculating a lot of stuff on subsequent updates that should not be done.

What are you doing with IB and their API? Running automated strategies or simply extracting data from their feed?

 

[zwangerz]

If you need live data then API connection is the way to go no question. The code should be fairly simple once you are familiar with the API, probably less complex than what you have now. I ditched IQFeed because of the cost (around 150$ vs 15$/month at IB for futures and options data)

 

[zwangerz]

Thanks.

Oh, I wasn't aware they charged that. Anyway, 200$ isn't a big deal.

Well, QCollector seems to upload data instantly. But it's the calculation steps inside my application which is the problem. For one, I suspect it's re-calculating a lot of stuff on subsequent updates that should not be done.

What are you doing with IB and their API? Running automated strategies or simply extracting data from their feed?

This was 2 years ago maybe they dropped the charge in the meantime.
I retrieve live tick data from IB and do some math(around 100 lines of code) for every data point I receive. It's only a couple of instruments but with heavy volume and my app is coded in Java and my CPU load is around 0.1% on a 8 year old PC. The system then places around 10 orders a day without manual intervention.

 

[nooby_mcnoob]

What is the problem if it is too slow, what negative issue do you experience?

The problem is that I'm using the application in a live, moving market. Speed is of the essence.

This is not a correct answer to the question. What are you unable to do that you need to do which causes problems for you?

From the sounds of it, you could probably hire someone to optimize your code and be done with it.

 

[zwangerz]

I've got some time to kill, you can send me the part you suspect being the bottleneck and I'll look over it. Make sure to delete all the parts you are uncomfortable sharing

 

[Howard]

This is not a correct answer to the question. What are you unable to do that you need to do which causes problems for you?

Can it be more obvious?

I already said speed is of essence. If you ever day traded, you know that it's key to be able to act fast on new information.

Obviously, if my way of retrieving new information is slower than it can be, I'm at a disadvantage.

From the sounds of it, you could probably hire someone to optimize your code and be done with it.

Already did that and went from a nearly useless application to something that's actually workable and fairly stable.

But the programmers I've used say I've maxed out performance as it is and that I'll need to start from scratch if I want any real improvements beyond this.