It's basically a program which runs on my computer. I can't chop it up and just send you parts.
But I appreciate the offer. Really.
Connecting to API (IQ Feed) to retrieve historical/live data VS loading data from QCollector?
- Thread starter Howard
- Start date
It's basically a program which runs on my computer. I can't chop it up and just send you parts.
But I appreciate the offer. Really.
It's not clear that you are already day trading. The reason I'm being picky about it is because you sound like you just want it to be faster but haven't really decided what you're going to do with it once it's faster.
Quite often, if you really lay it out, you can solve problems much more cost effectively or avoid touching it altogether.
For example, possible solutions could be run the app in the background every 5 minutes. But we don't know that this is feasible if you already know you want it faster but without understanding exactly what you're trying to do that is faster.
One of the characteristics of good software developers is that they want to solve problems. You should take the offers of one of the guys here and do a screenshare about the problems you're having. No need to send code. Then they can advise you.
Okay. I did specifically talk about intraday updates and second updates, so I thought it would be clear that the main purpose is day trading then.
The application is already running in the background, if I want to. It's automated. The problem is that it's too slow updating data and there's also steps I have to take to make that data available to me which should be automated. Compare that with QCollector which runs in the background and updates data seamlessly.
I already spent a lot of money optimizing this. What several independent programmers have told me now is that in order to get any improvements beyond what I currently have - I should re-write the entire thing from scratch.
Part of the reason for this is because the entire thing has grown very complex over the years and it's very poorly documented. One programmer spent an entire day just understanding the how the thing was built. Thus, it's hard to do upgrades/changes. The latter also because of how it was built. I had a programmer do a modification for me easily, but he declined some other request I had because he would not want to 'mess' with that part of the application since it could botch up everything else.
One programmer said that the code is really good and that the programmer writing it clearly knows what he's doing, but that it's complex and poorly documented. Another one flat out said that the code is really bad...
Anyway, we're going off topic as well, I think?
Mostly curious if it's worth setting up direction connection to an API or if the solution with retrieving data from QCollector is perhaps good enough that it's not worth the extra trouble to do so.
I just checked Qcollector to see if I could need it. What do you need it for? I really see no reason to use it at all. You can get the data from IQfeed directly and store it in a free SQL database if you want to use it later for back-testing or even simply dump the data in a text file if you don't need to run SQL queries on it.
To give you an answer if it's worth to do a rewrite one has to know the architecture of your program and how much the delays bother you so you won't get a definitive answer here
To give you an answer if it's worth to do a rewrite one has to know the architecture of your program and how much the delays bother you so you won't get a definitive answer here
Last edited:
Okay. Cool.
I use QCollector to store 1-minute intraday OHLC data in a *.csv file. My application in turn loads this *-csv-file for further calculations.
Are you saying that I can download this data directly from IQ Feed without an additional program?
I assume this wasn't the case back in the day - hence why the QCollector was created.
The program is currently working sufficiently, while the delay is bothersome.
But I want to do a re-write for various other reasons also - including implementation of new stuff/algorithms and adding in more flexibility.
Yes you can skip Qcollector. One reason for the bottle-neck could be since the programs are sharing data via files which is a terrible idea Qcollector could be locking the file while it is writing data and your program has to wait for a spot where the file is unlocked which is dependent on the volume/number of instruments and could take a long time.
Only benefit I can see for Qcollector is a simple way to collect data for backtesting. From my understanding it knows to subscribe and unsubscribe symbol request so you don’t hit your symbol limits.
If your trading futures with IQFeed doubt you’d hit 500 symbol limit or if you add to your package you can get up to 1500, don’t think they recommend past that. So then you need your program to know how many real-time request it’s making for how many symbols, then in subscribe. Can get messy.
I’d recommend learning some language yourself, especially if you plan on considering any automation of sending orders. Theirs a lot of issues that could pop up and if you have no clue how the program reacts to data connection losses. Bad ticks, flash crashes. No one one is going to make themselves responsible for a run away algo.
If your trading futures with IQFeed doubt you’d hit 500 symbol limit or if you add to your package you can get up to 1500, don’t think they recommend past that. So then you need your program to know how many real-time request it’s making for how many symbols, then in subscribe. Can get messy.
I’d recommend learning some language yourself, especially if you plan on considering any automation of sending orders. Theirs a lot of issues that could pop up and if you have no clue how the program reacts to data connection losses. Bad ticks, flash crashes. No one one is going to make themselves responsible for a run away algo.
Okay. I would need to look into how I can do that. Can IQ Feed store their data in *.csv format? As of now, my application only loads *.csv files, I think. Might be that I do it like this for now, but consider other solutions for a future upgrade.
Still, the same question arises:
Connect directly to API and load/rip off data directly or get the data from a file I download locally to my computer and skip the API Connection?
You're right about what you're thinking: My application does have to wait for QCollector to update the file. However, that happens very fast and is not a problem at all for me. I've tried doing the same kind of update after-hours/manually and it's the same.
Best regards,
Howard