Trying to get a snapshot of realtime pricing of 600+ equities

Quote from dholliday:

Actually IQ Feed has an 1800 symbol limit, you just pay more.
I have my limit at 1300 symbols and I regularly watch 1000.
If you need more than 1800 they have another (more expensive) product.

-David
More...

well here we go. but since OP is needed few requests for only 600-1000 tickers into EOD-he can use just regular subscription.
having bigger limit probably not going to help him,cause you have to subscribe,wait for response,unsubscribe etc. takes time to process all this. but like i said-few reuests might be ok,but if one makes too many-they are not going to be happy. cause the feed is no designed for it(at least-that how it was back in a day). it's designed for stream.not in and out.
 
Quote from braincell:

Oh... ok that's interesting. I might pull the data for a few months then just to see if it's any good. It's incredibly complicated to get daily OHLC data very quickly with a lot of the data providers, even the expensive ones, so this might be useful if nothing else as an alternative data check / data source. Thanks.

Edit: note that the historical data i tested a while ago showed quite a few differences when compared to some paid sources (IB included). It's often very hard to say which one is correct, or properly cleaned though.
More...

i do use yahoo EOD data for cross reference with other providers every single day. there is rarely any differences(talking about close price in particular). i wouldn't use yahoo as a source of historical data,cause i'm not sure how they handle splits,dividends etc.



but OP is looking real time data,not EOD OHLC and again-yahoo is very good choice(if you need real time OHLC and volume and not abusing it)
one should be specially careful with YQL,cause they will ban you,if you placing too many requests too fast
 
Quote from Bob111:

well here we go. but since OP is needed few requests for only 600-1000 tickers into EOD-he can use just regular subscription.
having bigger limit probably not going to help him,cause you have to subscribe,wait for response,unsubscribe etc. takes time to process all this. but like i said-few reuests might be ok,but if one makes too many-they are not going to be happy. cause the feed is no designed for it(at least-that how it was back in a day). it's designed for stream.not in and out.
More...

You are of course correct, IQFeed is not meant for polling historical data during market hours. I was just pointing out that you can stream tick data for up to 1800 symbols. Checking the documentation, it appears the tick stream includes fields for OHLCV. Since I don't use these fields I don't know if they are actually populated.

I don't want to infer that I think this is the best solution for the OP, only that it is a possible solution. I don't have experience with any of the free solutions like Yahoo, but obviously free and easy to implement is probably the best solution for the OP.
I, and I'm sure others, appreciate your sharing your experience.
Thanks,
-David
 
Quote from dholliday:

You are of course correct, IQFeed is not meant for polling historical data during market hours. I was just pointing out that you can stream tick data for up to 1800 symbols. Checking the documentation, it appears the tick stream includes fields for OHLCV. Since I don't use these fields I don't know if they are actually populated.

I don't want to infer that I think this is the best solution for the OP, only that it is a possible solution. I don't have experience with any of the free solutions like Yahoo, but obviously free and easy to implement is probably the best solution for the OP.
I, and I'm sure others, appreciate your sharing your experience.
Thanks,
-David
More...

1800 tickers streaming unfiltered data-it's a lot of data to digest :p

i've tried 200-300(with few liquid ones) and my computer starts choking on it. and i wasn't processing anything , i just inserting the data into a tables with ticker name.
i'm having hard time to see how one can do anything else with it,if you subscribe for 100's of tickers at once and data is flowing into single application.
can anyone share their examples of how they handle streaming data from let say IQ feed and manipulate it on the fly?

with IB-it's easy,cause each ticker has unique ID and you can assign prices from stream into array without any loops. but in case of IQ feed-if you have array and need to assign prices in it as the data flow-you have to go through the loop,to find the ticker first and then put the data from stream into appropriate fields. i'm wrong?
is there is any other way?
or maybe there something new in their client and they now handle the data differently? i tried IQ long long time ago with really old client version.

Thank you!
 
Quote from Bob111:

1800 tickers streaming unfiltered data-it's a lot of data to digest :p

i've tried 200-300(with few liquid ones) and my computer starts choking on it. and i wasn't processing anything , i just inserting the data into a tables with ticker name.
i'm having hard time to see how one can do anything else with it,if you subscribe for 100's of tickers at once and data is flowing into single application.
can anyone share their examples of how they handle streaming data from let say IQ feed and manipulate it on the fly?

with IB-it's easy,cause each ticker has unique ID and you can assign prices from stream into array without any loops. but in case of IQ feed-if you have array and need to assign prices in it as the data flow-you have to go through the loop,to find the ticker first and then put the data from stream into appropriate fields. i'm wrong?
is there is any other way?
or maybe there something new in their client and they now handle the data differently? i tried IQ long long time ago with really old client version.

Thank you!
More...

Bob- I agree, it would be more efficient if IQFeed used the same reqId scheme as IB. Interestingly, that option is available when making historical requests, just not in the live stream. That said, I trade 1000 symbols (which usually includes largest ETFs and SP500 stocks) and have no problem at all keeping up with the live feed using Java and a Symbol-to-Security hashmap for directing a tick to the correct array. CPU never breaks a sweat.

Edit: I should add that my app is multithreaded with 5 - 10 socket connections to IQConnect (depending on which machine I'm running it on).
 
Quote from jtrader33:

Bob- I agree, it would be more efficient if IQFeed used the same reqId scheme as IB. Interestingly, that option is available when making historical requests, just not in the live stream. That said, I trade 1000 symbols (which usually includes largest ETFs and SP500 stocks) and have no problem at all keeping up with the live feed using Java and a Symbol-to-Security hashmap for directing a tick to the correct array. CPU never breaks a sweat.

Edit: I should add that my app is multithreaded with 5 - 10 socket connections to IQConnect (depending on which machine I'm running it on).
More...

yes.that's what tried back then-basically running bunch of instances of same application that are connected to their client via sockets.
hashmap!mmm! i have to try it (i'm not a programming pro and know just few basic things)
 
Quote from Bob111:

1800 tickers streaming unfiltered data-it's a lot of data to digest :p

i've tried 200-300(with few liquid ones) and my computer starts choking on it. and i wasn't processing anything , i just inserting the data into a tables with ticker name.
i'm having hard time to see how one can do anything else with it,if you subscribe for 100's of tickers at once and data is flowing into single application.
can anyone share their examples of how they handle streaming data from let say IQ feed and manipulate it on the fly?

with IB-it's easy,cause each ticker has unique ID and you can assign prices from stream into array without any loops. but in case of IQ feed-if you have array and need to assign prices in it as the data flow-you have to go through the loop,to find the ticker first and then put the data from stream into appropriate fields. i'm wrong?
is there is any other way?
or maybe there something new in their client and they now handle the data differently? i tried IQ long long time ago with really old client version.

Thank you!
More...

IQFeed comes in as a single stream.
I look at each line and determine what kind of message it is. If it is a new tick message I get the symbol out of the string and pass the message to the queue for that symbol/system.
I have a thread pool that takes the messages off the queues and passes the data to that systems compute engine.

On my very old dual-core lap top CPU usage got up to around 30% watching every tick for 1000 symbols and doing lots of calculations. Half of that was the IQFeed client component. I now run on a three year old 1.7ghz i7 laptop. I just checked task manager. I'm running live, watching about 400 symbols. CPU usage is 2-5 %.
-David
 
David Thank you for your response! i'm no expert in this and i'm using old VB6(the only thing i know on hobby level)
when i tried IQ real time streaming-yes,it give you a single stream

here is what i did-
first -create array of tickers and datafields you need. for example-400 tickers-then array will be 400 items

then i use loop to subscribe to a data from lbound to ubound-etc

now data starts streaming through and i got basically a string from IQ,containing different fields such as ticker,last price,time and so on

in order to assign this data to appropriate place in array i have to go trough the loop again and find in array item,where ticker = ticker
in streaming data from IQ. it's unique identifier i use

so each time i got string of data from IQ-i have go trough the loop to find appropriate spot for this ticker in array and assign whatever data i got into it

this is where i stuck,because there is so much data flowing that i can't keep up with it (also on dual core with plenty of memory and 10K HD)
my question is-is there is another way around or what i'm doing wrong or what other developers do to assign data into array(or any other object )and work with it?
i'm reading about hashtables right now,but i see no difference between them and indexed array. looks like you still have to go through the loop to find matching ticker
can anyone clarify this part for me? or this is the only way to work with IQ Streaming quotes?

Thank you!

found example here-

http://www.freevbcode.com/ShowCode.asp?ID=8581

but it makes things even more confusing. specially this sentence:
---
Hash tables are used to store data in them, like arrays, but the difference is this: to find an item in an array, it has an order of O(n), but in hash tables, the order is O(1).--
 
Quote from Bob111:

David Thank you for your response! i'm no expert in this and i'm using old VB6(the only thing i know on hobby level)
when i tried IQ real time streaming-yes,it give you a single stream

here is what i did-
first -create array of tickers and datafields you need. for example-400 tickers-then array will be 400 items

then i use loop to subscribe to a data from lbound to ubound-etc

now data starts streaming through and i got basically a string from IQ,containing different fields such as ticker,last price,time and so on

in order to assign this data to appropriate place in array i have to go trough the loop again and find in array item,where ticker = ticker
in streaming data from IQ. it's unique identifier i use

so each time i got string of data from IQ-i have go trough the loop to find appropriate spot for this ticker in array and assign whatever data i got into it

this is where i stuck,because there is so much data flowing that i can't keep up with it (also on dual core with plenty of memory and 10K HD)
my question is-is there is another way around or what i'm doing wrong or what other developers do to assign data into array(or any other object )and work with it?
i'm reading about hashtables right now,but i see no difference between them and indexed array. looks like you still have to go through the loop to find matching ticker
can anyone clarify this part for me? or this is the only way to work with IQ Streaming quotes?

Thank you!

found example here-

http://www.freevbcode.com/ShowCode.asp?ID=8581

but it makes things even more confusing. specially this sentence:
---
Hash tables are used to store data in them, like arrays, but the difference is this: to find an item in an array, it has an order of O(n), but in hash tables, the order is O(1).--
More...

O(n) should be read as "On the Order of n" meaning that the time it takes to preform the lookup increases linearly with the number of items in the array. It there are 400 items in the array, it takes up to 400 times as long to find the item you are looking for if you are looping through every item.
A Hash table takes the same amount of time to look up an item whether there is one item in the table or 400 items in the table. O(1). It's not important to understand how it does this.

A Hash table should be thought of as a a table of key value pairs with the symbol as the key and the value being where you want to send the information. For example; I put a reference to the queue that I want the data to go into.

Sorry I'm not familiar with VB.

Hope this helps,
-David
 
it does. Thank you for an explanation! basically it's like a table in database and when you have to update certain fields-you just update one that you need,based on unique identifiers(such as primary key,id,ticker,whatever). but unlike DB it does it on the fly,in RAM.
and like you said-in array-you have to go though the loop, until you find your ticker. the bigger the list-the longer it will take.
i get it now. it's looks like it's only available in .net,not in old VB6

Thank you!
 
You must log in or register to reply here.
Back
Top