Advice on IQFeed + Java set up

[jtrader33]

I currently have my Java ATS set up to receive live data from IQ's socket and then distribute it to various LinkedBlockingQueues, each of which has it's own consumer thread. Each of ~1000 symbols is assigned to one of the queues. Doing it this way (and not using a thread pool) simplified a lot of concurrency headaches. I realize this has a cost in terms of efficiency but I'm okay with that for now.

That said, I'm considering switching from this approach to one where I set up multiple sockets (each with their own reader thread) instead of using queues, and having the IQFeed app do the work of marshalling the ticks to the correct socket. Is there an obvious answer to which approach would have better performance?

The other motivation for doing this is that I'm assuming if IQFeed moves to a multithreaded live feed at some point, it will be implemented the same way as the historical currently is with multiple sockets concurrently fed.

 

[2rosy]

if i understand correctly.

1) you currently have a queue for each stock
2) you want to change to have a socket connection for each stock

i would group the stocks into categories and broadcast those messages over either messaging layer or multicast. The consumers can grab what they want. That way you have one connection to iqfeed and many consumers

 

[jtrader33]

Quote from 2rosy:

if i understand correctly.

1) you currently have a queue for each stock
2) you want to change to have a socket connection for each stock

i would group the stocks into categories and broadcast those messages over either messaging layer or multicast. The consumers can grab what they want. That way you have one connection to iqfeed and many consumers

No, sorry...say I have 1000 symbols...currently I set up 5 queues and assign 200 stocks to each. So instead of 5 queue / consumer pairs, I would be switching to 5 sockets / reader threads. That make sense?

 

[2rosy]

Quote from bs2167:

No, sorry...say I have 1000 symbols...currently I set up 5 queues and assign 200 stocks to each. So instead of 5 queue / consumer pairs, I would be switching to 5 sockets / reader threads. That make sense?

dont you still need a queue (buffer) in the reader thread

 

[jtrader33]

I believe the ticks will queue at the individual socket level. Here is my understanding of the way IQFeed works:

There is a single threaded IQFeed app running on the local machine through which all ticks are processed. The app then routes ticks to one or more sockets which are read by my ATS. Currently, I have it set up to send everything to one socket where a single socket reader thread in my ATS grabs the ticks and then further reroutes them to the appropriate queues...from there each queue has it's own consumer thread. However, if I have the IQFeed app send the ticks to different sockets, I can forgo the reroute to queues on my side and just have the socket reader threads carry out bar building / strategy execution / etc..

My guess is that having the IQFeed app do the 'tick routing' would probably be more efficient that doing it myself...but honestly I'm new to Java and don't know enough about programming in general to make that judgment confidently...thus the question.