Need to handle websocket data for 10,000 symbols.

H

hardik dhankecha

Hi,
here I have add some points which I need to solve for move ahead in development. I am using Polygon.io for get data feed.

Point 1 :
I have 10,000 symbols of my selection. Now I need to get live records of those symbols only. When I am connecting to websocket using channel.* then it's providing me all symbols data without any issue. But when I make query string for fetch live data for 10,000 symbols then it's giving me message for connection close. Also when I make query string for 4500 symbols then it's working fine and websocket run smoothly.
So, how can I fetch live data for 10,000 symbols in websocket ? in a single query string.

Point 2 :
When I call any api for get historical data then it's taking 3 to 5 seconds for response. Which is very high for development. I am working for strategies of live stock market and for that I need to call historical data API for 10,000 symbols on regular basis and if single API call takes 5 seconds then for 10,000 symbols it will take very long time.
So is there any way to make api call response very fast ? my expectation is nearly 500 milliseconds.

so please check this points and if possible then guide me with proper solution so it can help me in development. I am using c# for development.
 
hardik dhankecha said:
So, how can I fetch live data for 10,000 symbols in websocket ?
More...
I am not using them but generally speaking:
A) Don't request all at once ... Send requests 1000 at a time with 1 second delay in between.
B) if they allow, use multiple sockets where each is responsible for a range (A-L, M-Z)
All initialization needs to be done prior to market open.
 
hardik dhankecha said:
Point 1 :
I have 10,000 symbols of my selection. Now I need to get live records of those symbols only. When I am connecting to websocket using channel.* then it's providing me all symbols data without any issue. But when I make query string for fetch live data for 10,000 symbols then it's giving me message for connection close. Also when I make query string for 4500 symbols then it's working fine and websocket run smoothly.
So, how can I fetch live data for 10,000 symbols in websocket ? in a single query string.
More...

Sounds like there is a limit.
I would just make an array or vector whatever and compare if the symbol is in that array of 10,000. If it is not then ignore the incoming msg.
 
qlai said:
I am not using them but generally speaking:
A) Don't request all at once ... Send requests 1000 at a time with 1 second delay in between.
B) if they allow, use multiple sockets where each is responsible for a range (A-L, M-Z)
All initialization needs to be done prior to market open.
More...

Please can you guide me that how can I use multiple websocket at a time ?
Right now I am using connection like this.
Code: 
private void websocket_Opened(object sender, EventArgs e)
{
       createSymbolsQueryString(); // using for get symbols list
       Console.WriteLine("Connected!");
       this.websocket.Send("{\"action\":\"auth\",\"params\":\"123456789123456789\"}");
       this.websocket.Send("{\"action\":\"subscribe\",\"params\":\"" + queryString + "\"}");
       this.websocket.Send("{\"action\":\"subscribe\",\"params\":\"" + queryString1 + "\"}");
       this.websocket.Send("{\"action\":\"subscribe\",\"params\":\"" + queryString2 + "\"}");
}
 
Last edited by a moderator:
otctrade said:
Sounds like there is a limit.
I would just make an array or vector whatever and compare if the symbol is in that array of 10,000. If it is not then ignore the incoming msg.
More...

Yes, I have tried for this solution but every time it's checking our list of 10000 symbols for those symbols which we don't need to handle so it's not solving our issue. I have tried with it. Polygon providing more then 25000 symbols so for 15000 symbols are not in list and loop will work for those also.
 
I use Polygon.io, but for realtime quotes, not historical data. During my development I noticed that after about 1000 symbols were being monitored, the websocket would close. So I switched to the NATS interface instead of websocket, and all socket problems disappeared. I do not know if there is a NATS interface for historical data.
 
You must log in or register to reply here.
Back
Top