Need to handle websocket data for 10,000 symbols.

[DaveV]

Are you on Windows or Linux?

Windows, using C#

 

[carrer]

Windows, using C#

Windows are known for not being able to handle many socket connections.
Linux might be able to accomplish what you are trying to do. I am not sure how a C# application behaves on Wine. You might want to rewrite it in Java/Python.

 

[trend2009]

what kind of realtime data does the company provide? realtime level 1 quote for all US stocks? I can not find the information from their website. it only says realtime data.

 

[ZBZB]

www.worldtradedata.com and www.tiingo.com are competitors.

 

[otctrade]

I don't think subscribing to 10000 symbols separately will work.

The way I did it in node is to build an object of all the symbols I want to track. Here is a simplified version.Mine had about 5,000 symbols.
let symbols = {
'AAPL' : true,
'GOOG' : true
};

if (symbols['AAPL']) console.log('exists'); else console.log('does not exist');


symbols['AAPL'] you would replace with symbols[msg.sym] or whatever the field is for the incoming symbol

I'm sure there is a way to do this in C# too

 

[cruisecontrol]

Yes, I have tried for this solution but every time it's checking our list of 10000 symbols for those symbols which we don't need to handle so it's not solving our issue. I have tried with it. Polygon providing more then 25000 symbols so for 15000 symbols are not in list and loop will work for those also.

The top 2000 symbols have 99% of the volume so there is effectively no difference in performance between subscribing 2000 or 25000.

 

[cruisecontrol]

I use Polygon.io, but for realtime quotes, not historical data. During my development I noticed that after about 1000 symbols were being monitored, the websocket would close. So I switched to the NATS interface instead of websocket, and all socket problems disappeared. I do not know if there is a NATS interface for historical data.

Do you get an error 1006 when the websocket closes? If so I am getting the same error.

 

[hardik dhankecha]

Are you on Windows or Linux?

I am on windows and using C# for development.

 

[carrer]

I am on windows and using C# for development.

Windows are known for not being able to handle many socket connections.
Linux might be able to accomplish what you are trying to do. I am not sure how a C# application behaves on Wine. You might want to rewrite it in Java/Python.

 

[DaveV]

Do you get an error 1006 when the websocket closes? If so I am getting the same error.

Sorry I didn't record the error code, and I have totally replaced the WebSocket code with NATS so I have no way to retest.