Hello,
----
I started a new thread under "Trading Software", but found this to be more appropriate as there is discussion around development. I don't know how to remove the other discussion.
----
I have started looking into trade and time & sale data for equities with mixed results. My problems seem technical in nature and hence sharing what I have and get feedback here.
My setup: 8 core CPU, decent GPU, 16GB RAM, tons of SSD storage.
Using python to digest incoming stream - using Tradier for data. This basically requires me to open up multiple streams while passing the symbols I am interested in the request. I have a multithreaded system retrieving this data and puts them in influxdb (making sure data with same timestamp also gets written - incrementing timestamp by 1ns). Now, have a Grafana plugin to digest the data and build relationship with data.
My questions: 1) Is there a framework to allow me to do this so I can focus on actual data and number crunching? 2) If not, what are the best tools to digest the data on a local machine - I am looking at about 500K to 1.5M lines of records to digest based on my filter and if I scan the entire US equity market
----
I started a new thread under "Trading Software", but found this to be more appropriate as there is discussion around development. I don't know how to remove the other discussion.
----
I have started looking into trade and time & sale data for equities with mixed results. My problems seem technical in nature and hence sharing what I have and get feedback here.
My setup: 8 core CPU, decent GPU, 16GB RAM, tons of SSD storage.
Using python to digest incoming stream - using Tradier for data. This basically requires me to open up multiple streams while passing the symbols I am interested in the request. I have a multithreaded system retrieving this data and puts them in influxdb (making sure data with same timestamp also gets written - incrementing timestamp by 1ns). Now, have a Grafana plugin to digest the data and build relationship with data.
My questions: 1) Is there a framework to allow me to do this so I can focus on actual data and number crunching? 2) If not, what are the best tools to digest the data on a local machine - I am looking at about 500K to 1.5M lines of records to digest based on my filter and if I scan the entire US equity market