If you want to speed up, you may run the correlation computations in Kubernetes with Docker containers.
Divide up the work data in suitable blocks/segments and run it simultaneously, how many in parallel that is needed to get meet the time constraint you. Either on-premise at home or on some public cloud vendor.
At HFT frequency the correlation between stocks at tick level will be effectively zero, and not informative (since the price changes will be dominated by the noise of the bid/ask bounce). Probably five minute data is the lowest frequency at which correlations make sense.
Hi @globalarbtrader I wanted ask your opinion. I currently trade FX and look at the correlation between the absolute value of pair prices. I was thinking that it might be better to measure the correlation of a change in PnL for 1 lot of each pair; A and B.
And since you are looking for future relationships, you could take the correlation based on change in PnL and do a simulation to find the likely future range of those correlation values within a given future time, using the worst case value in that range say at sigma 1 or sigma 2, to help decide how much risk to take on a new position.
