Folks,
It's rather fascinating all that goes into handling automated connectivity with data providers for a trading platform. Add to that the broker connectivity and it's quite complex.
Just consider some of these issues that tickzoom has uncovered and solved.
The biggest challenge of connectivity to brokers and providers is "reconnecting" after any outage however brief or long it might be.
So at a high level that means properly geting back fill data. And it also means resynchronizing the automated strategy with the trades.
But there's far more involved in the details.
For example, it makes little sense to have the automated trading system directly connect to brokers. Instead there must be execution/data service(s) which are separate from the main platform.
That makes it possible to run them on the same or a different computer to share data and connectivity with different instances of the platform.
That adds another level of potential connectivity issues.
Now when there is an outtage anywhere in the connection, those parts which are still functioning must gracefully handle it and continue retrying at short intervals to reconnect.
When it does reconnect, it must synchronize. For a data feed, it must pull the back fill data to update the local data.
But is that all? Of course not, it must also switch the automated strategy back to "historical mode" meaning that it can't place real trades. This it runs the back fill data through the strategies or portfolio to bring everything current with "right now".
What about trades? Of course, during a back fill period and any outage, there must be an "emergency stop" active.
As soon as back fill and updating the strategy complete, the strategy must compare the positions and orders to synchronize those that need to be canceled, replaced, closed, opened, etc.
Then it must switch back to real time mode to continue trading.
Is that all?
There's more.
The data service is collecting data to a file. Well during a reconnection and back fill, the data server needs to "read" from the file to which it normally writes. That creates conflict.
So it has to pause in writing, queue the data to memory while the back fill process is reading the file.
Can that take very long? No! Otherwise, it would fill up memory or increase risk that a hardware failure could lose data not yet written.
So that file must remain rather small.
That means another process must "sweep" these data files from a cache to a more permanent data store as the begin greater than X size.
Also, you advanced traders want to have more than one provider as a backup. So those data must be collected separately. And in order to automatically switchover, the platform has to recognize timeouts and delays to switch to another provider. And then switch back when necessary.
Add to all this that it must handle multiple symbols, providers, and brokers at the same time.
Certainly not a walk in the park.
TickZoom will release an update in a couple days which has all these facilities.
The only downside right now is that it still, at the moment, only supports MB Trading as a broker. But here's the plan to fix that very quickly.
A plan was established to integrate with the open source TradeLink project which has free open source connections to a variety broker servers via a common interface.
TradeLink already supports several but without all the facilities above.
So this will be a "match made in heaven" to have TradeLink as the standard API to brokers and data vendors while TickZoom handles the rest of the platform.
Josh has done great work over time in supporting and maintaining his broker connections with the standard TradeLink API. That's really a lot of work since the brokers are continually upgrading and changing stuff.
His system is to use automating testing and keep subscribed and current on notices from brokers and vendors about upcoming changes to their API and required upgrades.
So it will be good to support him financially to continue with that since TickZoom is commercial open source rather than free open source.
It's just satisfying to have a plan to support some free open source in the trading industry.
TickZOOM knows, all too well, that programmers have to feed their kids too!
Sincerely,
Wayne
It's rather fascinating all that goes into handling automated connectivity with data providers for a trading platform. Add to that the broker connectivity and it's quite complex.
Just consider some of these issues that tickzoom has uncovered and solved.
The biggest challenge of connectivity to brokers and providers is "reconnecting" after any outage however brief or long it might be.
So at a high level that means properly geting back fill data. And it also means resynchronizing the automated strategy with the trades.
But there's far more involved in the details.
For example, it makes little sense to have the automated trading system directly connect to brokers. Instead there must be execution/data service(s) which are separate from the main platform.
That makes it possible to run them on the same or a different computer to share data and connectivity with different instances of the platform.
That adds another level of potential connectivity issues.
Now when there is an outtage anywhere in the connection, those parts which are still functioning must gracefully handle it and continue retrying at short intervals to reconnect.
When it does reconnect, it must synchronize. For a data feed, it must pull the back fill data to update the local data.
But is that all? Of course not, it must also switch the automated strategy back to "historical mode" meaning that it can't place real trades. This it runs the back fill data through the strategies or portfolio to bring everything current with "right now".
What about trades? Of course, during a back fill period and any outage, there must be an "emergency stop" active.
As soon as back fill and updating the strategy complete, the strategy must compare the positions and orders to synchronize those that need to be canceled, replaced, closed, opened, etc.
Then it must switch back to real time mode to continue trading.
Is that all?
There's more.
The data service is collecting data to a file. Well during a reconnection and back fill, the data server needs to "read" from the file to which it normally writes. That creates conflict.
So it has to pause in writing, queue the data to memory while the back fill process is reading the file.
Can that take very long? No! Otherwise, it would fill up memory or increase risk that a hardware failure could lose data not yet written.
So that file must remain rather small.
That means another process must "sweep" these data files from a cache to a more permanent data store as the begin greater than X size.
Also, you advanced traders want to have more than one provider as a backup. So those data must be collected separately. And in order to automatically switchover, the platform has to recognize timeouts and delays to switch to another provider. And then switch back when necessary.
Add to all this that it must handle multiple symbols, providers, and brokers at the same time.
Certainly not a walk in the park.
TickZoom will release an update in a couple days which has all these facilities.
The only downside right now is that it still, at the moment, only supports MB Trading as a broker. But here's the plan to fix that very quickly.
A plan was established to integrate with the open source TradeLink project which has free open source connections to a variety broker servers via a common interface.
TradeLink already supports several but without all the facilities above.
So this will be a "match made in heaven" to have TradeLink as the standard API to brokers and data vendors while TickZoom handles the rest of the platform.
Josh has done great work over time in supporting and maintaining his broker connections with the standard TradeLink API. That's really a lot of work since the brokers are continually upgrading and changing stuff.
His system is to use automating testing and keep subscribed and current on notices from brokers and vendors about upcoming changes to their API and required upgrades.
So it will be good to support him financially to continue with that since TickZoom is commercial open source rather than free open source.
It's just satisfying to have a plan to support some free open source in the trading industry.
TickZOOM knows, all too well, that programmers have to feed their kids too!
Sincerely,
Wayne
jk