WIDE SCALE SERVICE DISRUPTION
On 28 December at approximately 10:24 EST, IB experienced a system failure leading to a significant service disruption affecting all US exchanges and all clients connecting to our US data center. The disruption lasted in an acute state approximately 35 minutes until 11:00 EST. Core functionality was reestablished at 11:05 although some minor order routing issues persisted thereafter.
While the reasons for the problem are still under investigation, our initial analysis has determined that there was a hardware failure in equipment responsible for linking our primary data network to the backup network. Our data centers are designed to be "hot switching" meaning that a failure in one segment triggers an instantaneous activation of services in the backup segment without any impact on real-time operations. In principal, the design is multi-path so there is substantial fault tolerance. In this event, it appears that the nature of the triggering failure compromised the synchronization process between the networks leading to an inability for them to automatically recover.
We recognize that a fault tolerant, robust system is of paramount important to our clients. We have invested heavily, and continue to invest, to provide a minimal downtime framework. Unfortunately, complex systems such as IB's, which link thousands of trading terminals to many dozens of exchanges and trading venues, still pose substantial challenges to achieving complete fault tolerance. We regret the impact the problems caused by the system failure had on our clients and trading partners. We would like to assure all affected parties that we are committed to identifying and addressing any area that can compromise our ability to deliver fast and reliable service.
Interactive Brokers Management
*from System Status page on IB website.
On 28 December at approximately 10:24 EST, IB experienced a system failure leading to a significant service disruption affecting all US exchanges and all clients connecting to our US data center. The disruption lasted in an acute state approximately 35 minutes until 11:00 EST. Core functionality was reestablished at 11:05 although some minor order routing issues persisted thereafter.
While the reasons for the problem are still under investigation, our initial analysis has determined that there was a hardware failure in equipment responsible for linking our primary data network to the backup network. Our data centers are designed to be "hot switching" meaning that a failure in one segment triggers an instantaneous activation of services in the backup segment without any impact on real-time operations. In principal, the design is multi-path so there is substantial fault tolerance. In this event, it appears that the nature of the triggering failure compromised the synchronization process between the networks leading to an inability for them to automatically recover.
We recognize that a fault tolerant, robust system is of paramount important to our clients. We have invested heavily, and continue to invest, to provide a minimal downtime framework. Unfortunately, complex systems such as IB's, which link thousands of trading terminals to many dozens of exchanges and trading venues, still pose substantial challenges to achieving complete fault tolerance. We regret the impact the problems caused by the system failure had on our clients and trading partners. We would like to assure all affected parties that we are committed to identifying and addressing any area that can compromise our ability to deliver fast and reliable service.
Interactive Brokers Management
*from System Status page on IB website.
