Quote from IBsoft:
We believe that our Sun nights problems were caused by a fault inside one of our high-availability-file-servers (HAFS). The purpose of such systems to serve files in a reliable and redundant manner. They are implemented as combinations of dual/mirrored disk-array and a purpose-built hardware and operating system. We are contacting the manufacturers of various components of the system about the failure.
The connectivity issue that some of our customers experienced on Thu/Fri of last week had to do w/ a combination of two factors:
1) An HAFS event (similar to the one described above) caused thousands of TWS's to simultaneously reconnect.
2) Some of the reconnects were handled by our gateways that were upgraded from Linux OS 2.4 to 2.6. The version 2.6 of the OS has a bug that manifests itself when a number of simultaneous reconnects exceeds 1024. We have taken these gateways out of the system until we obtain a required OS patch or until we modify the gateway application to work around the OS bug.
