For a brief period last night and again early this morning, a small percentage of the users on our system had trouble logging into their email accounts. Our engineering team worked to resolve this issue which ultimately affected about 3% of our users.
Here is what happened:
There was an IP address conflict between two of our servers that caused a problem with our automatic failover routines. These routines normally allow servers to temporarily take on the responsibilities of others when a problem is detected, but in this case the conflict caused those particular servers to take over for the wrong machines. The result: some of our customers ended up logging into machines that did not actually host their email. Once we resolved the IP conflict and restarted the necessary services, everything went back to normal with ZERO loss of data. The one good thing here is that 97% of our customers were never even aware that there was a problem.
We regret that any of our customers experienced an interruption of services and we are constantly taking steps to prevent things like this from happening in the future. In this case, we’ll now be comprehensively monitoring the IP addresses on all of our servers and we will now be able to preemptively detect such conflicts before they lead to problems.
Again, we apologize for any inconvenience this may have caused.