Brief Update From Amazon on S3 Downtime and Outage

As most of the Internet world knows (or saw), yesterday Amazon’s S3 cloud storage was down for over 8 hours. We reported on the S3 downtime scene as updates were provided.  We are waiting for official word from Amazon on what happened, why it happened and what the Amazon Web Services team is doing to prevent future issues from taking down the ultra-popular storage service. My guess is that we won’t hear anything until mid-week.

In the meantime, Om Malik of GigaOm has been able to get an update from Amazon. Here’s the statement:

As a distributed system, the different components of S3 need to be aware of the state of each other.  For example, this awareness makes it possible for the system to decide which redundant physical storage server to route a request to.

We experienced a problem with those internal system communications, leaving the components unable to interact properly, and customers unable to successfully process requests.  After exploring several alternatives, the team determined it had to take the service offline to restore proper communication and then bring service online again.

These are sophisticated systems and it generally takes a while to get to root cause in such a situation—we will be providing our customers with more information when we’ve fully investigated the incident.  We’re proud of our operational performance in operating S3 for almost 2.5 years, and our customers have generally been pleased with the reliability and performance of the service. But any downtime is unacceptable and we won’t be satisfied until it is perfect.

Amazon S3 is used heavily by a number of services behind Amazon’s retail websites.  Those services were impacted, but the retail website did not show noticeable problems because it mostly uses cached data.

So what it sounds like is that people stand on each corner and yell from one to another. If the 3rd person in line doesn’t hear the 2nd, they take the service down because something might be wrong at the 2nd person’s station.

It’s good to hear that Amazon uses S3 for storage on their own sites. They should feel the same pain that other publishers do. Check out all of our Amazon S3 coverage.