CATEGORIES
- NYC COVERAGE
- WEB STARTUPS
- WEB NEWS
- CONFERENCES
- WEB TECH JOBS
- VENTURE CAPITAL
- MICROSOFT
- INTERVIEWS
- ADVERTISING
- VIDEO
- ALL TOPICS
- ALL COMPANIES
CONTRIBUTORS
- ADRIAN CHAN
- ALICIA NAVARRO
- ALLEN STERN
- CORSIN CAMICHEL
- DRAMA 2.0
- DARREN HERMAN
- HANK WILLIAMS
- MARK DAVIS
- RICK TUROCZY
- SANFORD DICKERT
- SHANNON CLARK
- Comment on YouTube Down by DVS01
- Comment on Twitter COO Costolo: Advertising Coming To Twitter Soon by Satoshi Nakajima
- Comment on Twitter COO Costolo: Advertising Coming To Twitter Soon by OMG Stop the Web! Twitter is gonna run ads ? and Scoble says you?ll love it
- Comment on What?s Up With Yahoo Mail Delivery? by MJ
Amazon S3 Down
Latest update from Amazon 7:30pm Eastern – "US service has been partially restored. We continue to work to fully restore the service."
Amazon’s S3 storage service appears to be down. CenterNetworks images are broken because of it and I had to move the style sheet back so the site at least renders correctly. Sites like Twitter have massive broken images currently because Amazon S3 is down.
We will keep updating this post until the service has recovered. As of Noon Eastern time, the S3 service is down.
Amazon’s S3 service was also down this past February and Amazon explained the reasons for the outage and downtime a few days later. There has to be a way to failover when S3 is down.
As always report in if you are having issues with Amazon S3.
Update: Amazon S3 clients are now posting outage messages on the forums. It appears EC2 is working ok.
Update 2: Amazon is now reporting on their health dashboard that Amazon S3 has "elevated error rates".
Update 3 – 12:55pm: Now down over an hour, Amazon says they are "pursuing several paths of corrective action. Sites affected include SmugMug and Twitter. Twitter is up with no images while SmugMug had to take the service down since so much of it relies on Amazon’s services.
Update 4 – 1:55pm: Amazon now reporting, "10:33 AM PDT A quick update that we believe this is an issue with the communication between several Amazon S3 internal components. We do not have an ETA at this time but will continue to keep you updated."
Update 5 – 3:45pm: We are now down over 3.5 hours – Amazon’s latest update, "12:25 PM PDT We have restored communication between additional hosts and are continuing this work across the rest of the fleet. Thank you for your continued patience."
Update 6 – 6:00pm: Here’s Amazon’s last update, "2:36 PM PDT We have restored all internal communication across Amazon S3 hosts. We have started the multi-step process to begin accepting requests across Amazon S3 locations."
We use SecoBackup to backup our servers to Amazon S3. The nice thing about the SecoBackup product is it runs in the background, and only delays back ups by 6 hours, does not skip or miss them. We sailed smoothly through this. Some downtime is something we expect, and for a service like backup we dont mind it, esp. for the cost of using Amazon’s AWS.
Here’s a blog that talks about how it affected some users –
http://secobackup.com/blog/2008/07/21/secobackups-resilience-and-todays-amazon-downtime/
Steve
Congrats on solid reporting today, Allen. You were on top of this story.
Thanks Nate!
Solid reporting on a bad service, we should make S3 outages a national holiday considering they come around every few months, how about another outage right before Christmas? Or better yet Halloween? We can have people dress up in Scary Amazon S3 masks and run around screaming at IT managers. I’m done with these guys, i’m going to check out Nirvanix or another cloud service.
Our site’s images were completely offline as well – http://www.collarfree.com
We use SecoBackup to backup our servers to Amazon S3. The nice thing about the SecoBackup product is it runs in the background, and only delays back ups by 6 hours, does not skip or miss them. We sailed smoothly through this. Some downtime is something we expect, and for a service like backup we dont mind it, esp. for the cost of using Amazon’s AWS.
Here’s a blog that talks about how it affected some users –
http://secobackup.com/blog/2008/07/21/secobackups-resilience-and-todays-amazon-downtime/
Steve
Our service (PhotoShare, a photo sharing service for iPhone users) also uses S3, and we had a very hectic Sunday. Here is a quick report.
http://satoshi.blogs.com/uie/2008/07/amazon-s3-issue.html
Those of you complaining are a bunch of whining babies. You probably have no idea technically what it takes to run that infrastructure. They keep you informed and probably offer cost-credit. When someone sees your service out, you say, “I use Amazon and they were down for a while.” What are they going to say? It’s Amazon for God’s sake. I bet electricity sucked in its early days too, and I’m sure your ancestors were sitting around bitching about that. It’s early in this technological moonscape. Host it yourself or deal with it.
Would you buy a parachute in BETA? Do your risk acceptance homework. You’re a bunch of Twitter Fan Boys.
Twitter fan boys? Funny insult, but totally off target. If we were twitter fan boys, we’d accept downtime as twitter is down at least 5 times a day!
Hi. I can’t get on Amazon. It always tells me that the connection won’t work. What can I do?
Gene
Shelley – you can read the books you already purchased right? I don’t have a Kindle so I am unsure how the storage aspect works but great find nonetheless
_________________
sohbet
It’s still not working.
I had to manually add “https://www.amazon.com/” to my cookies, just to login.
Even then, the site takes 2 to 3 minutes to load between pages.
When I checked my wish list, everything was listed as “No longer available”.
I also had a window pop up that asked me if I wanted to except a cookie from a third party site.
It appears to me that they’ve most likely been hacked.
Thanks for the heads up!
wtf – still down nearly 2 hours later!
Twitpic is also randomly down, example this one celebrating today’s event (http://twitpic.com/4tls) – while S3 comes up look at the thumbnail (http://twitpic.com/show/thumb/4tls.jpg)
Wow 3.5 hours, that’s worse than Rackspace.
yep – crazy!
how much longer can it take ????
Muxtape songs won’t load either. They timeout trying to reach s3.amazonaws.com
It would seem that we Kindle users are also impacted by the S3 downtime.
Shelley – you can read the books you already purchased right? I don’t have a Kindle so I am unsure how the storage aspect works but great find nonetheless!
Servers are touchy sometimes. I have a major ISP where the server had a glitch. Took 3 days for them to fully restore service.
You have access to the books and what not you’ve already downloaded, but we can’t download subscriptions or purchased books.
We’ll be able to once S3 has been fixed.
I imagine this is also impacting on Amazon’s brand new beta testing video service, too. Or perhaps the video service is what impacted on S3.
I don’t think they want to get people upset either. I read somewhere else that it could take up to 5 days for all to be back to normal.