There Were No Plain Bagels Today

When I head into Manhattan in the morning, I start nearly every day at a cafe where I get the same items each time. The clerk knows what I like and when they have no more of my items left, he tells me so I don’t need to wait on line and I can go elsewhere even though I enjoy their food and the locations (free wifi, lots of tables, etc). This morning he shouted to me, “sir we have no plain bagels today, I’m sorry.” While this rarely happens, it got me thinking about some of the interesting events over the past week and the blog posts discussing these events.

Last week the Gmail email service was down for a few hours. Our post about the outage received over 1,500 comments. One of the interesting posts I read came from a blogger who was outraged and mad that Gmail was down and that the company hadn’t posted reasons for the outage and a time for recovery. What was interesting about the post is that this same blogger allowed the customers of his startup (and investors) to go for months with no information about his service before it was shut down with no notice.

From my perspective, everything will go down or fail at some point. Last month ceiling tiles at a subway station in NYC fell and the station had to be closed for over a week. The MTA worked 24 hours a day to get things fixed and while it was a mess for people who needed to travel through that area, I saw very few complaints about how the MTA handled the incident. The truth is that when things fail, it’s important to consider how the company handles the issue and protects their users from the same issue in the future.

If a company handles the outage well and/or we know that the team is working on fixing the problem, we need to all take a deep breath and realize that humans are working to get things fixed. Sure it’s frustrating, maybe it could cost some amount of business but those are costs that should be taken into account as part of doing business. From what I saw last week with Google, they handled the Gmail outage well. As you read this, I am sure you (and I) are thinking that Twitter is another story with regards to how they handle outages.

Last weekend the big story was about a widespread hacker attack against WordPress blogs. If you read CN, you know that my sites have been exploited so many times, the hackers are sharing some of their link revenue with me. My sites have also lost a lot of Google juice and I am not even sure that all of the sites are back to normal in the Google index.

One of those affected by the most recent hacker wave was Rackspace employee Robert Scoble. Robert lost two months of posts after a hacker decided Robert posted too often about Friendfeed and removed the posts (heh). Many blogs attacked him for not having backups and that he should have been prepared for this. I liked Robert’s reply to those who wrote about his mistake:

“So I am a bad blog administrator. But no one died. I am not in jail. I don’t owe millions. My sons still hug me. There are worse mistakes!”

Sometimes you have to touch the hot stove to learn that it’s hot.  Somehow I am guessing Robert will never have to worry about a backup again. Naturally there are worse outages and some that are just unrecoverable. Even in those circumstances, there is still the same “hot stove learning”.

No matter how good the technology gets, there will always be outages and there will always  be human error. It’s how we deal with the outages and handle the mistakes that makes all the difference.