When’s The Right Time For Production Policies for Startups?

During my 12+ years building and managing very large corporate Web sites, one thing was always clear. There were defined policies and procedures about how files move to production and "go live". Typically files required multiple signatures depending on the type of file and which app they belonged to. As I moved up the ranks into management, it was my responsibility to make sure that nothing ever made it onto the live servers inadvertently. With Sarbanes-Oxley in place, this required even more documentation for file moves and definable policies for production servers. It was also important to keep all of our servers current – some of the farms I’ve worked with had hundreds of servers in them.

Yet with many startups I speak with, there is (many times) no application documentation, nor commented source code. Over the past week I’ve seen two situations where some policies and procedures could have saved time and money. Dogster CEO Ted Rheingold speaks about his search engine mishap and my hosting provider Mosso accounts for the other mishap.

Ted has a great post documenting the importance of checking files that are moved to production and keeping things in order. His group moved a search engine robots file into production that shouldn’t have gone up. What this did was cause a huge drop in search engine related traffic to Dogster. Ted says it didn’t account for a large monetary loss but had the issue continued unnoticed, it sure could have.

Ted concludes by noting, "it’s about being vigilant in keeping a holistic view of your whole web app and keeping it as small and efficient as possible even as your project gets more and more complex." I’d say I agree and disagree. It’s about realizing that as a startup grows, it’s not as easy to keep track of the files and the people behind those files. I am never pushing extra paperwork and documentation, but there is always some minimum required level so that simple things like this robots file don’t happen again with something more important.

Our hosting company Mosso had an issue last Friday with one of my Web sites. I won’t go into the entire issue now but my site was down for nearly 10 daylight hours. This cost me money on two ends. On one side I was trying to fix the issue all day and on the other side, I couldn’t write content here on CN because I was working on the code issue.

After the issue was resolved, Mosso staff posted the following message:

We did not realize at the time that a change was made to php.ini over 6 weeks ago, but never applied because we had simply not restarted Apache since then. When we made other configuration changes and forced Apache to restart it picked up on the rogue, untested change that was made to php.ini many weeks ago. But initially the warning messages went away – so we thought we were on to a fix. Site owners were unprepared to see a bunch of new messages that "broke" their sites, and it certainly looked fatal.

Again, someone made a change that was probably not documented and was left in development. So when the site was moved into production, all hell broke loose. Some percentage of Mosso’s customers were in some way affected by this issue which ranged from just showing warning messages all day through sites that were not functioning.

I had a great conversation once the issue was resolved with Rob La Gesse who is the new Mosso Director of Software Development about how they plan to make sure this never happens again. I will have more about that conversation when I post my full Mosso review.

As your startup grows, it’s important to remember that as more people touch the code, the servers and the Web services, policies need to be put in place to make sure issues like the ones above never arise.