It's not easy to do this stuff right.

Actually, I already knew that, but these two events definitely give me higher profile (and in the case of Ma.gnolia, more horrifying) anecdotes to share when I stare incredulously at someone who's just proposed, with a dismissive wave of their hand (indicating it'll be easy), adding in some feature or running fast and loose with site or feature development on a project. While I do play pretty loose with updates and the introduction of features on my own network of sites (really, I've got a network,) often going from concept to production in as long as it takes me to design/code it; when playing with other people's money I tend to be the guy pointing out the scary, often ignored, dark underbelly of the grand scheme. That's because (a) it's part of my job and (b) I've done this often enough to know that there's a possibility for things to blow up at every turn. Will they? Probably not, otherwise I'd be stark, raving mad, but as these two cases have shown, things can go wrong and when they do they can be embarrassing or, in the case of Ma.gnolia, catastrophic.

Of course, none of this should be stopping anyone from doing fun and interesting things with web sites. You simply can't innovate and create great sites and applications without some risk. Risk/reward and all that. It's just a really good thing to acknowledge and minimize that risk wherever possible and prepare for negative events so as to soften their impact on your business if they do occur.

These two incidents illustrate that point as well. Google, with teams and processes in place to deal with just such emergencies, were able to restore service to normal across their network in what they're saying was under an hour (most people I know saw it for only 20 minutes or so.) Ma.gnolia, on the other hand, seems to have taken a somewhat riskier approach and are still down, suffering what appears to be a complete loss of user data* (and, I imagine, basically the end of the site.)