It Was Sad When the Great Blog Went Down …

Oh, they built the blog RegenAxe, to sail the internet blue.
For they thought it was a blog that spam would never go through.
It was on its maiden trip, that a virus hit this blog.
It was sad when the great blog went down.

It was sad, so sad.
It was sad, so sad.
It was sad when the great blog went down
Uncles and aunts, little children lost their pants.
It was sad when the great blog went down.

OK, it is Thursday night, what should I blog about?  Oh I know!  Wait my blog isn’t working.  What’s going on?  [An hour later]  Hmm, it’s still not working.  I know, I’ll check for news on Google … Ah, per The Huffington Post

During the service outage, WordPress sent out the following tweets:

  • We are back running at full capacity now. Closely monitoring services for any aftershocks. – Matt Mullenweg
  • We’re investigating the source & most expedient fix. I hope to have everyone’s blogs back & running as soon as possible. – Matt Mullenweg
  • WordPress.com is down, we’re working on restoring service now.

RegenAxe has been up and running on WordPress for about twenty-one months now.  This is the first service outage that I have experienced.  I think that is pretty good service.  Just so long as it doesn’t happen again.

The bird pictures with this post are all from San Francisco.  The Thayer’s Gull was photographed in Fisherman’s Wharf.  Go figure.  The Greater Scaup, pictured above, was seen in Crissy field.  Actually, it could be a Lesser Scaup, they have the same coloration.  There is only a size difference and there is interbreeding between the two species, anyway, Greater Scaup sounded better.  The following photo of a Great Blue Heron is also from Crissy Field.  Today’s header shows a pelican silhouetted against the sky.  This picture is from Moss Beach.

Even before the service outage, I had planed a technology themed post.  I had googled RegenAxe and beyond this website and the associated YouTube, Flickr and Vodpod accounts, I found an interesting (at least to me) collection of sites.  They all refered back to this site.  They were broadly speaking statistical in nature.  Some of them rate the value of RegenAxe.  Their various estimates of the value careen between a high of over a thousand dollars to only forty-six dollars.  More importantly though, some seem to have rather detail knowledge of the traffic on this website.  Even better, you can look up your friends and family member’s websites too.  😈

Hear is an update on the service outage from WordPress:

Today WordPress.com was down for approximately 110 minutes, our worst downtime in four years. The outage affected 10.2 million blogs, including our VIPs, and appears to have deprived those blogs of about 5.5 million pageviews.

What Happened: We are still gathering details, but it appears an unscheduled change to a core router by one of our datacenter providers messed up our network in a way we haven’t experienced before, and broke the site. It also broke all the mechanisms for failover between our locations in San Antonio and Chicago. All of your data was safe and secure, we just couldn’t serve it.

What we’re doing: We need to dig deeper and find out exactly what happened, why, and how to failover more gracefully next time and isolate problems like this so they don’t affect our other locations.

I will update this post as we find out more, and have a more concrete plan for the future.

I know this sucked for you guys as much as it did for us — the entire team was on pins and needles trying to get your blogs back as soon as possible. I hope it will be much longer than four years before we face a problem like this again.