Topic: About Last Night's Outage

[Dunrobin]

As many of you know, the main site was unavailable last night, starting around 6:30 PM EDT.  Ironically, once I was able to get in touch with a tech support person, the problem turned out to be what I thought was an extremely minor modification to the code that runs the site.

There are a handful of episodes in the filmography that feature two of the Stooges as solo players.  I use codes to identify episodes by which team of Stooges were featured - for example, "MLC" indicated Moe, Larry and Curly.  For solo appearances I use a code like "SMH" which stands for "Solo Moe Howard", and when there are two Stooges as solo actors in an episode I used both codes, separated by a comma ("SMH,SCH", for example.)

Yesterday new member mbruno pointed out that the links to the filmography for these solo combinations that appear on the Home page were not working, and when I investigated I realized it was because of the comma in the Stooge Team.  Commas weren't included in the list of approved characters in the site's URLs, so I modified the list to include the comma as an allowed character, and when I tried the links again it seemed to work just fine.  I thought "Hey! Problem solved! That was easy!" 

Unfortunately, that little change apparently started causing the site to over-tax the CPU on the hosting server, which started affecting other web sites that are hosted on the same machine.  As the problem grew worse, the tech support admin at the hosting company made the logical decision to suspend operation of the site's core script to eliminate the drain on the CPU.  I don't fault them for doing that - I would have done the same thing.  The only problem I have with the hosting company is that it seemed to take forever for someone to get back to me with the info I needed to find the problem and fix it.

In defense of the hosting company, though, that was the first time we've ever had that kind of an issue with them; they are usually very prompt and efficient.  Part of the problem was just that I am not the person who usually deals with them, and it took a while to verify my authenticity, etc., and I can't fault them for that either.  When I finally was able to talk with one their people this morning he was able to help me identify just when the problem started, and I realized it had to be my little comma "fix" that started it.  It only took me a minute to go into the database and change the handful of episodes to use an underscore (_) instead of a comma, and back out the change I had made to the script, and eureka! - the site was working again.

Ironically, I now also realize that the stupid comma in those few episodes may have been what brought down the site a couple of years ago.  I thought it was just my crappy, helter skelter coding that originally ran the site, which is part of the reason why I separated the forum and moved it to a different domain (which in turn caused other issues.)  The coding I use now is much more structured and organized, which made it much easier to identify the problem this morning.

So know you know why I seem to dawdle at making updates to the site.  Sometimes even a tiny change can really fuck things up!