[Leica] Narrative about the extended LUG outage

[r.s.taylor at comcast.net (Richard Taylor)]

Oy vey!, as we used to say in the old country (Flatbush).  Been there done 
that on some systems I?ve worked on.  I feel your pain.  

Best, 

Dick




On Jul 02, 2014, at 12:24 AM, Brian Reid <reid at mejac.palo-alto.ca.us> 
wrote:

> In case you care.
> 
> Server computers that are engineered for reliability have two power 
> supplies and two power cords. Power supplies are the most frequent 
> component to fail in server computers, so having two of them makes it 
> survive the outage of one.
> 
> The server computer that had supported the LUG had two power supplies. 
> They were stacked vertically, one on top of the other. Both power supplies 
> had been running 24x7 for about 9 years, and their fans had sucked in a 
> certain amount of lint. Lint is flammable. The bottom power supply failed, 
> and the lint caught fire. The flame rose to the upper power supply and 
> ignited its stored-up lint also. Like firestarters in a Franklin stove, 
> the 20-second burst of flame was enough to ignite the various flammable 
> items (including lint) in the main enclosure. The flash fire probably only 
> lasted 40 or 50 seconds, but it was hot enough to destroy most of the 
> solder traces that were near the power supplies on the circuit boards. 
> There were various plastic tags on some of the cables, which added 
> flammable material.
> 
> You can go to the store and buy a laptop or a desktop computer, but you 
> really can't go buy a server computer. Yes, this being silicon valley, 
> there are stores around that sell server computers (Central Computer is 
> the best of the lot) but buying a server computer at a retail store is 
> like buying a bicycle at a department store. It's just not the same thing. 
> Server computers are special-order, because there are so many variations 
> on how they are built that no one can afford to keep good ones in 
> inventory.
> 
> The fire was on a Saturday morning, and I knew that the soonest I could 
> even place an order for a replacement server was Monday, and even at 
> rush-rush prices I wouldn't get it until Thursday. At the time a 
> Saturday-to-Thursday outage seemed unconscionable. So I decided to move 
> the LUG and its supporting software to the newest and emptiest of my 
> half-dozen servers. It wasn't exactly a spare--it was running a few little 
> things--but mostly it was idle.
> 
> The LUG server had been running software from the era of its installation, 
> about 2005. The new server was built with chips and components that the 
> old software didn't understand, so I couldn't just restore the LUG server 
> backups onto the new server. They wouldn't run. I had to get the new 
> software working on the replacement server and then manuall move over each 
> piece.
> 
> I made the mistake of believing the operating system documentation, which 
> detailed a function called "system upgrade". It was supposed to work they 
> way Mac or Windows updates work--you let it do its thing for a while, and 
> then you reboot and all is well. After running the system upgrade, nothing 
> worked any more, including the few services that had been on that machine. 
> After asking the experts, I realized that I was going to have to wipe the 
> machine, do a clean install, get all of the necessary apps installed, and 
> then restore both sets of backups (LUG server and previous contents of 
> that server) to the clean system.
> 
> So far this is not a crazy plan. I've done things like it many times 
> before, though the 9-year software update gap made for a few challenges.
> 
> Once I got all of the apps installed and the backups restored, I 
> immediately typed the command to turn it all on
>       /local/mailman/bin/mailmanctl start
> and nothing happened. The error log showed a preposterous, deeply hard to 
> believe error message.
> 
> The wise person's first step in debugging strange failures on computers is 
> to type the error message into a search engine (I use Bing) to see if 
> other people had asked about it. To my great astonishment, no one had. 
> This never happens. Somebody else *always* has the same problem and has 
> asked about it.
> 
> I then started reading the source code of Mailman, trying to see what 
> circumstances would cause it to generate that message.  Mailman is written 
> in a language called Python. When you are having trouble like this, a good 
> step is to explore "version skew". Mailman Version XXX works only with 
> Python Version YYY. The versions of Python that are extant just now are 
> 2.5, 2.6, 2.7, 3.2, 3.3, and 3.4.  This is an abnormally large spread of 
> "current" versions, which usually means that the language developers have 
> made incompatible changes and have to keep old versions around for apps 
> that have come to depend on them.
> 
> I tried all 6 of those Python versions. I got the same odd error in the 
> 2.* versions, and absolute chaos in the 3.* versions. Since the version of 
> Mailman that I wanted to use (2.1.18) failed the same way with all of the 
> 2.* Python versions, I wiped the slate clean one last time and installed 
> Python 2.7.
> 
> Gonna have to find this problem the old-fashioned way.
> 
> Many days pass as I read documentation, run tests, explore the software, 
> use debuggers, create and read log files, all to no avail.
> 
> Then I decided to instrument and log what was happening when 
> Mailman/Python started up. Figuring out how much information to put in a 
> log file is a black art. If you log too much, you will never find what you 
> are looking for in the swamp of details. If you log too little, you 
> probably won't log what you're looking for.
> 
> After far too much time staring at the logs, I saw that Python was 
> initializing from a library that was not listed in the Mailman 
> docdumentation.
> 
> An aside: language systems like Python tend to be aggressive in how they 
> find libraries. They look around and if they find something that looks 
> like a library, they use it. I'm sure the Python designers (none of whom 
> is named Monty) thought they were doing the world a favor by making it go 
> out and find its own libraries. "Autoconfiguration" run amok. Bad idea.
> 
> This library was obsolete. In the 9 years of not upgrading, the Mailman 
> software had changed the place where it kept certain library functions, 
> and both of them were present in the version I was trying to run. The 
> "wipe clean and reinstall" function only wiped the directories that it 
> knew about, and this obsolete directory was not on its list -- it had been 
> retired years ago -- so it didn't get removed by the "wipe clean" function.
> 
> If I had run all 12 of the upgrades between Mailman 2.1.6 and 2.1.18, one 
> of them would surely have deleted that newly-obsolete directory. But I 
> didn't, so it was still there.
> 
> When a complex computer system is using two different versions of the same 
> library, with creation dates 7 years apart, it doesn't stand a chance of 
> working.
> 
> I typed the Unix command "rm -rf /local/mailman/Mailman/pythonlib/email"
> which got rid of the ancient and incompatible library
> and everything started working. Perfectly.
> 
> There were hundreds of loose ends, and I spent the next week hunting them 
> down, but it wasn't taking 18 hours a day and LUG mail was flowing while I 
> did it.
> 
> Thanks for listening.
> Brian Reid
> LUG Saloonkeeper and server wrangler
> 
> 
> 
> 
> 
> _______________________________________________
> Leica Users Group.
> See http://leica-users.org/mailman/listinfo/lug for more information