[Leica] 1 more week in the December LUG photo contest

[reid at mejac.palo-alto.ca.us (Brian Reid)]

> suddenly things are noticeably peppier,
>
> are we on a new server?

No such luck. Here's an explanation of what happened that I sent to a tech 
group.

From: Brian Reid <reid@justus.anglican.org>
Subject: Re: [Technical] spam assassin daemon??
Date-Sent: 26 December 2007 11:41:45am

This stuff is FAR too fragile and interdependent.
Here's my current understanding of what happened.
At midnight GMT (at the start of Christmas day) an extraordinary avalanche 
of spam was sent from Russia.

My spam filter uses, as one of its principal methods for spotting spam, 
recognition that the same message (perhaps with systematic variations) is 
being sent simultaneously to many recipients. There are several dozen 
mailboxes, all of which at one time or another were real, that now serve as 
honey pots; all mail sent to them is assumed to be spam.

The honey-pot mail all gathers in a mailbox on the anti-spam server, and 
every 10 minutes a batch processing program gets run to sweep over the 
contents of the honey-pot mailbox and uses it to update the Bayesian 
database in the spam filter.

The anti-spam server uses Postfix; it was only the second time I had ever 
installed Postfix (J2C was the first) and I didn't know about all of the 
configuration options. There is a configuration option that limits the size 
of a mailbox, and the factory settings on Postfix set that to 50 megabytes. 
Once a mailbox is bigger than 50 megabytes it can not receive new mail.

During the 10-minute period from 12 to 12:10 GMT on Christmas, the honey-pot 
mailbox got more than 50 megabytes of spam sent to it, which caused it to 
fill, which caused it to lock. (This is the mysterious part: I don't see why 
that would have locked it, but it did. Maybe it filled in mid-message, and 
had plans to release the lock once the message delivery was done).

When the 12:10 processing job came along, it couldn't get a lock on its 
input file, so it waited for that lock to clear, which of course that lock 
never would. Since the author of the 12:10 processing job has read Dijkstra, 
he knew to set the locks in lexical order lest there be a deadly embrace. 
The honey-pot processor had set the lock on the spam detector before it went 
to get the lock on the mailbox. And, of course, that mailbox lock never 
opened up, so the whole system froze.

Eventually the spam detector system, which was waiting for its lock to open 
so that it could start processing again, crashed. That meant that the spam 
detector was not running at all. Within a few minutes, undetected spam 
started filling the Mailman input queues, and within the hour Mailman shut 
itself off from overload.

As I type this there are about 1000 legitimate Mailman messages being 
unspooled, and some honey-pot processing taking place, so I think it's all 
working.

The only remediation I can think of is to reduce the number of mailboxes 
that feed into the honey-pot, and also to find the Postfix configuration 
parameter that limits the size of that mailbox, and increase it or remove it.

Brian