Archived posting to the Leica Users Group, 2007/12/26
[Author Prev] [Author Next] [Thread Prev] [Thread Next] [Author Index] [Topic Index] [Home] [Search]On Dec 26, 2007, at 1:26 PM, Brian Reid wrote: > >> suddenly things are noticeably peppier, >> >> are we on a new server? well out here at least, it has never been faster, Steve >> > > No such luck. Here's an explanation of what happened that I sent to > a tech group. > > From: Brian Reid <reid@justus.anglican.org> > Subject: Re: [Technical] spam assassin daemon?? > Date-Sent: 26 December 2007 11:41:45am > > This stuff is FAR too fragile and interdependent. > Here's my current understanding of what happened. > At midnight GMT (at the start of Christmas day) an extraordinary > avalanche of spam was sent from Russia. > > My spam filter uses, as one of its principal methods for spotting > spam, recognition that the same message (perhaps with systematic > variations) is being sent simultaneously to many recipients. There > are several dozen mailboxes, all of which at one time or another > were real, that now serve as honey pots; all mail sent to them is > assumed to be spam. > > The honey-pot mail all gathers in a mailbox on the anti-spam server, > and every 10 minutes a batch processing program gets run to sweep > over the contents of the honey-pot mailbox and uses it to update the > Bayesian database in the spam filter. > > The anti-spam server uses Postfix; it was only the second time I had > ever installed Postfix (J2C was the first) and I didn't know about > all of the configuration options. There is a configuration option > that limits the size of a mailbox, and the factory settings on > Postfix set that to 50 megabytes. Once a mailbox is bigger than 50 > megabytes it can not receive new mail. > > During the 10-minute period from 12 to 12:10 GMT on Christmas, the > honey-pot mailbox got more than 50 megabytes of spam sent to it, > which caused it to fill, which caused it to lock. (This is the > mysterious part: I don't see why that would have locked it, but it > did. Maybe it filled in mid-message, and had plans to release the > lock once the message delivery was done). > > When the 12:10 processing job came along, it couldn't get a lock on > its input file, so it waited for that lock to clear, which of course > that lock never would. Since the author of the 12:10 processing job > has read Dijkstra, he knew to set the locks in lexical order lest > there be a deadly embrace. The honey-pot processor had set the lock > on the spam detector before it went to get the lock on the mailbox. > And, of course, that mailbox lock never opened up, so the whole > system froze. > > Eventually the spam detector system, which was waiting for its lock > to open so that it could start processing again, crashed. That meant > that the spam detector was not running at all. Within a few minutes, > undetected spam started filling the Mailman input queues, and within > the hour Mailman shut itself off from overload. > > As I type this there are about 1000 legitimate Mailman messages > being unspooled, and some honey-pot processing taking place, so I > think it's all working. > > The only remediation I can think of is to reduce the number of > mailboxes that feed into the honey-pot, and also to find the Postfix > configuration parameter that limits the size of that mailbox, and > increase it or remove it. > > Brian > > > > _______________________________________________ > Leica Users Group. > See http://leica-users.org/mailman/listinfo/lug for more information