The mailing list for listmasters using Sympa

[Miles Fidelman <address@concealed>]

Hi Soji,

Thanks so much for the quick reply.  I guess time zone differences help! :-)

I seem to have fixed the problem - see a couple of in-line notes, below, 
and resolution at the end.


IKEDA Soji wrote:
> Hi,
>
> 2015/09/06 10:47、Miles Fidelman <address@concealed> のメッセージ:
>
> > Hi Folks,
> >
> > Maybe somebody can help me diagnose this:
> >
> > - our server went wonky the other day, starting with huge load spikes that I 
> > think came from a DDoS attack, but also might have been a new kind of spam 
> > jamming up our spam filter (lot of disk i/o from amavisd-new)
> >
> > - it backed up mail - lots of stuff just queued up, going nowhere
> >
> > - I have the load back under control, but....
> >
> > i) it seems like a lot of list traffic has stalled - some queues are pretty 
> > big (auth, msg), while there's nothing in the moderation or outgoing queues
> >
> > ii) sympa.pl seems to be running a heavy load, then it dies
> >
> > I'm wondering two things:
> > i.  Is Sympa choking on one particular message - and if so, how to figure out 
> > which one it is?
> > ii. Is there a way to prompt Sympa to process it's queues (akin to a postfix 
> > flush)?
> There are no way to flush queue.  Instead, you can move the messages in msg/ 
> directory to other place (such as new directory msg.backup/), put a small 
> part of them back into msg/ directory, check if they are delivered, and 
> repeat until all messages are delivered.  This decreases disk load to read 
> large number of directory entries.

I ended up simply deleting lots of entries, though....

> Also, to decrease CPU load and memory usage, you, if possible, can stop 
> unnecessary processes for a while: HTTPd and task_manager.pl can be stopped 
> (make sure to restart them when the system works normally).
>
> To know what happened when Sympa crashes, system logs, and error outputs 
> saved under tmp/ queue directory can help.

The funny thing is that the *.stderr files were all empty, and the dump 
files weren't particularly helpful.

I think the issue was that there was a zombie copy of sympa.pl running - 
I saw it in top, even though I'd just done ../init.d/sympa status and it 
said that sympa.pl had died.  Sure enough, when I stopped the system, 
there was that sympa.pl process running - killed it, restarted, and 
things started working again.

BUT..... the issues had survived across several reboots, so I think one 
of the messages in the queue caused sympa.pl go to wonky, so it was the 
combination of deleting messages, and THEN killing the process that 
solved the problem.  I kind of wish the log files had been more useful, 
but I think the zombie process had locked up logging somehow.  Sigh.....

Thanks again!

Miles

--
In theory, there is no difference between theory and practice.
In practice, there is.   .... Yogi Berra