The mailing list for listmasters using Sympa

[Micah Anderson <address@concealed>]

* Serge Aumont <address@concealed> [2008-10-29 22:26-0400]:
> Micah Anderson a écrit :
>> I'm actually looking to do the opposite of what was originally posted on
>> this thread.
>>
>> In our setup, we have a few lists with a very large number of
>> subscribers. When a post is sent to one of these lists, any other
>> sympa processing is blocked until it completes delivery to the MTA (in
>> our case, postfix). On the below example, there are almost 60k
>> subscribers, and the hand-off from sympa to postfix takes 2843 seconds
>> (about 47 minutes):
>>
>> If I am correct, what this means is that for 40 minutes there are no
>> other sympa operations, so all other messages that are in the queue,
>> or that arrive in the queue during this period are blocked by this
>> process.
>>   
>
> Sympa.conf distribution_mode fork option will make sympa fork, one 
> process analyse incomming messages forward messages to editor, answers to 
> command message etc. The second distribute message. It better but only 
> one message distribution process is running at time. 

I was surprised to hear of 'distribution_mode fork' option in
sympa.conf, and set it immediately the other day to see if this would
help. After a few days of data gathering, I noticed that we were getting
sporadic complaints about messages not being delivered. So I had a look
and it looked like there were about 4k messages that had piled up in the
'distribution' spool directory that were not being picked up for
whatever reason. Two sympa.pl processes were running, and I tried to
restart them, but it seemed as if whichever process was supposed to be
parsing that distribution spool directory was not. 

I need to look into this more to find out why not, but until I can spend
that time, I had to turn the "distribution_mode fork" off, and manually
move those files that were in the distribution directory up to the msg
spool directory to get processed. (I haven't upgraded to the latest
sympa release yet, so this perhaps might be an issue that has been fixed
there, or perhaps a local configuration problem).

>>
>> Why does it take so long, and how can I either speed up this process
>> or parallelize it?
>>
>> I have increased sympa.conf's maxsmtp 200 from 100 (as well as
>> postfix's smtp process limits), but that didn't change anything in the
>> times.
>>   
> The requested delay is comming from the number of system call to  
> sendmail or postfix. maxsmtp if the maximum number of  "sendmail"  
> processus called by sympa.pl. Increase this number will make  
> distribution faster only if many of thoses processus stay a lot of time  
> in the machine. This is possible when using sendmail and when thoses  
> processus are managing very slow smtp sessions. In your case (postfix)  
> the maxsmtp parameter is not the problem because postfix just queue  
> messages and finish very fast.

I see, that explains why I did not see a noticable difference when I
doubled that value.

> You should increase nrcpt parameter. nrcpt is the number of recipient as  
> argument of each sendmail/postfix call. The number of processus that  
> sympa have to run is #subscribers divided by nrcpt . (if you don't use  
> verp).

What if I am using verp? We are here, so I need to figure out what the
correct setting for the nrcpt parameter should be. 

> Sendmail as a limit for that parameter. If you put a hight value for it  
> each process become too complex and slow. Postfix is better for that so  
> you may increase it. Let's say from 25 to 100 or more.
> Is there some postfix guru that can confirm the value we can use with  
> postfix ?

I already had increased this value to 50 in my sympa.conf file, but I
will increase it to 100 to see if that helps.

>
> We did a major change in Sympa bulk mailer this summer. Next Sympa  
> version 6.0 will parallelize this process and it will be possible to  
> start a new message distribution for a hight priority list even if so  
> hugue distribution process is started.

Great, this is exciting to hear!

> Let us known if nrcpt change is a good solution for you.

Will do so, I've made this change and will report back my findings.

Micah

Attachment: signature.asc
Description: Digital signature