The mailing list for listmasters using Sympa

[Olivier Salaun - CRU <address@concealed>]

We know there is a memory problem with Sympa processes when managing 
lots of lists. Unfortunately we were not able to find out what data 
structures were using so much memory and what was making the process 
grow. We beleive that the Perl interpreter might have memory leaks, not 
giving back all the memory when data are no more referenced.

When have been looking for sufficient high-level debugging CPAN modules 
without success. If you know of a good module for going through all 
alocated vars and getting strcture size, let us know.

Shared memory : we don't have experience in programming this in Perl. If 
someone proposes us a prototype we will do our best to validate it 
against Sympa.

mod_perl : what would be the benefit compared to FastCGI ?

ReiserFS : it's a good idea to use it anyhow....

Elijah Saxon wrote:

> I am very impressed with sympa's performance processing and delivering mail. 
> I am less impressed with wwsympa's performance. Even with fastcgi, I have 
> always found wwsympa sluggish, even on a 2 ghz machine with 1 meg or ram and 
> no load. Regardless, it beats the pants off any alternatives and
> is acceptably sluggish.
>
> Lately, however, I have been wondering about wwsympa scalability with a large 
> number of lists. Sympa may work beautifully with lists of hundreds of 
> thousands of subscribers, but what about wwsympa with thousands of lists?
>
> In particular, because the configs for each list are kept in memory (is this 
> correct?), will wwsympa become memory bound? Currently, our wwsympa processes 
> are running at 150 megs each, but that is steadily climbing. Even now, we 
> would quickly run out of our 2 gig swap if we did not kill
> off wwsympa processes nightly.
>
> I have a couple questions:
>
> - is anyone running a box with over 5000 lists? how?
>
> - do wwsympa.fcgi processes share memory? could they?
>
> - how much work would it be to port to mod_perl?
>
> - what do people think about simply making swap really really big? does 
> wwsympa routinely access all the memory it keeps, or are there parts that 
> will safely get swapped out with disuse?
>
> - if you have multiple virtual domains, will splitting the fastcgi directives 
> into each domain improve performance? i guess the trade off is less memory 
> per process, but smaller pool of processes which can handle any given 
> request. i could be confused as to how this works. obviously, it would be 
> better to just put the domains on a different machine.
>
> - what can be done to improve wwsympa performance? (other than increasing the 
> number of processing or buying faster and dedicated hardware). i have found 
> that once over 1000 lists, a file system with b-tree directory storage, like 
> reiserfs, is a must.
>  
--
Olivier Salaun
Comite Reseau des Universites