Developers of Sympa

[Olivier Salaun - CRU <address@concealed>]

Hi,

I'm responding to your message in the sympa-dev ML because it's rather a 
development concern.

Elijah Saxon wrote:

> We have run into a bit of a problem with wwsympa. With three thousand lists, 
> the caching of all the configs into memory is becoming a problem: it takes 
> over 20 minutes to start up and would quickly eat up all memory if we did not 
> restart them every night (although related, this might be more to do with the 
> rumor of a memory leak). The server is 1.7 ghz, 1 gig memory. Also, it would 
> be quite nice to be able to kick apache without creating 20 minutes of down 
> time.
>
> So I would like to write a patch for wwsympa which did not load the configs into memory. 
You should validate first that it is the config loading that takes so 
much time (we know other sites that have that many ML and they never 
reported such problems). Here is the piece of code that loads list 
config in wwsympa :
   if ($wwsconf->{'use_fast_cgi'}) {

       foreach my $l ( &List::get_lists('*') ) {
           my $list = new List ($l);
       }
   }
You should add traces at the beginning and end of this code to find out 
how long it takes. If it appears to last 20 minutes, then go down to 
List::get_lists() and check that the readdir() does not take too  much 
time. In the loop above in wwsympa, you could also check how long each 
'new List' take because a couple of lists could slow dow the whole 
process. If you eventually decide to store list config elsewhere or 
using a different format, all you should have to change are the 
List::_load_admin_file() and List::_save_admin_file() subroutines. Maybe 
these subroutines could be optimized. We are also considering storing 
authorization scenarios in a single hash in memory ; scenario hash 
structure in currently stored in every list admin hash. Changing this 
should make the 'new List' a bit faster and also save some memory space.

We could also consider having the list config as a tied variable (read 
http://www.perl.com/pub/a/2001/09/04/tiedhash.html), tied to a DB file 
for instance, but :
   1/ It would make data access slower
   2/ We would have to deal with DB files locking
   3/ We previously used DB_File tied vars for the 'include' 
user_data_source but we had some inconsistancies

List config could also be stored in MySQL but the DB definition would be 
somehow tricky because of the parameter occurences (0-n and 1-n).

> Before I begin, I have a few questions:
>  
>
> 1) should I bother? is this totally insane?
> 2) what features would I lose? I assume I would then have to
>   pre-render the directory of lists, come up with a new way
>   to approve pending lists and remove closed lists, and do without
>   searching by admin email address on the sympa admin page.
>   If that is all, it seems like a small price to pay. But maybe
>   there are some really important reasons to have the configs in memory.
>   I guess at a minimum, each list name and subject should be in memory
>   so that you can find lists by searching.
> 3) what areas of the code should I look at to do this?
> 4) does it take long enough to load each config that it would
>   be prohibitively slow to attempt to parse the file each time it is
>   needed? would a pool of recently loaded configs be the best way to go?
>
> -elijah
>  
--
Olivier Salaun
Comite Reseau des Universites