Developers of Sympa

[elijah <address@concealed>]

We are finally upgrading our sympa to 5.3.1. Unfortunately, things are
going badly.

issue #1 -- each process is using 1gb of ram
---------------------------------------------

On our old machine, we have about 11 thousand lists. This causes sympa
to use a lot of memory:

> 19552 sympa  571m 507m wwsympa.fcgi       
> 19596 sympa  571m 505m wwsympa.fcgi       
> 19494 sympa  571m 500m wwsympa.fcgi       
> 11992 sympa  607m 397m task_manager.pl    
> 16244 sympa  571m 379m wwsympa.fcgi       
> 11986 sympa  834m 329m sympa.pl           
>  4653 sympa  142m 110m bounced.pl         
> 11989 sympa  152m  22m archived.pl  

Unfortunately, when these lists are imported into 5.3.1, the processes
use double the ram! wwsympa.fcgi is using 1.2gb, and sympa.pl almost as
much. It is not a memory leak issue: it uses this much right away on
startup. Our old server required 4gb just to keep it from constantly
thrashing, so it is likely our new one will need 8gb.

My questions are thus: what can we do to fix this? Have others seen the
same behavior? What has changed to dramatically increase memory
consumption? (perhaps the config.bin stuff?)

issue #2 -- wwsympa.fcgi is dramatically slower
-----------------------------------------------

Here is an example of the time it takes to perform a single request (on
a server with plenty of ram, zero load, and two 2.2ghz opterons):

> Jul  5 12:11:52 parrot wwsympa[21302]: List::get_lists() START 
> List::get_lists(lists.riseup.net)
> Jul  5 12:11:55 parrot wwsympa[21302]: List::get_lists() END 
> List::get_lists(lists.riseup.net)
> Jul  5 12:11:55 parrot wwsympa[21302]: List::get_lists() START 
> List::get_lists(lists.riseup.net)
> Jul  5 12:12:03 parrot wwsympa[21302]: List::get_lists() END 
> List::get_lists(lists.riseup.net)
> Jul  5 12:12:03 parrot wwsympa[21302]: List::get_lists() START 
> List::get_lists(lists.riseup.net)
> Jul  5 12:12:06 parrot wwsympa[21302]: List::get_lists() END 
> List::get_lists(lists.riseup.net)
> Jul  5 12:12:06 parrot wwsympa[21302]: [robot lists.riseup.net] [client 
> 0.0.0.0] [user address@concealed] main::do_home() do_home

14 seconds! It is enough to make me cry. With debug logging turned on,
it can take a minute, but here I have just added a 'notice' debug to the
 List::get_lists function. As you can see, it is spending most of the 14
seconds in the three calls to List::get_lists() (made by
List::get_which(), made by main loop of wwsympa.fcgi--i think).

Is it possible that the change to read the binary config.bin has
actually made it slower? I have thought of trying somehow to put all the
config.bin files in shared memory, like /dev/shm, but I think that the
disk cache is big enough that they are mostly all in memory already,
especially on this server with nobody using it yet.

I do not understand the code at all, but I am puzzled by how
wwsympa.fcgi calls List::get_lists(*) on startup, and then three times
again for every request. Can't these configs be cached somehow and not
be read again every time there is a request?

I must apologize for my lack of perl knowledge. Our collective is highly
proficient in ruby, C, python, java, php, and even bash--but we are all
at a loss when working with perl.

thank you,
-elijah