Developers of Sympa

[address@concealed]

On Tue, Jul 31, 2007 at 12:01:08PM -0700, Sergiy Zhuk wrote:
> hi
> 
> On Tue, 31 Jul 2007 address@concealed wrote:
> 
> > Very good to hear.  I am now looking to optimize taskmanager.pl - I have
> > managed to trim the memory usage for wwsympa by adding db caching, and
> > will now try to do the same for Task.pm.
> 
> The culprit here is that it doesn't technically affect memory use, since
> gradually (as more lists are accessed) it will grow to its usual size.

With the latest set of my patches installed, wwsympa has a footprint of 48M 
-- and it has remained that way for three days now.

I will keep track of this sympa process to see how much it balloons -- it is 
my guess (and, frankly, my hope) that it will remain this same size.

> Task manager is scanning all list configs, which is why it's that large from
> the start.
> I think we need to change fundamentally how we read and store list data.
> Right now there'a a lot of stuff stored in a list object, which could be
> shared like scenarios.
> Another way would be limiting the memory use, so we only update list cache
> for most frequently used lists.

Hmm - the list_of_lists cache itself does not use that much memory - it was 
the calls to get_list which caused the great memory usage.  The only reason I 
commented out the initial call to get_list (which pre-populates the list 
cache) 
was to improve the start up time of sympa.  I am aware that this cache will 
eventually get populated, but that is not at all a concern.

> Any other ideas ?

For taskmanager.pl and Task.pm...  I feel that I need to better understand
how this script operates -- An idea for reducing memory usage by removing 
the embedded list_object inside of a Task did not turn out as expected.

in Task.pm I changed:
            $task->{'list_object'} = new List ($task->{'object'});
            $task->{'domain'} = $task->{'list_object'}{'domain'};
to:
             my $list = $task->{'list_object'};
             $task->{'domain'} = $list->{'domain'};

And then in task_manager.pl from: 
            my $list = $task->{'list_object'};
to:
            my $list = new List ($task->{'object'}, $task->{'domain'});



The result?

18973 sympa     16   0 1012m 965m 3868 R   23 24.3   1:28.30 task_manager.pl  
  

A whopping 300MB in memory savings...  yet only a 'drop in the bucket', so 
to speak.  :) 

The data structures inside of Task and task_manager which are taking up 
space can be optimized with DB caching.  I will need to better understand 
how this module operates first.

Much regards Serge,
Charles