Skip to Content.
Sympa Menu

devel - [sympa-dev] Re: Noise in the db / list sync diffs

Subject: Developers of Sympa

List archive

Chronological Thread  
  • From: address@concealed
  • To: Sergiy Zhuk <address@concealed>
  • Cc: address@concealed
  • Subject: [sympa-dev] Re: Noise in the db / list sync diffs
  • Date: Tue, 31 Jul 2007 15:12:01 -0700

On Tue, Jul 31, 2007 at 12:01:08PM -0700, Sergiy Zhuk wrote:
> hi
>
> On Tue, 31 Jul 2007 address@concealed wrote:
>
> > Very good to hear. I am now looking to optimize taskmanager.pl - I have
> > managed to trim the memory usage for wwsympa by adding db caching, and
> > will now try to do the same for Task.pm.
>
> The culprit here is that it doesn't technically affect memory use, since
> gradually (as more lists are accessed) it will grow to its usual size.

With the latest set of my patches installed, wwsympa has a footprint of 48M
-- and it has remained that way for three days now.

I will keep track of this sympa process to see how much it balloons -- it is
my guess (and, frankly, my hope) that it will remain this same size.

> Task manager is scanning all list configs, which is why it's that large from
> the start.
> I think we need to change fundamentally how we read and store list data.
> Right now there'a a lot of stuff stored in a list object, which could be
> shared like scenarios.
> Another way would be limiting the memory use, so we only update list cache
> for most frequently used lists.

Hmm - the list_of_lists cache itself does not use that much memory - it was
the calls to get_list which caused the great memory usage. The only reason I
commented out the initial call to get_list (which pre-populates the list
cache)
was to improve the start up time of sympa. I am aware that this cache will
eventually get populated, but that is not at all a concern.

> Any other ideas ?

For taskmanager.pl and Task.pm... I feel that I need to better understand
how this script operates -- An idea for reducing memory usage by removing
the embedded list_object inside of a Task did not turn out as expected.

in Task.pm I changed:
$task->{'list_object'} = new List ($task->{'object'});
$task->{'domain'} = $task->{'list_object'}{'domain'};
to:
my $list = $task->{'list_object'};
$task->{'domain'} = $list->{'domain'};

And then in task_manager.pl from:
my $list = $task->{'list_object'};
to:
my $list = new List ($task->{'object'}, $task->{'domain'});



The result?

18973 sympa 16 0 1012m 965m 3868 R 23 24.3 1:28.30 task_manager.pl


A whopping 300MB in memory savings... yet only a 'drop in the bucket', so
to speak. :)

The data structures inside of Task and task_manager which are taking up
space can be optimized with DB caching. I will need to better understand
how this module operates first.

Much regards Serge,
Charles





Archive powered by MHonArc 2.6.19+.

Top of Page