Subject: Developers of Sympa
List archive
- From: address@concealed
- To: Sergiy Zhuk <address@concealed>
- Cc: address@concealed
- Subject: [sympa-dev] Re: Noise in the db / list sync diffs
- Date: Tue, 31 Jul 2007 15:12:01 -0700
On Tue, Jul 31, 2007 at 12:01:08PM -0700, Sergiy Zhuk wrote:
> hi
>
> On Tue, 31 Jul 2007 address@concealed wrote:
>
> > Very good to hear. I am now looking to optimize taskmanager.pl - I have
> > managed to trim the memory usage for wwsympa by adding db caching, and
> > will now try to do the same for Task.pm.
>
> The culprit here is that it doesn't technically affect memory use, since
> gradually (as more lists are accessed) it will grow to its usual size.
With the latest set of my patches installed, wwsympa has a footprint of 48M
-- and it has remained that way for three days now.
I will keep track of this sympa process to see how much it balloons -- it is
my guess (and, frankly, my hope) that it will remain this same size.
> Task manager is scanning all list configs, which is why it's that large from
> the start.
> I think we need to change fundamentally how we read and store list data.
> Right now there'a a lot of stuff stored in a list object, which could be
> shared like scenarios.
> Another way would be limiting the memory use, so we only update list cache
> for most frequently used lists.
Hmm - the list_of_lists cache itself does not use that much memory - it was
the calls to get_list which caused the great memory usage. The only reason I
commented out the initial call to get_list (which pre-populates the list
cache)
was to improve the start up time of sympa. I am aware that this cache will
eventually get populated, but that is not at all a concern.
> Any other ideas ?
For taskmanager.pl and Task.pm... I feel that I need to better understand
how this script operates -- An idea for reducing memory usage by removing
the embedded list_object inside of a Task did not turn out as expected.
in Task.pm I changed:
$task->{'list_object'} = new List ($task->{'object'});
$task->{'domain'} = $task->{'list_object'}{'domain'};
to:
my $list = $task->{'list_object'};
$task->{'domain'} = $list->{'domain'};
And then in task_manager.pl from:
my $list = $task->{'list_object'};
to:
my $list = new List ($task->{'object'}, $task->{'domain'});
The result?
18973 sympa 16 0 1012m 965m 3868 R 23 24.3 1:28.30 task_manager.pl
A whopping 300MB in memory savings... yet only a 'drop in the bucket', so
to speak. :)
The data structures inside of Task and task_manager which are taking up
space can be optimized with DB caching. I will need to better understand
how this module operates first.
Much regards Serge,
Charles
-
[sympa-dev] Re: Noise in the db / list sync diffs,
epsas, 07/31/2007
-
[sympa-dev] Re: Noise in the db / list sync diffs,
Sergiy Zhuk, 08/01/2007
- [sympa-dev] Re: Re: Noise in the db / list sync diffs, Olivier Salaün, 08/01/2007
-
[sympa-dev] Re: Noise in the db / list sync diffs,
Sergiy Zhuk, 08/01/2007
Archive powered by MHonArc 2.6.19+.