Developers of Sympa

[address@concealed]

> Can you post your mods to this list ?
> BTW, the first get_list call is to cache list configs in memory.
> sympa used to store scenarios with every list, which is partly why it's
> using so much memory.

Hi Serge,

It was a pleasure to see your response.  I have posted the diff to the sympa 
SourceSup at 
https://sourcesup.cru.fr/tracker/index.php?func=detail&aid=3219&group_id=23&atid=170
 . Please tell me what you think of these changes.  


As for the first call to List::get_lists inside of wwsympa - I know see how 
this will populate the %list_of_lists used by save and load. 

I will experiment with using this %list_of_lists hash in combination with 
Cached::Memcached::Tie to see if it helps wwsympa performance.  It is my 
strong suspicion it will not, as the main culprit for fcgi performance seems 
to be the @all_lists structure returned by get_lists...  


My current thinking for optimizing wwsympa.fcgi involve either: 

  1. Implementing calls to get_lists which request specific lists to return 
(as below with the $userlifts argument).  Is this done best by making queries 
to a database table kept in sync with list configs on the filesystem -- or by 
merely creating and maintaining seperate %list_of_pending, and etc..?   

  2. Developing a reasonable and efficient method for wwsympa to access the 
%list_of_list result cache.  Maybe by means of wwsympa accessing to a tie'd 
memcache share, rather than a conventional 200mb array.  Right now, 
list_of_lists is used only inside of List.pm -- it would be nice for wwsympa 
to have access to that SAME memory space.  Ideas?  

The current method of loading the values from %list_of_lists into an array 
is, to put it bluntly, puting the servers through much pain.  

Thanks again for your response Serge.  I totally appreciate your attention to 
this matter!
 
Salut,
Charles

--- List.pm 2007-06-12 02:34:38.000000000 -0700
+++ List.pm.Jul19   2007-07-19 04:18:09.000000000 -0700
@@ -10233,6 +10233,7 @@
 sub get_lists {
     my $robot_context = shift || '*';
     my $options = shift;
+    my $userlists = shift;

     my(@lists, $l,@robots);
     do_log('debug2', 'List::get_lists(%s)',$robot_context);
@@ -10257,7 +10258,18 @@
        do_log('err',"Unable to open $robot_dir");
        return undef;
    }
-   foreach my $l (sort readdir(DIR)) {
+    my (@files, $len);
+    if ( defined($userlists)){
+        @files = sort @$userlists;
+        $len = @files+0;
+        &do_log('debug2', "we are using $len directories");
+    }
+    else {
+        @files = sort readdir(DIR);
+        $len = @files+0;
+        &do_log('debug2', "we are using $len directories");
+    }
+   foreach my $l (@files) {
        next if (($l =~ /^\./o) || (! -d "$robot_dir/$l") || (! -f 
"$robot_dir/$l/config"));

        my $list = new List ($l, $robot, $options);
@@ -10463,12 +10475,26 @@

     ## WHICH in Database
     my $db_which = {};
+    my $all_lists;

+    # If using the db, extract listnames from db, and pass that to
+    # &get_lists - cep
     if (defined $Conf{'db_type'} && $List::use_db) {
-   $db_which = &get_which_db($email,  $function);
+       $db_which = &get_which_db($email,  $function);
+        my @lists;
+        foreach my $r (keys %{$db_which}){
+            foreach my $l (keys %{$db_which->{$r}}){
+                &do_log('debug2', "\$l is $l");
+                push @lists, $l;
+            }
+        }
+        $all_lists = &get_lists($robot, undef, \@lists);
+    }
+
+    unless (defined($all_lists)){
+        $all_lists = &get_lists($robot);
     }

-    my $all_lists = &get_lists($robot);
     foreach my $list (@$all_lists){

    my $l = $list->{'name'};




On Sat, Jul 21, 2007 at 12:02:58PM -0700, Sergiy Zhuk wrote:
> hi
> 
> On Thu, 19 Jul 2007 address@concealed wrote:
> 
> > I have made optimizations to get_list() and get_which() that result in a
> > far smaller $all_lists array.  Reducing the size of the list-of-Lists
> > returned by get_list saves a considerable amount of memory.
> 
> 
> -- 
> rgds,
> serge