Developers of Sympa

[Harald Wilhelmi <address@concealed>]

Hello,

we are working at the moment on a larger Sympa installation, which
extensivly uses LDAP. Here some issues we found:

1) We found the well known problem with <SPACE> in LDAP attributes.
   Strangly the fix for the problem (add a few '"' in _include_users_ldap
   from List.pm) seems not to be in the official CVS tree?!

2) List::_include_users_ldap fails in the case of a anonymous bind.

3) We found it worth the effort to optimize the do_lists routine
   for the case, that a topic is selected in the Web-Interface.
   In our case this reduces the time needed for this operation
   from typically 30s to 5s. 
   
4) We found a few minor issues in the templates/NLS which I will
   discuss in seperate EMail to the sympa-translation mailing list.

5) We needed to write a new startup script. It's a valid HP-UX startup
   but is mostly generic and should be a good base for most other
   UNIX systems too. However it operates directly on the pidfiles.
   So it needs some manual configuration.

We tried to include a diff for issues 1), 2) and 3) in this EMail. However
our working copy of Sympa contains extra debug code, the old urlizes patch
and other modifcations. So my diff is manunally tuned and my be somewhat
strange... The diff is against "3.3 alpha ldap.4". The startup script is
included too. Feel free to include it in future Sympa relases or supply
it as contributed software. 

In addition we made a few observations with LDAP in general: Our LDAP-Servers
are under heavy use and are relatively slow. As far as we have observed
wwsympa will not operate effectivly before it has done a List::load() for
all configured lists. When using LDAP this takes >5h. In addition the
wwsympa processes had grown to about 590MB. This happened obviously
because Sympa will load the subscriber data into the RAM for "includes".

So we decided to cache the LDAP data in a SQL-Database. That reduced the
process size of the Sympa processes to 130MB and the starup time (the
time to do the List::load() calls) to just a few minutes. For the moment
the syncronisation (LDAP->DB) is not integrated in Sympa, but we thinking
about doing so.

Have you made such observations yourself? Is a "cache LDAP in DB" data
source planned? Would you integrated such a design in the official Sympa
release if we would implement it?

Best regards
Christoph Stock
Harald Wilhelmi


-- 

----------------------------------------------------------------
Harald Wilhelmi                         Betastr. 9A
TNG Technology Consulting GmbH          85774 Unterfoehringen
Partner                                 Tel: +49(89)21589960
address@concealed             Web: www.tngtech.com
Index: sympa/src/List.pm
diff -u sympa/src/List.pm:1.1.1.1 sympa/src/List.pm:1.5
--- sympa/src/List.pm:1.1.1.1   Wed Oct 24 14:47:52 2001
+++ sympa/src/List.pm   Fri Nov  2 15:11:50 2001
@@ -4355,18 +4390,24 @@
     }
 
     do_log('debug', "Connected to LDAP server $host:$port") if 
($main::options{'debug'});
-
-    unless ($ldaph->bind ($user, password => "$passwd")) {
-       do_log ('notice',"Can\'t bind with server $host:$port as user '$user' 
: $@");
-       return undef;
+    if ( defined $user ) {
+      unless ($ldaph->bind ($user, password => "$passwd")) {
+         do_log ('notice',"Can\'t bind with server $host:$port as user 
'$user' : $@");
+         return undef;
+      }
+    } else {
+      unless ($ldaph->bind ) {
+         do_log ('notice',"Can\'t do anonymous bind with server $host:$port 
: $@");
+         return undef;
+      }
     }
 
     do_log('debug', "Binded to LDAP server $host:$port ; user : '$user'") if 
($main::option{'debug'});
 
     do_log('debug', 'Searching on server %s ; suffix %s ; filter %s ; attrs: 
%s', $host, $ldap_suffix, $ldap_filter, $ldap_attrs) if 
($main::options{'debug'});
-    unless ($fetch = $ldaph->search ( base => $ldap_suffix,
-                                      filter => $ldap_filter,
-                                     attrs => $ldap_attrs)) {
+    unless ($fetch = $ldaph->search ( base => "$ldap_suffix",
+                                      filter => "$ldap_filter",
+                                     attrs => "$ldap_attrs")) {
         do_log('debug',"Unable to perform LDAP search in $ldap_suffix for 
$ldap_filter : $@");
         return undef;
     }
@@ -4768,6 +4809,58 @@
    return @lists;
 }
 
+sub grep_for_topic {
+    my $file = $_[0];
+
+    unless(open GREP_FILE, "<$file") {
+       do_log('debug', "List::grep_for_topic: can't access file %s", $file);
+       return undef;
+    }
+    while(<GREP_FILE>) {
+       split;
+       if($_[0] eq "topics") {
+               close GREP_FILE;
+               return $_[1];
+       }
+    }
+    close GREP_FILE;
+    do_log('debug', "List::grep_for_topic: no topic found in %s", $file);
+    return undef;
+}
+
+## List of lists with given topic
+sub get_lists_by_topic {
+   my $topic = $_[0];
+   my(@lists, $l);
+   do_log('debug2', 'List::get_lists_by_topic()');
+
+   unless (-d $Conf{'home'}) {
+       do_log('debug',"no such directory $Conf{'home'}");
+       return undef ;
+   }
+
+   unless (opendir(DIR, $Conf{'home'})) {
+       do_log('debug',"unable to open $Conf{'home'}");
+       return undef;
+   }
+   if($topic eq 'topicsless') {
+      foreach $l (sort readdir(DIR)) {
+         next unless (($l !~ /^\./o) and (-d $l) and (-f "$l/config"));
+         if(grep_for_topic("$l/config") eq undef) {
+             push @lists, $l ;
+         }
+      }
+   } else {
+      foreach $l (sort readdir(DIR)) {
+         next unless (($l !~ /^\./o) and (-d $l) and (-f "$l/config"));
+         if(grep_for_topic("$l/config") =~ /^$topic/) {
+             push @lists, $l ;
+         }
+      }
+   }
+   return @lists;
+}
+
 ## List of lists in database mode which e-mail parameter is member of
 sub get_which_db {
     my $email = shift;
Index: sympa/wwsympa/wwsympa.fcgi
diff -u sympa/wwsympa/wwsympa.fcgi:1.1.1.1 sympa/wwsympa/wwsympa.fcgi:1.2
--- sympa/wwsympa/wwsympa.fcgi:1.1.1.1  Wed Oct 24 14:47:53 2001
+++ sympa/wwsympa/wwsympa.fcgi  Fri Nov  2 10:51:21 2001
@@ -1555,48 +1555,69 @@
         }
     }
 
-    foreach my $l ( &List::get_lists() ) {
-        my $list = new List ($l);
+    if($in{'topic'}) {
+        my $composed_topic=$in{'topic'};
+       if($in{'subtopic'}) {
+           $composed_topic.="/".$in{'subtopic'};
+       }
+        foreach my $l ( &List::get_lists_by_topic($composed_topic) ) {
+            my $list = new List ($l);
+           &wwslog('debug', '   ... do_lists (topic) %s', $l);
 
-        my $sender = $param->{'user'}{'email'} || 'nobody';
-        my $action = &List::request_action 
('visibility',$param->{'auth_method'},
+            my $sender = $param->{'user'}{'email'} || 'nobody';
+            my $action = &List::request_action 
('visibility',$param->{'auth_method'},
                                             {'listname' =>  $l,
                                              'sender' => $sender,
                                              'remote_host' => 
$param->{'remote_host'},
                                              'remote_addr' => 
$param->{'remote_addr'}});
 
-        next unless ($action eq 'do_it');
+            next unless ($action eq 'do_it');
 
-        my $list_info = {};
-        $list_info->{'subject'} = $list->{'admin'}{'subject'};
-        $list_info->{'host'} = $list->{'admin'}{'host'};
-        if ($param->{'user'}{'email'} &&
-            ($list->am_i('owner',$param->{'user'}{'email'}) ||
-             $list->am_i('editor',$param->{'user'}{'email'})) ) {
-            $list_info->{'admin'} = 1;
-        }
-        if ($param->{'user'}{'email'} &&
-            $list->is_user($param->{'user'}{'email'})) {
-            $list_info->{'is_subscriber'} = 1;
-        }
+            my $list_info = {};
+            $list_info->{'subject'} = $list->{'admin'}{'subject'};
+            $list_info->{'host'} = $list->{'admin'}{'host'};
+            if ($param->{'user'}{'email'} &&
+                ($list->am_i('owner',$param->{'user'}{'email'}) ||
+                $list->am_i('editor',$param->{'user'}{'email'})) ) {
+                $list_info->{'admin'} = 1;
+            }
+            if ($param->{'user'}{'email'} &&
+                $list->is_user($param->{'user'}{'email'})) {
+                $list_info->{'is_subscriber'} = 1;
+            }
 
-        ## no topic ; List all lists
-        if (! $in{'topic'}) {
             $param->{'which'}{$list->{'name'}} = $list_info;
-        }elsif ($list->{'admin'}{'topics'}) {
-            foreach my $topic (@{$list->{'admin'}{'topics'}}) {
-                my @tree = split '/', $topic;
+        }
+    } else {
+        foreach my $l ( &List::get_lists() ) {
+            my $list = new List ($l);
+           &wwslog('debug', '   ... do_lists %s', $l);
+
+            my $sender = $param->{'user'}{'email'} || 'nobody';
+            my $action = &List::request_action 
('visibility',$param->{'auth_method'},
+                                            {'listname' =>  $l,
+                                             'sender' => $sender,
+                                             'remote_host' => 
$param->{'remote_host'},
+                                             'remote_addr' => 
$param->{'remote_addr'}});
 
-                next if (($in{'topic'}) && ($tree[0] ne $in{'topic'}));
-                next if (($in{'subtopic'}) && ($tree[1] ne $in{'subtopic'}));
+            next unless ($action eq 'do_it');
 
-                $param->{'which'}{$list->{'name'}} = $list_info;
+            my $list_info = {};
+            $list_info->{'subject'} = $list->{'admin'}{'subject'};
+            $list_info->{'host'} = $list->{'admin'}{'host'};
+            if ($param->{'user'}{'email'} &&
+              ($list->am_i('owner',$param->{'user'}{'email'}) ||
+              $list->am_i('editor',$param->{'user'}{'email'})) ) {
+                $list_info->{'admin'} = 1;
+            }
+            if ($param->{'user'}{'email'} &&
+                $list->is_user($param->{'user'}{'email'})) {
+                $list_info->{'is_subscriber'} = 1;
             }
-        }elsif ($in{'topic'} eq 'topicsless') {
+
             $param->{'which'}{$list->{'name'}} = $list_info;
         }
     }
-
     return 1;
 }
 
#! --SH--
#

#
# This script starts/stops Sympa.
#
# Harald Wilhelmi TNG Technology Consulting GmbH
#       address@concealed
#
# ATTENTION:
#
# This skript works only if the following variable is kept in sync
# with a explicit configuration of the pidfile (sympa.conf),
# bounced_pidfile and archived_pidfile (wwsympa.conf) parameters!
#

pidfilepath="/dist/sympa/run"

#
# It should be quite generic and work on most UNIX-Systems. Esspecially
# it's a vailid HPUX 10.x/11.x startup skript.
#
# Arguments: start, stop, verify_up, verify_down, restart
#
# Return values (start and verify_up/stop verify_down):
# 0: Success (all are up/all are down)
# 1: Failure (including partial)
# 2: Skipped
#

sympadir="--BINDIR--"
sympaconf="--CONFIG--"
wwsympaconf="--WWSCONFIG--"

start_wait=3
stop_wait=5

export PATH=/usr/bin:/bin:/usr/sbin:/sbin

# Gets a module name and check is it is up.
check_up() {
        if [ -r $pidfilepath/$1.pid ]
        then
                ps -p `cat $pidfilepath/$1.pid` | grep -q $1.pl 
                if [ $? -ne 0 ]
                then
                        echo "   XXX $1.pl has crashed! (pidfile without 
proc)"
                        return 1
                else
                        return 0
                fi
        else
                echo "   XXX $1.pl is down! (no pidfile)"
                return 1
        fi
}

# Gets a module name and check is it is down.
check_down() {
        if [ -r $pidfilepath/$1.pid ]
        then
                ps -p `cat $pidfilepath/$1.pid` | grep -q $1.pl
                if [ $? -ne 0 ]
                then
                        echo "   XXX $1.pl has crashed! (pidfile without 
proc)"
                        return 0;
                else
                        echo "   XXX $1.pl is still up!"
                        return 1;
                fi
        else
                return 0;
        fi
}

# See how we were called.
case "$1" in
  start)
        if [ ! -d $sympadir ]; then
                echo "XXX $sympadir directory not found!"
                exit 2
        fi
        if [ ! -f $sympaconf ]; then
                echo "XXX $sympaconf not found!"
                exit 2
        fi
        if [ ! -f $wwsympaconf ]; then
                echo "XXX $wwsympadir not found!"
                exit 2
        fi

        if [ ! -f $pidfilepath/sympa ]; then
                echo "Starting Sympa subsystem: "
                echo '   ***  sympa.pl...'
                $sympadir/sympa.pl
                echo '   ***  archived.pl...'
                $sympadir/archived.pl
                echo '   ***  bounced.pl...'
                $sympadir/bounced.pl
                touch $pidfilepath/sympa
                sleep $start_wait
        else
                echo "   Sympa seems active. No action will be taken."
        fi
        $0 verify_up
        exit $?
        ;;
  stop)
        echo "Stopping Sympa subsystem: "
        if [ -f $pidfilepath/sympa ]; then
                kill `cat $pidfilepath/sympa.pid`
                kill `cat $pidfilepath/archived.pid`
                kill `cat $pidfilepath/bounced.pid`
                sleep $stop_wait
        else
                echo "   Sympa seems down. No action will be taken."
        fi
        $0 verify_down
        value=$?
        if [ $value -eq 0 -a -r $pidfilepath/sympa ]
        then
                rm $pidfilepath/sympa 
        fi
        exit $value
        ;;
  restart)
        $0 stop && $0 start
        exit $?
        ;;
  verify_up)
        some_are_down=0
        echo "Verifying Sympa is UP:"
        check_up sympa || some_are_down=1
        check_up bounced || some_are_down=1
        check_up archived || some_are_down=1
        if [ $some_are_down -eq 0 ]; then 
                echo "   All fine."
                exit 0
        else
                echo "   *** FAILURE!"
                exit 1
        fi
        ;;
  verify_down)
        some_are_up=0
        echo "Verifying Sympa is DOWN:"
        check_down sympa || some_are_up=1
        check_down bounced || some_are_up=1
        check_down archived || some_are_up=1
        if [ $some_are_up -eq 0 ]; then 
                echo "   All is down."
                exit 0
        else
                echo "   *** FAILURE!"
                exit 1
        fi
        ;;
  start_msg)
        echo "Starting Sympa"
        exit 0
        ;;
  stop_msg)
        echo "Stopping Sympa"
        exit 0
        ;;
esac

echo "Usage: $0 start|stop|start_msg|stop_msg|verify_up|verify_down|restart"
exit 1