Developers of Sympa

[Olivier Salaun <address@concealed>]

Hi Harald,

Sorry for responding so late ; we were quite busy with Sympa these days...

Harald Wilhelmi wrote:

> we are working at the moment on a larger Sympa installation, which
> extensivly uses LDAP. Here some issues we found:
> 
> 1) We found the well known problem with <SPACE> in LDAP attributes.
>    Strangly the fix for the problem (add a few '"' in _include_users_ldap
>    from List.pm) seems not to be in the official CVS tree?!

Lynda discovered this bug while developing new LDAP features, but we had not
fix it in _include_users_ldap().
I've just applied your fix to CVS.

> 2) List::_include_users_ldap fails in the case of a anonymous bind.

Also applied your patch.

> 3) We found it worth the effort to optimize the do_lists routine
>    for the case, that a topic is selected in the Web-Interface.
>    In our case this reduces the time needed for this operation
>    from typically 30s to 5s.
> 
> 5) We needed to write a new startup script. It's a valid HP-UX startup
>    but is mostly generic and should be a good base for most other
>    UNIX systems too. However it operates directly on the pidfiles.
>    So it needs some manual configuration.

We could provide your script in Sympa distribution ; we just need to
check what system is and install it if 'hpux'. What does `uname` returns
on hpux ? Where should the script be installed ?

> We tried to include a diff for issues 1), 2) and 3) in this EMail. However
> our working copy of Sympa contains extra debug code, the old urlizes patch
> and other modifcations. So my diff is manunally tuned and my be somewhat
> strange... The diff is against "3.3 alpha ldap.4". The startup script is
> included too. Feel free to include it in future Sympa relases or supply
> it as contributed software.

Thanks for these contributions.

> In addition we made a few observations with LDAP in general: Our 
> LDAP-Servers
> are under heavy use and are relatively slow. As far as we have observed
> wwsympa will not operate effectivly before it has done a List::load() for
> all configured lists. When using LDAP this takes >5h. In addition the
> wwsympa processes had grown to about 590MB. This happened obviously
> because Sympa will load the subscriber data into the RAM for "includes".
> 
> So we decided to cache the LDAP data in a SQL-Database. That reduced the
> process size of the Sympa processes to 130MB and the starup time (the
> time to do the List::load() calls) to just a few minutes. For the moment
> the syncronisation (LDAP->DB) is not integrated in Sympa, but we thinking
> about doing so.
> 
> Have you made such observations yourself? Is a "cache LDAP in DB" data
> source planned? Would you integrated such a design in the official Sympa
> release if we would implement it?

A LDAP (actually any 'include' type source) cache is now (in current CVS 
version)
managed by Sympa. Instead of managing the cache in memory, Sympa now has a DB 
structure
on-disk ('subscribers.db' file). This way, all processes (sympa.pl, 
wwsympa.fcgi, 
bounced.pl, archived.pl) benefit from this cache which is updated by any 
process
BUT wwsympa.fcgi. This should solve your problems.

Therefore, I don't think your (3) point is required Sympa anymore, because the
List::new() should be far shorter now.

We also have plans to have a LDAP cache in DB because it would :
        1/ allow users to set reception options
        2/ allow bounces management for them
        3/ make it possible to have mixed MLs, with both included
           and subscribed users

In our mind, the main problem was syncronisation : how to perform an 
incremental
update of the DB. Could you describe your LDAP->DB cache implementation ?

Thanks.

--
Olivier Salaün
Comité Réseau des Universités